Background

Failures and retries are inevitable in a distributed system. A pipeline step times out, a network call drops, a worker is rescheduled, and the same step runs again. In this post I’ll discuss why every workflow step a platform exposes should be safe to retry, and where that responsibility starts and stops.

When a platform primitive is not idempotent, the retry does damage. A blind create called twice leaves you with duplicate records, or two of a resource that was meant to be one, or state that no longer matches what the consumer thinks it provisioned. The burden of correctness then quietly moves onto every team that uses the platform, which is exactly backwards.

The guarantee

A platform owned step must be safe to run more than once with the same input and end in the same state. The consumer should be able to retry without reading the internals, because retrying is what orchestrators and humans both do when something looks stuck.

Implementation

Two habits cover most of it.

Query before creating. Check whether the resource already exists; if it does, treat the step as a no-op rather than an error. The failure mode this avoids is the unconditional create:

# not idempotent: second run fails or duplicates
aws s3 mb s3://my-platform-bucket

# idempotent: create only when absent
aws s3api head-bucket --bucket my-platform-bucket 2>/dev/null \
  || aws s3 mb s3://my-platform-bucket

Upsert for data writes. Prefer a write that converges rather than one that assumes a clean slate. The retry has to land on the same row, not a second copy of it.

INSERT INTO resource_state (id, status)
VALUES (:id, :status)
ON CONFLICT (id) DO UPDATE SET status = excluded.status;

The point is that platform owned primitives, the pipeline steps, the provisioning calls, the integration glue, handle this internally. Consumers should not have to wrap every call in their own existence check; if they do, the guarantee was never really provided.

Scope

This applies to the layers the platform owns: data pipelines, API and event integrations, and infrastructure provisioning steps. It is a claim about primitives, not about business logic.

Caveats

Application level idempotency is a different problem and it stays with the consumer. Deduplicating business events, deciding whether two “order placed” messages are the same order, reasoning about at-least-once delivery in their own handlers, that is theirs to own because only they know what “the same” means for their domain. The platform guarantees the infrastructure and integration layers are safe to retry; it does not, and cannot, decide semantics for the code running on top of it.