In distributed systems, a timeout does not tell a client whether the server completed its work. When an API client sends an HTTP POST /v1/payments request and the connection times out after 5 seconds, the client cannot know what happened:

Did the request packet drop before reaching the server?
Did the server execute the payment, but the response packet dropped on the return path?
Did the server process crash mid-transaction?

If the client retries blindly, it risks charging the customer twice. If it does not retry, the user's order remains unfulfilled. To make a retry safe, an API can record a client-generated idempotency key and the outcome associated with it. When the same logical request arrives again, the server recognizes it and avoids repeating the side effect. Idempotency means repeated execution has the same intended effect on system state; it does not require every response to be byte-for-byte identical.

Figure 1: The service scopes a key to the caller and operation. A new key and its database mutation are committed together with the replayable result; a duplicate uses the unique constraint to find and return that result without repeating the mutation.

A payment retry, step by step

A customer submits a payment and the client creates one unique key for that payment attempt.
The server stores the key with the request state and performs the payment operation.
The response is lost, so the client retries with the same key and the same request.
The server finds the stored result and returns it instead of creating another charge.

The key is not a magic network guarantee. The record must be durable for the period clients may retry, concurrent requests using the same key must be coordinated, and the server should reject reuse of a key with a different request body. For workflows spanning independent services, each step still needs its own idempotency or deduplication strategy.

Delivery attempts are not business effects

“At-most-once” and “at-least-once” describe delivery or processing attempts within a particular system and its failure model; neither means that a business operation is guaranteed to happen exactly once across arbitrary services. Retrying tends to create at-least-once attempts, so consumers should tolerate duplicates. Durable deduplication can provide an effectively-once effect only for the state changes covered by the deduplication transaction. TCP provides reliable, ordered bytes for a live connection, not a guarantee that a remote handler ran exactly once.

Anatomy of an idempotency key

An idempotency key is an opaque, unique token generated by the client representing one logical business intent.

1. Key generation and scoping

Format: An opaque, high-entropy token such as a UUID. Idempotency-Key is a common API convention and is the subject of an IETF Internet-Draft; define its scope and behavior in your own API contract.
Strict multi-tenant scoping: Never key solely on the client-provided token. An idempotency key must be compound-scoped to the authenticated identity and route:

Storage Key = Tenant_ID + Principal_ID + Endpoint_Route + Idempotency_Key

Scoping prevents malicious tenants from guessing or squatting on keys to inspect another tenant's cached responses or cause denial of service.

2. Request payload hashing (Detecting mutation drift)

What happens if a buggy client sends two requests with the same idempotency key, but changes the request body from 10.00 USD to 100.00 USD?

Silently returning the 10.00 USD response would mislead the caller.
Executing the 100.00 USD charge violates idempotency.
Rule: Store a digest of the request fields that affect the operation, using a stable representation. Hashing arbitrary JSON bytes is not necessarily semantic canonicalization: whitespace, field order, and default values can differ. If the same key arrives with a different operation or payload, reject it with the status your API documents (commonly 409 Conflict).

Atomicity and concurrent duplicate requests

For a database-only operation, the simplest robust design is to commit the idempotency result and the business mutation in the same transaction:

[ Incoming Request ]
         │
         ▼
    [ Begin transaction ]
         │
         ├── New key ──> Apply mutation + save result ──> Commit
         └── Existing key ──> Compare request + replay result

Race condition: Simultaneous duplicate requests

Consider two identical payment requests arriving at two different gateway servers at the exact same millisecond (e.g., user double-clicks submit or an aggressive client timeout fires).

If both servers query SELECT status FROM idempotency_keys WHERE key = :k simultaneously, neither finds a record. Both proceed to execute the payment, resulting in a double charge.

The solution: Enforce a unique constraint on the documented key scope and make the insert, business mutation, and saved result one database transaction. A process-local “check then insert” is not sufficient across servers.

Production database schema (PostgreSQL)

CREATE TABLE idempotency_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id VARCHAR(64) NOT NULL,
    principal_id VARCHAR(128) NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL,
    endpoint_path VARCHAR(255) NOT NULL,
    request_payload_hash CHAR(64) NOT NULL,
    status VARCHAR(20) NOT NULL
        CHECK (status IN ('PENDING', 'RESOLVED')),
    response_code INTEGER NULL,
    response_headers JSONB NULL,
    response_body JSONB NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at TIMESTAMPTZ NOT NULL,
    CONSTRAINT uq_tenant_idempotency UNIQUE
        (tenant_id, principal_id, endpoint_path, idempotency_key)
);

CREATE INDEX idx_idempotency_expires_at
    ON idempotency_records (expires_at);

Atomic transaction execution pattern

BEGIN;

-- A unique constraint arbitrates concurrent requests across servers.
-- If this inserts a row, this transaction owns the new key.
INSERT INTO idempotency_records (
    tenant_id,
    principal_id,
    idempotency_key,
    endpoint_path,
    request_payload_hash,
    status,
    expires_at
) VALUES (
    'tenant_abc',
    'user_123',
    'client_req_998',
    '/v1/payments',
    :request_digest,
    'PENDING',
    NOW() + INTERVAL '24 HOURS'
)
ON CONFLICT (tenant_id, principal_id, endpoint_path, idempotency_key)
DO NOTHING
RETURNING id;

If the insert returns a row, apply the database mutation and update the idempotency row to RESOLVED with the response to replay, then commit. If it returns no row, read the existing record after the conflicting transaction resolves: compare the request digest and either replay the saved result or report that the same key is still in progress, according to the API contract. Do not execute the business mutation on that path.

-- Continue the transaction that inserted the new key.
UPDATE accounts
SET balance = balance - 100.00
WHERE id = 'acc_42';

INSERT INTO payments (
    id, account_id, amount, status
) VALUES (
    'pay_1', 'acc_42', 100.00, 'COMPLETED'
);

-- B. Atomically record the idempotency result
UPDATE idempotency_records
SET status = 'RESOLVED',
    response_code = 201,
    response_body =
      '{"payment_id": "pay_1", "status": "ok"}',
    response_headers =
      '{"Content-Type": "application/json"}'
WHERE tenant_id = 'tenant_abc'
  AND principal_id = 'user_123'
  AND idempotency_key = 'client_req_998'
  AND status = 'PENDING';

COMMIT;

If the process or database transaction fails before commit, both the mutation and the newly inserted record roll back. A retry can then attempt the operation. This pattern assumes the business mutation and idempotency record use the same transactional database. A durable, separately committed PENDING state requires a different design with leases, recovery, and explicit handling for abandoned work.

Multi-service architectures and distributed sagas

When a business operation must span external payment gateways (e.g., Stripe) or multiple microservices, a single local database transaction cannot span across network boundaries.

[ API Gateway ]
       │
       ▼ (ACID Transaction)
[ Database: State + Outbox Table ]
       │
       ▼ (CDC / Asynchronous Worker)
[ External Downstream Gateway / Payment API ]
  (Forward Client's Idempotency-Key)

In multi-service systems:

Use downstream deduplication deliberately: If the downstream API supports idempotency, send a stable key in the format and scope it documents. A correlation ID alone does not deduplicate work, and key support differs by provider.
The Transactional Outbox Pattern: Store outgoing external events in an outbox table within the local database transaction. A background CDC worker polls the outbox and dispatches events with retries.
Idempotent Consumers: Every downstream service receiving an event or message must check its own local idempotency table before processing side effects.

Retry algorithms: Exponential backoff with jitter

When an API call fails with a retryable condition (HTTP 503, a connection reset, or an ambiguous timeout), a client may retry if the operation is safe to repeat and its deadline or retry budget allows it. Clients that retry on the same fixed interval can synchronize and overload a recovering service.

The Full Jitter algorithm

One common policy is exponential backoff with full jitter:

sleep = random(0, min(cap, base * 2^attempt))

function calculateBackoffWithJitter(attempt, baseMs = 100, capMs = 10000) {
  // Exponential backoff ceiling
  const temp = Math.min(capMs, baseMs * Math.pow(2, attempt));
  // Uniform random distribution between 0 and temp (Full Jitter)
  const sleepMs = Math.floor(Math.random() * temp);
  return sleepMs;
}

Attempt 0: 0 to 100 ms
Attempt 1: 0 to 200 ms
Attempt 2: 0 to 400 ms
Attempt 3: 0 to 800 ms
Attempt 4: 0 to 1600 ms

Randomizing the delay reduces synchronized retry bursts; it does not guarantee a smooth arrival rate or make an unsafe operation safe to retry.

Operational safeguards for retries

Retry budgets: Bound retry volume relative to normal traffic and service capacity. Choose the budget from measured behavior; a single percentage is not suitable for every system.
Circuit breakers: Treat thresholds and sampling windows as service-specific policy, and remember that opening a breaker does not undo work that may already have completed.
Idempotency-key retention: Keep records at least as long as clients are allowed to retry. Expiration changes semantics: a late retry after deletion may execute again. Use indexed cleanup or time partitioning only when it fits the database and workload.

Key takeaways

Network timeouts are ambiguous; never assume a timed-out request failed to mutate backend state.
Scope keys to the authenticated caller and operation as defined by the API contract.
Store enough request identity to detect key reuse with different operation parameters.
Co-locate business state changes and idempotency record resolution inside the exact same local ACID transaction.
Pair client retries with exponential backoff, Full Jitter, and retry budgets to prevent destructive retry storms.

Idempotency, Retries, and Duplicate Processing