The shape
A client sends a request. The response never comes back. The client cannot tell whether the operation happened, so it retries. Without help, you now have two charges, two posts, or two orders.
An idempotency key is the client naming the operation before performing it, so the server can recognise a second arrival as the same operation rather than a new one. The client generates a unique value, sends it with the request, and repeats the same value on every retry of that same intent. The server stores the key with the result and, on seeing it again, returns the stored result instead of doing the work twice.
Two things about that are easy to get backwards, and both matter more than the mechanism.
The key names the intent, not the request. Two different taps on "post" are two intents and must carry different keys, even if the payloads are byte-identical. One tap that gets retried four times is one intent and must carry the same key all four times. So the key is generated at the moment the user acts, not at the moment the HTTP call is made. It has to be the client's job, because only the client knows which of those two situations it is in.
Not everything needs one. An operation that is naturally repeatable does not need a key. Writing the same bytes to the same object storage key twice is the same as writing them once, and PUT /users/42 {name: "x"} sets a value rather than changing one. Reach for a key exactly where the operation stops being naturally repeatable, which in practice means creation and anything that increments, appends or charges.
The mechanism
Looking the key up is trivial. Deciding what to do with each state is not, and that is where this gets built wrong.
The row you store per key holds four things: the key, a state, a fingerprint of the request, and the response once there is one.
The insert must be atomic with the work. Put the key row and the effect in the same transaction, or use the insert's unique constraint as the lock. If you check-then-act with two round trips, two concurrent retries both see "not found" and both proceed, which is the exact race the key existed to prevent.
The in-progress state is the hard part. A retry arriving while the first attempt is still running cannot be given a result, because there is not one. Returning 409 and letting the client retry with backoff is the right answer. Blocking and waiting is worse: it ties up a connection on an operation whose duration you do not control, and it turns one slow request into two.
Store a fingerprint of the request. If the same key arrives with a different body, that is a client bug or an attack, and returning the original response silently would hide it. Hash the meaningful fields and return 422 on a mismatch.
Store the response, not just the fact of success. The retry must receive what the first call received, including the resource id. Returning 200 with an empty body on a replay leaves the client unable to proceed.
How to tell you have this problem
Your support queue contains "I was charged twice" or "my post appeared twice." This is the symptom that reaches you last and matters most.
A mobile client retries on timeout. Every mobile client does, because a request that times out on a train is far more likely to have failed than succeeded. If any endpoint it retries creates something, you need keys on that endpoint.
A queue consumer performs an external side effect. Every durable queue is at-least-once. If a consumer sends an email, charges a card, or calls a partner API, redelivery duplicates that effect and there is no transaction spanning your database and their system.
The query that finds existing damage: group your creation table by everything except id and timestamp, where the timestamps are within a few seconds of each other, and count. Near-duplicate rows separated by seconds are retries that got through.
What it costs
Storage, kept for as long as a client might retry. Mobile clients retry after being offline for hours, so the key has to outlive the request by a long way. 24 hours is a common floor. That is a table with a write rate of its own, because every creation writes a row whether or not it is ever retried.
Keys stranded in-progress. If a process dies mid-operation the key stays in-progress and every retry gets 409 forever. You need a background job, usually called a reaper, that resolves rows older than the longest an operation can legitimately take. It then has to decide whether to fail them or go and check whether the effect actually happened, and checking is not always possible.
It only covers the boundary you put it on. A key on your API does not protect a downstream service you call. Either propagate the key onward, so the whole chain de-duplicates on the same value, or accept that your retry to them can duplicate their effect.
Clients get it wrong, and you cannot fix it from the server. The most common client bug is generating the key inside the retry loop, which makes every retry a new operation and quietly defeats the entire mechanism. There is no server-side detection for this. It looks exactly like legitimate traffic, so it belongs in your client SDK rather than your documentation.
Where it shows up in the course
| article | the form it takes there |
|---|---|
| Idempotency and Exactly-Once Delivery | the key as one mechanism inside the wider question, with the conditional insert that makes it safe |
| on post creation and nowhere else, because the media upload is naturally repeatable and the post is not | |
| Message Queues vs Event Streams | the consumer side, where delivery is at-least-once and the key is what makes reprocessing harmless |
| URL Shortener | on creation, as the safe alternative to collapsing identical URLs onto one code, which looks like the same saving and is not |
| WhatsApp and Chat Systems | a client-generated message id that both makes the send retryable and lets the receiving device drop the duplicate |
Getting it wrong
Letting the server generate the key. Then it is not an idempotency key, it is a request id. Only the client knows whether this is a new intent or a retry of an old one.
Checking before inserting. Two round trips is a race. Use the unique constraint on the insert as the mutual exclusion, and let the failure tell you it is a duplicate.
Returning 200 with no body on a replay. The retry needs the original response. Store it.
Treating a key mismatch as a duplicate. The same key with a different payload is a bug you want to see, not a replay to hide.
Waiting on the in-progress state. It converts a duplicate into a held connection, and under retry storms that is how you exhaust a connection pool.
Assuming it gives you exactly-once. It gives you at-least-once with de-duplication. The distinction matters the moment somebody asks what happens when the client never sees any response at all.