A customer taps pay. The request crosses a network you do not control, the response times out, and your app does the reasonable thing and retries. The card gets charged twice. Or a processor sends you a webhook that says a payment settled, delivers it twice because that is what at-least-once delivery means, and you pay a vendor out twice. Neither one looks like a failure in your logs. Both are the most expensive kind of payment bug there is.
Here is the plain version. A payment is not a function call that either works or throws. It is a conversation with a system you do not own, over a network that drops messages, where the dangerous outcome is not the charge that fails. It is the charge that quietly happens a second time. So the entire job is making sure that when the same intent arrives twice, your system recognizes it as the same intent and refuses to act on it again. That recognition cannot live in hopeful application code. It has to live in the database.
The database decides whether it already happened
Every operation that moves money carries an idempotency key the caller supplies, and the table that records those operations has a unique constraint on it. The flow is boring on purpose: claim the key, and if it was already claimed, return the result you recorded the first time instead of doing the work again. An insert that conflicts on the key does nothing. The guarantee comes from the database enforcing uniqueness under concurrency, not from your code checking and then acting, because between the check and the act is exactly where two requests race and both win. And the key has to come from your backend, not be generated on the client. The version of this that catches teams is subtle: a mobile app mints the idempotency key on the device to save a round trip, it passes every test and every demo, and then a week of bad cell coverage and automatic retries hands the backend three different keys for one tap of pay. The guard sees three separate payments and waves them all through, and customers start getting charged two and three times for a single order. No fraud, no breach, just a key created in the wrong place.
This is the one pattern that survives a retry, a double-click, a redelivered webhook, and a timeout-then-retry, all of which are the same event wearing different clothes. Build it once at the money layer and most of the double-charge class disappears, because the second attempt finds the key already taken and hands back the original answer.
Rather than take the argument on faith, run it. Same tap, same flaky network, two architectures:
Webhooks lie about how many times they happened
A webhook is the processor telling you something moved. The trap is that every major provider delivers at least once, which is a polite way of saying sometimes more than once, and some retry for days. Treat each delivery as a fresh fact and you double-process: a duplicate payout, a duplicate email, a duplicate ledger entry. The discipline has four parts and they are not optional. Verify the signature against the raw request bytes before you parse anything, with a timing-safe comparison and a short timestamp window so a captured payload cannot be replayed. Persist the raw event keyed on the provider plus its event id, with a unique constraint, so a redelivery is recognized and dropped by the database. Acknowledge fast, in well under a second, by doing only that verify-and-persist before returning, then hand the real work to a background worker, because providers time out in seconds and a slow handler becomes a redelivery storm. And when the work fails, retry with backoff, then move the event to a dead-letter table and alert a human rather than letting it vanish or loop forever. The same reconciliation you do between your own records, the processor, and the bank is what catches anything that still slips through, and that is its own piece.
A charge is a state machine, not a boolean
The other quiet source of payment bugs is modeling a charge as a flag that says paid. A real charge moves through states: authorized, then captured, then maybe refunded, sometimes partially, sometimes disputed, sometimes reversed. Each transition has rules about what can follow it, and webhooks about those transitions arrive out of order, so the settled event can land before the created one. The fix is to process every event against the current persisted state of the charge, never against the order the events arrived in, and to guard each transition so an out-of-order or repeated event becomes a no-op instead of corruption. Declines deserve the same care. A soft decline like insufficient funds is worth a retry on a sensible cadence; a hard decline like a stolen card must never be retried; and blindly hammering both is how you turn a recoverable customer into a flagged merchant account.
One thing you minimize rather than solve is the card data itself. Tokenization and hosted fields keep the raw numbers off your servers entirely, which shrinks your PCI scope from a compliance project into a much smaller one. You do not want to be the system of record for a primary account number if you can possibly avoid it.
What's still standing in 2028
The reason this matters more every year, not less, is that the parts of a payment flow AI is good at are the parts that were never the hard part. A language model will generate a checkout screen and a charge call in an afternoon, convincingly. What it does not generate is exactly-once, because exactly-once is not surface, it is invariants and edge cases and the discipline to put the gate in the database where races cannot get around it. A payment is a trust problem you reconcile, not an integration you call once and forget, and that distinction is the whole job. By 2028 the checkout screen is a commodity. The system that can prove it never charged a customer twice is not.
FAQ
Why do customers get charged twice? Almost always because a timeout was treated as a failure. The first request landed, only its response was lost, and the retry created a second charge because nothing recognized it as the same intent. A double-click, a redelivered webhook, and a timeout-then-retry are the same event wearing different clothes.
How does an idempotency key prevent double charges? Every money operation carries a key, and the table recording operations has a unique constraint on it. A retry claims the same key, conflicts, and gets back the original result instead of doing the work again. The guarantee comes from the database enforcing uniqueness under concurrency, not from application code checking and then acting.
Why do payment webhooks arrive more than once? Because providers deliver at least once, which in practice means sometimes more than once, and some retry for days. The discipline: verify the signature on raw bytes, persist the event keyed on provider plus event id with a unique constraint, acknowledge fast, process in the background, and treat event order as unreliable.
Is a charge just paid or unpaid? No. A real charge moves through states, authorized, captured, refunded, disputed, and the events about those transitions arrive out of order. Process every event against the persisted state of the charge, never against arrival order, and guard each transition so a repeat becomes a no-op instead of corruption.
What 2muchcoffee covers
We build and harden the money-movement layer underneath fintech products, the idempotency, the webhook discipline, the charge state machine, the reconciliation that proves it all tied out. If you are wiring up payments and the words "at least once delivery" just made you slightly nervous, that is the right instinct and the right time to talk, before a retry in production teaches you the same lesson the expensive way. The plain way in is the AI and engineering work we do.
One concrete action
Pick your charge endpoint and write down what happens if the exact same request arrives twice, a hundred milliseconds apart. If the honest answer is "two charges," add the idempotency key and the unique constraint before you add anything else, then prove it by firing the request twice and watching exactly one charge come out. That single gate is the foundation the rest of building fintech software stands on.
