A customer taps pay. The request crosses a network you do not control, the response times out, and your app does the reasonable thing and retries. The card gets charged twice. Or a processor sends you a webhook that says a payment settled, delivers it twice because that is what at-least-once delivery means, and you pay a vendor out twice. Neither one looks like a failure in your logs. Both are the most expensive kind of payment bug there is.
Here is the plain version. A payment is not a function call that either works or throws. It is a conversation with a system you do not own, over a network that drops messages, where the dangerous outcome is not the charge that fails. It is the charge that quietly happens a second time. So the entire job is making sure that when the same intent arrives twice, your system recognizes it as the same intent and refuses to act on it again. That recognition cannot live in hopeful application code. It has to live in the database.
The database decides whether it already happened
Every operation that moves money carries an idempotency key the caller supplies, and the table that records those operations has a unique constraint on it. The flow is boring on purpose: claim the key, and if it was already claimed, return the result you recorded the first time instead of doing the work again. An insert that conflicts on the key does nothing. The guarantee comes from the database enforcing uniqueness under concurrency, not from your code checking and then acting, because between the check and the act is exactly where two requests race and both win.
And the key has to come from your backend, not be generated on the client. The version of this that catches teams is subtle: a mobile app mints the idempotency key on the device to save a round trip, it passes every test and every demo, and then a week of bad cell coverage and automatic retries hands the backend three different keys for one tap of pay. The guard sees three separate payments and waves them all through, and customers start getting charged two and three times for a single order. No fraud, no breach, just a key created in the wrong place.
This is the one pattern that survives a retry, a double-click, a redelivered webhook, and a timeout-then-retry, all of which are the same event wearing different clothes. Build it once at the money layer and most of the double-charge class disappears, because the second attempt finds the key already taken and hands back the original answer.
Rather than take the argument on faith, run it. Same tap, same flaky network, two architectures:
Even the thousand-engineer banks ship this bug
If this failure mode sounds like a startup problem, the record says otherwise. In January 2018, Capital One customers found their debit purchases posted twice, sometimes many more times, across the country; CNBC and the Washington Post covered it, and the bank attributed it to an internal technical issue, fixed within about a day. In June 2023, Chase customers alleged Zelle payments were debited twice in a network glitch; the duplicates were reversed the next day, and a class action followed anyway, over the overdraft and late fees the duplicates had triggered in between. Institutions with compliance departments larger than most startups still double-charge people, because the mechanics in this article are genuinely hard, and because the cost of the bug is never just the duplicate. It is everything the duplicate knocks over.
The dangerous outcome is not the charge that fails. It is the charge that quietly happens twice.
Webhooks lie about how many times they happened
A webhook is the processor telling you something moved. The trap is that every major provider delivers at least once, which is a polite way of saying sometimes more than once, and some retry for days. Engineers have been quoting Tyler Treat's 2015 essay for a decade on this: you cannot have exactly-once message delivery, and anyone selling it otherwise is wrong about distributed systems. What you can have is at-least-once delivery plus idempotent processing, which is the entire webhook discipline compressed into one sentence.
The discipline has four parts and they are not optional. Verify the signature against the raw request bytes before you parse anything, with a timing-safe comparison and a short timestamp window so a captured payload cannot be replayed. Persist the raw event keyed on the provider plus its event id, with a unique constraint, so a redelivery is recognized and dropped by the database. Acknowledge fast, in well under a second, by doing only that verify-and-persist before returning, then hand the real work to a background worker, because providers time out in seconds and a slow handler becomes a redelivery storm. And when the work fails, retry with backoff, then move the event to a dead-letter table and alert a human rather than letting it vanish or loop forever. The same reconciliation you do between your own records, the processor, and the bank is what catches anything that still slips through, and that is its own piece.
A charge is a state machine, not a boolean
The other quiet source of payment bugs is modeling a charge as a flag that says paid. A real charge moves through states: authorized, then captured, then maybe refunded, sometimes partially, sometimes disputed, sometimes reversed. Each transition has rules about what can follow it, and webhooks about those transitions arrive out of order, so the settled event can land before the created one. The fix is to process every event against the current persisted state of the charge, never against the order the events arrived in, and to guard each transition so an out-of-order or repeated event becomes a no-op instead of corruption. Declines deserve the same care. A soft decline like insufficient funds is worth a retry on a sensible cadence; a hard decline like a stolen card must never be retried; and blindly hammering both is how you turn a recoverable customer into a flagged merchant account.
One thing you minimize rather than solve is the card data itself. Tokenization and hosted fields keep the raw numbers off your servers entirely, which shrinks your PCI scope from a compliance project into a much smaller one. You do not want to be the system of record for a primary account number if you can possibly avoid it.
What's still standing in 2028
The rails themselves are moving too. In the UK, commercial variable recurring payments started flowing in 2026, with the UK Payments Initiative's first pricing framework setting the access fee at 5.5p per successful transaction, a shape of cost that undercuts card fees for recurring collection. In the EU, the final texts of PSD3 and the Payment Services Regulation were agreed in April 2026, with the rules generally applying about twenty-one months after official publication. New rails, same law of nature: every one of them will deliver events at least once, and the discipline in this article is what turns that into exactly-once processing.
The reason this matters more every year, not less, is that the parts of a payment flow AI is good at are the parts that were never the hard part. A language model will generate a checkout screen and a charge call in an afternoon, convincingly. What it does not generate is exactly-once, because exactly-once is not surface, it is invariants and edge cases and the discipline to put the gate in the database where races cannot get around it. A payment is a trust problem you reconcile, not an integration you call once and forget, and that distinction is the whole job. By 2028 the checkout screen is a commodity. The system that can prove it never charged a customer twice is not.
FAQ
Why do customers get charged twice? Almost always because a timeout was treated as a failure. The first request landed, only its response was lost, and the retry created a second charge because nothing recognized it as the same intent. A double-click, a redelivered webhook, and a timeout-then-retry are the same event wearing different clothes.
How does an idempotency key prevent double charges? Every money operation carries a key, and the table recording operations has a unique constraint on it. A retry claims the same key, conflicts, and gets back the original result instead of doing the work again. The guarantee comes from the database enforcing uniqueness under concurrency, not from application code checking and then acting.
Why do payment webhooks arrive more than once? Because providers deliver at least once, which in practice means sometimes more than once, and some retry for days. The discipline: verify the signature on raw bytes, persist the event keyed on provider plus event id with a unique constraint, acknowledge fast, process in the background, and treat event order as unreliable.
Is a charge just paid or unpaid? No. A real charge moves through states, authorized, captured, refunded, disputed, and the events about those transitions arrive out of order. Process every event against the persisted state of the charge, never against arrival order, and guard each transition so a repeat becomes a no-op instead of corruption.
Do big banks still double-charge customers? Yes, and publicly. Capital One customers saw debit purchases post twice across the country in January 2018, attributed to an internal technical issue, and Chase customers alleged double-debited Zelle payments in a June 2023 network glitch that drew a class action over the fees the duplicates triggered. Scale does not repeal the mechanics.
What is changing about payment rails in 2026? In the UK, commercial variable recurring payments began flowing in 2026 with an access fee of 5.5p per successful transaction under the first UKPI pricing framework, and the EU agreed final PSD3 and Payment Services Regulation texts in April 2026, applying roughly twenty-one months after publication. Every new rail still delivers events at least once.
What 2muchcoffee covers
We build and harden the money-movement layer underneath fintech products, the idempotency, the webhook discipline, the charge state machine, the reconciliation that proves it all tied out. If you are wiring up payments and the words "at least once delivery" just made you slightly nervous, that is the right instinct and the right time to talk, before a retry in production teaches you the same lesson the expensive way. The plain way in is the AI and engineering work we do.
One concrete action
Pick your charge endpoint and write down what happens if the exact same request arrives twice, a hundred milliseconds apart. If the honest answer is "two charges," add the idempotency key and the unique constraint before you add anything else, then prove it by firing the request twice and watching exactly one charge come out. That single gate is the foundation the rest of building fintech software stands on.
