Commercial risk
At-least-once is not an edge case — it is the explicit contract of every serious payment provider and message bus. Build as if delivery were exactly-once and the first busy day turns retries into duplicate charges, duplicate enrolments, and duplicate fulfilment, each one a refund and a support ticket.
The damage lands in money and in other people's trust: a customer's card charged twice for one purchase, a course granted twice, an order shipped twice. Every duplicate is a manual reversal, and enough of them is a chargeback rate that threatens the payment account itself.
Two facts about distributed systems collide here. First, payment processors retry webhook delivery — for days — until your endpoint acknowledges with a 2xx, because they would rather deliver an event twice than lose it. Second, message queues like Kafka or SQS guarantee at-least-once delivery, which is a polite way of saying "sometimes twice." Any handler that assumes it sees each event exactly once is built on an assumption the infrastructure explicitly refuses to make.
The correct defence is idempotency: processing the same event twice must produce the same result as processing it once. It sounds simple and is routinely implemented in ways that don't hold under the concurrency that triggers the problem. Here is where the naive versions fail and what actually makes a handler safe to run twice.
1. Check-then-act: the race hiding in “if not exists, insert”
The instinctive dedup is: look up whether we've seen this event; if not, process it. Between the lookup and the write there is a window, and under redelivery that window is exactly when the duplicate arrives. Two deliveries of the same event both run the lookup, both find nothing, both proceed to process — two payments recorded, two enrolments created. The check was real; it just wasn't atomic with the act it was guarding.
Teams often try to patch the race by catching the resulting unique-constraint violation — but frequently by matching on the database's error message string. That match is driver- and vendor-specific: a phrase that works on Postgres silently fails to match on MySQL, so after a database upgrade or a failover to a differently-configured replica, the catch stops catching and the exception propagates as a 500. The provider, seeing a non-2xx, redelivers — amplifying the very duplication the code was trying to prevent.
// Racy: the gap between exists() and create() is where the duplicate lands.
if (! WebhookEvent::where('signature_hash', $hash)->exists()) {
WebhookEvent::create(['signature_hash' => $hash]); // two deliveries both reach here
$this->process($event);
}
// And the fragile "fix" — matching a vendor-specific error string:
} catch (QueryException $e) {
if (str_contains($e->getMessage(), 'webhook_events_signature_hash_unique')) {
// works on one database, silently fails on another
}
}2. Dedup on the provider's ID, and make the insert the gate
Idempotency must key on an identifier the provider guarantees is stable per event — Stripe's `evt_...` event ID, not a hash of the payload. Payload hashing conflates two genuinely-identical-but-distinct events (the same amount charged twice legitimately) and, depending on serialisation, can differ for byte-level variations of the same event. The provider's event ID is the canonical dedup key; use it.
Then let the database do the mutual exclusion the check-then-act couldn't. Put a unique constraint on the event ID and make the insert itself the gate: attempt to insert the event row first, in the same transaction as the effect it authorises. If the insert succeeds, this is the first time — do the work. If it violates the unique constraint, this is a duplicate — stop. Detect that violation by catching the specific typed exception or using an upsert primitive (`INSERT ... ON CONFLICT DO NOTHING` and checking the affected-row count), never by reading an error string. The race is gone because insertion is atomic; there is no window.
// The insert IS the lock. First writer wins; duplicates no-op.
DB::transaction(function () use ($event) {
$inserted = DB::table('processed_events')->insertOrIgnore([
'provider_event_id' => $event->id, // Stripe's evt_..., the canonical key
'received_at' => now(),
]);
if ($inserted === 0) {
return; // already processed — idempotent no-op, still return 2xx
}
// Effect and its dedup record commit together, or not at all.
$this->applyPaymentAndEnrolment($event);
});3. The queue leg: commit the offset with the effect, not before
The provider's webhook is only the first hop. Frequently the handler records the payment and then publishes to an internal queue that a second service consumes to grant the entitlement — and that internal leg is at-least-once too. If the consumer creates the enrolment and then crashes before committing its queue offset, the event is redelivered and the enrolment is created again. The dedup discipline has to extend to every consumer, not just the edge.
The rule for each consumer is the same: make the effect idempotent on a stable key, and tie the acknowledgement to the effect. Don't commit the offset until the effect (including its dedup-row insert) has durably committed, so a crash replays into an idempotent no-op rather than a duplicate. Exactly-once delivery is a promise the infrastructure won't make; exactly-once effect is one you build, per consumer, by combining at-least-once delivery with idempotent, transactionally-gated processing.
Retries and at-least-once delivery are not failures of your providers — they are how they avoid losing your events, and they are guaranteed to happen. A handler that treats each delivery as unique will, on its first busy day, charge a card twice and enrol a student twice, then amplify the problem by 500-ing on a mismatched error string and inviting yet another redelivery.
Key idempotency on the provider's event ID, make a unique insert the atomic gate for the effect it authorises, detect duplicates by typed constraint violations rather than error text, and extend the same discipline to every downstream consumer by committing the offset only with the effect. Build for exactly-once effect on top of at-least-once delivery — because the second one is the only guarantee you actually get.