Integration6 min read
Your webhooks will arrive twice
ARQON EngineeringARQON Singapore
Any integration works on the happy path. Whether you can forget it exists depends on how it behaves on a bad day.
Most of the integration bugs we're asked to fix weren't bugs on launch day. The code did exactly what it was written to do. What changed was the world around it. A vendor resent a webhook. A token expired over a long weekend. Two people edited the same customer record a second apart.
An external API is a distributed system you have no control over. Treating it like a function call that always returns is the most common reason integrations quietly corrupt data.
Assume every message arrives twice, or never
Most webhook providers promise at-least-once delivery, which is a polite way of saying duplicates are your problem. Every handler we write is idempotent: record the event ID before processing, and do nothing if the same ID shows up again. It costs one table and one index. Skip it and, now and then, an invoice goes out twice, and the first person to notice is your customer.
- Store an idempotency key for every incoming event and every outbound write.
- Space retries with exponential backoff and jitter, and stop at a ceiling.
- Send messages you can't process to a dead-letter queue, where a person can inspect and replay them.
- Never let a sync overwrite a newer value with an older one. Compare a version or timestamp before writing.
Drift will happen, so reconcile
Even a well-built integration drifts. Webhooks get dropped during a vendor outage. Someone edits the database by hand. A nightly job that compares both sides and reports the differences turns silent data corruption into a short list you can read over coffee. On a freight dispatch project, this report is why the planners came to trust the new system. Every difference arrived in writing, every morning.
Make failure visible
An integration without a dashboard hears about its problems from customers. At a minimum, track events received, processed and failed, queue depth, and the age of the oldest unprocessed message. If you set one alert, set it on that last number. It starts climbing before anything else goes wrong.
Filed under Integration · 2026.05.27
Back to the journal