If you’ve ever tried pushing thousands of translation updates from Lokalise all at once and ended up in a web of failed webhooks, you’re not alone. Things can get messy fast. But today, we’re going to explain what went wrong and how something called the *Replay + Idempotency pattern* saved the day. Don’t worry—it sounds complex, but it’s actually pretty cool.
TL;DR
When doing bulk pushes from Lokalise, some webhooks failed due to timeouts and overload. Instead of manually tracking and fixing those, a pattern called Replay + Idempotency made it possible to safely retry them without causing duplicate data. Each request was uniquely identified and safely ignored if already processed. The perfect combo of *retry logic* with *no double trouble*.
What happened: A webhook traffic jam
It all started with a bulk push. Imagine thousands of translation strings getting shipped out to your app via Lokalise webhooks. Smooth, right? Well, not quite.
The receiving system had a hard time keeping up. Webhooks started to fail. Some got timeouts. Others were dropped completely. Lokalise saw red. Logs were full of errors. And worst of all — translations were going missing.
This is how a bulk push went from helpful to hazardous. It wasn’t that the data was bad. It just came too fast.
Why do webhooks fail during a bulk push?
Here are the common reasons:
- API rate limits: Too many requests hit the server at once.
- Timeouts: Processing takes too long, and Lokalise gives up.
- Network errors: Temporary blips, slow connections, or packet loss.
- Backpressure: The receiver can’t keep up, and drops requests.
The result? Some translation strings arrived. Others didn’t. Re-running the push seemed risky. What if the system ended up with duplicates? That would be even worse.
Attempt #1: The Band-Aid Approach
At first, the fix was manual. Failed webhooks were logged. Devs retried them one by one. It was painful and slow. Also prone to human error.
Then came an idea: what if we could re-send everything—but also make sure no duplicates were created? Enter the power duo: Replay and Idempotency.
What is “Replay + Idempotency”?
Let’s break this down simply.
- Replay: Try again! Resend the same request if it fails. This is your safety net.
- Idempotency: Make requests “safe” to repeat. If you’ve already processed it once, ignore it the next time. No double import. No duplicate damage.
If your translation import system supports this, you can hammer it with repeated webhook calls. No stress. No mess.
How idempotency works in this setup
Every webhook from Lokalise includes data about the translation string. To make it idempotent, the receiving server needed to do two things:
- Generate a unique ID for each import request. This could be a combination of:
- Translation key
- Language code
- Lokalise project ID
- A version or timestamp
- Check if the ID has already been processed before doing anything.
If the request is new? Process and import it. If not? Skip it.
This system worked wonders. Failed webhooks could be retried as many times as needed. And the destination app stayed clean. One translation per key. Always.
So what does the replay pattern look like in practice?
Replay isn’t just hitting a button labeled “Try again.” It has rules. Good replay patterns include:
- Fail logs: Know which webhooks didn’t go through.
- Dead-letter queues: Automatic holding zones for failed attempts.
- Retry logic: Set intervals before trying again, maybe exponential backoff.
- Monitoring: Alerts or dashboards showing if retries are working.
The replay only works safely if paired with that magical idempotency behavior. Otherwise, it can become a “duplicate emitter.” Nobody wants that.
Lessons learned
The team realized a few things during this webhook disaster that might help others:
- You can’t trust bulk operations to always deliver every webhook.
- Replay + Idempotency should be designed in from the beginning.
- Handling failed webhooks manually is madness. Automate it.
- Monitoring matters. Visibility makes debugging 10x easier.
Bonus: What would have happened without idempotency?
This is where things really get spicy.
If a failed webhook was simply retried without checking if it was already processed, some apps imported the same translation twice. Or worse, the second import overwrote newer content. Data inconsistency was the nightmare scenario.
Duplicate records. Confused translators. Angry users.
This is why idempotency was more than a nice-to-have. It was damage control.
Going forward: the improved webhook strategy
Based on this webhook chaos, the system now uses the following pattern:
- Async webhook processing: Webhook data is stored first, then processed in the background.
- Unique import keys: Each webhook includes a unique deduplication token.
- Dead-letter queue: Failed webhooks are captured for retry without loss.
- Auto retry with limits: Webhooks are retried automatically up to 3 times, with backoff.
- Monitoring dashboards: Any failure spikes are visible in real time.
Now, even if there’s a big bulk push from Lokalise, the system stands tall. Fast, clean, and reliable.
Conclusion
Those webhook failures during a bulk push from Lokalise were frustrating, but they sparked some brilliant defenses. The key to survival was pairing *retries* with *idempotency*. Together, they allowed for unlimited safe replays without any unwanted surprises.
So the next time your system has to eat a spaghetti bowl of webhooks, remember: replay is fine, as long as it’s idempotent.
That’s not just smart engineering. That’s peace of mind.
Image not found in postmeta