At 14:00 UTC on October 10, both connection poolers in front of Inngest’s primary database dropped their connections and could not get them back. The evening before, a data center provider had lost its network link between one of Inngest’s three data centers and the other two for 44 minutes. Inngest is a durable-execution service. You hand it a function written as steps, and it runs the steps, retries the ones that fail, and remembers which ones finished so a restart doesn’t redo them.
The incident report is a good one. Three clauses in its Root Cause section carry the whole thing, and I’ll take them one at a time.
“In both incidents, these internal failures counted against functions’ configured retries.”
A retry allowance belongs to the function’s author. They set it for the failures their own step can have: a flaky third-party API, a timeout, a bad deploy on the far end. The counter it feeds has one column, attempts. Nothing in it records who used the attempt up. For 44 minutes on the 9th and in two windows on the 10th, the platform used them. A step configured with three retries could run out of them with its own code never having failed once.
Temporal keeps the same kind of counter. Its server can’t see a worker crash. It waits for the start-to-close timeout to expire, schedules the activity again, and the attempt count goes up by one. A crashed worker and a bug in the activity each cost one attempt. Step Functions goes further than either. It names its errors, so a retrier can treat a Lambda service exception differently from an exception your own function threw. Read down that list, though, and every name describes something that happened to a state. Timeout, TaskFailed, Permissions, Runtime. None of them is a name for the engine being unavailable.
“a step that failed permanently on an internal error left its run Running instead of marking it failed”
A run’s status is a short list of words: finished, failed, still going, and a few others. The sentence turns on two of them. The engine died while holding the run, and the status it left behind said the function was still going. The field had no word for “I never got to try,” so it used the word for “taking a while.”
“so onFailure handlers and functions waiting on it with step.invoke were not notified”
This is where the first two clauses reach the tenant. A failure handler is the code the author wrote for the day something breaks, and it fires on failed. A function that called this one with step.invoke waits for a terminal status. Both were watching a field that said Running, and the field never changed. From inside the tenant’s own code, the platform’s outage and a slow step looked the same. Every signal the tenant could read pointed at their own function.
The fix list answers the second and third clauses. “A run whose step fails permanently because of an internal error is now marked as failed,” so the handlers fire. “If the run’s state cannot be read at that moment, the work stays queued and resumes once the state is available again.” Put those together and the status has grown a distinction it didn’t have before: failed, with the handlers told, or not yet attempted and still in the queue. Neither of those is Running.
The first clause gets no fix on the list. The report mentions retries once, in that sentence, and the fixes don’t come back to it.
In June I argued that the run dying partway is the median failure, and that every step has to survive being run twice. The report’s instruction for the stranded runs is to cancel and replay them, and “replayed runs start again from the beginning, so steps that already completed will run again.” That is the case I had in mind. What I had in mind was my steps. The thing that runs the steps has steps of its own, and when it fails at one of them, the record it keeps counts the failure against the step it was carrying.
So, a test for any pipeline that retries on your behalf, a vendor’s or one you built. Take last month’s retries and sort each one by whose failure it was: the step’s, or the thing running the step. Then open the run status enum and look for a value that covers the second pile. A platform that had internal failures last month and has no such value used up its tenants’ retries on itself, and its records say the tenants were slow.
By the only field a handler could read, the platform never went down on October 10. A lot of functions took a while.