Never mark a failed send permanent

A send fails. Your worker has the error in hand and has to make one decision: try again later, or give up on this message for good.

Most systems answer this by sorting errors into two buckets. Permanent errors are hopeless, so stop. Transient errors might work next time, so retry. It looks like a classification problem, and the instinct is to classify carefully.

That instinct is wrong, and it took me a while to see why. The two buckets are not symmetric. Getting one wrong costs you almost nothing. Getting the other wrong destroys a message and nobody finds out.

What each mistake actually costs

Suppose you mark something transient that was really permanent. The worker retries. It fails again. It retries a few more times, spends the attempt budget, and moves the row to failed. Total damage: four or five pointless attempts and a little log noise. The message still ends up exactly where it belonged.

Now suppose you mark something permanent that was really transient. The message is gone immediately. There is no retry, no second chance, and no alarm, because as far as the system is concerned it behaved correctly. You find out when a customer asks why they never got their password reset, if you find out at all.

Once you look at it that way, the design rule falls out on its own. Retrying is the safe default. Permanent is the exception, and the bar for it should be high.

The rule I settled on

Mark a failure permanent only when retrying it is incoherent, not merely unlikely to help.

Incoherent means the retry cannot possibly do anything different, no matter who does what in the meantime. Two examples from our own send workers:

• The queued row names a transport shape the code has no adapter for. There is no code path that could send it. Retrying runs the same missing branch forever.
• The row names a provider the tenant no longer uses. The credentials it needs are gone by definition. A retry is not waiting for anything.

Everything else stays retryable, including a lot of things that look hopeless.

The cases that feel permanent but are not

This is where it gets interesting, because the tempting mistakes all look like clean permanent failures.

A credentials key was rotated and the stored secret no longer decrypts. That will fail every single time until a human fixes it. It is still retryable, because a human might fix it. The repair probably will not land inside the retry window, and that is fine. We would rather burn the attempts and dead-letter the row than throw the message away in the five minutes before someone notices.

A sending domain is still waiting on verification. Same reasoning. Nothing about the message is wrong. Something about the account is temporarily incomplete.

The tenant has not configured a transport at all. Also retryable. An empty configuration is a setup step someone has not done yet, not a statement about this message.

The one that surprises people

A bad recipient address feels like the most obviously permanent failure there is. If the mailbox does not exist, why would you ever try again?

We deliberately do not treat it as permanent, for a simple reason: at that layer we cannot tell the difference between a bad address and a provider having a bad minute. Both surface as a rejection at send time. Providers return soft failures that read like hard ones, especially under load.

The authority on whether an address is real is the provider's own asynchronous feedback, the bounce and rejection events that arrive later and say so explicitly. That signal is trustworthy. A rejection during a send attempt is not, so it gets the same benefit of the doubt as everything else.

What retrying should actually look like

Defaulting to retry only works if retrying is well behaved. Three things make it so.

Back off. Our failed rows come due again after roughly one minute, five minutes, fifteen minutes, then an hour. A tight loop against a struggling provider makes the outage worse and burns the budget before the provider recovers.

Add jitter. We spread each delay by about a quarter either way. Without it, an outage that fails a thousand rows in one minute produces a thousand rows that all come due in the same instant, and the recovery attempt becomes a second outage.

Bound it. Five attempts, then the row is dead-lettered. Retrying forever is not kindness, it is a queue that never drains and a signal you can never act on.

Make failed mean failed

One last piece, which matters more than it looks. In our system a failed row is terminal. It is never claimed again by anything.

That sounds like a small consistency point. It was actually what let us add retries to a queue that was already running in production. Every row already sitting in failed kept exactly the meaning it had before the change. Nothing woke up and re-sent a two week old notification to a confused customer.

If failed sometimes means "we might get back to this", you can never reason about the state, and you certainly cannot change the retry logic underneath a live queue.

The short version

• Retryable is the default. Permanent is the exception you have to argue for.
• Permanent means a retry is incoherent, not that it is unlikely to work.
• If a human could fix the cause, it is transient, even if they probably will not fix it in time.
• Do not let the send attempt decide whether an address is real. The provider's feedback decides that.
• Back off, jitter, and bound the attempts.
• Failed is terminal, always.

None of this is clever. It just comes from asking a different question. Not "is this error permanent?", which is often unknowable, but "which way do I want to be wrong?", which usually has an obvious answer.