Retry Policy

What is a retry policy, and how is it different from simply re-running on failure?

A retry policy is a set of rules deciding whether an automated workflow should automatically retry on failure, how many times, and how long to wait between each attempt — not unconditionally re-running immediately every time a failure occurs. A complete retry policy answers at least three questions: is this failure worth retrying at all (some failures won't be fixed by retrying), what's a reasonable retry limit, and how should the interval between retries be structured.

The biggest difference from simply re-running on failure lies in interval design and a cap mechanism. A simple re-run usually retries immediately on failure — if the problem is temporary (a brief network blip), retrying immediately might fail again before the network has had time to recover. A retry policy instead designs progressively widening intervals, giving the problem time to resolve on its own, and sets a retry cap — once exceeded, it stops and switches to notifying a human instead of retrying indefinitely.

What are the limitations of a retry policy, and which one is most commonly overlooked?

The most commonly overlooked thing is that not every failure is suited to being solved with a retry policy. A retry policy handles temporary failures with a real chance of self-recovery — a brief network interruption, the other end's server being temporarily overloaded. If the failure's root cause is broken logic — a judgment condition written backward, a data format that was wrong from the start — no number of retries changes the result; it's the same error every time. Retrying in this case just wastes time and compute; what actually needs fixing is the logic itself, not a bigger retry count.

The second commonly overlooked limitation is that more retries isn't always safer. If the retry count is set too high and intervals are stretched too long, a problem that should have been noticed immediately can end up dragged out and only genuinely discovered much later, because the system is still quietly retrying in the background. This kind of delay can be costly in some situations — a failed customer payment, for instance. A retry policy needs to balance giving the problem a chance to resolve on its own against letting someone know quickly if it persists, not simply maximizing retry count.

When should you design a retry policy, and when isn't it necessary?

The core situation suited to a retry policy is when an action depends on an external system, and that external system occasionally has temporary, non-fundamental issues. Calling a third-party API, connecting to a database, sending a network request — these actions occasionally fail from network fluctuations or the other end's system being briefly overloaded, but the problem usually resolves itself within a few minutes. In this situation, a retry policy automatically handles most temporary failures that don't need human intervention.

A retry policy isn't needed when the failure's cause is fundamentally something that won't change over time. Failing because input data was in the wrong format, or getting a wrong result because logic was written incorrectly — these failures produce the same result no matter how long you wait or how many times you retry. A retry policy is completely useless here; what actually needs to happen is fixing the root problem directly. A simple test: ask whether this failure could plausibly disappear on its own after a few minutes. If yes, a retry policy fits. If not, fix the logic directly instead.

How should advanced users design a retry policy to balance automatic recovery with timely detection of genuine problems?

The key move for advanced users is using exponentially increasing retry intervals, rather than a fixed or linearly increasing one. In practice, this means the first retry waits a short time (one minute, say); if it still fails, the second wait multiplies by some factor (five minutes); the third multiplies again (thirty minutes). This exponentially increasing interval design lets the system retry quickly right when a problem first occurs, when it's most likely to be a brief fluctuation. If failures persist, that signals the problem probably isn't brief, and progressively widening the interval both gives the problem more time to recover and avoids overloading the same system with a burst of retry requests in a short window.

Another advanced technique is designing 'retries exhausted but the problem persists' into an explicit escalation path, rather than letting it end in silence. In practice, this means pairing the retry policy with an escalation mechanism — once the retry cap is exhausted, it automatically triggers a notification including the specific error message from each failed retry during that window, letting whoever picks it up see at a glance where the problem likely lies, instead of just receiving 'this task failed' and having to dig through logs themselves to figure out what happened.