When an automated step fails: retry, correct and run again
SynaptaGrid8 min read
In short
An automated process calls other systems, and other systems sometimes stop answering: a service is down, a record is missing, a limit is reached. A process that holds up shows where it stopped and has a way to continue for each kind of problem. A temporary failure, such as a timeout, can be retried after a delay. A permanent one, such as a missing field, needs a person to fix the data and run again. In both cases the failed step must be visible, the item must not be treated as approved, and the history must show what happened. Once the cause is fixed, the run continues from the failed step, or from an earlier one. Decide in advance which steps may be repeated safely, because a retry that runs twice can charge or send twice.
Two kinds of failure, two responses
Most failed steps fall into one of two groups. Telling them apart is what decides whether to wait or to ask a person.
| Option | Temporary | Permanent |
|---|---|---|
| Example | A timeout, a service that is briefly down, a rate limit | A required field is missing, the record does not exist, the credentials are wrong |
| Does repeating help? | Usually, after a delay | No, the same input fails again |
| Right response | Retry with a growing delay and a limit on attempts | Stop, show why, let a person correct the data, then run again |
| Who is told | Nobody, unless retries run out | The owner of the process |
Recovering from a failed step
Four rules that prevent the worst outcomes
- A failure never approves
- If a check could not run, the item is not approved. The safe default for an approval is to stop, not to pass.
- Make steps safe to repeat
- A step that creates a payment or sends a message should carry an identifier so that running it twice has the effect of running it once.
- Limit retries
- Retry a few times with a growing delay, then stop and tell a person. Retrying forever hides the problem.
- Show the cause
- The failed step and the error it returned should be visible to the person who has to fix it, without reading logs.
The old way and the new way
The table compares how failures are handled when a process runs by hand or by an unmonitored script with a process that shows its failures.
- A step fails
- The old way: Nobody notices until someone asks why nothing happened
- The new way: The run shows the failed step and its error
- The item
- The old way: Left in an unknown state, sometimes approved by mistake
- The new way: Not approved, with the reason visible
- Recovering
- The old way: Edit files and redo the work by hand
- The new way: Fix the record, then restart the failed step or run again
- Checking what happened
- The old way: Read logs or ask around
- The new way: A run history with the result of every step
How we decided
In the example we built and ran in our own workspace, a supplier approval failed at a step that checks a company register, because the register service returned an error. The run showed the failed step and the error, and the supplier record was not approved. After the record was corrected and saved, a new run started and passed the same step.
In that example we corrected the record and started a new run, which passed the same step. The run page also lets a person with the right permission continue without a new run: restart the failed step in the same run, rewind to an earlier completed step (the steps after it are marked as superseded), or replay the whole run with its original input as a new run.
We chose to show the failure and stop, not to pass the item on and warn later. The cost is that a person has to act when a check cannot run. The benefit is that no item is approved on the strength of a check that did not happen.
Standards and accepted practice behind this
These are the practices the guide rests on. We align with them. This page does not claim a certification.
- Fail closed
- When a control cannot run, deny and stop. Do not continue as though it had passed.
- Idempotency
- An operation that can be repeated with the same effect as running it once, usually by sending an identifier with it.
- Source: IETF: RFC 9110, section 9.2.2 (idempotent methods) (opens in a new tab)
- Exponential backoff
- Retry after a delay that grows, with a limit, so a struggling service is not overwhelmed.
- Dead-letter handling
- Keep what could not be processed, visibly, so a person can deal with it.
Where SynaptaGrid fits
SynaptaGrid Automation keeps a run history with the result of each step. A failed step is visible, and the item it was working on is not approved.
From the run page, a person with the right permission can restart a failed step in the same run, rewind to an earlier completed step, replay the run, or cancel it. The supplier approval example on the site shows a failed check and the recovery by correcting the record and running again.
Frequently asked questions
- What happens when a workflow step fails?
- A well-designed workflow stops at the failed step, shows the error, and does not treat the item as approved. A person then fixes the cause and restarts the failed step, or runs the process again.
- Should a failed step be retried automatically?
- Only if the failure is likely to be temporary, such as a timeout. Retry a limited number of times with a growing delay, then stop and tell a person. A permanent failure, such as a missing field, will fail again.
- How do I stop a retry from running an action twice?
- Make the step safe to repeat by sending an identifier with the action, so that a second attempt has the same effect as the first.
- Can I see why a run failed?
- You should be able to. In SynaptaGrid Automation the run history shows each step and its result, including the failed step.
SynaptaGrid
See a failed step and the recovery
Walk through the supplier approval, including what happens when a check fails.