When an automated step fails: retry, correct and run again

SynaptaGrid8 min read

#workflow#reliability

In short

An automated process calls other systems, and other systems sometimes stop answering: a service is down, a record is missing, a limit is reached. A process that holds up shows where it stopped and has a way to continue for each kind of problem. A temporary failure, such as a timeout, can be retried after a delay. A permanent one, such as a missing field, needs a person to fix the data and run again. In both cases the failed step must be visible, the item must not be treated as approved, and the history must show what happened. Once the cause is fixed, the run continues from the failed step, or from an earlier one. Decide in advance which steps may be repeated safely, because a retry that runs twice can charge or send twice.

Two kinds of failure, two responses

Most failed steps fall into one of two groups. Telling them apart is what decides whether to wait or to ask a person.

OptionTemporaryPermanent
ExampleA timeout, a service that is briefly down, a rate limitA required field is missing, the record does not exist, the credentials are wrong
Does repeating help?Usually, after a delayNo, the same input fails again
Right responseRetry with a growing delay and a limit on attemptsStop, show why, let a person correct the data, then run again
Who is toldNobody, unless retries run outThe owner of the process

Recovering from a failed step

The failed step is shown and the item is not approved. After the cause is fixed, the step is restarted in the same run, or a new run is started, and it passes the same step.

Four rules that prevent the worst outcomes

A failure never approves
If a check could not run, the item is not approved. The safe default for an approval is to stop, not to pass.
Make steps safe to repeat
A step that creates a payment or sends a message should carry an identifier so that running it twice has the effect of running it once.
Limit retries
Retry a few times with a growing delay, then stop and tell a person. Retrying forever hides the problem.
Show the cause
The failed step and the error it returned should be visible to the person who has to fix it, without reading logs.

The old way and the new way

The table compares how failures are handled when a process runs by hand or by an unmonitored script with a process that shows its failures.

A step fails
The old way: Nobody notices until someone asks why nothing happened
The new way: The run shows the failed step and its error
The item
The old way: Left in an unknown state, sometimes approved by mistake
The new way: Not approved, with the reason visible
Recovering
The old way: Edit files and redo the work by hand
The new way: Fix the record, then restart the failed step or run again
Checking what happened
The old way: Read logs or ask around
The new way: A run history with the result of every step

How we decided

In the example we built and ran in our own workspace, a supplier approval failed at a step that checks a company register, because the register service returned an error. The run showed the failed step and the error, and the supplier record was not approved. After the record was corrected and saved, a new run started and passed the same step.

In that example we corrected the record and started a new run, which passed the same step. The run page also lets a person with the right permission continue without a new run: restart the failed step in the same run, rewind to an earlier completed step (the steps after it are marked as superseded), or replay the whole run with its original input as a new run.

We chose to show the failure and stop, not to pass the item on and warn later. The cost is that a person has to act when a check cannot run. The benefit is that no item is approved on the strength of a check that did not happen.

Standards and accepted practice behind this

These are the practices the guide rests on. We align with them. This page does not claim a certification.

Fail closed
When a control cannot run, deny and stop. Do not continue as though it had passed.
Idempotency
An operation that can be repeated with the same effect as running it once, usually by sending an identifier with it.
Source: IETF: RFC 9110, section 9.2.2 (idempotent methods) (opens in a new tab)
Exponential backoff
Retry after a delay that grows, with a limit, so a struggling service is not overwhelmed.
Dead-letter handling
Keep what could not be processed, visibly, so a person can deal with it.

Where SynaptaGrid fits

SynaptaGrid Automation keeps a run history with the result of each step. A failed step is visible, and the item it was working on is not approved.

From the run page, a person with the right permission can restart a failed step in the same run, rewind to an earlier completed step, replay the run, or cancel it. The supplier approval example on the site shows a failed check and the recovery by correcting the record and running again.

Frequently asked questions

What happens when a workflow step fails?
A well-designed workflow stops at the failed step, shows the error, and does not treat the item as approved. A person then fixes the cause and restarts the failed step, or runs the process again.
Should a failed step be retried automatically?
Only if the failure is likely to be temporary, such as a timeout. Retry a limited number of times with a growing delay, then stop and tell a person. A permanent failure, such as a missing field, will fail again.
How do I stop a retry from running an action twice?
Make the step safe to repeat by sending an identifier with the action, so that a second attempt has the same effect as the first.
Can I see why a run failed?
You should be able to. In SynaptaGrid Automation the run history shows each step and its result, including the failed step.
Share

SynaptaGrid

See a failed step and the recovery

Walk through the supplier approval, including what happens when a check fails.