Build a Failure-to-Control Map before you automate

Three purple workflow samples sit in inspection bays: one intact, one split, and one amber-marked under a hinged check arm beside an evidence rail.

Before you automate a workflow, inspect the last three times a human rescued it.

Use those rescues to decide which controls your workflow needs and what evidence would justify keeping them.

Key Takeaways

An agent harness is the operating layer around a model that supplies tools, context, state, permissions, checks, and recovery.
A better model doesn't choose your authoritative records or approval rights. Those decisions need explicit controls outside its answers.
A Failure-to-Control Map links a rescued handoff to a testable check, a named owner, pass evidence, and a stop-and-recovery path.

Find the email, spreadsheet, or approval where work stopped. Name who stepped in, what they had to reconstruct, and what the delay cost. Start with the work your team already struggles to finish.

A Failure-to-Control Map connects an observed failure to the smallest check worth testing, a named owner, and evidence of a pass. Each entry also records where work must stop and how a person can recover it.

The map gives an operations leader a way to decide what to fix before buying more automation. It also gives the implementer a testable requirement. “Make the workflow reliable” becomes “pause when the required source record is missing, and show the reviewer which record couldn't be found.”

Every added control should address a specific failure.

Why a better model can leave the failure intact

A stronger model can improve extraction, reasoning, and drafting. Test that possibility when the output itself is wrong. Don't assume every failure belongs outside the model.

But a model change doesn't decide which company record is authoritative. It doesn't grant data access, appoint an approver, or prove that a requested action reached the destination system.

Consider an assistant preparing a customer response from an old policy. Better writing leaves the stale rule intact. The required change is to retrieve the applicable policy and check the answer against it.

An instruction to seek approval also isn't an approval boundary. If the system can send without a recorded decision, the instruction leaves a route around the rule. Put the stop at the action itself.

Diagnose the missing capability before changing the model. Otherwise, you can't tell whether the next successful run came from better reasoning, fresher records, or a person quietly repairing the output again.

Trace the handoff before choosing the control

Step 1. Bound the job

Choose a repeated handoff with accessible records, a visible consequence, and a human owner. Define its trigger and its end state. Preparing a packet for approval is a different job from approving it or sending it externally.

Keep the first scope narrow enough to inspect end to end.

Step 2. Reconstruct the rescues

Use the last three rescues as a practical starting sample, not a statistical benchmark. Read the underlying records with the person who intervened. Include a normal successful run for contrast.

Record what was available when work stopped, what the person supplied, and what happened afterward. Separate evidence from recollection. If nobody measured the delay, mark it unknown or estimated rather than inventing a baseline.

Step 3. Write the failure precisely

Replace “the AI got confused” with an observable event. “The draft used a superseded policy” suggests a different control from “the current policy was available but interpreted incorrectly.”

Ask what the proposed check could have detected at the time. Don't require information that only became available after the rescue.

Step 4. Define the pass and the stop

Choose a control from the matrix below. Name the person responsible for the rule and the person who handles a blocked case. Specify the expected result before testing.

Include a deliberate failure case. A missing approval should pause the case, not provide a plausible reason to proceed. Test the boundary in an isolated environment without risking a live external action.

Step 5. Compare complete runs

Replay the saved case safely, then try the control on a bounded live run with human oversight. Track rework, elapsed delay, review effort, and recovery effort. Record unnecessary stops as well as missed failures.

A faster draft that makes the reviewer reconstruct everything hasn't completed the job faster. Keep, revise, or remove the control based on the whole handoff. Don't confuse one passing case with proof of general reliability.

The control and evidence matrix

Use the following entries as a starting matrix, not a shopping list. Each keeps the same fields so an operator can compare failures without translating between technical feature names.

For every real entry, attach the run identifier, trigger, affected systems, source links, observed consequence, rule version, and test date. Restrict those records to authorized reviewers. Preserve the source version or permitted snapshot used in the decision, not just a link that changes later.

Wrong rule, template, or policy

Control: Retrieve the applicable source and require the output to cite the rule it used.
Owner: The policy owner approves which version applies to the task.
Pass evidence: Source version, cited passage, output, and owner verdict agree on the applicable rule. A citation alone doesn't prove correct interpretation.
Stop and recovery: Pause on conflicting rules. The owner resolves applicability before work resumes.

Missing or stale record

Control: Read the authoritative system and check required fields and freshness against the owner's standard.
Owner: The record owner defines what must be present and how current it must be.
Pass evidence: The packet includes the source link, last-updated time, retrieval time, and field-check result.
Stop and recovery: Missing or stale required data blocks progression. Route the missing item to its owner without guessing a replacement.

Work crossed an approval boundary

Control: Prevent the external action until the designated approver records approval for the exact proposed action.
Owner: A named approver holds the decision right.
Pass evidence: The approval record identifies the reviewed version and action. The test without approval shows that execution was blocked.
Stop and recovery: Missing approval means no action. Changed content returns for approval.

The same exception is returned

Control: Save the approved correction as a versioned rule and a reusable test case.
Owner: The domain owner approves the correction and its scope.
Pass evidence: The saved exception passes without re-teaching. An ordinary case still passes without triggering the exception rule.
Stop and recovery: Send unmatched cases to the owner. Don't turn one unusual correction into a universal rule.

Work stopped or duplicated an action

Control: Record workflow state and check for an existing completion before retrying. Use a stable action identifier so a retry refers to the same intended action.
Owner: The workflow owner defines recovery with the system maintainer.
Pass evidence: A safely interrupted test resumes with one confirmed completion and no duplicate action.
Stop and recovery: An uncertain destination result triggers reconciliation before retry. A timeout isn't proof that nothing happened.

Review required full reconstruction

Control: Assemble a review packet containing source citations, proposed output, exceptions, and the requested decision.
Owner: The reviewer defines the evidence needed to decide.
Pass evidence: The reviewer can approve or reject from the packet. Record review time and any sources they still had to reopen.
Stop and recovery: Return incomplete packets with the specific missing evidence. Don't bury uncertainty in a polished summary.

Worked example

An operations team prepares supplier renewal packets. The purchasing system holds the contract, a shared folder holds supporting documents, and an operations manager approves the packet before submission.

During a recent handoff, an assistant copied an insurance expiry date from a spreadsheet. The underlying certificate had expired. A coordinator caught the mismatch during review and rebuilt the packet from the source documents.

The failure statement is precise: the packet treated a copied date as current evidence. The first control to test is a source check before the packet reaches approval. Submission remains outside the assistant's authority.

The hypothetical map entry reads:

Source of truth: The certificate in the approved document folder, matched to the supplier record.
Control: Compare the certificate's expiry date with the required coverage date. Missing evidence or insufficient coverage blocks the packet.
Owner: The supplier-record owner obtains a replacement. The operations manager retains approval authority.
Evidence: Supplier identifier, certificate link and version, retrieval time, both compared dates, check result, and reviewer decision.
Acceptance test: The saved expired-certificate case blocks. A valid-certificate case reaches review with its evidence attached. Neither case submits anything externally.

Replay the original case without writing to the purchasing system. Check that the packet exposes the expired certificate and identifies the recovery owner. Then test missing evidence and a normal valid certificate.

During the bounded live trial, record preparation and review effort separately. If the reviewer still opens every document to establish what happened, the packet needs work. If valid certificates keep blocking, inspect the rule or extraction before expanding the workflow.

No new platform is required to test this decision. The team can begin with a checklist and source-linked packet.

When to use it, and what not to build

Use the map when repeated handoffs require manual rescue and the team can inspect the relevant evidence. It is especially useful before expanding an assistant from drafting into actions with consequences.

Don't force it onto a one-off task, inaccessible records, or work nobody owns. Resolve ownership and access first. If a single model call plus human review meets the quality and cost requirements, stop there.

Recent failures also leave blind spots. Three rescues won't expose every rare or severe risk. Keep mandatory security, legal, and approval controls even when no incident has occurred.

The map guides targeted improvements, not permission to wait for preventable harm.

Avoid these failures in the implementation:

Prompt-only controls: Reminders don't replace enforced permission checks or independent verification.
A second source of truth: Keep links and decision evidence without creating an unsynchronized copy of the source records.
Automatic exception learning: Require owner approval before a correction changes future behavior.
A platform built from the matrix: Don't add memory, multiple agents, or custom orchestration merely to fill every row.

Before the first trial, confirm the implementation checklist:

The trigger, end state, and human owner are explicit.
The failure record distinguishes observed facts from estimates.
The control has a pass case, a block case, and saved evidence.
The action boundary and recovery owner are tested.
Review, false stops, and recovery count toward the cost.
The owner will decide whether to keep, revise, or remove the control after the trial.

Start with one rescued handoff. Write down what broke, choose the smallest control worth testing, and decide what evidence would justify keeping it. Build from that result.