Don’t ship an AI workflow you can’t defend with versioned evidence

A capable AI demo can still be unsafe in production.

That gap is why technical teams need a private AI release packet.

Executives deciding whether to approve production exposure or more budget should start with What leaders should require before approving AI for production. This companion is the implementation side of that standard.

A release packet turns a company’s definition of acceptable AI-assisted work into versioned evidence. It shows what was tested, how it was scored, who reviewed it, where the system failed, what changed between versions, and what would happen if the workflow had to stop.

Private evals provide the measurement half of that packet. Runtime permissions, approvals, isolation, and rollback provide the control half.

Keep both halves visible.

The demo is not the operating standard

A polished pilot samples a narrow path. The input is clean, the task is familiar, and an engineer watches the run. When something goes wrong, the team corrects it before the result reaches a customer, employee, ledger, or production system.

Production removes that protection. An invoice arrives with a missing field. A support ticket contains instructions copied from an untrusted page. A policy changed last month, but the retrieval index still serves the prior version. An agent selects the right tool with the wrong account ID, and a plausible answer triggers an unauthorized write.

The final text may still look excellent.

Visible capability can coexist with weak operating evidence, and nobody notices until the cost lands somewhere that matters.

Agentic systems widen this gap because the output is only one part of the run. OpenAI’s agent-evaluation guidance describes traces that include model calls, tool calls, guardrails, and handoffs.[1] Microsoft’s observability guidance treats tool invocation, task completion, dependencies, latency, and production behavior as distinct signals.[2]

A release decision therefore needs more than a handful of impressive answers. It needs proof that the system reached the correct end state, used allowed tools with valid arguments, preserved user and tenant boundaries, escalated when policy required a person, stayed inside the approved cost and latency envelope, obtained approval for irreversible actions, and left a trace complete enough to reconstruct and reverse the run.

Public benchmark scores and vendor demos answer none of this. They can inform model selection. They cannot establish that an accounts-receivable agent reconciles your ledger, follows your approval policy, or avoids sending an unsupported explanation to a customer.

The operating standard has to come from the work itself.

A private eval measures your work

A private eval is a company-controlled, company-specific measurement system. Private describes control over the cases, purpose, rubrics, access, versions, and stewardship.

It does not automatically mean confidential, anonymized, on-premises, or inaccessible to the vendor. Data sensitivity, API retention, application state, observability, annotation, connectors, backups, region, and third-party processing still need separate review.[3]

For this operating model, a release-evidence case has six parts:

1. A unit of work.
2. Versioned input and context.
3. An expected result or acceptable range.
4. Scorers.
5. Thresholds and consequences.
6. Evidence.

Open-ended work rarely has one perfect golden paragraph. Consider an AI system that drafts a monthly revenue-variance explanation. Two drafts can use different wording, and both can be acceptable. The required standard is more concrete: every stated number reconciles with the ledger, material drivers are supported, uncertainty is labeled, customer data stays inside policy, and variances above the approved threshold are routed to the finance owner.

That standard is checkable.

The wording is not the point.

For an agent, the eval also records the path. A correct final answer can follow an unsafe tool call. A completed task can hide a missed approval. A cheap run can create hours of rework.

One overall quality score buries exactly these failures, which is why the scorecard should preserve five separate decision lines:

• Task quality: correctness, completeness, grounding, and acceptable end state.
• Security hard fails: unauthorized actions, data exposure, prohibited tool calls, missing approvals, and failed escalation.
• Evaluator validity: human agreement, judge agreement, false passes, false failures, and known bias.
• Cost and latency: tokens, tool calls, retries, completion time, and reviewer burden.
• Business outcomes: accepted work, rework, resolution, complaints, downstream loss, or another workflow-specific result.

The lines will conflict. A cheaper model may miss more exceptions. A safer policy may escalate so often that the workflow loses economic value. A fluent answer may score well even though it produces the wrong state transition.

Private evals make those tradeoffs visible.

The judge needs an eval too

Every score depends on a measurement instrument. When the instrument is unreliable, automation produces false confidence faster.

Use the simplest reliable evaluator for each criterion. Schema validation, arithmetic reconciliation, allowed-state checks, citation resolution, and permission allow-lists belong in deterministic code. Explicit policies can score required escalation or prohibited content. Model judges can help with groundedness, completeness, relevance, and other judgments that require interpretation. Subject-matter experts remain necessary for contested, high-consequence, or newly discovered cases.

Anthropic’s evaluation guidance draws the same ladder: code-based grading is fast and reliable for objective properties, human grading is flexible but expensive, and model grading scales only after its reliability has been tested.[4]

A model judge needs a contract. Pin the judge model and version, prompt, rubric, evidence supplied, and generation settings where the provider supports them. Keep the calibration cases separate from the cases used to write the judge prompt. Then test the judge against independently labeled human decisions.

Does it agree on ordinary cases?
Where does it pass a blocker that an expert rejects?
Does reversing the order of two candidate answers change the result?

Research on LLM judges has documented position, verbosity, self-preference, and reasoning limitations in the studied settings.[5] These findings do not create a universal correction factor. They create a control requirement: automated judgment must earn trust against the errors that matter in the workflow.

Calibration, done credibly, looks like ordinary measurement discipline. Blind the human labels where practical, anchor the rubric with explicit pass, borderline, and fail examples, and swap answer order on pairwise comparisons. Analyze false passes and false failures by criterion and harm tier, review a sample of automatic passes in high-risk groups, and record both human-human and judge-human agreement. When disagreement exposes an ambiguous standard, fix the rubric, not the reviewer. Then re-audit whenever the judge, policy, traffic, or target system changes.

Agreement statistics can help assess evaluator consistency, but their interpretation depends on the sampling design, category prevalence, weighting, and consequences of disagreement.[6] No single case count, correlation target, or audit percentage certifies an AI evaluator. Set the required evidence from the release decision, the frequency and cost of errors, the risk tier, and reviewer capacity.

Security and change control need separate gates

Quality and security answer different questions. A quality score asks whether the system did useful work. A security gate asks whether any tested behavior crossed a boundary the organization treats as non-negotiable.

Never allow an aggregate score to offset an unauthorized action.

The adversarial suite should test direct and indirect prompt injection, sensitive-data exfiltration, confused-deputy behavior, malformed or hostile tool output, unsafe arguments, cross-tenant access, loops, retry storms, and side effects that bypass approval.[7] Report the observed result by risk category: failures divided by tested cases, with the case design and coverage visible. Build a targeted adversarial suite to find dangerous behavior. Do not present its failure rate as the natural incident rate in production.

Keep measurement and control distinct here. An eval can detect that an agent attempted an unauthorized transfer. Scoped credentials, tool-side permission checks, human approvals, isolated test environments, network restrictions, and rollback mechanisms prevent or contain the action. A scorer that reports the transfer after execution is evidence of failure, not protection.

Security hard fails need an independent release gate and independent runtime enforcement.

The same discipline must govern system changes. Model routing, prompt edits, retrieval changes, new tools, policy revisions, and fine-tuning can improve one criterion while damaging another. Run the same versioned suite against the incumbent and the candidate. Keep authoring and development cases separate from held-out release cases, and when you use a held-out case to tune the system, move it into development and replace it with a fresh equivalent. NIST’s AI Risk Management Framework supports the broader practice of incorporating trustworthiness into AI design, development, use, and evaluation.[8]

Build one credible loop in 30 days

Start with one recurring, bounded, reversible workflow. Name the business owner, technical owner, release authority, rollback owner, source of truth, risk tier, and measurable outcome before collecting cases. Treat the sequence below as an operating recommendation, not a universal certification standard.

Days 1 to 5: define the boundary.

Map the inputs, outputs, tools, data sensitivity, approvals, prohibited actions, downstream systems, stop conditions, and rollback path.

Days 6 to 10: prove the plumbing.

Create five to ten smoke-test cases. Use them to verify the case schema, trace capture, versioning, a deterministic scorer, a rubric, and the reviewer workflow. These cases prove the measurement pipeline runs. They do not create credible release evidence.

Days 11 to 18: build a stratified suite.

Add representative traces, known failures, complaints, reviewer overrides, policy boundaries, subject-matter expert cases, and clearly labeled synthetic edge cases. Cover each common workflow type, mandatory escalation, prohibited action, and high-severity failure family.

Days 19 to 23: protect the test.

Separate authoring, development, held-out release, and sealed audit cases. Define required outcomes, acceptable ranges, hard fails, and thresholds per criterion and harm tier.

Days 24 to 27: calibrate the evaluators.

Run deterministic checks first, then policy rules, model grading where justified, and human adjudication. Compare automated judgments with blinded human labels. Inspect false passes on severe cases. Revise the rubric when disagreement reveals an unclear standard.

Days 28 to 30: rehearse the release decision.

Compare the incumbent and candidate on the held-out suite. Run the targeted adversarial cases separately. Test the approval, deny, alert, stop, and rollback paths in an isolated environment. Record cost, latency, reviewer load, and downstream outcomes without blending them into one score.

Production monitoring then begins a different evidence stream. Use production-representative sampling to estimate normal behavior when the sampling design supports it, and targeted review to discover rare, severe, novel, or adversarial failures. Keep the two reports separate.

Every confirmed incident, complaint, reviewer correction, override, or controlled injected defect should end with a disposition: add a case, change a policy or tool control, clarify the rubric, or document why the event cannot be reproduced. That loop turns the suite from a test artifact into a compounding asset.

At the end of day 30, the release owner should have one packet: the versioned suite, held-out results, security hard fails, judge calibration, cost and latency, reviewer capacity, business outcome measure, named decision rights, and a tested rollback record. If that packet does not support the decision, keep the workflow bounded and withhold the release or budget expansion until the missing evidence exists.

Sources

1. OpenAI, “Evaluate agent workflows”:
   https://developers.openai.com/api/docs/guides/agent-evals
2. Microsoft, “Observability in generative AI”: https://learn.microsoft.com/en-us/azure/foundry/concepts/observability
3. OpenAI, “Data controls in the OpenAI platform”: https://developers.openai.com/api/docs/guides/your-data
4. Anthropic, “Define success criteria and build evaluations”: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
5. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”: https://arxiv.org/abs/2306.05685
6. McHugh, “Interrater reliability: the kappa statistic”: https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
7. OWASP GenAI Security Project, “LLM01:2025 Prompt Injection”: https://genai.owasp.org/llmrisk/llm01-prompt-injection/
8. NIST, “AI Risk Management Framework”:
   https://www.nist.gov/itl/ai-risk-management-framework