Building Private AI Evals: Why Your Company Needs Them and How to Start

I've been fine-tuning language models since 2019. In that time, I've built document intelligence systems (Document Oracle - classifying over 60M documents), shipped evaluation tooling (RubricLLM), and worked in Accounts Receivable (AR) and ERP workflows where a wrong number costs real money.

Here's the practical lesson from enterprise AI: model choice matters, but teams cannot make a defensible workflow decision without measurement.

Most companies buy a model, wire up a workflow, watch the demo, and then stall. The agent produces output that's 90% right in the wrong direction. It drafts a customer email that's grammatically perfect and strategically wrong. Nobody catches it for weeks.

The problem isn't the AI. The problem is that nobody defined what "good" means for this specific workflow, and nobody built a repeatable way to check.

That's what an evaluation does. For finance leaders who care about controls, audit trails, and cost discipline, that measurement layer is a critical foundation for enterprise AI.

What an evaluation actually is

An evaluation, or eval, is a versioned measurement system that encodes your company's specific definition of acceptable work.

Think of it as a structured exam for your AI workflow. Each case in the exam has five parts:

a specific input: a task, a document, a request
the tools and context the agent is allowed to use
a reference answer or acceptable range
a rubric for scoring the output
a pass or fail result with evidence

Run all the cases, and you get a scorecard. Change the model, the prompt, or a tool, and you re-run the same cases to see whether quality improved, held steady, or regressed.

Not a vibe check. Not a demo. A repeatable, versioned measurement applied to a specific business task.

Public benchmarks vs your private eval

Public benchmarks tell you which model scores highest on math, coding, and reasoning problems designed by AI labs.

They don't tell you whether the model understands your customer's short-pay deduction reason well enough to correctly validate or reject it. They don't tell you whether the model will hallucinate a refund policy that doesn't exist. They don't tell you anything about your workflow, your data, or your definition of correct.

A benchmark is a public scoreboard. A private eval is an executable judgment about your specific work.

One critical distinction: "private" means company-controlled and company-specific. It doesn't automatically mean confidential, on-premises, or vendor-inaccessible. You control the purpose, the cases, the rubric, the access, and the stewardship. Sensitivity of individual data fields is a separate classification decision you make per field, not a blanket property of the suite.

Why this matters

Five business reasons make private evals worth funding. Each one maps to a question you're already asking.

Release confidence. Before you let an agent touch a real workflow, you need evidence that it meets or exceeds your standard. An eval suite gives you a repeatable gate: run the cases, review the failures, decide. Without it, every deployment is a gut call dressed up as a decision.

Cost control. The frontier model is expensive and often overkill for routine work. The obvious move is to route simple tasks to cheaper models and keep the expensive one for hard cases. But that only works if you can prove the cheaper model performs well enough on your tasks. Without evals, model routing is guessing. With evals, it's a cost decision backed by evidence.

Model switching. Vendor pricing changes. Models get deprecated. New providers enter the market. When you have a private eval suite, switching becomes a disciplined comparison: run both models against your cases, compare scores, and decide. Without it, switching is a blind swap that degrades quality quietly until someone catches an error six weeks later.

Auditability. Finance leaders need evidence trails. An eval suite produces versioned records: which cases ran, which model version, which rubric, what scores, who reviewed, and what decision was made. That's an audit trail. When auditors or regulators ask how you validated the AI before deployment, you have something concrete to show them.

Compounding institutional knowledge. Every production failure, customer complaint, and reviewer correction can become a new test case. Over time, the suite can become a defensible learning asset when it remains tied to real workflow traces, governance, and sustained use. New team members inherit the accumulated judgment of everyone who came before them.

A concrete example: dispute disposition in manufacturing AR

Let me make this tangible. I've worked in AR and payments workflows long enough to know that one of the most painful, high-volume processes in mid-market manufacturing is dispute resolution.

Here's the scenario. You're a discrete make-to-order industrial-component manufacturer. Revenue between $25M and $250M, 50 to 500 employees. You run Epicor Kinetic, Infor CloudSuite, IFS, or Plex. You maintain formal ISO 9001 or AS9100 quality processes. Your customers are OEMs.

A customer short-pays an invoice. They include a deduction reason: quality defect, quantity shortage, delivery damage, or pricing variance. Now someone in your AR team has to adjudicate that dispute. They pull the production order, the quality non-conformance record, the shipment and proof-of-delivery records, the bill of lading, and the pricing agreement. They compare the customer's claim against the actual evidence. Then they decide: validate the deduction, reject it with proof, or escalate.

This takes time. It takes institutional knowledge. And it takes getting right, because every wrong validation is a write-off and every wrong rejection risks a customer relationship.

Selecting the workflow

The agent reads the relevant ERP, QMS, and WMS records and produces one structured draft disposition:

The agent may research and draft. It may not send a response, post a payment, issue a credit memo, or adjust credit. System permissions disable those actions, and only authorized people may approve them.

Building the cases

You build an initial test library from historical disputes. Each test is a single closed dispute replayed from the initial short payment through the draft decision. You include the full evidence set: invoice, production order, quality record, production yield, bill of lading, proof of delivery, and shipment records.

For this example, the 84 tests form a pilot library, not a universal standard. The right number depends on the workflow's risk, the kinds of failure you need to detect, the financial importance of the decision, how cases are selected, and how much reviewer time is available.

The library needs to cover both common disputes and dangerous exceptions. That means valid and invalid quality-defect claims. Supported and unsupported shortage claims. Delivery-damage cases. Pricing-variance disputes. Cases with missing or conflicting evidence. Disputes above the materiality threshold. Bad-linkage traps, where similar IDs or mismatched remittances, could confuse the agent. Indirect prompt-injection cases are hidden in the retrieved content. Cross-customer confusion cases.

Deliberately difficult cases show whether the controls catch known risks. Do not use their pass rate to claim how often failures occur in normal production.

Ground truth

Two people, an AR specialist and the finance owner, review each historical case independently using the same records available to the system. They do not see the system's answer. After resolving any disagreement, their decision becomes the answer the eval uses for comparison.

For structured fields, you use exact labels. For the explanation, you don't write one golden paragraph and fail at anything that differs. You define required facts, prohibited claims, and evidence requirements. Two acceptable explanations may use different wording. The test is whether the output is grounded, complete, and decision-useful.

The rubric

Each scoring criterion names the business reason, the observable evidence, the scale, and the severity:

A hard fail is an error serious enough to block release. If the agent validates a deduction the finance team rejected, that's a blocker. If the dollar amount doesn't reconcile to the ledger, that's a blocker. If a cited record doesn't exist, that's a blocker. If the agent tries to use a forbidden tool or take an external action, that's a blocker.

Scoring and test groups

You divide the test library by customer and time, not randomly. Similar disputes from the same customer stay together. Otherwise, the system might appear to pass simply because it was tested on a near-copy of something used during development.

Use four groups. The team uses authoring cases to build the instructions and rules. It uses development cases to compare changes. Reserved release cases are used only for the final decision before deployment. A small sealed audit group is kept from the builders and used for periodic independent checks.

If the team examines a reserved release case and changes the system as a result, that case is no longer independent. Move it to the development group and create a new case.

Pilot

You start by replaying historical disputes in a safe test environment. The system can read approved copies of past records, but it has no production credentials and cannot change data, spend money, send notifications, or contact customers.

Every customer-facing response requires human approval during the pilot. Every dispute above the customer's materiality threshold requires the finance owner's approval. The AI system cannot issue write-offs or credit memos, post payments, or contact customers.

Monitoring and the incident-to-eval loop

Once the pilot begins, keep a secure record of every case the system handles. For each case, record what the system received, the information it relied on, which model and instructions were used, any tools it called, the result it produced, how long it took, what it cost, who approved it, and what happened afterward. Review a representative sample of live cases to make sure quality and safety hold up in normal use. Separately review high-risk cases, such as large-dollar disputes, missing evidence, or attempted unauthorized actions.

Then turn what you learn back into tests. For every confirmed incident, complaint, reviewer correction, or near miss, decide whether to add a new eval case, change a policy or tool, clarify the scoring rules, or document why the issue could not be reproduced.

"Noted in Slack" is not a learning infrastructure.

Five cases test the plumbing, not the reliability

Here's something I've seen teams repeatedly get wrong. They build five or ten test cases, watch them pass, and declare victory.

Five cases prove the measurement system runs. They prove the schema works, the scorer fires, the trace captures, and the rubric distinguishes pass from fail.

That's plumbing. Not reliability.

Here's why it matters mathematically. If you observe an 80% pass rate on 10 cases, the 95% confidence interval runs from roughly 49% to 94%. That's not a release signal. That's a coin flip with error bars.

To make a credible release decision, choose the test library size based on the workflow's risk, the errors you expect, the smallest change that would matter to the business, the cost of approving a bad system, the cost of rejecting a good one, and the reviewer's available time. Even 50 to 100 tests can miss rare but severe failures.

The 84-test library in this example is a starting point, not a certification. Revisit the scoring requirements after the first 50 real disputes have been independently reviewed.

A failed test is not a safety control

This distinction matters more than any other in this article.

When an eval reports a release-blocking error, it tells you that the system would have done something unacceptable in that test. The result does not prevent the same action from happening in live operations.

Prevention requires a separate permission or approval control at the point where the action would occur.

These controls operate at different layers. A test that reports an unauthorized action after it happened is not protection.

In the dispute workflow, the eval checks whether the system tries to post a payment or send an email. Actual prevention occurs through permissions: the write-off function is unavailable, email sending is blocked, and payment posting is blocked. The eval measures behavior. The permission system limits behavior.

Test both. Don't confuse one for the other.

Implementation checklist

If you're starting from zero, here's the sequence:

Select one recurring workflow. Name the business owner, the result that matters, the records that establish the truth, and the way to return to the manual process if needed.
Decide which company data the evaluation may use before collecting anything. Document the minimum fields, who may access them, how long they are kept, and how they are deleted.
Create a safe test environment using historical records or a simulation. Prove that the system cannot change production data, send messages, spend money, or use production credentials.
Collect 5 to 10 simple cases to prove the test format, activity record, scoring, and review process work correctly.
Expand the test library to cover common work, known failure modes, mandatory escalations, and actions the system must never take.
Have two independent reviewers score a sample. Resolve disagreements and update the written scoring rules.
Define the score required for each criterion, which errors block release, and the severity of each error. Report normal-use cases separately from deliberately difficult safety cases.
If an AI model helps grade subjective work, compare its decisions with those of human reviewers first. Track how often it incorrectly passes or fails each case type, not just its average agreement score.
Before live use, test alerts, emergency-stop authority, blocked actions, and the process for returning to manual work.
Begin with a small, reversible group of live cases only after the business owner approves the evidence. Keep human approval for decisions above the system's proven level of authority.
Turn every confirmed incident, complaint, and reviewer correction into a new test, a clearer rule, or a system change.
Run the full test library after every meaningful change and on a regular schedule. Refresh the independent audit cases when the workflow, customers, or policies change.

What this buys you

I built evaluation tooling because I kept seeing the same failure mode. Teams would demo a workflow, get excited, deploy it, and then discover weeks later that the agent was quietly producing output nobody trusted.

Private evals close that gap. They turn "we think it works" into "we have evidence it works, and here's what happens when we change something."

For a company leader, that's the difference between a speculative cost and a controlled, auditable, improvable system. The eval suite is the asset that compounds. Every cycle sharpens the definition of good. Every incident becomes a permanent regression test. Every new team member inherits the accumulated judgment of the people who built it.

The companies that build this layer keep their AI deployments alive. The companies that skip it stay stuck in pilot mode, wondering why the demo never made it to production.

Start with one workflow. Build the cases. Grade them. Then build more.