Evals and Human-in-the-Loop: How I Ship AI Features Without Gambling Trust

The fastest way to burn an AI product is to confuse model fluency with product readiness. Fluency ships demos. Trust ships businesses.

I treat evals and human-in-the-loop (HITL) as core product engineering — the same category as auth, payments, and on-call — not as a research nice-to-have. This essay is the playbook I use when taking AI features from customer request to production ownership.

Autonomy is earned, not configured

A useful ladder:

1. Assist — draft / suggest; human always commits
2. Prepare — create pending objects; human approves
3. Act in sandbox — side effects only in reversible environments
4. Act with policy — narrow auto-actions with kill switches
5. Broad autonomy — rare; requires strong evals + monitoring

Most 0→1 products should live in 1–2 for longer than founders want. That restraint is a feature.

Start from the customer failure mode

Before writing an eval set, write the nightmare:

• Wrong owner assigned on a customer commitment
• Email sent with confidential data
• Fake “decision” becomes a roadmap item
• Medical / legal / financial hallucination phrased confidently

Your HITL boundary should sit directly in front of that nightmare. Everything else can be more automated.

Offline evals: make regressions boring

Minimum viable eval harness:

• Golden inputs — real anonymized samples, not toy sentences
• Expected structured outputs — JSON fields that matter to the product
• Graders
  • schema validity (hard fail)
  • exact / fuzzy match on critical fields
  • citation presence
  • refusal correctness when evidence is missing
• Baselines — last prompt/model that shipped

Run on every change to prompts, tools, routers, or packing logic.

What good golden sets look like

• Include ugly reality: interruptions, crosstalk, multilingual snippets, empty docs
• Label unknown cases where the correct behavior is to abstain
• Keep a “shark tank” set of historical production failures
• Version datasets like code

Quantity helps; representative failure diversity helps more.

Online metrics: the product’s truth

Offline scores can look great while users hate you. Track online:

• Acceptance rate — approved without edits
• Edit distance — how much humans rewrite
• Override reasons — taxonomy beats free text alone
• Time-to-approve — friction signal
• Escape rate — users abandoning the AI path
• Cost per accepted outcome
• Incident rate — severe wrongness classes

These metrics close the ownership loop after deploy. They tell you whether to widen autonomy or tighten the contract.

HITL UX is part of the model system

A review screen is not a form. It is a reasoning prosthesis:

• Show evidence next to claims
• Highlight low-confidence / changed fields
• One-click approve for clean cases; surgical edit for messy ones
• Capture why the human changed something (feeds the shark tank)

If review is painful, humans rubber-stamp — and you lose the safety you thought you bought.

Policy gates beat prompt pleas

Do not rely on “please be careful” in the system prompt for dangerous actions. Use deterministic gates:

• Allowlists of tools per workflow stage
• Max spend / max sends per run
• Required citations for external communication drafts
• Hard block on actions when schema validation fails
• Environment flags: HITL_REQUIRED=true until metrics clear a bar

Prompts suggest. Policies enforce.

Model-graded evals: use carefully

LLM-as-judge is useful for fuzzy quality (tone, summary usefulness). It is dangerous for factual correctness about your domain unless tightly constrained.

My stance:

• Deterministic graders for fields that must be right
• Model graders for subjective quality — with spot human audits
• Never let a judge silently become the only release criterion

Shipping cadence that protects trust

A release train I like for AI features:

1. Shadow mode — model runs, humans don’t see it; compare offline
2. HITL mode — model proposes; humans approve
3. Partial auto — only high-confidence narrow actions
4. Expand — when acceptance metrics hold for a defined window

Skipping steps is how “we’ll fix it later” becomes a churn event.

Connecting evals to product discovery

Customer requests often sound like capability asks (“auto-update Jira”). Evals help you renegotiate:

• If golden-set accuracy on assignee extraction is 71%, the honest product is propose assignees, not auto-assign.
• If humans accept 95% of summaries but rewrite action items 60% of the time, invest in action-item evals — not a flashier chat UI.

This is AI product engineering: design the feature to the measured capability, then improve the capability with a feedback loop.

A concrete example shape

For meeting → tasks:

| Stage | Automation | Gate | |---|---|---| | Transcript normalize | Auto | schema | | Candidate tasks | Auto | schema + evidence span | | Task create in workspace | HITL | human approve | | Notify channel | Auto after approve | idempotent send |

Evals focus on candidate quality and evidence linking. Online metrics focus on approval/edit rates. Autonomy expands only when both stay healthy.

Anti-patterns

• Evaluating only happy-path demos
• Using win-rate chatbots as product QA
• Hiding uncertainty to look “smart”
• No shark tank of past incidents
• Removing HITL because “the model got better” without metric proof
• Logging prompts with PII into eval sets carelessly

Ownership after launch

When feedback arrives, I want a path:

complaint → reproduce on golden/shark set → fix packer/prompt/tool/policy → eval gate → staged rollout

That path is the difference between a team that ships AI and a team that operates AI products.

Closing

Evals tell you what the system can do. HITL decides what the product is allowed to do today. Together they let you move from 0→1 without gambling the user’s trust for a demo metric.

If you are hiring for Senior AI Product Engineering, this is the muscle to interview for: not who can call the newest model API, but who can design capability contracts, prove them, and own the aftermath when reality disagrees.