An AI agent finishes the weekly operations report before the manager opens her laptop. The draft uses an old target, attributes a revenue shortfall to a customer delay without showing its contribution, and leaves out an eligible account.
In this hypothetical workflow, the manager discovers the problems while preparing for a leadership meeting. She retrieves the approved target, reconciles account records and rewrites the explanation. The generation trace has already ended.
An evaluation that stops at draft completion misses that work. To assess whether the agent reduces workload, follow the report through review, acceptance and the questions it creates afterward.
Keep elapsed time and human effort separate
Start the measurement when the reporting assignment begins. Record draft availability and acceptance as separate events. Define acceptance against the reporting requirements: reconciled figures, the correct comparison period and explanations a reviewer can substantiate.
Elapsed time measures how long the assignment takes. Human effort measures the active time people spend collecting inputs, reviewing, correcting and resolving questions. A report waiting overnight for a reviewer increases elapsed time without consuming a night's labor. Two people investigating different discrepancies consume both people's time, even when they work simultaneously.
Record those measures separately. For human effort, sum active time across contributors without counting the same person's overlapping activity twice. Keep generation latency and model usage alongside them as diagnostic measures.
Make the comparison fair. Use equivalent reporting assignments and the same acceptance requirements for the manual and AI-assisted processes. A faster report with unreconciled figures has changed the quality standard. Also retain abandoned drafts and unsuccessful runs; counting only accepted outputs conceals the effort spent on failures.
Acceptance needs a timestamp and the accepted document version. If management later challenges a figure, attach the follow-up work to the same assignment. For periodic analysis, choose and disclose a follow-up window so both processes receive comparable observation. Mark reports that still have unresolved questions when that window closes.
Give each intervention enough context to investigate
A review record should help someone reproduce the problem. The following TypeScript type is a proposed starting point for evaluation, not a description of a deployed Coryntas feature:
type ReviewEntry = { reportId: string; draftVersion: string; contributorRef: string; activeMinutes: number; work: "verify" | "correct" | "business_decision"; finding: string; affectedRecordIds: string[]; }; activeMinutes records the contributor's active work on that entry. Waiting belongs in the assignment timeline. draftVersion identifies the document they inspected; otherwise a later rerun can make the original discrepancy impossible to reproduce. The work category describes the intervention, while the finding explains what happened. One report can require several entries.
Keep the relevant input versions available through the report's existing access controls. Record identifiers alone will not reproduce an error if the underlying records change. Avoid copying confidential account data into a broadly accessible analytics log.
For the wrong target, the finding should identify the target used and the approved reference that contradicts it. Then investigate whether retrieval selected an obsolete document, an instruction pointed to the wrong source, or the business had never settled which target applied. Those causes require different changes.
For the customer-delay explanation, a citation establishes where a claim came from. It does not establish how much of the shortfall the delay explains. Give the reviewer the affected amount, relevant dates, comparison basis and any remainder the explanation does not cover. If the evidence cannot support the attribution, leave the cause unresolved rather than turning a relevant note into a confident conclusion.
The omitted account requires a different check. Reconcile the report's included accounts against the eligible input set and record intentional exclusions. A polished narrative cannot reveal a record it never mentions. A second model reading only that narrative faces the same missing evidence.
This distinction builds on the review problem examined in Where the Work Goes After an AI Draft. The engineering task here is to make the interventions observable enough to distinguish a source problem, a reasoning gap and a decision that belongs to the business.
Test the correction on the next report and its exceptions
Fixing the document closes today's discrepancy. Before changing the recurring process, establish how far the correction should apply.
Suppose the manager excludes an account from one leadership discussion. That decision does not automatically change eligibility for every future report. Preserve its scope. If two owners disagree about the reporting target, record an unresolved business decision and route it to the responsible owner. A prompt edit cannot establish authority.
Use the three failures as evaluation cases. Supply an obsolete target alongside the approved one and check which reference the report uses. Supply a delay that explains only part of the shortfall and check that the remainder stays unresolved. Remove an eligible account from the draft and check that reconciliation detects it.
After a process change, rerun the relevant case and an exception where the correction should not apply. Include an earlier reporting period if the approved target changed over time. Passing the current week's case alone can hide a new historical error.
Recording every mouse movement would create another workload. Start with brief review entries and report-level timestamps; acknowledge that recalled time is approximate. Periodically compare those estimates with a directly observed session before treating small differences as meaningful.
The result may be less review effort, more capacity for analysis, or a workflow that still costs too much to check. Establish which outcome occurred at the same quality standard.
How do you measure the human work between an agent's first draft and a result your team accepts?
