Compass was built around a commitment to look at every customer conversation and understand what happened in it. For most of the history of customer-facing operations, that was operationally impossible, so sampling and scorecards did the work that could be done. Those tools are still useful. Scoring is how most QA programs enter the category and how leadership tracks performance over time. What is changing is what the scores are built on. More of the conversations happening inside customer-facing operations are now handled by AI agents, assisted by AI agents, or routed by AI agents. The AI agent performance metrics most teams track today were inherited from a human-agent rubric, and the scores leadership relies on only hold up if the signals underneath them see what AI agents actually do.
Most teams deploying an AI agent in a customer-facing role start evaluation in the same place. They take the QA rubric they already use for human agents and point it at the new one. Script adherence. Empathy. Resolution. Handle time. The rubric is familiar, the reports look consistent, and the AI agent produces scores that fit cleanly into the dashboards leadership already reads.
Much of that rubric does carry over. The underlying operational question is the same: did the agent handle this conversation in a way that served the customer and the business. Resolution still matters. Scope still matters. Tone still matters. The categories transfer. What has to change is how each category is measured when the thing being measured is a system rather than a person.
Script adherence is a weak signal for a system that is incapable of going off-script the way a human can, but it is still worth tracking as a baseline. Empathy scoring picks up tone, which an AI agent produces consistently regardless of whether the underlying response is correct, so it has to be paired with a signal that catches the moments where consistent tone masks an incorrect response. Resolution counts a closed conversation, which an AI will claim more often than a human will, so the definition of a closed conversation has to be tightened. The rubric does not need to be thrown out. It needs new signals added where the old ones go blind, and existing signals interpreted differently where AI behavior is not comparable to human behavior.
A human agent who doesn't know the answer behaves in observable ways: slower cadence, hold time, visible escalation. AI agents do not produce those tells. AI agents fail differently in ways that have been well established: repeated failure patterns, confidence without accuracy, and failures that surface later. What follows here is the signal layer: the specific detectors an evaluation pipeline needs to catch those failure modes while they are happening, not after the complaints arrive.
These signals sit beneath six familiar evaluation criteria: conversation quality, accuracy, policy adherence, task completion, escalation, customer experience. Each signal detects an AI-specific failure mode within one or more of those criteria. A useful AI-agent evaluation layer produces both scores and evidence: scores let QA teams and leadership compare agents, intents, and time windows the way they already do; evidence shows what sits underneath each score, so a drop in a number is traceable to a specific conversation and a specific thing the agent said. For AI agents the evidence layer carries more weight than it does for human agents, because the failure modes are harder to catch by ear. Eight signals matter most.
Every substantive factual claim the agent makes gets traced back to a source. Policy text, product documentation, account data, prior interaction history, whatever the agent was given access to. A claim that cannot be traced is flagged. Grounding fidelity is a conversation-level score reflecting what fraction of the agent's assertions were anchored in retrievable context versus generated without clear basis. Low grounding fidelity on a call does not always mean the agent was wrong. It means the risk of being wrong was high and nothing in the system was in a position to catch it. This signal and the next one, hallucination exposure, together make the Accuracy criterion observable for AI agents.
Every AI agent is deployed with a defined scope. Handles billing questions. Does not handle disputes. Can quote a policy. Cannot interpret a policy. Scope discipline tracks moments where the agent crossed those lines, or came close. A disputed charge accepted as a routine refund. A medical question answered as if it were a product question. A request for legal advice given a response instead of a handoff. Scope breaches are among the most common AI-agent failures, and the human rubric has no equivalent for them because human agents are expected to use judgment at the edges. This signal is what makes the Policy adherence criterion observable: it detects when required checks were skipped or when the agent took a restricted action without the right triggers in place.
For human agents, escalation is a soft signal: did the agent recognize they were out of their depth and transfer appropriately. For AI agents the stakes are inverted. The question is not whether the agent escalated. It is whether the agent escalated at the right moment. Escalation accuracy has to measure both missed escalations and overcautious ones. The second failure mode is as expensive as the first. An AI agent that escalates everything is a routing layer, not an agent.
Distinct from grounding fidelity, which asks whether a claim had a basis, hallucination exposure asks whether any claim contradicts the sources the agent was given. A grounded claim can still be wrong if the agent misread the source. Hallucination exposure flags contradictions, invented policies, invented product capabilities, invented dates, invented promises. Each flagged instance needs evidence attached: the exact utterance and the source that contradicts it. This is the signal that maps most directly to customer-facing risk, and it is the one most closely tied to AI agent hallucination as a failure mode in production systems.
The failure mode here is confidence without accuracy: an AI agent delivers inaccurate information with the same tone and cadence as accurate information, so the failure is invisible to surface-level review. Confidence calibration is the signal that makes this failure mode observable. It compares what the agent expressed in words to what the system was actually confident about. A conversation where the agent sounded definitive while the system was uncertain is a conversation worth reviewing. A conversation where the agent hedged on something the system was confident about is a different kind of inefficiency, but still a signal.
When an AI agent transfers a conversation to a human, the human inherits context. What has been discussed, what has been promised, what has been verified, what remains open. Handoff quality evaluates the completeness and accuracy of that transfer. A transferred conversation where the human has to re-ask the customer what the issue is scores low. A transfer that includes a wrong summary of what the AI promised scores worse. The handoff is where most blended AI/human operations lose customer trust, and it is almost never measured because the human rubric has no slot for it.
Resolution is a dangerous metric for AI agents because they will claim resolution more often than humans will. Scope-aware resolution weights the claim against what was in scope to resolve. It pairs with the Task completion criterion, which asks whether the relevant action actually succeeded in the system record. Scope-aware resolution asks a different question in the same territory: whether the claim of success was within what the agent was authorized to make. An agent that marks a call resolved after answering a question it was authorized to answer gets credit. An agent that marks a call resolved after telling the customer "that's handled" on a request it was never equipped to action gets flagged. Together, task completion and scope-aware resolution catch both the system-record failure and the authorization failure that often sit behind the same conversation.
AI agents maintain tone well in neutral conversations and often drift in adversarial ones. A customer who repeats themselves, pushes back, or expresses anger will sometimes receive a response that mirrors frustration, cuts off empathy mid-sentence, or defaults to procedural language that reads as dismissive. Tone drift under pressure flags those moments specifically, measured against the conversation's emotional trajectory rather than its baseline. A human-agent empathy score averaged across a call hides this. The signal has to catch the inflection point.
None of these signals is useful in isolation. A single hallucination on a thousand-call sample is noise. A repeated failure pattern on a specific intent, with grounding fidelity dropping and escalation accuracy drifting down over the same window, is a finding. The signals compose into a view of where the agent is operating reliably, where it is operating close to the edge of its competence, and where it has moved past that edge without anyone noticing. AI failure modes repeat identically until a root cause is fixed, which is why pattern-level composition matters more here than it does for human-agent evaluation. A single human agent's lapse does not predict the next human agent's lapse. A single AI agent's lapse often predicts the next ten thousand.
The composite is also where human-agent and AI-agent evaluation start to look similar again at the top layer. The underlying signals are different. The operational question is the same: what actually happened in this conversation, and what evidence do we have for what we think happened. For a human agent, the evidence is behavioral. For an AI agent, the evidence is a mix of what the agent said, what it was given to work with, and what the system knew about its own uncertainty while it was saying it.
Three things tend to shift once an AI-agent evaluation layer is in place.
Scores recalibrate. Conversations that looked strong on a human rubric look different once scope breaches and ungrounded claims are visible. The new numbers are closer to the real operational picture. The old numbers reflected what the rubric was built to see, not what the AI agent was actually doing.
The review queue changes shape. Instead of surfacing calls that failed on handle time or script adherence, QA starts surfacing calls that failed on hallucination exposure or scope discipline. Those are the calls where customer-facing risk was actually accumulating. They are also the calls that give product and operations the clearest instructions for what to adjust: what to add to the agent's grounding corpus, where to tighten scope, where to improve the handoff template.
Evaluation starts pointing forward. Grounding fidelity on today's calls tells you something about the shape of tomorrow's incidents. Scope discipline trending down on a specific intent tells you something is drifting in the prompt or the routing before a customer complaint arrives. The signals move from scoring the past to warning about the near future, which is what evaluation has always wanted to be and what sampling never allowed it to become.
In practice a human agent and an AI agent handling similar intents get compared on resolution, scope, and customer outcome, and diverge on the signals that only apply to one or the other. The AI agent gets scored on grounding fidelity and hallucination exposure. The human does not. The human gets scored on judgment at the edges of policy. The AI does not, because judgment is not what we are asking it for. The composite tells leadership which conversations each kind of agent is handling well, where each one is accumulating risk, and where the handoffs between them are breaking down. A single evaluation framework across human and AI conversations is what makes this view possible; the signal layer here is what makes that framework operational for AI-handled calls.
The question evaluation has to answer has not changed. It is still "how is this agent performing," and scoring is still the honest shorthand for the answer. What is changing is what the score has to be built on. For a human agent, a score built on behavioral signals usually tracks reality closely enough that the number can stand on its own. For an AI agent, the number only means what it says if the signals underneath it see the failure modes that matter. Grounding, scope, escalation accuracy, hallucination exposure, handoff quality. The score is still the headline. The evidence is what makes the headline true.
Understanding every conversation was always the goal behind the scores. It stays the goal when the voice on the other end of the call is an AI agent instead of a person. The signals change because the evidence has to. The composite still gives leadership a number to track. The evidence underneath it is what the operation acts on when the number moves.
The categories carry over: resolution, scope, tone, escalation, handoff. The signals underneath each category have to change. A human agent produces visible uncertainty when they do not know the answer. An AI agent does not. A human rubric built around those observable tells will miss failure modes specific to AI agents: ungrounded claims, scope breaches, overconfident wrong answers, incomplete handoffs. An AI-agent evaluation layer adds signals that catch those failure modes directly, measured against evidence from the conversation and the system's own confidence while it was handling the call.
