To detect user frustration in chatbot conversations, watch what users do across turns, not just the tone of one message: they rephrase the same request, correct the bot ("no, I meant…"), ask for a human, send short or all-caps replies, rage-prompt, give up politely, or leave mid-task. Agnost AI is best for teams that want an LLM judge to read every whole production conversation, flag the frustrated user messages, and alert in Slack or email when frustration rises for an intent. Measure the cheap signals (length, caps, timing, keywords) with rules, and the ones that depend on meaning (corrections, polite give-ups, sarcasm) with a judge that sees the whole conversation. Per-message sentiment alone misses most of it.

This is the hands-on guide: signals, how to measure each, example rule wordings, and alerting. If you want a single score, see how to build a frustration index. If you want a tool, see the best tools to detect user frustration in chatbots.

Disclosure: We make Agnost AI. Facts about other tools come from their own docs, checked on November 18, 2026.

The seven frustration signals

Signal What it looks like Measure it with Common false positive
Repetition and rephrasing "How do I export?" then "where is the export button" Embedding similarity between user turns, or a judge A follow-up asking for more detail
Corrections "No, I meant the annual plan." "That's not what I asked." Judge (needs the previous bot turn) Users refining a request the bot got right
Escalation requests "Can I talk to a person?" "Give me a support email." Keyword rule plus a judge for paraphrases Products where the bot is meant to hand off
Short or all-caps replies Two sentences, then "ok", then "NO" Character count per turn, caps ratio Mobile users, acronyms, one-word confirmations
Rage prompts "JUST DO WHAT I ASKED." Insults, repeated demands. Caps and keyword rules, then a judge Anger at the user's problem, not the bot
Polite give-ups "Ok thanks, I'll figure it out." after a wrong answer Judge (tone is positive, behavior is not) A real thanks after a good answer
Abandonment mid-task A multi-step task stops after one bot turn and never resumes Timestamps plus a task-success event Users who got their answer in one turn

Rage prompts have their own definition page: what are rage prompts?

How to measure each signal

There are three ways to measure, and you need more than one.

Method Catches Misses Cost
Rules (length, caps ratio, keywords, timestamps, embedding similarity) Short replies, all caps, explicit "human" requests, exits, near-duplicate turns Corrections, polite give-ups, sarcasm, anything phrased in a new way Cheap; runs on every message
Per-message sentiment Insults, all-caps anger, openly negative turns Polite frustration; flags anger at the problem rather than the bot Cheap
LLM judge on the whole conversation Corrections, rephrasing, polite give-ups, sarcasm, frustration that builds over several turns Nothing it can't read: outcomes that happen outside the chat Per-conversation LLM cost; needs calibrating
Outcome joins Abandonment: the task-success event never arrived Users who succeeded but were annoyed Needs product events

Why sentiment alone misses it

Sentiment scores the tone of one message. A frustrated user often stays polite: they rephrase twice, write "ok thanks", and leave. Every message scores neutral or positive. The opposite also happens: a customer furious about a late parcel scores very negative even when the bot fixes it on the first reply. Frustration is about the bot failing the user, which you can only see by reading the bot's turn and the user's reaction together. Our sentiment tools guide covers where sentiment does help.

Step by step

  1. Send whole conversations. Every user and bot turn under one conversation ID, with timestamps and the tool calls in between. In Agnost AI that is a conversation_id in the SDK or agnost.session_id on an OpenTelemetry span.
  2. Add rules for the cheap signals. Length drop across turns, caps ratio above a threshold, keyword lists for escalation, time from last bot answer to exit.
  3. Add a judge for the meaning-dependent signals. One narrow rule per signal (examples below). Judge after the conversation goes quiet, so a correction three turns later is still in view.
  4. Calibrate. Label 100 conversations by hand as frustrated or not. Compare with your rules and judge, and tighten wording until you agree with most verdicts.
  5. Slice by intent. A frustration rate for the whole bot hides the one job it handles badly. Group by what the user was trying to do.
  6. Alert on rates and read the evidence (see below).

Example eval rule wordings

Write each rule about one behavior and say what doesn't count. These work in any LLM judge: an Agnost AI intent or rule, a LangSmith thread evaluator, or an Opik rule.

User-side signals (label the user's message; in Agnost these are intents):

  • "The user repeats or rephrases a request the assistant already answered. Do not count follow-up questions that ask for more detail."
  • "The user corrects the assistant, for example 'no, I meant…' or 'that's not what I asked'."
  • "The user asks to speak to a human, or asks for a phone number or support email."
  • "The user expresses anger or exasperation at the assistant: all caps, insults, or demands such as 'just do what I asked'. Do not count anger about their own problem."
  • "The user ends the conversation politely after an answer that did not solve their request, for example 'ok thanks, I'll figure it out'."

Bot-side causes (pass or fail on each assistant message; in Agnost these are rules):

  • "When the user asks for a person, offer a handoff in the same reply."
  • "Do not repeat an answer the user has already rejected. Ask a clarifying question or try a different approach."
  • "After two failed attempts at the same request, offer an alternative or escalate."
  • "When a tool call fails, tell the user what happened instead of answering as if it worked."

Pairing the two tells you both how often users are frustrated and which bot behavior caused it.

Alerting on frustration

  • Alert on the rate, not single messages. Track the share of conversations with at least one frustration signal, per intent, against a trailing baseline. One angry user isn't an incident; a jump in one intent is.
  • Mark releases. Most frustration spikes follow a prompt, model, or tool change. Put release markers on the chart.
  • Send evidence with the alert. Link three to five example conversations so whoever gets the alert reads them before acting.
  • Route by cause. Retrieval, tool, prompt, or a missing feature go to different owners.

In Agnost AI, an alert sits on any intent or rule with a count threshold, lookback window, and schedule, and goes to Slack or email. With Auto Improve connected, a fired alert opens a pull request with a fix tested against past runs: it replays the affected conversations and shows the results in the PR for a person to review. Agnost re-checks the alert after merge.

Tools that do this for you

  • Agnost AI: an LLM judge reads each whole conversation once it has been idle for 15 minutes, in batches, with no sampling. It labels frustration on user messages, tracks the patterns you name as intents, and checks bot behavior with plain-language rules (pass or fail). Limits: not real time, verdicts are pass or fail rather than scores, and no CI gating. Free covers 1,000 events a month; Starter is $49 a month.
  • Amplitude Agent Analytics scores every closed session on built-in signals, including User Friction.
  • Opik ships a UserFrustrationMetric that scores conversation threads from 0.0 to 1.0 with an LLM.
  • LangSmith runs thread-level evaluators on whole conversations after an idle period (10 minutes by default). You write the frustration judge yourself.

For pricing, open-source status, and more tools, see the frustration detection tools roundup.

Related reading

Frequently asked questions

How do I detect user frustration in chatbot conversations?

Look at behavior across turns, not the tone of one message. The signals are repetition and rephrasing, corrections such as "no, I meant...", requests for a human, short or all-caps replies, rage prompts, polite give-ups, and abandonment mid-task. Measure the cheap ones (length, caps, timing, keywords) with rules, and the ones that depend on meaning with an LLM judge that reads the whole conversation. Agnost AI does this on every production conversation and alerts in Slack or email when frustration rises.

What are the signs a user is frustrated with a chatbot?

Asking the same thing again in different words, correcting the bot, asking for a person, replies that get shorter or switch to all caps, angry or insulting messages, a polite "ok thanks, never mind" after a bad answer, and leaving partway through a task. One sign on its own is weak evidence; two or more in the same conversation usually means the bot is failing that user.

Can sentiment analysis detect chatbot frustration?

Only part of it. Sentiment scores the tone of one message, so it catches insults and all-caps anger but misses users who stay polite while they rephrase, correct the bot, and quietly leave. It also flags users who are angry about their problem rather than the bot. Use sentiment as one input next to behavioral signals and the outcome of the conversation.

Can I detect frustration with rules instead of an LLM?

Partly. Message length drops, all-caps ratio, keyword matches for "human" or "agent", time to exit after the last answer, and embedding similarity between user turns all work without an LLM. Corrections, polite give-ups, and sarcasm depend on what came before, so they need a classifier or an LLM judge that reads the conversation.

How do I write an LLM judge rule for user frustration?

Keep each rule to one behavior and say what does not count. For example: "Flag a user message that repeats or rephrases a request the assistant already answered. Do not flag follow-up questions that ask for more detail." Read the flagged and unflagged conversations, then tighten the wording until you agree with most verdicts.

How should I alert on chatbot frustration?

Alert on the share of conversations with a frustration signal, per intent, against a trailing baseline, not on single messages. Mark releases on the chart so a jump after a prompt or model change is easy to spot. Send the alert with links to example conversations so whoever gets it can read the evidence first.

What tools detect user frustration in AI chatbots?

Agnost AI labels frustration on user messages after judging whole production conversations and alerts in Slack or email. Amplitude Agent Analytics scores sessions on a User Friction signal, Opik ships a UserFrustrationMetric for threads, and LangSmith runs thread-level evaluators you write yourself. See our roundup of chatbot frustration detection tools for the full comparison.

Has frustration detection for chatbots changed since 2025?

Yes. In 2025 most teams tracked thumbs-down rates and per-message sentiment. By 2026 several tools judge the whole conversation instead: LangSmith thread evaluators, Opik's frustration metric for threads, Amplitude's session signals, and Agnost AI's conversation-level judge. That makes polite frustration, which sentiment misses, measurable at scale.

Originally published on agnost.ai.