Rules, classic ML or an LLM: a decision ladder

"Should we use an LLM for this?" is usually asked as if there were two options: do it by hand, or call a large model. There are at least four, and they differ in cost, speed, and how well you can test them. The simplest way to choose is a ladder: start at the bottom, and climb only when the rung below fails a target you wrote down in advance.

Rung 0: do not automate yet

If the process is rare, unstable or poorly understood, the right move is to document it and measure the manual baseline. Automation of a process nobody can describe produces a fast version of confusion.

Rung 1: rules and lookups

Regexes, validation, tables, thresholds, if-statements.

Pick it when: the decision can be explained in a few lines and the inputs have a stable shape.
You get: exact repeatability, zero per-call cost, a one-line answer to "why did it do that?".
It fails when: the input is free text with many phrasings, or the rules multiply faster than anyone can maintain them.

Rung 2: classic machine learning

TF-IDF plus logistic regression, gradient boosting, small classifiers trained on your labelled examples.

Pick it when: you have (or can get) a few hundred to thousands of labelled examples and the output is a category, a number or a ranking.
You get: fast, cheap, runs on a CPU, calibrated probabilities you can threshold, easy to evaluate.
It fails when: you have no labels, the answer is a free-form sentence, or the task needs broad world knowledge.

In my public demos, a plain TF-IDF model with a calibrated threshold handled about 76% of support tickets automatically at 97.7% accuracy on a held-out set (public Banking77 data, test numbers). It was enough because the task is classification into 77 known intents.

Rung 3: an LLM, constrained

A language model with a tight prompt, a fixed output schema, validation of the result, and a fallback to a human.

Pick it when: the input is varied text, labels are scarce, or the output must be generated or extracted flexibly.
You get: wide coverage with little training data, flexible handling of new phrasings.
You pay with: per-call cost, latency, non-determinism, provider terms and data handling, and a harder testing job.
Make it testable: structured output, validators (checksums, ranges, allowed values), a golden set, and a confidence or consistency signal that routes doubtful items to a person.

Rung 4: an LLM with tools or an agent

Multi-step behaviour, calling APIs, taking actions. Climb here last, and only with a narrow scope, permissions you can audit, and a human approving anything irreversible.

The question at every rung

What is the target, as a number?
What does the cheapest rung that could reach it score on a golden set?
If it falls short, why? Add the missing capability, not the biggest model.

A useful habit is to run the lower rung as a baseline even when you plan an LLM. If a regex plus a classifier scores within a point or two, you have saved a lot of money and risk. If it scores far below, you now have a number that justifies the step up.

Combine them

The best systems are often layered: rules for the easy and the forbidden cases, a classifier for the bulk, an LLM for the long tail, and a human queue behind all three.