Every e-commerce support team has the same problem: thousands of free-text customer reviews and no time to read them. Which ones are about a late delivery? Which customers are angry? Which ones need someone to act today?
I wanted to know whether a small language model could do that triage as well as a much larger one, at a fraction of the cost. So I set up a controlled experiment with real Brazilian Portuguese reviews, a hand-labeled test set, confidence intervals and a budget of zero.
Short answer: a Qwen3-1.7B fine-tuned with QLoRA ties a few-shot Qwen3-8B on two of three fields, loses 4 accuracy points on sentiment, and runs 3.7× faster in batch. The whole experiment ran on Kaggle's free T4 GPUs; at on-demand cloud prices it would cost about $1.30.
The most interesting findings are in the details: where the models fail, and the field where a plain TF-IDF baseline is statistically as good as both LLMs.
The task
Each review becomes one line of JSON:
{"categoria": "nao_recebido", "sentimento": "negativo", "urgencia": "alta"} | Field | Values |
|---|---|
categoria (main problem) | atraso_entrega (late delivery), nao_recebido (not received), produto_errado_ou_incompleto (wrong or incomplete product), defeito_ou_qualidade (defect or quality), atendimento_vendedor (seller service), elogio_sem_problema (praise, no problem), outro (other) |
sentimento (overall tone) | positivo, neutro, negativo |
urgencia (does the company need to act?) | alta (high), media (medium), baixa (low) |
Keys and labels stay in Portuguese on purpose: they are the data contract a Brazilian product team would consume. A written labeling guide defines each class and the tie-break rules: if a review mentions several problems, the most severe wins; alta urgency only when the customer asks for an action.
The setup
The data is the Olist public e-commerce dataset: 99,224 reviews, 40,977 with a comment, and 36,709 after removing duplicates. The texts are short (median of 62 characters), informal and full of typos.
I compared three approaches:
| Approach | What it is | Trained on |
|---|---|---|
| A. Baseline | TF-IDF + logistic regression, one classifier per field | 5,000 silver labels |
| B. Teacher | Qwen3-8B in 4-bit, system prompt + 6 examples (few-shot) | nothing |
| C. Student | Qwen3-1.7B with QLoRA, system prompt only | 5,000 silver labels |
The most important decision in the project was about labels:
- Gold: 300 reviews I labeled by hand, blind (text only, no star rating and no model output), split into 100 dev and 200 test.
- Silver: 5,000 other reviews labeled by the teacher.
The baseline and the student learn from the same silver labels. That makes the student a distillation of the teacher and ensures the comparison isolates the model, not the data. All three are judged against the human gold labels, never against silver: measuring the student on the same kind of labels it trained on and calling it accuracy is the easiest way to fool yourself.
One more detail: gold and silver have the same number of reviews per star rating. In the raw data, 48% of reviews are 5 stars, so complaints are over-represented on purpose. The metrics describe a balanced mix, not real production traffic.
Evaluation discipline
This is what separates an experiment from a demo.
- The test set was used exactly once, after every decision (prompt, model, training run) was frozen. All iteration happened on the 100 dev reviews.
- Paired bootstrap confidence intervals. With 200 test reviews, each review is worth 0.5 accuracy points, and small differences can be noise. I resampled the test set 2,000 times (the same reviews for every approach) and computed a 95% CI for each difference.
- One shared parser for all three approaches, so "valid JSON" means the same thing for each.
- Checks against facts no labeler saw. The star rating and the real delivery dates were hidden from me and from the models. That makes them an independent check, even on the 5,000 silver reviews, which have no human reference.
Results
Gold test set, n = 200. Accuracy / macro-F1:
| Approach | categoria | sentimento | urgencia | Valid JSON |
|---|---|---|---|---|
| A. TF-IDF baseline | 0.775 / 0.70 | 0.795 / 0.73 | 0.745 / 0.74 | 100% |
| B. Qwen3-8B teacher | 0.870 / 0.78 | 0.840 / 0.77 | 0.820 / 0.81 | 100% |
| C. Qwen3-1.7B student | 0.855 / 0.77 | 0.800 / 0.69 | 0.825 / 0.82 | 100% |
The table alone invites over-reading. Here are the differences with 95% CIs (bold when the interval excludes zero):
| Difference in accuracy (points) | categoria | sentimento | urgencia |
|---|---|---|---|
| Teacher − student | +1.5 [−1.5, +4.5] | +4.0 [+1.0, +7.5] | −0.5 [−4.0, +3.0] |
| Teacher − baseline | +9.5 [+4.0, +15.0] | +4.5 [−1.5, +10.5] | +7.5 [+2.0, +13.0] |
| Student − baseline | +8.0 [+3.0, +13.5] | +0.5 [−4.5, +5.5] | +8.0 [+3.0, +13.5] |
What this says:
- The student ties the teacher on category and urgency. Its only clear loss is sentiment.
- Both LLMs beat TF-IDF by 7 to 10 points on category and urgency. Those fields require some understanding of the text.
- On sentiment, TF-IDF is statistically as good as either LLM. That was the surprise. Extra model capacity buys little here, and the error analysis below shows why: the bottleneck is how
neutrowas defined and labeled, not model size.
The student agrees with the teacher on 93–94% of test reviews in each field, and not a single output failed to parse as JSON (zero failures across 300 gold and 5,000 silver reviews), without constrained decoding.
Speed and cost
| Teacher 8B | Student 1.7B | |
|---|---|---|
| Hardware | 1× T4, 4-bit | 1× T4, 4-bit |
| Batched throughput | 0.71 reviews/s | 2.6 reviews/s |
| Latency, 1 review (p50) | 3.82 s | 3.09 s |
| GPU cost per 1,000 reviews* | $0.137 | $0.037 |
* GPU time only, at Google Cloud's on-demand price of $0.35 per T4-hour. The runs themselves used Kaggle's free quota. One-off costs at the same price: labeling the 5,000 silver reviews with the teacher, $0.69; training the student (96 minutes), $0.56.
A caveat I did not expect: single-request latency barely moves (3.8 s → 3.1 s). On a T4, a ~30-token answer is dominated by the fixed cost of generating each token, not by model size. Part of the batch gain also comes from the shorter prompt: fine-tuning replaces the teacher's few-shot examples. The small model pays off in batch processing, not in one-at-a-time requests.
Where the models fail
I read every test review where at least one model got a field wrong (84 of 200). A few patterns came up again and again.
Keyword shortcuts. The models react to words, not meaning:
"Recebi antes do prqzo" ("Got it before the deadline", original typo included)
Gold:
elogio_sem_problema· Teacher:atraso_entrega· Student:atraso_entrega
The word prazo (deadline) pulls toward late delivery even when the package arrived early. The student got a similar review right and fell for another one: distillation copies the teacher's bias, not each individual answer.
neutro is the hardest class. Of the 40 neutral test reviews, the teacher gets 16 right and the student only 10. Plain confirmations like "Recebi corretamente." ("Received it correctly.") become positivo; calm requests like "Gostaria de um vale troca! Aguardando número de postagem." ("I'd like an exchange voucher! Waiting for the shipping number.") become negativo. Six of the eight reviews behind the student's 4-point sentiment gap are here: it learned the teacher's bias and amplified it.
This boundary is genuinely ambiguous; you could defend positivo for "it arrived as promised". It is a labeling-guide problem as much as a model problem, and that is exactly why TF-IDF keeps up with the LLMs on sentiment.
Urgency rises with tone. The rule says alta only when the customer asks for an action. About 20 of the 68 media urgency reviews are predicted alta by each model. "Quase um mês de atraso na entrega" ("Almost a month late") is serious, but it asks for nothing. In a support queue the error leans to the safe side, but the predicted high-urgency queue ends up about 45% larger than the real one.
The negativity bias shows up at scale. The star-rating check confirms it on the 5,000 silver reviews, with no human in the loop: the teacher labels 4–5 star reviews negativo 15% of the time, against 9% in my gold labels. Overall, sentiment still tracks the stars well (a mean of 2.1 stars for negativo and 4.4 for positivo), so the silver labels are sound in aggregate, just tilted.
What did not work
- My first prompt was bad at
neutro. It got 1 of 8 neutral dev reviews right (0.69 macro-F1 on sentiment). Rewriting the neutral definition with concrete example phrasings brought it to 0.82 on dev. The test set shows the problem was only partly solved. - I did not try a smaller student. The plan was to choose between two sizes with a quick dev test. To save GPU quota, I went with Qwen3-1.7B without trying Qwen3-0.6B, which might have been good enough on category and urgency at a third of the size.
- One training run, no hyperparameter search. That was a deliberate choice, but it means the student's numbers are a floor, not a ceiling.
- The public demo got cut. I planned a free Hugging Face Space, but Gradio Spaces now require a paid plan. The demo runs locally.
- A single labeler. A second blind pass over 30 gold reviews matched the first on all 30. That sounds great, but it was done the same day (memory helps), with 30 reviews the 95% lower bound is around 88%, and it measures consistency, not correctness. On re-reading, at least one gold label looks wrong, and I would have labeled it the same way both times. The real test would be a second, independent labeler.
Lessons
- The student's ceiling is the teacher. The student cannot be much better than the labels it learns from. Better silver labels (teacher self-consistency, or a few hundred human labels mixed in) would likely help more than any hyperparameter tuning.
- Class definitions matter more than model size. The biggest gap in the whole experiment (
neutrosentiment) is not solved by a bigger model. It needs a sharper definition and better labels. - Always run the boring baseline. Without TF-IDF in the table, I would have concluded that "LLMs are better at sentiment" and been wrong.
- Report intervals, not just point estimates. Four of the nine differences in the accuracy table are within noise. The bootstrap is cheap and it changes the conclusions.
- Small models win on throughput, not latency. If your workload is one request at a time on a T4, a 4.7× smaller model saves less than you would think.
Links
- Code, data splits and every prediction: github.com/SoaresCRF/ai-engineer-portfolio/tree/main/review-triage-llm
- Model (LoRA adapter): huggingface.co/SoaresCRF/qwen3-1.7b-review-triage-ptbr-lora
The reviews and derived labels come from the Olist dataset (CC BY-NC-SA 4.0), so the gold labels and the adapter are released for non-commercial use, with attribution.

