How to Get an LLM to Classify Things Reliably

Most classification failures are label problems wearing a prompt costume. If two people on your team disagree about which tag an input should get, no prompt will make the model consistent. Run this test before anything else. Take 20 real inputs, hand them to two colleagues, and have them label independently without talking. Every disagreement is a category defect. You will usually find three kinds: - Overlapping labels. "Bug" and "Technical issue" are the same thing to a model and to half your team. - Labels at different levels. "Payments" is a domain. "Refund request" is an intent. Mixing them in one list forces an impossible choice. - Missing labels. The input genuinely does not fit, so the model picks the nearest neighbour and you never find out. Fix the set, not the wording. Merge overlaps, split anything doing two jobs, keep each list to one level of abstraction, and write a one-line definition plus an exclusion for each label. The exclusion matters more than the definition: "Billing, but not refunds, which go to Refunds" removes an entire class of error. Keep the list short, too. A flat list of forty tags asks the model to hold forty definitions in mind at once. Two chained calls, one picking among six domains and one picking among the tags inside that domain, is almost always more accurate and easier to debug.

Do not ask the model for a category and then regex the answer out of a sentence. Give it a JSON schema with your labels as an enum and let the API guarantee the shape. This has been a solved problem since OpenAI shipped Structured Outputs on 6 August 2024. Their announcement reports that on their evals of complex JSON schema following, gpt-4o-2024-08-06 with Structured Outputs scores 100%, while gpt-4-0613 scores less than 40%. The technique is constrained decoding: the schema is compiled to a grammar and invalid tokens are masked at each step, so an off-list label is not merely unlikely, it is unreachable. Every major provider now offers a version of this. The payoff is narrower than it sounds. Structured output kills format errors. It does not kill wrong answers. OpenAI say so directly in the same post: "Structured Outputs doesn't prevent all kinds of model mistakes. For example, the model may still make mistakes within the values of the JSON object." Schema compliance and semantic accuracy are different metrics and you need both. There is also a cost. Tam and colleagues, in "Let Me Speak Freely?" (arXiv, August 2024, revised October 2024), found a significant decline in LLM reasoning ability under format restrictions, with stricter constraints producing greater degradation. The workaround is to give the model somewhere to think inside the schema: a short free-text "reasoning" field before the constrained label field, so it works the problem before it commits.

When a category keeps failing, add examples of that category before you add another rule. Instructions describe the boundary in the abstract. Examples show the model exactly where it sits. This is OpenAI's own advice for exactly this situation: when developers find mistakes in the values, they recommend "providing examples in the system instructions or splitting tasks into simpler subtasks". It matches what happens in practice. A paragraph explaining the difference between a complaint and a feature request moves accuracy a little. Four real examples of borderline cases move it a lot. Pick your examples from failures, not from the easy middle. An input that any human classifies instantly teaches the model nothing. What you want are the near-misses: the ones your two colleagues disagreed about, the ones the model got wrong last week, the ones sitting right on the line between two labels. Include at least one example per label, and extra examples wherever your error counts say you are weakest. How much this buys you depends on the model. Chae and Davidson, writing in Sociological Methods & Research (February 2026) on ten models across zero-shot, few-shot, fine-tuning and instruction-tuning, found the largest models generally offer the best predictive performance even with little or no training data, while fine-tuning smaller models stays competitive on accuracy and cost. On a frontier model, examples are your main lever. On a small model at high volume, fine-tuning gets cheaper once you have labelled data.

You cannot improve a classifier you have not measured. Thirty to fifty real, hand-labelled examples is enough to stop guessing, and it is a couple of hours of work. Use real inputs from your product, not inputs you invented. Invented examples are written by someone who already knows the answer, so they are unrealistically clean. Label them yourself or with the one person who best understands the domain. Hamel Husain, in his LLM Evals FAQ, argues for exactly this: a single domain expert as the definitive voice on quality rather than a committee, with Cohen's Kappa only when multiple annotators are genuinely needed. Then do error analysis properly. Husain's process is to read traces, write open-ended notes on what went wrong, group those notes into a failure taxonomy, and count the categories. His rule of thumb is to review at least 100 traces, and to stop once roughly 20 more turn up no new failure category. He also recommends binary pass or fail judgements over one-to-five scales, because the gap between a 3 and a 4 is subjective and annotators drift to the middle to avoid deciding. Report accuracy per label, never as a single number. An overall 88% hides the fact that your biggest category is at 97% and the three that actually need routing are at 55%. The per-label view tells you which examples to add next. That measure-then-change loop is the core of what we teach in buildAcademy.

A classifier that says "I do not know" is more useful than one that quietly guesses. Give the model a way out, and route those cases to a human. The research area is called abstention. Wen and colleagues, in "Know Your Limits: A Survey of Abstention in Large Language Models" (arXiv, July 2024, published in TACL 2025), frame a model declining to answer as a way to reduce hallucination and improve safety in production systems. For a classifier it is cheap to implement and pays for itself quickly. Three things to build: - An explicit escape label. Add "unclear" or "needs review" to your enum with a definition, so the model has a correct answer available when nothing fits. Without it, the model must pick a wrong label. - A confidence field. Ask for high, medium or low alongside the label. Treat it as a coarse signal, not a probability, and validate it against your eval set before trusting any threshold. If low-confidence predictions are no less accurate than high-confidence ones, the field is noise. - A queue with a feedback loop. Escalated items go to a person, that decision is stored, and those decisions become your next batch of examples. The escalation path is not overhead, it is the training data pipeline. Set the threshold from the cost of error, not from a round number. A misrouted support ticket costs a few minutes. A misrouted safety report costs far more, and should escalate aggressively.

Does structured output guarantee the right answer? No. It guarantees the right shape. OpenAI state plainly that Structured Outputs "doesn't prevent all kinds of model mistakes" and the model can still get values wrong. Schema compliance and semantic accuracy are separate metrics, and only the second one is what your users experience. How many examples should I put in the prompt? Start with one or two per label, then add examples only where your eval set shows failures. Prioritise borderline cases over obvious ones. If you are past roughly twenty examples and accuracy has flatlined, the problem is your label definitions, not your example count. Should I use a big model or fine-tune a small one? Start with the largest model available and examples in the prompt. Chae and Davidson found large models perform well with little or no training data, while fine-tuned smaller models stay competitive on accuracy and cost. Move to fine-tuning when volume makes per-call cost the binding constraint. How big does my eval set need to be? Thirty to fifty real labelled examples gets you measuring. Hamel Husain recommends reviewing at least 100 traces for error analysis, and stopping when about 20 more reveal no new failure category. Grow the set with every escalated and misclassified case you see in production. Can I use an LLM to grade my classifier? Yes, for narrow binary checks. Husain recommends scoping judges to pass or fail tasks and measuring true positive and true negative rate against a held-out set you labelled by hand. A judge you have not validated against human labels is a second opinion, not a measurement.