What if You Don't Need an LLM?

The model is not the product

It is easy to start an AI project by asking which language model to use. That question feels practical because it names a vendor, a context window, a benchmark, and a price. It also skips the more important question: what kind of intelligence does the product actually need?

Many useful AI features are closer to retrieval, ranking, labeling, or scoring than open-ended generation. The job may be finding the right document, grouping similar tickets, labeling an incoming request, detecting a risky message, checking whether text follows a pattern, or measuring whether a user is upset. A large language model can often handle those tasks, but a narrower tool may be faster, cheaper, and easier to trust.

The best AI systems usually route each task to the simplest tool that can meet the quality bar instead of reaching for the largest model by default.

Search is usually an embedding problem

If the product needs to retrieve relevant knowledge, embeddings should be considered before an LLM. Embedding models turn text into vectors so related ideas sit near each other. That makes them a natural fit for semantic search, duplicate detection, clustering, recommendation, and knowledge base retrieval.

An LLM may still be useful after retrieval, especially if the user needs a synthesized answer. The retrieval problem comes first: index the right source material, compare queries against that index, and return evidence the system can use.

This distinction matters because retrieval failures are often blamed on the model when the real issue is the search layer. If the right document never reaches the prompt, even the best model is forced to guess.

Good retrieval systems usually need:

• clean document boundaries;
• metadata that supports filtering;
• chunking that preserves meaning;
• ranking that reflects the user’s task;
• evaluation queries with expected sources.

Once that foundation exists, the LLM becomes an optional layer on top of search, not the whole architecture.

Classification does not always need generation

Text classification is another place where LLMs are often overused. Routing a support ticket, detecting spam, assigning a topic, flagging policy violations, or predicting sentiment may work better with embeddings, a small classifier, or a fine-tuned lightweight model.

The reason is simple: classification needs consistency more than eloquence. A model that writes beautifully but changes its label when the prompt is phrased differently can create expensive downstream behavior. In many production systems, a boring classifier with stable probabilities is more valuable than a brilliant general model with variable answers.

This is especially true when the label set is small and the business has historical examples. If thousands of support tickets have already been tagged by humans, that data can train or validate a classifier directly. An LLM may help bootstrap labels or handle ambiguous cases, but it does not need to own the entire workflow.

Rules are underrated

Some problems should not use AI at all. If a policy says invoices over a certain amount require approval, that is a rule. If a form is invalid when a field is missing, that is validation. If a customer is ineligible because a contract expired, that is a database lookup.

Replacing deterministic logic with an LLM can make the product less reliable. It adds cost, latency, nondeterminism, and a new failure mode where the system explains the wrong answer with confidence.

Rules are often the clearest expression of business knowledge. A mature architecture uses rules for facts and constraints, embeddings for similarity, classifiers for labels, and LLMs for language tasks that actually require flexible reasoning or generation.

Smaller models can be better products

Even when AI is needed, a large frontier model may not be. Grammar checking, sentiment analysis, intent detection, entity extraction, moderation, autocomplete, and translation can often be handled by smaller models or specialized services. These systems can be faster, cheaper, easier to monitor, and less likely to surprise users.

Cost shows up in more places than the model invoice. Latency, rate limits, evaluation complexity, incident response, data governance, and reliability work all become part of the bill. A feature that responds instantly and costs almost nothing can shape user behavior in ways a slower, more expensive feature never will.

A large model can probably solve the task. The useful question is whether the extra capability changes the product outcome enough to justify the operational cost.

Use evals to choose the architecture

An eval set should help with a bigger decision than prompt or vendor selection. It should tell you whether an LLM belongs in the system at all.

A useful eval set should include examples that a simpler system can attempt. For a classification workflow, compare an LLM against embeddings plus a classifier. For a search workflow, compare generated answers against retrieval quality and source coverage. For a grammar feature, compare a large model against a specialized writing model or deterministic checks. For a sentiment feature, compare it against a smaller model that returns stable scores.

Then judge the systems the way the product will be judged:

• accuracy on real examples;
• consistency across repeated runs;
• latency at expected traffic;
• cost per successful task;
• ease of debugging;
• failure behavior when the input is messy;
• human review effort created downstream.

This moves the decision away from model preference and toward evidence. Sometimes the LLM wins. Sometimes it only wins on demos.

Hybrid systems are usually the answer

The strongest production architectures often use an LLM sparingly. A support product might use embeddings to retrieve similar tickets, rules to enforce refund limits, a classifier to route urgency, and an LLM to draft a response for human review. A sales tool might use deterministic CRM fields for eligibility, embeddings for account research, and an LLM only to produce a personalized email.

That routing makes the system easier to maintain. Each component does the work it is naturally good at, and each layer can be tested on its own.

It also gives teams a better path to improvement. If retrieval is bad, improve the index. If labels are inconsistent, improve the classifier or training data. If generated text is weak, improve the prompt or model. Without that separation, every problem becomes “the AI is bad,” which is not a useful debugging category.

The practical test

Before committing to an LLM, ask what the output needs to be. If the answer is a label, ranking, score, match, lookup, validation, or extraction from a predictable format, test a simpler approach first. If the answer requires open-ended language, multi-step reasoning, synthesis across messy sources, or adaptation to unusual user intent, an LLM may be the right layer.

Then ask whether the business needs the best possible answer or the most reliable useful answer. Those are not always the same thing.

LLMs are extraordinary tools, but they should not become architectural autopilot. After you develop your eval set, use it to check the uncomfortable possibility that the product does not need an LLM. A smaller system that is cheaper, faster, easier to explain, and good enough for the user’s job is often the better engineering choice.