Why Not Every AI Task Needs an LLM
decision model(dee-SIZH-un model)
A model that answers a narrow, structured question.
LLM(el-el-em)
An AI that generates text and handles open-ended tasks.
threshold(THRESH-ohld)
A cutoff score that triggers or blocks an action.
What happened
PostHog, a software company, says every AI task does not need an LLM. An LLM creates free-form text and handles open-ended reasoning. A decision model answers a structured question with a type and a probability. It may return yes, no, a choice, or a score. If its probability estimates are well tested, a score of 0.8 should be correct about 80 percent of the time. The article drew 104 points and 42 comments on Hacker News, a technology news site. That measures attention, not truth.
Background
PostHog says decision models can be up to 444.6 times cheaper and 193.6 times faster than leading models. Those figures come from TypeSafe AI, which supplied the comparison. The article does not give enough conditions to generalize them. The practical gap is between fixed rules and a full LLM. Rules are too rigid for some fuzzy checks. LLMs are too slow or costly for simple checks. A decision model can handle a narrow question in the middle.
Why it matters
In Replay Vision, a product that watches and summarizes user session recordings, PostHog checks each summary for personal information. Two yes-or-no questions ask whether it names a real person or shows contact details or an ID. If either score reaches 0.4, Gemini, Google's AI model, rewrites the summary. Otherwise, the pipeline skips the LLM. PostHog says 98.7 percent of production summaries skip that call.
The Slack bot uses a stricter threshold. Slack is a workplace messaging service. The bot checks whether a message asks for information and whether PostHog can answer. Both scores must reach 0.85 or higher. The reason is different error costs. An extra call is annoying or expensive. A missed privacy problem is worse. This is why one threshold cannot fit every task.
What is confirmed
These are concrete PostHog production examples, not a broad independent trial. The company also uses a decision model to rerank a short list of search results. It checks whether each summary matches the user's query. PostHog reports a 25 percent improvement across 20 test queries. Another tool picks charts from a database query and the first three result rows. The common pattern is simple: prepare the hard context first, then ask one narrow question.
What remains unknown
Decision models break down when the task needs several reasoning steps. PostHog says an earlier multi-step job sent large requests and blocked its model server for hours. An internal test that ranked code-review findings was inconclusive. Open-ended work still belongs with an LLM. The article also says thresholds changed between versions of the same model. Teams needed labeled examples and shadow testing to recalibrate. A score of 0.85 still allows about 150 errors per 1,000 checks.
What to watch next
The useful lesson is not that one model replaces another. It is that teams should measure each task separately. They should track false alarms, missed cases, human review time, cost, and speed. Decision models fit narrow judgments. LLMs fit writing and deeper reasoning. PostHog's author expects this article may become outdated within two months, so the next evidence should come from repeated measurements, not slogans.
Sources: PostHog's article and the Hacker News discussion.
Use the Simplest AI That Fits the Job
📰 Full story: Why Not Every AI Task Needs an LLM
A decision model can screen simple tasks before an LLM handles harder work.
decision model(dee-SIZH-un model)
An AI that answers one clear kind of question.
LLM(el-el-em)
An AI that writes text and handles harder questions.
threshold(THRESH-ohld)
A score that decides whether something happens.
💡 The gist
- PostHog, a software company, uses decision models for narrow questions.
- An LLM, a text-making AI, handles writing and harder reasoning.
- Hacker News, a tech news site, showed attention. The page had 104 points and 42 comments. That does not prove the claims.
An LLM often gives a long, flexible answer. That helps with open-ended work. It wastes effort on a tiny check. A decision model asks one clear question. It can return yes, no, a choice, or a score. The score shows how likely the answer is right.
PostHog uses this method in Replay Vision. Replay Vision summarizes user activity records. It checks each summary for personal information. Two questions ask whether it names a real person. They also ask about contact details or an ID. If either score reaches 0.4, Gemini, Google's AI model, rewrites the summary. Otherwise, the system skips the LLM. PostHog says 98.7 percent of production summaries skip it.
The PostHog bot in Slack uses a higher threshold. Slack is a workplace messaging service. The bot checks every message for two conditions. Is the message asking for information? Can PostHog answer it? Both scores must reach 0.85. This reduces unwanted replies. A threshold must match the cost of each mistake. Missing private information can be worse than making an extra AI call.
PostHog also uses a decision model for search results. Another system first finds possible matches. The decision model checks each match. It asks whether the result fits the user's search. PostHog reports better ranking on 20 test queries. The improvement was about 25 percent.
This method has limits. Multi-step reasoning can overload the system. An internal code-review test was inconclusive. Open-ended writing still needs an LLM. The best rule is simple. Use a decision model for narrow judgments. Use an LLM for harder thinking.
Hacker News attention is useful for spotting interest. It is not a test of accuracy. Teams still need their own measurements.
Sources: PostHog's article and the Hacker News discussion.
💬 Decision models may help, but testing comes first
HN commenters agree that a small model with limited choices can help with simple decisions. They disagree about its name and warn that poor tests can make a cheaper system fail in production.
- People simplify complicated things to save effort. A decision model also limits its answers to an approved list, so it is less likely to invent an unexpected answer than a generative LLM.
- Some commenters say this is still a kind of LLM or classifier, so “not an LLM” may be mostly a label.
- A team that cannot run good evaluations may replace a strong LLM and lose accuracy without noticing. One commenter reported that tests missed p99 customers, used unrealistic synthetic data, and left out Japanese, bilingual, or device-specific cases.
- The same commenter reported broken light or music controls and a Samsung TV problem missed for about 3 months, affecting 2–3% of customers. These are user reports, not independently verified results.
- A decision model used to choose other models may not understand a brand-new model. Some commenters also say to use normal programs, regexes, or a calculator when those tools fit, while others see LLMs as an easier interface to computers.
- Some commenters worry about putting customer data into such systems and see the practice as surveillance-like.
initial digest at 42 comments (revision 1). We fetched 42 comments and sampled 42 across the thread. These are HN users’ reports, not independently verified facts.
One AI Checks. One AI Writes.
📰 Full story: Why Not Every AI Task Needs an LLM
A simple checker can save the writing AI for harder work.
decision model(dee-SIZH-un model)
An AI that answers small, clear questions.
LLM(el-el-em)
An AI that writes long text.
PostHog, a software company, gives different jobs to different AI systems.
A decision model answers simple questions. It can say yes or no. It can also choose from answers.
An LLM writes long pieces of text. It handles harder questions.
Replay Vision is a tool that shortens activity records. It checks each record first. Only tricky records reach the writing AI. This saves work for the writing AI.
Hacker News is a technology news site. People discussed the article there. Discussion means attention. It does not mean every claim is true.
Simple questions can use a simple checker. Hard questions still need the writing AI. That is the main idea.
Sources: PostHog and Hacker News.
💬 A small chooser and a big word machine
People are talking about using a small tool for simple choices. They also say the tool must be tested carefully.
- A decision model chooses from a short list, so it is less likely to make a surprising answer. Some people say it may still be a kind of LLM with a different name.
- If the tests are poor, real lights, music, or TVs may stop working. One commenter reported that a TV problem lasted about 3 months and affected 2–3% of customers; this was not independently checked.
- A chooser may not understand a brand-new model, and sometimes a normal program or calculator is enough. Some people worry about private customer data, while others think LLMs make computers easier to use.
initial digest at 42 comments (revision 1). We fetched 42 comments and sampled 42 across the thread. These are HN users’ reports, not independently verified facts.
💬 HN debates decision models, LLMs, and evaluation
Commenters see a case for using a constrained decision model for simple choices, but they also raise concerns about evaluation quality, terminology, model routing, new model capabilities, and customer data.
initial digest at 42 comments (revision 1). We fetched 42 comments and sampled 42 across the thread. These are HN users’ reports, not independently verified facts.