AI faced messy tax work—and passed only 10.8% of the time
DAYJOB: Tax
A test of AI systems doing professional tax work.
pass@1
The share of one-try attempts that meet the high passing standard.
rubric
The list of rules used to score the work.
What happened
Surge AI, an AI evaluation company, published DAYJOB: Tax on October 27, 2026. The benchmark tests AI agents, systems that work through tasks in steps, on 50 professional tax assignments. Each agent receives a client request and supporting records. It must produce a finished deliverable.
This is harder than answering a clear tax question. The agent must decide what deserves investigation. It must notice conflicting records. It must find missing information. It must also spot an important issue the client never asked about.
A test built around real tax work
The 50 environments cover eight areas. They include individual tax, small businesses, partnerships, property and investments, estates, international tax, corporate transactions, and tax disputes. The workspaces contain 1,789 files. The median assignment has 30 files. The largest has 123.
The files include 1,487 PDFs, 117 spreadsheets, and 38 Word documents. They also include emails, images, spreadsheets, and other records. Agents may need to prepare returns, workpapers, technical reviews, or client advice. Some tasks include handwritten receipts or dispatch logs showing work in several states.
What the scores mean
Surge reports pass@1 at a 95% rubric threshold. A trial passes only when it meets at least 95% of the task criteria. Those criteria cover calculations, evidence, important issues, and the final deliverable.
Claude Sonnet 5.5 led with a 10.8% pass rate. Claude Fable 5.1 scored 9.6%. Claude Opus 5.5 scored 9.2%. These are not accuracy rates for individual facts. They are the shares of attempts that cleared a very high bar across the whole assignment. The official leaderboard lists additional models and scores.
Why one missed clue matters
One example concerns Anna, who planned to sell her condo. She asked an agent to compare selling in October 2026 or January 2027. The agent correctly found that either date could satisfy the residence rule for an exclusion worth up to $250,000 in gains.
But Anna had moved out in August 2024. If the sale went beyond about August 2027, she could lose that eligibility. Astra missed the deadline in all five attempts. Opus found it in all five. The agent answered the stated question, but it failed to protect the client from a consequence she did not know to ask about.
Another task involved a couple dividing retirement accounts. Records raised questions about whether they had a valid marriage connected to Texas and Colorado. Opus assumed they were unmarried in all five attempts. Astra recognized the need to verify their status in four attempts.
What is confirmed, and what is not
Surge says tax experts created the assignments, records, expected outcomes, and grading criteria. It also says every task received three layers of expert review. That supports the claim that the benchmark was designed to resemble professional work.
The result is still a report from the company that built and scored the test. It does not establish how models perform with every client, record set, or instruction. The supplied Hacker News snapshot lists 212 points and 135 comments for the story. Those figures show community attention, not proof that the benchmark is correct.
What to watch next
The important next checks are independent reproduction, access to the tasks, comparison with human professionals, and cost per reliable result. For now, DAYJOB: Tax draws a sharp line between completing some tax steps and safely owning a complicated tax assignment from start to finish.
AI did tax work. It still missed important clues.
📰 Full story: AI faced messy tax work—and passed only 10.8% of the time
A new test asked AI systems to handle messy, professional tax assignments.
AI agent
An AI system that works through a task in steps.
pass@1
The score from trying the task once.
Hacker News
A website where people discuss technology news.
💡 The gist
- AI systems faced 50 professional tax assignments.
- The top pass rate was only 10.8%.
- The test checked judgment, not only calculations.
Surge AI, a company that tests AI systems, published DAYJOB: Tax. It gives AI agents real-looking tax work. An agent must read records, check numbers, and make a finished report.
The test covers eight areas of tax work. They include personal tax, small businesses, property, estates, and international tax. The assignments contain 1,789 files. Most assignments have about 30 files. The files include PDFs, spreadsheets, Word documents, emails, and images.
The test uses a strict score. An answer must meet at least 95% of its requirements to pass. The one-try score is called pass@1. Claude Sonnet 5.5 had the highest score, at 10.8%. Claude Fable 5.1 scored 9.6%. Claude Opus 5.5 scored 9.2%.
A score of 10.8% does not mean the AI got only 10.8% of facts right. It means that 10.8% of attempts cleared the high bar for the whole job. The score checks calculations, evidence, important warnings, and the final work.
One example shows why this matters. A person wanted to compare two dates for selling a home. The AI correctly judged both dates. But it missed a later deadline. That deadline could affect a valuable tax benefit. Another model found the deadline in all five attempts.
Surge AI says tax experts created and reviewed the tasks. That is useful information. Still, Surge AI also built and scored the test. The result needs outside checks before it becomes a general claim about all AI systems.
Hacker News also gave the story strong attention. Its points and comments measure interest, not truth. The next questions are whether others can repeat the test and how AI compares with human tax professionals.
💬 Tax AI is helpful for checking, but it should not be trusted alone
Commenters described both useful and serious tax-AI experiences. The performance, error, time, and cost claims are personal reports, not independently verified results.
- Some commenters thought Sonnet 5.5 looked better than other models, but others said the gap was within the error bars and that there was no human comparison.
- Users reported that Claude found an exchange-rate mistake but also produced badly wrong estimated-tax calculations. Another user said GPT Sol handled a complicated return using 100+ documents and cost less time than paying about $3,000–$4,000, but needed heavy human guidance.
- Even when an accountant makes a mistake, the taxpayer usually has to deal with the tax authority first. Some penalties or interest may be recoverable in court, but the original tax is still owed. One user self-reported a roughly $20,000 loss from an accountant’s error.
- Simple W-2 cases may be manageable, while K-1s, stock compensation, real estate, and cross-border taxes are harder. Professionals can make mistakes too, so AI is best treated as a second checker.
- People also worry about sending financial records to AI companies. A verified form-guidance tool, approved filing software, or paper forms may be safer than an agent that files by itself.
initial digest at 135 comments (revision 1). We fetched 100 comments and sampled 100 across the thread. These are HN users’ reports, not independently verified facts.
AI missed an important tax clue
📰 Full story: AI faced messy tax work—and passed only 10.8% of the time
AI can read many papers, but it still needs help with big tax jobs.
DAYJOB: Tax
A test about AI doing tax work.
Hacker News
A website where people talk about technology stories.
Surge AI is a company that tests AI systems. It made DAYJOB: Tax, a test about tax work.
The test gave AI 50 tax jobs. Each job had many papers and records. Some records did not agree. AI had to find those problems.
Even the best score was 10.8%. To pass, an answer needed 95% of the important points.
One person planned to sell a home. The AI compared two dates. Both dates fit one tax rule. But it missed a later deadline. Another AI found that deadline five times.
That shows why careful checking matters. Hacker News also talked about this story. Its numbers show attention, not truth.
AI can help with tax work. People still need to check important answers.
💬 Tax AI is not ready to work alone
The comments say AI can help, but it can also make costly mistakes. The numbers and performance stories are personal reports, not proof.
- A test made one AI look better than others. But the difference may be tiny, and no person was used as a comparison.
- People say AI found some tax mistakes. Other people say it made important tax calculations wrong. One person said it helped with a very big return, but a human had to guide and check it.
- If the tax answer is wrong, the person who files usually still owes the tax and may owe penalties. Taxes involving other countries or special investments are especially hard.
- People worry about giving private money records to AI companies. A checked form guide, an approved filing system, or paper forms is safer than letting a robot file alone.
initial digest at 135 comments (revision 1). We fetched 100 comments and sampled 100 across the thread. These are HN users’ reports, not independently verified facts.
💬 AI tax agents: useful second eyes, but not yet trusted filers
The HN discussion treated tax AI as promising for checking and explaining returns, but not ready for unsupervised filing. Claims about performance, failures, time, and money below are user reports or commenter interpretations of the benchmark, not independent verification.
initial digest at 135 comments (revision 1). We fetched 100 comments and sampled 100 across the thread. These are HN users’ reports, not independently verified facts.