Anthropic’s safety bill would reserve model shutdowns for catastrophic risks
Anthropic
The company that makes Claude.
evaluator
An outside person or group that checks the AI lab.
Department of Commerce
The U.S. government department given the proposed stopping power.
What happened
Anthropic, the company that makes Claude, is lobbying U.S. lawmakers around a draft AI safety bill, according to a Politico report cited by Tom’s Hardware. The proposal would let the government block a model’s release only when a foreseeable, imminent risk could kill or seriously injure more than 50 people, or cause more than $1 billion in damage. This is a proposal, not enacted law.
How the proposal would work
An independent third-party evaluator, registered with the Department of Commerce, would periodically assess the AI lab and its models. The lab could pay the evaluator, but only if the payment did not affect its independence.
The bill focuses on three kinds of danger: creating or releasing a bioweapon; AI-powered or AI-enabled cyberattacks; and a lab losing control of a model or agents. If the evaluator sees one of these risks becoming imminent, it must report the evidence to Commerce within 72 hours. The department would then take the evidence to a federal judge in the District Court for the District of Columbia. The judge would decide whether to halt the release.
The proposal also gives Commerce unusual authority. It says no other U.S. department, agency, or officer could order the lab to stop releasing its model. That would make the Commerce Department the central gatekeeper under this draft.
Why the threshold matters
The proposal does not describe every harmful AI outcome as enough to stop a release. It sets a high, specific threshold. That could make government action easier to define before a crisis, but harder to use when harm is serious yet falls below the line. The question is not only whether a model is dangerous. It is also who estimates the danger, how soon it must happen, and how the damage is counted.
The design also places a company’s proposed safety system inside a government process. Anthropic says the bill reflects its Advanced AI Framework and policies it has urged lawmakers to adopt since June. That explains the company’s position, but it does not mean Congress has accepted it.
What is confirmed, and what is not
The reported draft has a named threshold, an evaluator, a 72-hour reporting rule, and a court decision. Anthropic confirmed to Politico that the proposal represents its framework. The linked Hacker News post received 158 points and 162 comments. Those numbers show community attention. They do not prove that the bill is correct, sufficient, or likely to pass.
Several details remain open. The public reporting does not establish the final bill text, the exact meaning of foreseeable and imminent, or the method for estimating deaths, injuries, and financial damage. It also does not show whether lawmakers will introduce the draft, change its thresholds, or reject it.
What to watch next
The next useful evidence is the bill’s actual text, if it is formally released. Watch for the evaluator’s independence rules, the evidence required by Commerce, and the judge’s standard for stopping a release. Also watch whether Congress gives another agency similar power. Until then, this is a company-backed proposal for a narrow intervention path, not a new safety law.
A proposed AI safety law would use a very high stop sign
📰 Full story: Anthropic’s safety bill would reserve model shutdowns for catastrophic risks
Anthropic wants a clear rule for when the U.S. government may stop an AI model.
draft
A proposed text that is not law yet.
evaluator
An outside person or group that checks the AI lab.
federal judge
A judge in the U.S. national court system.
💡 The gist
- Anthropic proposed a draft safety bill for powerful AI models.
- The government could act above 50 serious injuries or $1 billion damage.
- Hacker News gave the story 158 points and 162 comments. Attention is not proof.
Anthropic, the company that makes Claude, is discussing the proposal with lawmakers. The proposal is not law. It is a draft.
The draft creates a narrow path for stopping a model’s release. The danger must be foreseeable and imminent. People must be able to see it coming soon. The danger must also be very large.
The danger must involve more than 50 people killed or seriously injured. Or it must cause more than $1 billion in damage. Dollars are money used in the United States.
First, an independent evaluator would inspect the lab and its models. The evaluator would be registered with the Department of Commerce. Anthropic could pay it, if payment did not harm its independence.
The proposal names three dangers. They are creating or releasing a bioweapon, an AI-powered cyberattack, and losing control of an AI model or agents.
If one danger seems close, the evaluator reports it within 72 hours. The Department of Commerce then brings evidence to a federal judge. The judge decides whether to stop the release.
The proposal gives Commerce the main stopping power. Other U.S. departments could not order the lab to stop, under this draft.
Why does this matter? A clear rule could reduce confusion during an emergency. But a high line could miss smaller harms. The hardest questions involve counting damage and predicting danger. These details decide how much power the proposal gives the government.
The public report does not show the final bill text. It also does not show whether lawmakers will introduce or pass it. The next step is to watch for official text and debate.
Hacker News attention should stay separate from facts. Its score shows interest in the story. It does not show that the proposal is correct or good.
Source: Tom’s Hardware.
💬 What does the 50-death/$1B line mean?
HN commenters ask whether the proposal would improve safety or mainly help large AI companies.
- The proposal appears to wait until 50 people have died or damages reach $1 billion ($1B) before special action begins. Critics worry that harm just below the line could be ignored and encourage companies to stay under it.
- One commenter claims a single crypto hack could reach $1B and says Anthropic denied crypto defenders access to Mythos. Another says many smart contracts—programs that handle transactions—can be updated or repaired through a hard fork, a community-wide change to the rules. These are commenters' claims, not independently confirmed facts.
- Some commenters worry that insurance and audits would favor large labs and hurt small companies and open-weight models. Others say the rules would mainly target frontier labs and exempt smaller labs.
- The responsibility debate is whether to punish the person who misuses a general-purpose tool, like a computer, or also hold an AI company responsible for unsafe design, as with a faulty car. The law is still unclear.
initial digest at 162 comments (revision 1). We fetched 100 comments and sampled 100 across the thread. These are HN users’ reports, not independently verified facts.
A rule for stopping a very dangerous AI
📰 Full story: Anthropic’s safety bill would reserve model shutdowns for catastrophic risks
Anthropic made a plan for when people may stop an AI model.
checker
A separate person or group that looks for danger.
Commerce Department
A U.S. government department.
Hacker News
A website where technology stories get attention.
Anthropic, the company that makes Claude, has an idea. It wants a rule for very dangerous AI.
The idea is not a law yet. People are still thinking about it.
The rule could act if more than 50 people might die. It could also act if more than 50 people might be badly hurt. It could act if damage might pass $1 billion. A billion dollars is a very large amount of money.
First, a separate checker would look at the AI lab. The checker would tell the Commerce Department within 72 hours. Then a federal judge would decide about stopping the model.
The plan names three dangers: bioweapons, computer attacks, and losing control of AI agents.
Hacker News talked about the story. It had 158 points and 162 comments. Those numbers show attention. They do not prove the plan is right.
We still do not know if this plan will become law. We also do not know how people would count danger. Next, people will watch for the real bill. They will also watch what Congress decides.
Source: Tom’s Hardware.
💬 When does the safety rule start?
The proposal uses a very high line before special action begins. People disagree about whether that is safe.
- The rule starts only after 50 people die or damage reaches $1B. Critics worry that smaller harms will be forgotten, and that someone might try to stay just below the line.
- One commenter says one crypto attack might reach $1B and also says Anthropic refused Mythos access to crypto defenders. Another commenter says many smart contracts can be fixed. These are only commenters' claims.
- People worry that insurance and audits could help big labs more than small companies or open-weight models. Some say the rules would mainly cover frontier labs and spare smaller ones.
- Some people say the person using AI should be responsible, like with a computer. Others say the company should also make it safe, like a car maker. The law has not made this clear yet.
initial digest at 162 comments (revision 1). We fetched 100 comments and sampled 100 across the thread. These are HN users’ reports, not independently verified facts.
💬 Is the safety trigger a risk threshold—or a gate for big labs?
The article describes a proposal that activates its framework only when AI harm reaches 50 deaths or $1 billion in damages. HN commenters debate whether that is a workable safety line, or a rule that can overlook smaller harms and favor large labs.
initial digest at 162 comments (revision 1). We fetched 100 comments and sampled 100 across the thread. These are HN users’ reports, not independently verified facts.