Anthropic’s AI safety bill sets a high bar for stopping a model
Anthropic
The AI company that makes Claude.
Commerce Department
The US government office named in the draft.
foreseeable and imminent risk
A danger expected soon and considered likely.
What happened
Anthropic, the AI company that makes Claude, has pitched a draft federal AI safety bill to US lawmakers. TechCrunch reported the proposal after Politico described a confidential draft. Anthropic later confirmed that the draft reflects its Advanced AI Framework. The company says it has promoted that framework to federal and state lawmakers since June.
The proposal would let the government stop a powerful AI model only in a narrow set of cases. The reported threshold is a foreseeable and imminent risk of more than $1 billion in damage, or of killing or seriously injuring more than 50 people. The danger would also need to involve a biological weapon, an AI model escaping its developer’s controls, or a cyberattack.
How the proposal would work
The draft would use outside evaluators registered with the US Commerce Department. They would periodically examine an AI lab and its models. If an evaluator found one of the listed risks, the Commerce Department could ask a federal judge in Washington, DC, to suspend the model. The judge, not the department alone, would make the final decision.
A related report from Tom’s Hardware says the reported text would prevent other federal departments, agencies, or officers from independently ordering a model’s release or use to be stopped. That makes the proposal more than a list of safety tests. It is also a plan for who gets emergency power.
Why it matters
The idea has two sides. A single process could make government action easier to understand. Companies would know which risks matter, and a judge would review the evidence. Anthropic says the policies are sensible.
However, a single process could also slow action. A future risk may be severe without crossing 50 deaths or $1 billion in damage. Forecasting those numbers before an incident is hard. If only Commerce can seek a court order, another agency might see a danger but lack a direct way to pause the model.
The proposal also raises a conflict-of-interest question. The evaluator would be outside the lab, but reports say the lab could pay for the evaluation if that payment did not affect independence. The rule would need to make that independence believable to the public.
What is confirmed
The draft exists as a proposal presented to lawmakers. Anthropic has acknowledged its connection to the company’s framework. The 50-person and $1 billion thresholds resemble those in California legislation, so they are part of a wider US policy debate.
What has not happened is just as important. The proposal is not a federal law. It has no confirmed final text, vote, or date for taking effect. Reporting about it is based on a confidential draft, not a passed statute.
What remains unclear
The public still needs to see the complete language. It is not clear how officials would define an imminent risk, calculate likely deaths or damage, or judge that a model has escaped its developer’s control. It is also unclear how the system would interact with state laws and existing regulators.
What to watch next
The next clues are a bill number, a public draft, congressional sponsors, and responses from other agencies. The larger question is whether AI companies should help write the rules for the systems they sell. Any final law will need both a clear way to stop a dangerous model and a clear explanation of who can act before the damage becomes irreversible.
Anthropic suggests a narrow way to stop dangerous AI
📰 Full story: Anthropic’s AI safety bill sets a high bar for stopping a model
Anthropic, the AI company that makes Claude, shared a draft safety bill.
draft safety bill
A proposed law about keeping AI safe.
Commerce Department
The US government office named in the proposal.
federal judge
A judge who decides cases under US national law.
💡 The gist
- The draft suggests rules for stopping a dangerous AI model.
- Action would require over 50 deaths or serious injuries, or $1 billion damage.
- A Commerce Department office would ask a judge to pause the model.
A bill is a proposed law. The draft is not law yet.
What the draft says
The draft covers powerful AI models. The government could act only when a specific danger is expected soon. The danger would need to involve a biological weapon, lost developer control, or a cyberattack.
An outside evaluator would check the AI lab and its models. The evaluator would be registered with the Commerce Department. If it found a serious risk, it would report the risk. Commerce could then ask a federal judge to pause the model.
Other federal agencies could not order the pause by themselves. This creates one clear path. It also puts much power in one department and one court. The proposal therefore changes who can act during a crisis.
Why the numbers matter
The limits may make government action more predictable. Companies would know which risks matter. A judge would review the evidence.
But danger does not wait for a number. An AI could harm fewer than 50 people and still cause terrible damage. It could also cause harm before experts can estimate the cost. A narrow rule might protect companies from sudden action. It might also slow protection for the public.
Anthropic says the plan is sensible. Similar numbers appear in California AI legislation. That shows the idea is part of a wider US debate.
What is still unknown
This is a confidential draft. Congress has not passed it. The final wording, supporters, and vote date are unknown. It is also unclear how evaluators would stay independent. We do not know how the government would define “imminent risk.”
State rules could also overlap. A company might face different rules in different places. The draft does not explain that relationship. The next key steps are a public bill and a congressional debate.
A new rule for stopping a risky AI
📰 Full story: Anthropic’s AI safety bill sets a high bar for stopping a model
Anthropic, the company that makes Claude, suggested a new rule.
Anthropic
The company that makes Claude.
judge
A person who makes a legal decision.
cyberattack
An attempt to harm or enter computer systems.
The simple idea
Anthropic makes Claude. Claude is an AI tool. Anthropic showed a rule idea to people who make US laws. The idea is not a law.
When could AI be stopped?
The rule names two big numbers. More than 50 people could die or suffer serious injuries. Or the damage could pass $1 billion. Then a government office could ask a judge for help. The judge would decide.
The danger must come from a few listed causes. One cause is a weapon made with harmful germs. Another is AI escaping its maker’s control. A third is a cyberattack. Another government office could not stop it alone.
What do we know?
This is only a plan. Congress has not made it law. People still need to see the full plan. We do not know who will check the AI. We do not know if lawmakers will agree.