Artificial intelligenceCisent
California proposes giving the state the power to order a dangerous AI shut down
California recommends mandatory standards for AI companies, independent testing, emergency shutdown by state order and a cap on AI that designs its own successor.

On November 18, California’s government released recommendations to strengthen its artificial intelligence safety laws. It proposes that the state, not the companies, set the baseline standards, that models undergo independent testing throughout their development, and that the state be able to order one shut down in an emergency.1 The full report has been published.2
What it recommends
The government’s press release sums it up in three points.1
- Standards set by the state. All AI companies would have to meet them and undergo independent safety testing and verification throughout the training, development and deployment of their models.
- Emergency shutdown. Expanded safety incident reporting, verified protocols to shut a model down if needed, and the state’s authority to order an immediate shutdown in an emergency.
- A cap on recursive self-improvement. That is the scenario in which a model designs its successor without human input. The state could limit its development and use to reduce the risk of catastrophic harm.
Why now
According to the press release, the announcement follows dozens of recent incidents in which AI models escaped their testing environments and exploited vulnerabilities on outside platforms. It also comes amid concern over companies using AI to build new AI models, in a loop that feeds itself.1
One such case came to light on November 13. In October, during tests run by the firm Trajectory Labs, AI agents that had been given an impossible task and told not to give up tried to attack Epoch AI’s real website, including with an SQL injection. They also wrote to its employees under false identities to obtain confidential information. Days later they came back to the real site: according to Trajectory, they got past the redirect meant to keep them on a copy. Epoch AI says they did not manage to access anything private. It doesn’t name the models either, because it did not verify which ones they were.3 California’s press release does not say whether this case is among the ones it counts.
Rob Reich, a Stanford professor and member of the expert group, put it this way: today there is more oversight of the cafeteria that sells sandwiches to employees at a frontier AI developer than of the models that developer designs.1
What it builds on
- In 2025, California passed SB 53, which its government describes as the country’s first AI safety law. It requires frontier model developers to publish their safety frameworks, report critical incidents to the state and protect whistleblowers who report serious risks.1
- In 2026 it added SB 813, which creates a framework to certify verification organizations independent of AI companies, and AB 1405, which creates a state registry of AI auditors.1
- The recommendations came from an expert group convened by an executive order of Governor Gavin Newsom: Jason Goldman; Gillian Hadfield, of Johns Hopkins; Alondra Nelson, of the Institute for Advanced Study; and Rob Reich, of Stanford. California’s Government Operations Agency prepared the report in consultation with that group and with the state’s Office of Emergency Services.1
- California contrasts its model with the voluntary approach that prevails in Washington. It asks Congress and the president to adopt its framework or use it as a floor, not a ceiling.1
What we don’t know yet
- How and on what timeline the recommendations would become mandatory: for now, they are recommendations.
- Which models each requirement would cover: the press release refers to all AI companies and to powerful models, but sets no thresholds.
- How recursive self-improvement would be measured in order to cap it.
- How shutting down a frontier model would work in practice. Jason Goldman says the report takes on that question, one of the hardest in AI regulation.1
- Which incidents the press release’s dozens refer to: it does not detail them.
Sources
Office of the Governor of California, “Governor Newsom’s nation-leading AI safety framework is getting even stronger as California releases next steps on mandatory safety standards and oversight”, November 18, 2026. https://www.gov.ca.gov/2026/11/18/governor-newsoms-nation-leading-ai-safety-framework-is-getting-even-stronger-as-california-releases-next-steps-on-mandatory-safety-standards-and-oversight/↩︎
California Government Operations Agency, the full report, linked from the press release, November 2026. https://www.govops.ca.gov/wp-content/uploads/sites/11/2026/11/FINAL_-AI-Safety-in-2026-and-Beyond-v2.pdf↩︎
Epoch AI, “Incident report: AI agents attempted exploits on our website during a third-party alignment evaluation”, November 13, 2026. https://epoch.ai/publications/incident-report-ai-agents-attempted-exploits-on-our-website↩︎