This story was originally published by CalMatters. Sign up for their newsletters.
###
Gov. Gavin Newsom today released a wide-ranging set of policy recommendations to mitigate risks from artificial intelligence, including by letting state officials order an immediate emergency shutdown of AI systems they deem dangerous.
Other recommendations would mandate that independent auditors test AI models and establish safety and security standards.
The shutdown power, popularly known as a “kill switch,” would apply when companies like ChatGPT maker OpenAI and Claude maker Anthropic refuse to take action to contain systems that go awry.
The recommendations were drawn up by a panel of experts brought together by Newsom earlier this fall, following high-profile cyberattacks by AI agents on governments, banks, and private companies.
The governor touted them as a blueprint for state legislation during a recent visit to Washington, according to a report in Politico. Newsom, who is widely expected to mount a presidential run, said the state is filling a vacuum left by a lack of AI rules at the federal level. “Instead of protecting the public, Donald Trump shrugs his shoulders and leaves AI regulation up to the companies themselves,” he said in a press release announcing the recommendations.
California is home to some of the largest AI companies in the world, and the California Legislature has passed more bills to regulate AI than any other state in the country. Newsom signed dozens of bills passed by the California Legislature into law in September to do things like protect kids, workers, and people living near data centers from AI and related impacts. He also accelerated implementation of laws to create an AI auditor registry and set standards for auditors. Legislators can propose new bills when they return to Sacramento on December 7.
The recommendations come the same week as a lawsuit by Attorney General Rob Bonta against OpenAI in an attempt to hold the company legally responsible for cybersecurity attacks carried out by agents made by the company.
The group also recommends the state of California:
- License auditors — known as independent verification organizations — and have the power to revoke the licenses of those who fail to maintain sufficient independence or technical expertise. Auditors are “under soft pressure to maintain good relationships with the [AI] labs so that they want to continue to work with us,” a staffer at auditing nonprofit METR, Rif Saurous, said during a California Senate hearing last month.
- Consider having state officials assign evaluations to auditors instead of AI companies hiring auditors in order to maintain independence.
- Fund CalCompute, a public cloud initiative intended to provide equitable access to AI, with a surcharge on fees AI companies pay to auditors. Governor-elect Xavier Becerra supported CalCompute funding during his campaign.
- Expand existing protections for AI whistleblowers to include contractors and former employees.
- Expand reporting of AI safety incidents to government agencies and shorten the window to report critical safety incidents to 48 hours from 15 days.
- Allow government limits on the use of AI to design AI, since some experts believe this practice makes systems more dangerous.
- Require developers to disclose the amount of computing resources they’re using to automate AI development.
- Require developers of large AI systems to demonstrate the ability to shut down models in emergencies.
- Establish state standards for safety, security, and transparency while referring to national and international standards.
The four-person panel of experts includes Alondra Nelson, who ran the White House Office of Science and Technology Policy during the Biden administration; Stanford University professor Rob Reich, who studies the governance of science and technology; ; former Google and Twitter executive Jason Goldman, who was the first White House Chief Digital Officer; and Gillian Hadfield, a John Hopkins professor who studies various issues related to AI.
Becerra said in a written statement that he will move quickly with the Legislature to put “full incident reporting and real shutdown authority into law.”
Anthropic, which makes the Claude AI system, is encouraged to see the report move the conversation forward on issues like the independent evaluation of AI models, according to the company’s chief global affairs officer, Tino Cuéllar.
“We will continue working with leaders in California and throughout the world to make sure we keep making progress on these issues at a critical time,” he wrote.
The recommendations by the governor’s panel and by auditors in recent weeks provide fodder to an incoming class of lawmakers in the California Legislature and Congress.
State Senator Scott Wiener, who wrote a state law to mitigate catastrophic AI dangers and was recently elected to take Nancy Pelosi’s seat in Congress, has pledged to introduce AI safety regulation after he’s sworn in.
State law currently defines catastrophic risk as a single event capable of killing more than 50 people or $1 billion in theft or property damage through a cyber attack or activity of an autonomous system. The report out today focuses only on catastrophic risk and does not explore ways to protect people from other AI harms, such as automated discrimination, job loss, or threats to the safety or mental health of children. Coauthors of the recommendations argue that building a system for credentialed auditors to test models for catastrophic risk will make it easier to evaluate models for other forms of harm.
Recommendations released today are the latest ordered by Newsom since the start of the age of generative AI. A late 2023 executive order required the state compile and assessment of the opportunities and risks associated with government use of generative AI. A subsequent state report assessed the impact of generative AI on vulnerable and marginalized communities. A June 2025 expert panel report also concluded that in order to balance innovation and safety, the AI industry needs evaluations by an independent, outside party.