Page 1

SENATE THIRD READING
SB 1047 (Wiener)
As Amended August 22, 2024
Majority vote

# SUMMARY

Requires developers of the most advanced, costly artificial intelligence (AI) systems to
implement certain protocols and safeguards in order to mitigate the risk of catastrophic harms.
Requires operators of computing clusters to obtain specified information relating to customers.
Provides for whistleblower protections and enforcement by the Attorney General (AG). Creates
the Board of Frontier Models (board), which approves specified regulations and guidance to be
issued by Government Operations Agency. Provides, upon appropriation, for the creation of a
framework to create a public cloud computing cluster.

**Major Provisions**
1) Requires developers of "covered models"—AI models trained or fine-tuned with a specified

level of compute power and that meet specified cost thresholds—and certain derivative
models to, among other things:

a) Before training the model, implement certain cybersecurity protections, the capability

to promptly enact a full shutdown, a written safety and security protocol, and take
reasonable care to implement appropriate measures to prevent "critical harms"—
defined as mass casualties, at least $500 million in damage, or other comparable
harms.

b) Before using the model or making it publicly available: assess whether the model is

reasonably capable of causing or materially enabling a critical harm; record and retain
test results from the assessment; and take reasonable care to implement appropriate
safeguards.

c) Neither use, unless exclusively for training or evaluation, nor make publicly available

a model if there is an unreasonable risk it will cause or materially enable a critical
harm.

d) Beginning in 2026, annually retain a third-party auditor to perform an independent

audit of a developer's compliance with applicable duties.

e) Make public and provide to the AG redacted copies of safety and security protocol

and auditors' reports. Upon request, provide to the AG unredacted copies of those
documents, which are exempt from the California Public Records Act. Submit annual
compliance statements to the AG. Report to the AG safety incidents within 72 hours.

2) Requires a person operating a computing cluster to implement written policies and

procedures to take certain actions when a customer uses compute resources sufficient to train
a covered model, including obtaining information to identify the customer.

3) Authorizes the AG to bring a civil action for violations of the bill for specified civil penalties

and injunctive and declaratory relief.


-----

Page 2

4) Provides whistleblower protections for employees who have reasonable cause to disclose to

the AG or Labor Commissioner information regarding a developer's noncompliance with the
bill or regarding unreasonably dangerous models.

5) Creates the board within the Government Operations Agency, consisting of nine members,

five appointed by the Governor and four by the Legislature. Establishes certain qualifications
and requirements for members of the board.

6) Requires the Government Operations Agency, by 2027, to issue guidance for preventing

unreasonable risks and adopt regulations to update the definition of a "covered entity" and
establish auditing standards, subject to approval by the board.

7) Upon appropriation, establishes in the Government Operations Agency a consortium required

to develop a framework for the creation of a public cloud computing cluster to be known as
"CalCompute" that advances the development and deployment of AI, as prescribed.

# COMMENTS

With rapid advances in AI, industry insiders—often bound by punitive non-disclosure
agreements—have sounded the alarm regarding the potential risks posed by these technologies.
A group of current and former employees at frontier AI companies recently wrote: "These risks
range from the further entrenchment of existing inequalities, to manipulation and
misinformation, to the loss of control of autonomous AI systems potentially resulting in human
_extinction." (Emphasis added.)_

This bill seeks to mitigate the risk of catastrophic harms from AI models so advanced that they
are not yet known to exist. The bill would require developers of such models to create and
implement safety and security protocols before initiating training. Developers must also
implement the capability to promptly shutdown models. Following training, developers would be
required to perform risk assessments on their models and implement reasonable safeguards,
subject to third-party auditing, before using or releasing them. If there is an unreasonable risk a
model will cause or materially enable a critical harm—mass casualties, at least $500 million in
damage, or other comparable harms—developers are prohibited from releasing or using the
model.

The bill requires the Government Operations Agency to adopt certain regulations and guidance,
subject to approval by the Board of Frontier Models, which would be established under the bill.
The bill also adds whistleblower protections; requires operators of computer clusters to
implement "know your customer" requirements, including the ability to shut down any resources
being used to train an advanced AI model; and lays the groundwork for the creation of a public
computing cluster known as "CalCompute." The Attorney General is charged with civil
enforcement of the bill's requirements.

This bill has generated a great deal of commentary, consternation, and misconception. To set the
record straight: SB 1047's requirements are not onerous. For one, they only impact models that
cost over $100 million in computing power to train—this threshold is baked into the definition of
"covered model" provided by the bill and cannot be altered except by future legislative action.
Secondly, the safety and security protocols developers would be required to implement broadly
align with national and international guidelines, as well as procedures these developers already

l i i l Fi ll h bill l i h d l i l h d


-----

Page 3

capabilities for models they themselves control. The open-source community can rest easy
knowing the models they download will not contain an immutable kill switch.

This bill does not create a state-sponsored licensing regime for AI; it does not ban the creation
and use of AI above a certain compute threshold; it does not impose strict liability on developers
for downstream harms caused by third parties, nor alter baseline rules of tort and criminal
liability; and it does not create exorbitant costs for startups seeking to train large models.
Performing basic risk assessments and implementing reasonable safeguards before training,
using, or releasing powerful, generally-capable models—and prohibiting their use only in
extreme cases involving unreasonable risks of mass casualties or massive economic damages—is
the bare minimum that Californians should expect of an industry claiming to have their best
interests at heart.

**According to the Author**

Large-scale artificial intelligence has the potential to produce an incredible range of benefits
for Californians and our economy—from advances in medicine and climate science to
improved wildfire forecasting and clean power development. It also gives us an opportunity
to apply hard lessons learned over the last decade, as we've seen the consequences of
allowing the unchecked growth of new technology without evaluating, understanding, or
mitigating the risks. SB 1047 does just that, by developing responsible, appropriate
guardrails around development of the largest, most powerful AI systems, to ensure they are
used to improve Californians' lives, without compromising safety or security.

SB 1047 will also promote the growth of the AI industry by establishing CalCompute, a
public AI research cluster that will allow startups, researchers, and community groups to
participate in the development of large-scale AI systems. By providing a broad range of
stakeholders with access to the AI development process, CalCompute will help align largescale AI systems with the values and needs of California communities.

**Arguments in Support**
The Center for AI Safety Action Fund, a co-sponsor of this bill, writes:

If California does not act to establish a clear and sensible governance framework, the safety
of its citizens could be imperiled, the nation's security could be seriously harmed, and AI's
enormous potential to improve our world could be derailed. Placing sensible guardrails
around serious risks that the most powerful systems might pose, while taking significant
steps towards leveling the playing field for academics and startups, is the best way to ensure
that CA's citizens can realize the immense benefits of this technology.

Encode Justice, a co-sponsor of this bill, writes:

SB 1047 introduces essential safeguards for the creation of highly capable AI models, often
known as "frontier AI models." These models are defined in the bill as trained using over
10^26 floating-point operations. Models of this scope would cost at least $100 million to
develop and, notably, do not yet publicly exist but are anticipated to emerge soon as
technological advancements continue. These are advanced, resource-intensive projects that
have caught attention at the highest levels of government and are the focus of President
Biden's Executive Order on Artificial Intelligence for their significant national security and
public safety implications.


-----

Page 4

**Arguments in Opposition**
California Chamber of Commerce writes on behalf of a coalition of trade associations:

In addition to creating inconsistencies with federal regulations, the bill demands compliance
with various vague and impractical, if not technically infeasible, requirements for which
developers will be subject to harsh penalties, including potential criminal liability. We are
concerned that the bill regulates AI technology as opposed to its high-risk applications,
creates significant regulatory uncertainty and therefore high compliance costs, and poses
significant liability risks to developers for failing to foresee and block any harmful use of
their models by others – all of which inevitably discourages economic and technological
innovation. And while recent amendments take important steps in responding to the opensource community, we remain concerned about the impact of the bill on AI research and
development in California and the impact on startups. Overall, the bill still makes AI
business too risky in California, particularly given the potential penalties under SB 1047.

# FISCAL COMMENTS

1) Costs (General Fund) to GovOps of an unknown but potentially significant amount to

establish and operate the Board, and to issue and update regulations. The bill also requires
GovOps to, upon appropriation by the Legislature, establish a consortium and develop a
framework for the CalCompute public cloud computing cluster. Although these
responsibilities are contingent upon appropriation, they add additional, potentially significant
General Fund cost pressure to GovOps.

2) Costs (General Fund) to DOJ of an unknown but potentially significant amount to maintain

and review developer statements of compliance, and to bring civil enforcement actions.
Actual costs will depend in part on the number of violations, the number of actions filed, and
the amount of workload associated with each action.

3) Costs (Labor and Enforcement Compliance Fund) to the Labor Commissioner of an unknown

but amount to receive and act on whistleblower reports.

4) Costs (Trial Court Trust Fund, General Fund) to the courts to adjudicate violations of the bill

and whistleblower cases. Actual costs will depend on the number of violations, the number
of actions filed, and the amount of court time needed to resolve each case. Although courts
are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund
may create a need for increased funding for courts from the General Fund. The fiscal year
2024-25 budget provides $37.3 million ongoing General Fund to backfill declining revenue
to the Trial Court Trust Fund.

# VOTES

**SENATE FLOOR:** **32-1-7**
**YES: Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd,**
Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman,
Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg,
Wahab, Wiener, Wilk
**NO: Grove**
**ABS, ABST OR NV: Allen, Alvarado-Gil, Dahle, Jones, Nguyen, Niello, Seyarto**


-----

Page 5

**ASM PRIVACY AND CONSUMER PROTECTION:** **8-0-3**
**YES: Bauer-Kahan, Bryan, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson**
**ABS, ABST OR NV: Joe Patterson, Dixon, Hoover**

**ASM JUDICIARY:** **9-1-2**
**YES: Kalra, Bauer-Kahan, Bryan, Connolly, Haney, Maienschein, McKinnor, Pacheco, Reyes**
**NO: Dixon**
**ABS, ABST OR NV: Joe Patterson, Sanchez**

**ASM APPROPRIATIONS:** **11-3-1**
**YES: Wicks, Arambula, Bryan, Calderon, Wendy Carrillo, Mike Fong, Grayson, Haney, Hart,**
Pellerin, Villapudua
**NO: Sanchez, Dixon, Ta**
**ABS, ABST OR NV: Jim Patterson**

# UPDATED

VERSION: August 22, 2024

CONSULTANT: Josh Tosney / Slater Sharp / P. & C.P. / (916) 319-2200 FN: 0004462


-----

