Office of Senate Floor Analyses
(916) 651-1520  Fax: (916) 327-4478

THIRD READING

Bill No: SB 1047

Author: Wiener (D), et al.

Amended: 5/16/24
Vote: 21

SENATE JUDICIARY COMMITTEE: 9-0, 4/2/24
AYES: Umberg, Allen, Ashby, Caballero, Durazo, Laird, Min, Stern, Wahab
NO VOTE RECORDED: Wilk, Niello

SENATE GOVERNMENTAL ORG. COMMITTEE: 11-0, 4/23/24
AYES: Dodd, Wilk, Ashby, Bradford, Glazer, Ochoa Bogh, Padilla, Portantino,

Roth, Rubio, Smallwood-Cuevas

NO VOTE RECORDED: Alvarado-Gil, Archuleta, Jones, Nguyen, Seyarto

SENATE APPROPRIATIONS COMMITTEE: 5-0, 5/16/24
AYES: Caballero, Ashby, Becker, Bradford, Wahab
NO VOTE RECORDED: Jones, Seyarto

**SUBJECT: Safe and Secure Innovation for Frontier Artificial Intelligence**

Models Act

**SOURCE: Center for AI Safety Action Fund**
Economic Security Project Action
Encode Justice

**DIGEST: This bill requires developers of powerful artificial intelligence models**
and those providing the computing power to train such models to put appropriate
safeguards and policies into place to prevent critical harms. This bill establishes a
state entity to oversee the development of these models and calls for the creation of
a public cloud computing cluster.

**ANALYSIS:**

Existing law:


-----

Page 2

1) Establishes the California Department of Technology (CDT) within the

Government Operations Agency, under the supervision of the Director of
Technology (Director), also known as the State Chief Information Officer.
(Gov. Code Sec. 11545(a).)

2) Provides that persons are responsible, not only for the result of their willful

acts, but also for an injury occasioned to another by their want of ordinary
care or skill in the management of their property or person, except so far as
the latter has, willfully or by want of ordinary care, brought the injury upon
themselves. (Civ. Code § 1714(a).)

This bill:

1) Establishes the Safe and Secure Innovation for Frontier Artificial Intelligence

Models Act.

2) Provides definitions for all relevant terms:

a) “Covered model” means an artificial intelligence model that was trained

using a quantity of computing power greater than 10^26 integer or floatingpoint operations (FLOP) or a model that can reasonably be expected to have
similar performance capabilities as assessed by commonly used
benchmarks.

b) “Limited duty exemption” means an exemption with respect to a covered

model that is not a derivative model that a developer can reasonably
exclude the possibility that a covered model has a hazardous capability or
may come close to possessing a hazardous capability when accounting for a
reasonable margin for safety and the possibility of posttraining
modifications.

c) “Hazardous capability” means the capability of a covered model to be used

to enable any of the following harms in a way that would be significantly
more difficult to cause without access to a covered model:

i) The creation or use of a chemical, biological, radiological, or nuclear

weapon in a manner that results in mass casualties.

ii) At least $500 million of damage through cyberattacks on critical

infrastructure.


-----

Page 3

iii) At least $500 million of damage by a model that autonomously engages
in conduct that would be criminal if done by a human.

iv) Other comparably severe threats to public safety and security.

d) “Hazardous capability” includes the capabilities above even if they would

not manifest but for fine tuning and posttraining modifications performed
by third-party experts.

e) “Fine tuning” means the adjustment of the model weights of an artificial

intelligence model after it has finished its initial training by training the
model with new data.

f) “Covered guidance” means either of the following:

i) Guidance issued by the National Institute of Standards and Technology

and by the Frontier Model Division that is relevant to the management
of safety risks associated with artificial intelligence models that may
possess hazardous capabilities.

ii) Industry best practices, including safety practices, precautions, or testing

procedures undertaken by developers of comparable models that are
relevant to the management of safety risks associated with AI models
that may possess hazardous capabilities.

g) “Computing cluster” means a set of machines transitively connected by data

center networking of over 100 gigabits per second that has a theoretical
maximum computing capacity of at least 10^20 integer or floating-point
operations per second and can be used for training artificial intelligence.

3) Establishes the Frontier Model Division (FMD) within CDT and tasks them

with various duties, including:

a) Review annual certification reports from developers.

b) Advise the Attorney General on potential violations of this law.

c) Issue guidance, standards, and best practices necessary to prevent

unreasonable risks from covered models with hazardous capabilities.


-----

Page 4

4) Requires developers of covered models that are not subject to a limited duty

exemption, before training the model and until an exemption establishes, to
implement various safeguards. This includes:

a) Implements administrative, technical, and physical cybersecurity

protections to prevent unauthorized access to, or misuse or unsafe
modification of, the covered model, that are appropriate given the
associated risks.

b) Implements the capability to enact a full shutdown or a “kill switch.”

c) Implements all covered guidance.

d) Implements, and annually reviews, a written and separate safety and

security protocol, as specified, that details how it is adequate to prevent
critical harms and “hazardous capabilities” and specifies the testing
procedures incorporated therein. The protocol shall be provided to FMD.

e) Ensures that these protocol are implemented as written, including by

designating senior personnel responsible and conducting audits, as
appropriate.

f) Refrains from initiating training of a covered model if there remains an

unreasonable risk that an individual, or the covered model itself, may be
able to use the hazardous capabilities of the covered model, or a derivative
model based on it, to cause a critical harm.

5) Establishes specified processes for and imposes obligations on developers

throughout model training and the initiation of widespread use of the models,
as provided, including requiring specified reporting and implementation of
appropriate safeguards.

6) Requires persons that operate computing clusters to implement appropriate

written policies and procedures to do the following when a customer utilizes
compute resources that would be sufficient to train a covered model:

a) Obtains, and annually validates, a prospective customer’s basic identifying

information and business purpose for utilizing the computing cluster, as
specified.


-----

Page 5

b) Annually assesses whether a prospective customer intends to utilize the

cluster to deploy a covered model.

c) Maintains for seven years and provides to the Frontier Model Division or

the Attorney General, records of actions taken pursuant to this law.

d) Implements the ability to promptly enact a full shutdown in the event of an

emergency.

7) Authorizes the Attorney General to bring an action seeking recovery of

preventive relief, including a permanent or temporary injunction, restraining
order, or other order against the person responsible for the violation, including
deletion of the covered model and the weights utilized in that model, as
provided. Monetary damages, including punitive damages, to persons
aggrieved and a court order for a full shutdown are also available. However,
these remedies are only available in response to harm or an imminent risk or
threat to public safety. The Attorney General may also recover a civil penalty
in an amount not exceeding 10 percent of the cost, excluding labor, to develop
the covered model for a first violation and in an amount not exceeding 30
percent of the cost, excluding labor, to develop the covered model for any
subsequent violation. Specified remedies are not available until six months to
one year after enactment.

8) Clarifies that the duties and obligations imposed are cumulative with any other

duties or obligations imposed under other law and shall not be construed to
relieve any party from any duties or obligations imposed under other law and
do not limit any rights or remedies under existing law.

9) Tasks CDT with creating a public cloud computing cluster known as

CalCompute through the commissioning of consultants with specified
objectives, first of which is to study the safe and secure deployment of largescale AI models.

10) Authorizes CDT to receive private donations, grants, and local funds, in

addition to allocated funding in the annual budget, to effectuate the
establishment of CalCompute.

**Background**

Owing to recent advances in processing power and the rise of big data, artificial
intelligence’s (AI) capacity and the scope of its applications have expanded


-----

Page 6

rapidly, impacting how we communicate, interact, entertain ourselves, travel,
transact business, and consume media. It has been used to accelerate productivity,
achieve efficiencies, liberate us from drudgery, write our college essay, help us
understand and enjoy the world, upgrade the Pope’s fashion, connect with each
other, and live longer, fuller lives. It has also been used to constrain personal
autonomy, compromise privacy and security, foment social upheaval, exacerbate
inequality, spread misinformation, and subvert democracy. For good or ill, its
transformative potential seems boundless.

This bill seeks to establish guardrails for the development of the most powerful AI
models to avoid the more catastrophic possibilities about which experts have raised
alarms. It places a series of obligations on developers of “covered models” and
providers of the cloud compute for training such models. This bill also seeks to
establish a public cloud-computing cluster that facilitates equitable participation in
the development and deployment of responsible AI systems. This bill is cosponsored by the Center for AI Safety Action Fund, Economic Security California
Action, and Encode Justice. It is supported by a host of tech companies and labor
organizations, as well as the Los Angeles Area Chamber of Commerce, and
opposed by the Chamber of Progress and a coalition of industry associations.

**Comments**

According to the author:

Large-scale artificial intelligence has the potential to produce an
incredible range of benefits for Californians and our economy—from
advances in medicine and climate science to improved wildfire
forecasting and clean power development. It also gives us an
opportunity to apply hard lessons learned over the last decade, as
we’ve seen the consequences of allowing the unchecked growth of
new technology without evaluating, understanding, or mitigating the
risks. SB 1047 does just that, by developing responsible, appropriate
guardrails around development of the largest, most powerful AI
systems, to ensure they are used to improve Californians’ lives,
without compromising safety or security.

SB 1047 will also promote the growth of the AI industry by
establishing CalCompute, a public AI research cluster that will allow
startups, researchers, and community groups to participate in the
development of large-scale AI systems. By providing a broad range of
stakeholders with access to the AI development process, CalCompute


-----

Page 7

will help align large-scale AI systems with the values and needs of
California communities.

**FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes**

According to the Senate Appropriations Committee:

-  CDT indicates significant one-time costs and ongoing costs (General Fund) to

implement this bill.

-  Judicial Council estimates minor and absorbable costs.

-  Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to

the courts.

-  Potential non-reimbursable annual costs (local funds, General Fund) in the

hundreds of thousands of dollars to counties for increased incarceration costs
relating to the expansion of felony perjury in this bill.

**SUPPORT: (Verified 5/17/24)**

Center for AI Safety Action Fund (co-source)
Economic Security Project Action (co-source)
Encode Justice (co-source)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program
Denizen
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Elicit
Enh Alpha LLC
Far AI, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity


-----

Page 8

Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Lionheart Ventures
Los Angeles Area Chamber of Commerce
Loveable Labs Incorporated
MIT AI Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group

**OPPOSITION:** (Verified 5/17/24)

Association of National Advertisers
California Chamber of Commerce
California Manufacturers and Technology Association
Chamber of Progress
Civil Justice Association of California
Computer and Communications Industry Association
Insights Association
Software and Information Industry Association
Technet

**ARGUMENTS IN SUPPORT: Encode Justice, a co-sponsor of the bill, writes:**

SB 1047 introduces essential safeguards for the creation of highly capable
AI models, often known as “frontier AI models.” These models are defined
in the bill as trained using over 10^26 floating-point operations. Models of
this scope would cost at least $100 million to develop and, notably, do not


-----

Page 9

yet publicly exist but are anticipated to emerge soon as technological
advancements continue.

These are advanced, resource-intensive projects that have caught attention at
the highest levels of government and are the focus of President Biden’s
Executive Order on Artificial Intelligence for their significant national
security and public safety implications.

**ARGUMENTS IN OPPOSITION: Chamber of Progress writes:**

Unfortunately, SB 1047 forces model developers to engage in speculative
fiction about imagined threats of machines run amok, computer models
spun out of control, and other nightmare scenarios for which there is no
basis in reality.

Instead, SB 1047 forces developers operating in the real world to
proactively mitigate against every conceivable harm - and many
inconceivable ones - not just by the model itself, but subsequent third
parties who make use of the model.

Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
5/20/24 18:52:26

****** END ******


-----

