Office of Senate Floor Analyses
(916) 651-1520  Fax: (916) 327-4478

UNFINISHED BUSINESS

Bill No: SB 1047

Author: Wiener (D), et al.

Amended: 8/22/24
Vote: 21

SENATE JUDICIARY COMMITTEE: 9-0, 4/2/24
AYES: Umberg, Allen, Ashby, Caballero, Durazo, Laird, Min, Stern, Wahab
NO VOTE RECORDED: Wilk, Niello

SENATE GOVERNMENTAL ORG. COMMITTEE: 11-0, 4/23/24
AYES: Dodd, Wilk, Ashby, Bradford, Glazer, Ochoa Bogh, Padilla, Portantino,

Roth, Rubio, Smallwood-Cuevas

NO VOTE RECORDED: Alvarado-Gil, Archuleta, Jones, Nguyen, Seyarto

SENATE APPROPRIATIONS COMMITTEE: 5-0, 5/16/24
AYES: Caballero, Ashby, Becker, Bradford, Wahab
NO VOTE RECORDED: Jones, Seyarto

SENATE FLOOR: 32-1, 5/21/24
AYES: Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero,

Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón,
McGuire, Menjivar, Min, Newman, Ochoa Bogh, Padilla, Portantino, Roth,
Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NOES: Grove
NO VOTE RECORDED: Allen, Alvarado-Gil, Dahle, Jones, Nguyen, Niello,

Seyarto

ASSEMBLY FLOOR: 48-16, 8/28/24 - See last page for vote

**SUBJECT: Safe and Secure Innovation for Frontier Artificial Intelligence**

Models Act


-----

Page 2

**SOURCE: Center for AI Safety Action Fund**
Economic Security Project Action

Encode Justice

**DIGEST: This bill requires developers of powerful artificial intelligence models**
and those providing the computing power to train such models to put appropriate
safeguards and policies into place to prevent critical harms. This bill establishes a
state entity to oversee the development of these models and calls for the creation of
a consortium to develop a framework for a public cloud computing cluster.

_Assembly Amendments restructure the bill and drastically narrow the requirements_
imposed on developers.

**ANALYSIS:**

Existing law:

1) Establishes the California Department of Technology (CDT) within the

Government Operations Agency (GovOps), under the supervision of the
Director of Technology (Director), also known as the State Chief
Information Officer. (Gov. Code Sec. 11545(a).)

2) Provides that persons are responsible, not only for the result of their willful

acts, but also for an injury occasioned to another by their want of ordinary
care or skill in the management of their property or person, except so far as
the latter has, willfully or by want of ordinary care, brought the injury upon
themselves. (Civ. Code § 1714(a).)

This bill:

1) Establishes the Safe and Secure Innovation for Frontier Artificial Intelligence

Models Act.

2) Requires developers of "covered models"—AI models trained or fine-tuned

with a specified level of compute power and that meet specified cost
thresholds—and certain derivative models to, among other things:

a) Before training the model, implement certain cybersecurity protections, the

capability to promptly enact a full shutdown, a written safety and security
protocol, and take reasonable care to implement appropriate measures to


-----

Page 3

prevent "critical harms"—defined as mass casualties, at least $500 million in
damage, or other comparable harms.

b) Before using the model or making it publicly available: assess whether the

model is reasonably capable of causing or materially enabling a critical
harm; record and retain test results from the assessment; and take reasonable
care to implement appropriate safeguards.

c) Neither use, unless exclusively for training or evaluation, nor make publicly

available a model if there is an unreasonable risk it will cause or materially
enable a critical harm.

d) Beginning in 2026, annually retain a third-party auditor to perform an

independent audit of a developer's compliance with applicable duties.

e) Make public and provide to the Attorney General redacted copies of safety

and security protocol and auditors' reports. Upon request, provide to the
Attorney General unredacted copies of those documents, which are exempt
from the California Public Records Act. Submit annual compliance
statements to the Attorney General. Report to the Attorney General safety
incidents within 72 hours.

3) Requires a person operating a computing cluster to implement appropriate

written policies and procedures to take specified actions when a customer
utilizes compute resources that would be sufficient to train a covered model,
including obtaining a prospective customer’s identifying information and
information to assess whether the customer intends to use the cluster to train a
covered model.

4) Authorizes the Attorney General to bring a civil action for violations of the bill

for specified civil penalties and injunctive and declaratory relief.

5) Provides whistleblower protections for employees including by prohibiting a

developer of a covered model from preventing an employee from disclosing
information, or retaliating against an employee for disclosing information, to
the Attorney General or Labor Commissioner if the employee has reasonable
cause to believe the information indicates the developer is out of compliance
with certain requirements or that the covered model poses an unreasonable risk
of critical harm.

6) Creates the Board of Frontier Models within GovOps, consisting of nine

members, five appointed by the Governor and four by the Legislature and
establishes certain qualifications and requirements for members of the board.


-----

Page 4

7) Requires GovOps, by 2027, to issue guidance for preventing unreasonable risks

and adopt regulations to update the definition of a "covered entity" and establish
auditing standards, subject to approval by the Board of Frontier Models.

8) Establishes, upon appropriation, in GovOps, a consortium required to develop a

framework for the creation of a public cloud computing cluster to be known as
"CalCompute" that advances the development and deployment of AI, as
prescribed.

9) Provides definitions for the relevant terms.

**Background**

Owing to recent advances in processing power and the rise of big data, artificial
intelligence’s (AI) capacity and the scope of its applications have expanded
rapidly, impacting how we communicate, interact, entertain ourselves, travel,
transact business, and consume media. It has been used to accelerate productivity,
achieve efficiencies, liberate us from drudgery, write our college essay, help us
understand and enjoy the world, upgrade the Pope’s fashion, connect with each
other, and live longer, fuller lives. It has also been used to constrain personal
autonomy, compromise privacy and security, foment social upheaval, exacerbate
inequality, spread misinformation, and subvert democracy. For good or ill, its
transformative potential seems boundless.

This bill seeks to establish guardrails for the development of the most powerful AI
models to avoid the more catastrophic possibilities about which experts have raised
alarms. It places a series of obligations on developers of “covered models” and
providers of the cloud compute for training such models. This bill also seeks to
establish a framework for developing a public cloud-computing cluster that
facilitates equitable participation in the development and deployment of
responsible AI systems. This bill is co-sponsored by the Center for AI Safety
Action Fund, Economic Security California Action, and Encode Justice. It is
supported by a host of tech companies and labor organizations, as well as the Los
Angeles Area Chamber of Commerce, and opposed by the Chamber of Progress
and a coalition of industry associations.

**Comments**

According to the author:

Large-scale artificial intelligence has the potential to produce an
incredible range of benefits for Californians and our economy—from
advances in medicine and climate science to improved wildfire


-----

Page 5

forecasting and clean power development. It also gives us an
opportunity to apply hard lessons learned over the last decade, as
we’ve seen the consequences of allowing the unchecked growth of
new technology without evaluating, understanding, or mitigating the
risks. SB 1047 does just that, by developing responsible, appropriate
guardrails around development of the largest, most powerful AI
systems, to ensure they are used to improve Californians’ lives,
without compromising safety or security.

**FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No**

According to the Senate Appropriations Committee:

-  CDT indicates significant one-time costs and ongoing costs (General Fund) to

implement this bill.

-  Judicial Council estimates minor and absorbable costs.

-  Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to

the courts.

-  Potential non-reimbursable annual costs (local funds, General Fund) in the

hundreds of thousands of dollars to counties for increased incarceration costs
relating to the expansion of felony perjury in this bill.

According to the Assembly Appropriations Committee:

1) Costs (General Fund) to GovOps to establish and operate the FMD and the

Board. GovOps reports ongoing annual costs between $5 million and $10
million.

2) Costs (General Fund) to CDT to establish and operate CalCompute. In the first

year of implementation, CDT reports costs of $508,000 for two temporary
positions, $6 million for external consultants, and $1 million for GenAI talent
practices to train state employees. In subsequent years, CDT anticipates costs
of $3.7 million for 16 additional positions, $3 million for external consultants,
and $700,000 for GenAI talent practices. The bill authorizes CDT to receive
private donations, grants, and local funds to support CalCompute, which may
offset General Fund costs by an unknown amount.


-----

Page 6

CDT notes it is challenging to hire GenAI talent in the public sector since
salaries are often two or three times higher in the private sector, and private
sector employers can offer bonuses, equity, and stock options. As a result, CDT
reports, its fiscal estimates for this bill include significant funding for external
consultants, and actual costs for consultants may be higher if the state does not
provide sufficient funding for training of state employees. GovOps will likely
also face these challenges and additional cost pressures.

3) Costs (General Fund, Labor and Enforcement Compliance Fund) of an

unknown but potentially significant amount to the Department of Justice (DOJ)
and Labor Commissioner to enforce violations of the bill. Costs to DOJ may be
in the high hundreds of thousands to low millions of dollars annually. Actual
costs will depend on the number of violations, the number of actions filed, and
the amount of workload associated with each action.

4) Costs (Trial Court Trust Fund, General Fund) to the courts to adjudicate

violations of the bill and whistleblower cases. Actual costs will depend on the
number of violations, the number of actions filed, and the amount of court time
needed to resolve each case. Although courts are not funded on the basis of
workload, increased pressure on the Trial Court Trust Fund may create a need
for increased funding for courts from the General Fund. The fiscal year 202425 budget provides $37.3 million ongoing General Fund to backfill declining
revenue to the Trial Court Trust Fund.

5) Minor and absorbable costs to Judicial Council to review and process the model

jury instructions recommended by the FMD, as required by the bill.

According to the Legislative Analyst’s Office, the General Fund faces a structural
deficit in the tens of billions of dollars over the next several fiscal years.

**SUPPORT: (Verified 8/28/24)**

Center for AI Safety Action Fund (co-source)
Economic Security Project Action (co-source)
Encode Justice (co-source)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program


-----

Page 7


Denizen
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Elicit
Enh Alpha LLC
Far AI, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity
Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Los Angeles Area Chamber of Commerce
Loveable Labs Incorporated
MIT AI Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group

**OPPOSITION:** (Verified 8/28/24)

8vdx
Abundance Institute
Adacta Labs
Aidy


-----

Page 8


Alec Action
Algolia
Alloy Automation
American Consumer Institute
Andi
Andy AI
Anneal
Antifragile Research Dba Creatorml
Apriora
Argovox
Association of National Advertisers
Baseline AI
Bilanc Finance INC.
Brighterway
California Chamber of Commerce
California Manufacturers and Technology Association
Candoriq
Canonical AI INC
Center for Data Innovation
Chamber of Progress
Chima
Circle Medical
Citron Labs
Civil Justice Association of California
Clarum
Clerky
Cointracker
Competitive Enterprise Institute
Computer and Communications Industry Association
Constructable
Consumer Technology Association
Context Fund
Curtsy
Cyble
Deconvolute AI
Deferred
Dianahr
Digital First Project
Distill
Docsum


-----

Page 9


Double
Double Finance
Eden - Workplace Management Software
Edge
Ello
Elythea
Entori, INC
Envelope
Extensional, INC
Fetchflow
Figments Corp
Firebender Corp
Flexdesk
Focal INC
Fondo
Gigaml
Gleam
Greptile
Happyrobot INC.
Hazel
Hedgy Labs
Insights Association
International Center for Law & Economics
Invert INC
Invoke Labs
Jo
K-scale Labs
Keywords Ai INC.
Kontractify, INC
Lightski
Linc AI
Linum
Listening.com
Local Reg
Los Angeles Area Chamber of Commerce
Lovecast INC Dba Stably AI
Lumona
Markprompt
Marr Labs Technologies, INC
Martian Mobility INC. Dba Telematica


-----

Page 10


Mayor of City & County of San Francisco London Breed
Metriport
Miru
Movley
Octolane AI
Oneshop
Onward
Openmart
Openpod
Orange Collective
Pabio INC.
Patched Codes, INC
Patika Technology, INC
Patterns
Persist AI
Phonely
Picnichealth
Playground
Poll Everywhere
Preloop INC.
Q8 Empowering Change
Query Vary
R Street Institute
Rainforest Qa, INC.
Regology INC
Rejuvenation Technologies INC
Retell AI
Revamp AI
Reworkd AI
Rinsed
Rippling
Risotto
Rocketlit INC.
Rootly
Sela
Sepal AI
Sideguide Technologies INC (known As Firecrawl)
Silicon Valley Leadership Group
Simplify
Software and Information Industry Association


-----

Page 11

Sorting Robotics
Spoken
Sprx
Squire AI
Stack AI
Sterling Road
Storyquest, INC.
Storyworth
Sway Finance
Sync Labs
Synova Life Sciences
Tailor
Taxgpt INC
Technet
Tensor
Tesorio INC
Tracecat
Trainy
Truewind Technologies, INC.
Turbolayer INC. Dba Automatica
Turboprop
Turing Labs
Tusk
Twine
Two Dots
Up Dog
Upkeep
Viva Labs
Walrus Tech INC
Willow
Wordware
Y Combinator
Zapier
Zaymo
Zep AI

**ARGUMENTS IN SUPPORT: Encode Justice, a co-sponsor of the bill, writes:**

SB 1047 introduces essential safeguards for the creation of highly capable
AI models, often known as “frontier AI models.” These models are defined


-----

Page 12

in the bill as trained using over 10^26 floating-point operations. Models of
this scope would cost at least $100 million to develop and, notably, do not
yet publicly exist but are anticipated to emerge soon as technological
advancements continue.

These are advanced, resource-intensive projects that have caught attention at
the highest levels of government and are the focus of President Biden’s
Executive Order on Artificial Intelligence for their significant national
security and public safety implications.

**ARGUMENTS IN OPPOSITION: Chamber of Progress writes:**

Unfortunately, SB 1047 forces model developers to engage in speculative
fiction about imagined threats of machines run amok, computer models
spun out of control, and other nightmare scenarios for which there is no
basis in reality.

Instead, SB 1047 forces developers operating in the real world to
proactively mitigate against every conceivable harm - and many
inconceivable ones - not just by the model itself, but subsequent third
parties who make use of the model.

ASSEMBLY FLOOR: 48-16, 8/28/24
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Bauer-Kahan, Bennett, Berman,

Bonta, Bryan, Calderon, Juan Carrillo, Connolly, Mike Fong, Friedman, Gabriel,
Garcia, Gipson, Grayson, Haney, Hart, Holden, Jones-Sawyer, Kalra, Lee,
Lowenthal, Maienschein, Mathis, McKinnor, Muratsuchi, Pacheco, Papan,
Pellerin, Ramos, Rendon, Reyes, Blanca Rubio, Santiago, Schiavo, Soria, Ting,
Waldron, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas

NOES: Alanis, Bains, Davies, Dixon, Hoover, Irwin, Low, McCarty, Jim

Patterson, Joe Patterson, Petrie-Norris, Luz Rivas, Sanchez, Ta, Valencia, Wallis

NO VOTE RECORDED: Boerner, Wendy Carrillo, Cervantes, Chen, Megan

Dahle, Essayli, Flora, Gallagher, Jackson, Lackey, Stephanie Nguyen, Ortega,
Quirk-Silva, Rodriguez, Villapudua

Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
8/28/24 22:40:24


-----

Page 13


****** END ******


-----

