# SENATE COMMITTEE ON APPROPRIATIONS

**Senator Anna Caballero, Chair**

**2023 - 2024 Regular Session**

**SB 1047 (Wiener) - Safe and Secure Innovation for Frontier Artificial Intelligence**
**Models Act**

**Version: April 30, 2024** **Policy Vote: JUD. 9 - 0, G.O. 11 - 0**

**Urgency: No** **Mandate: Yes**

**Hearing Date: May 6, 2024** **Consultant: Liah Burnley**


**Bill Summary: SB 1047 requires developers of artificial intelligence (AI) models to**
implement safeguards and policies to prevent specified hazards and threats to public
safety and authorizes the Attorney General (AG) to enforce these provisions. SB 1047
also establishes the Frontier Model Division within the Department of Technology (CDT)
to oversee the development of AI models and requires CDT to commission consultants
to create a public cloud computing cluster “CalCompute,” to conduct research into the
safe and secure deployment of large-scale AI models.

**Fiscal Impact:**

-  CDT indicates significant one-time costs and ongoing costs (General Fund) to

implement this bill. According to CDT one-time costs estimated over $7,000,000, to
recruit and retain staff and external consultants and ongoing costs estimated over
$9,000,000 to staff 24 positions and external consultants. CDT notes that this bill
would require technical, analytic, administrative, operational, programming,
research, and design work. CDT would need to establish a new team focused solely
on building out this new division and begin to define, scope, and implement initial
stages of CalCompute. As a result, CDT estimates that it would need 26 general
fund positions, two temporary program and project managers to design, develop,
and implement this new division, with 24 ongoing positions to adequately and
sustainably create, manage, and secure the Frontier Model Division and
CalCompute. CDT notes the importance of integrating in-house talent to ensure
proper oversight in responsible use of AI across strategy, design, and technical
functions. These foundational estimates include primarily full-time employees, with
an additional request for funding to source and award contracts for external
consultants necessary to provide expertise in the short-term within defined scopes of
work to be determined by CDT. In parallel, a recruitment strategy specifically built to
attract and retain talent is necessary to meet increased demand for AI expertise
within and outside of the CDT.

-  Judicial Council estimates minor and absorbable costs within existing resources to

review and process the model jury instructions recommended by the Frontier Model
Division, as required by this bill. According to Judicial Council, the language in this
bill allows sufficient flexibility for the Judicial Council to utilize existing procedures
including an advisory committee review, public comment period, and approval by the
Judicial Council prior to official adoption and dissemination of any new jury
instructions.


-----

-  Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to the

courts to adjudicate potential civil enforcement actions for violations of this bill
brought by the AG. The fiscal impact of this bill to the courts will depend on many
unknown factors, including the numbers of violations alleged to have occurred, if
parties settle the matter before the filing of an action, and the factors unique to each
case. While it is not known how many actions would be filed resulting from the
various changes that this measure would make, it generally costs about $8,300 to
operate a courtroom for one eight-hour day. Consequently, if reforms proposed by
this bill lead to the filing of cases that, combined, take 50 or more hours of court
involvement, the cost pressures of this measure to the courts would be significant.
This bill would further impact to the courts to adjudicate criminal perjury charges
brought against defendants who violate the bill’s perjury provisions. A defendant is
entitled to a no cost legal representation and a jury trial. If only four new perjury
charge is filed annually statewide and proceed to trial resulting in the use of two
days of court time the approximate cost to the trial courts is $66,400. While the
superior courts are not funded on a workload basis, an increase in workload could
result in delayed court services and pressure to fund additional staff and resources.
The Governor’s 2024-25 state budget proposes $83.1 million ongoing General Fund
to continue to backfill the Trial Court Trust Fund for expected revenue declines.

-  Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds

of thousands of dollars to counties for increased incarceration costs relating to the
expansion of felony perjury in this bill. Actual incarceration costs will depend on the
number of convictions, the length of each sentence. The average annual cost to
incarcerate one person in county jail is approximately $29,000. Based on the
average annual cost to incarcerate a defendant in county jail, if one defendant
statewide is sentenced annually to an average of three years in county jail for
perjury, the total cost to counties would be $87,000. California county jails are
increasingly overcrowded – particularly since the enactment of the Realignment Act
in 2011. Although new crimes are not considered reimbursable state mandates
pursuant to Proposition 30 (2012), overcrowding in county jails creates cost pressure
on the General Fund because the state has historically granted new funding to
counties to offset overcrowding since 2011.

**Background: In 2018, the Little Hoover Commission released a report titled, “Artificial**
Intelligence: A Roadmap for California” which detailed the possible benefits and risks of
AI. The report encouraged the Governor and the Legislature to: engage the public in
broad conversations about the responsible use of AI; identify the necessary training and
other educational opportunities to build the human infrastructure required for an AIfueled economy; explore opportunities to form and leverage public private partnerships
to use AI to address societal challenges; and examine legislative and policy strategies
to preserve and protect key public values, such as privacy, transparency and
accountability.

**Proposed Law:  This bill does all of the following:**

-  Includes numerous definitions, including, among others:


-----

```
o “AI model” means an engineered or machine-based system that infers,

```
from the input it receives, how to generate outputs that can influence
physical or virtual environments and that may operate with varying levels
of autonomy.
```
o “Computing cluster” means a set of machines connected by data center

```
networking that has a specified computing capacity and can be used for
training AI.
```
o “Covered model” means an AI model that was trained using a specified

```
quantity of computing power.

```
o “Developer” means a person or entity that creates, owns, or otherwise has

```
responsibility for an AI model.
```
o “Limited duty exemption” means an exemption, with respect to a covered

```
model that a developer can reasonably exclude the possibility has a
hazardous capability or may come close to possessing a hazardous
capability, as specified.

```
o “Hazardous capability” means the capability of a covered model to be

```
used to enable any of the following harms in a way that would be
significantly more difficult to cause without access to a covered model:

-  The creation or use of a chemical, biological, radiological, or

nuclear weapon in a manner that results in mass casualties;



-  At least five hundred million dollars $500,000,000 of damage

through cyberattacks on critical infrastructure via a single incident
or multiple related incidents;

-  At least $500,000,000 of damage by an AI model that

autonomously engages in conduct that would violate the Penal
Code if undertaken by a human; or,



-  Other threats to public safety and security that are of comparable

severity.

-  Provides that, before initiating training of a covered model, a developer may

determine whether the covered model qualifies for an exemption, as specified.
Upon determining that a covered model qualifies for an exemption, the developer
must submit to the Frontier Model Division a certification under penalty of perjury
that specifies the basis for that determination.

-  Provides that, before initiating training of a covered model that and is not the

subject of an exemption, and until that covered model is the subject of an
exemption, the developer of that covered model shall implement protections to
prevent unauthorized access to, or misuse or unsafe modification of, the covered
model, as specified.


-----

-  Provides that, upon completion of the training of a covered model that is not the

subject of an exemption, the developer shall perform capability testing sufficient
to determine if an exemption applies. Upon determining if an exemption applies
the developer shall submit to the Frontier Model Division, under penalty of
perjury, a certification of compliance within 90 days and no more than 30 days
after initiating the commercial, public, or widespread use of the covered model,
as specified.

-  Provides that, before initiating the commercial, public, or widespread use of a

covered model that is not subject to an exemption, the developer must implement
reasonable safeguards and requirements, as specified.

-  Requires a developer of a covered model that is not the subject of an exemption

to submit to the Frontier Model Division an annual certification under penalty of
perjury of compliance signed by the chief technology officer, or a more senior
corporate officer.

-  Requires developers to report all AI safety incidents affecting a covered model to

the Frontier Model Division in a manner prescribed by the Frontier Model
Division.



-  Requires a developer of a covered model that provides commercial access to

that covered model to provide a transparent, uniform, publicly available price
schedule for the purchase of access to that covered model.

-  Requires a person who operates a computing cluster to implement written

policies and procedures when a customer utilizes compute resources that would
be sufficient to train a covered model, as specified.



-  Requires a person that operates a computing cluster to provide a transparent,

uniform, publicly available price schedule for the purchase of access to the
computing cluster.

-  Authorizes the Attorney General to commence civil proceedings to enforce these

provisions and requires the court to disregard corporate formalities and impose
joint and several liability on affiliated entities, as specified.

-  Creates the Frontier Model Division within CDT to do all of the following:

```
o Compile annual certification reports received from developers and publicly

```
release summarized findings based on those reports;
```
o Advise the AG on potential violations of the provisions in this bill;
o Issue guidance, standards, and best practices necessary to prevent

```
unreasonable risks from covered models with hazardous capabilities;
```
o Establish an optional accreditation process and relevant accreditation

```
standards;

P bli h i d AI f t i id t t


-----

```
o Appoint and consult an advisory committee that shall, among other things,

```
advise the Governor on when it may be necessary to proclaim a state of
emergency relating to AI and advise the Governor on what responses may
be appropriate in that event and to provide technical assistance and
advice to the Legislature, upon request, with respect to AI-related
legislation;
```
o Levy fees, including a fee for the submission of a certification, in an

```
amount sufficient to cover the reasonable costs of administration; and,

```
   o Develop and submit to the Judicial Council proposed model jury

```
instructions for actions involving violations provisions of this bill, that the
Judicial Council may, at its discretion, adopt develop guidance to
developers regarding development of AI models, as specified.

-  Creates the Frontier Model Division Programs Fund within the General Fund. All

fees received by the Frontier Model Division shall be deposited into the fund. All
moneys in the account shall be available, only upon appropriation by the
Legislature, for purposes of carrying out the provisions of this bill relating to the
Frontier Model Division.



-  Establishes CalCompute, a public cloud computing cluster, within CDT with the

primary focus of conducting research into the safe and secure deployment of AI
and fostering equitable innovation.

-  Requires CDT to commission consultants for CalCompute that are

representatives of national laboratories, universities, and any relevant
professional associations or private sector stakeholders to meet specified
objectives.



-  Requires CDT to submit an annual report to the Legislature from the

commissioned consultants to ensure progress in meeting specified objectives.



-  Provides that CDT may receive private donations, grants, and local funds, in

addition to allocated funding in the annual budget.

**Related Legislation:**

-  SB 970 (Ashby, 2024) requires Judicial Council to review the impact of AI on

court proceedings and develop rules of court for claims that evidence has been
generated by or manipulated by AI. AB 970 is pending in this Committee.



-  SB 942 (Becker, 2024) establishes the California AI Transparency Act, which,

among other things, requires a provider, to create an AI detection tool by which a
person can query the covered provider as to the extent to which text, image,
video, audio, or multimedia content was created, in whole or in part, by a
generative AI system, as defined, provided by the covered provider that meets
certain criteria. AB 942 is pending in this Committee.


-----

-  AB 2930 (Bauer-Kahan, 2024) requires, among other things, a developer of an

automated decision tool to perform an impact assessment of the automated
decision tool. AB 2930 is pending in Assembly Appropriations Committee.

-  AB 3211 (Wicks, 2024) establishes the California Provenance, Authenticity and

Watermarking Standards Act, which requires an AI system provider, as defined,
to assist in the disclosure of data to mitigate harms caused by inauthentic
content. AB 3211 is pending in Assembly Appropriations Committee.



-  AB 1576 (Calderon, 2019) would have required the Secretary of Government

Operations to appoint participants to an AI working group to evaluate the uses,
risks, benefits, and legal implications associated with the development and
deployment of AI by California-based businesses. AB 1576 was held in this
Committee.

-  SB 348 (Chang, 2019) would have require CDT to devise a strategic plan

designed to aid departments and agencies with the incorporation of AI into state
information technology strategic plans, policies, standards, and enterprise
architecture to improve state services. SB 348 failed passage in Assembly
Appropriations Committee.



-  SB 752 (Stern, 2019) would have created the Commission on Tech Equity to

convene a public process to gather input and to understand the economic, social,
workplace, and technological landscape of innovation and technology in
California. SB 752 was held in the Senate Appropriations Committee.

-  AB 594 (Salas, 2019) would have required the Director of Technology to appoint

a Chief AI Officer within the CDT to evaluate the uses of AI in state government
and to advise the Director on incorporating AI into state information technology
strategic plans, policies, standards, and enterprise architecture. AB 594 was
vetoed.

**Staff Comments: This bill is one of a handful of bills related to AI technology in this**
Legislative Session. While all these bills are not focused on the same aspects of AI
technology, many of them would require duplicative work and could result in
inconsistent recommendations to the Legislature.

**-- END --**


-----

