**SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION**

**Senator Bill Dodd**

**Chair**

**2023 - 2024 Regular**

**Bill No:      SB** 1047 **Hearing Date:  4/23/2024**

**Author:** Wiener, et al.

**Version:** 4/16/2024 Amended

**Urgency:** No **Fiscal:** Yes

**Consultant:** Brian Duke


**SUBJECT: Safe and Secure Innovation for Frontier Artificial Intelligence Models**
Act

**DIGEST:  This bill, the Safe and Secure Innovation for Frontier Artificial**
Intelligence (AI) Systems Act, requires developers of powerful AI models and
those providing the computing power to train such models to place appropriate
safeguards and policies to prevent critical harms, as specified. Additionally, this
bill creates the Frontier Model Division within the California Department of
Technology (CDT) to oversee the development of those models and requires CDT
to create a public cloud computing cluster, to be known as CalCompute, as
specified.

**ANALYSIS:**

Existing law:

1) Establishes CDT, within the Government Operations Agency (GovOps), and

requires CDT to, among other things, identify, assess, and prioritize high-risk,
critical information technology services and systems across state government
for modernization, stabilization, or remediation.

2) Provides that persons are responsible, not only for the result of their willful acts,

but also for an injury occasioned to another by their want of ordinary care or
skill in the management of their property or person, except so far as the latter
has, willfully or by want of ordinary care, brought the injury upon themselves.


3) The Emergency Services Act authorizes the Governor to declare a state of

emergency, and local officials and local governments to declare a local
emergency, when specified conditions of disaster or extreme peril to the safety
of persons and property exist.


-----

4) Defines the term “state of emergency” and “local emergency” to mean a duly

proclaimed existence of conditions of disaster or of extreme peril to the safety
of persons and property within the state caused by, among other things, fire,
storm, riot, or cyberterrorism, as specified.

5) Requires a report that is required or requested by law to be submitted by a state

or local agency to the Members of either house of the Legislature to instead be
submitted as a printed copy to the Secretary of the Senate, as an electronic copy
to the Chief Clerk of the Assembly, and as an electronic or printed copy to the
Legislative Counsel. Each report shall include a summary of its contents, not to
exceed one page in length.


This bill:

1) Establishes the Safe and Secure Innovation for Frontier AI Systems Act, and

provides definitions for relevant terms, including:

a. “Advanced persistent threat” means an adversary with sophisticated levels of

expertise and significant resources that allow it, through the use of multiple
different attack vectors, as specified.

b. “AI model” means an engineered or machine-based system that, for explicit

or implicit objectives, infers, from the input it receives, how to generate
outputs that can influence physical or virtual environments and that may
operate with varying levels of autonomy.

c. “Computing cluster” means a set of machines transitively connected by data

center networking of over 100 gigabits per second that has a theoretical
maximum computing capacity of at least 10^20 integer or floating-point
operations (FLOP) per second and can be used for training AI.

d. “Covered guidance” means guidance issued by the National Institute of

Standards and Technology (NIST) and by the Frontier Model Division that
is relevant to the management of safety risks associated with AI models that
may possess hazardous capabilities or industry best practices, including
safety practices, precautions, or testing procedures undertaken by developers
of comparable models that are relevant to the management of safety risks
associated with AI models that may possess hazardous capabilities.

e. “Covered model” means an AI model that was trained using a quantity of

computing power greater than 10^26 integer or FLOP or a model that can
reasonably be expected to have similar performance capabilities as assessed
by commonly used benchmarks, as specified.

f. “Critical infrastructure” means assets, systems, and networks, whether

physical or virtual, the incapacitation or destruction of which would have a


-----

debilitating effect on physical security, economic security, public health, or
safety in the state.

g. “Derivative model” means an AI model that is a derivative of another AI

model, including a modified or unmodified copy of an AI model or a
combination of an AI model with other software. Derivative model does not
include an entirely independently trained AI model.

h. “Full shutdown” means the cessation of operation of a covered model,

including all copies and derivative models, on all computers and storage
devices within custody, control, or possession of a person, including any
computer or storage device remotely provided by agreement.

i. “Hazardous capability” means the capability of a covered model to be used

to enable any of the following harms in a way that would be significantly
more difficult to cause without access to a covered model:

i. The creation or use of a chemical, biological, radiological, or

nuclear weapon in a manner that results in mass casualties.

ii. At least $500 million of damage through cyberattacks on

critical infrastructure via a single incident or multiple related
incidents.

iii. At least $500 million of damage by an AI model that

autonomously engages in conduct that would violate the Penal
Code if undertaken by a human.

iv. Other threats to public safety and security that are of

comparable severity to the harms described above.

j. “Limited duty exemption” means an exemption with respect to a covered

model that is not a derivative model that a developer can reasonably exclude
the possibility that a covered model has a hazardous capability or may come
close to possessing a hazardous capability when accounting for a reasonable
margin for safety and the possibility of post-training modifications.

k. “Model weight” means a numerical parameter established through training

in an AI model that helps determine how input information impacts a
model’s output.

l. “Open-source AI model” means an AI model that is made freely available

and may be freely modified and redistributed.

m. “Post-training modification” means the modification of the capabilities of an

AI model after the completion of training by any means, including, but not
limited to, initiating additional training, providing the model with access to
tools or data, removing safeguards against hazardous misuse or misbehavior
of the model, or combining the model with, or integrating it into, other
software.

n. “Safety and security protocol” means documented technical and

organizational protocols that are used to manage the risks of developing and
operating covered models across their life cycle, including risks posed by


-----

enabling or potentially enabling the creation of derivative models or the
protocols specify that compliance with the protocols is required in order to
train, operate, possess, and provide external access to the developer’s
covered model.

2) Creates the Frontier Model Division, within CDT, and requires the division to

do all of the following:


a. Review annual certification reports received from developers and publicly

release summarized findings based on those reports, as specified.

b. Advise the Attorney General (AG) on potential violations of this bill, as

specified.

c. Issue guidance, standards, and best practices, as specified, and establish an

optional accreditation process and relevant accreditation standards, as
specified.

d. Publish anonymized AI safety incident reports received from developers, as

specified.

e. Establish confidential fora that are structured and facilitated in a manner that

allows developers to share best risk management practices, as specified.

f. Issue guidance describing the categories of AI safety events that are likely to

constitute a state of emergency, as specified, and responsive actions that
could be ordered by the Governor after a duly proclaimed state of
emergency.

g. Appoint and consult with an advisory committee to advise the Governor on

when it may be necessary to proclaim a state of emergency relating to AI
and advise the Governor on what responses might be appropriate in that
event.

h. Appoint and consult with an advisory committee for open-source AI to issue

guidelines for model evaluation and advise the Frontier Model Division, as
specified.

i. Provide technical assistance and advice to the Legislature, upon request,

with respect to AI-related legislation.

j. Monitor relevant developments relating to the safety risks associated with

the development of AI models and the functioning of markets for AI models.

k. Levy fees, including an assessed fee for the submission of a certification, as

specified.

l. Develop and submit to the Judicial Council proposed model jury instructions

for actions involving violations of this bill that the Judicial Council may, at
its discretion, adopt, as specified.

m. On or before July 1, 2026, issue guidance regarding technical thresholds and

benchmarks relevant to determining whether an AI model is a covered


-----

model, as specified, whether the covered model is subject to a limited duty
exemption, as specified.

n. At least every 24 months after initial publication of guidance, review

existing guidance in consideration of technological advancements, changes
to industry best practices, and information received, as specified.

3) Creates, in the General Fund, the Frontier Model Division Programs Fund, for

all fees received by the Frontier Model Division, and makes those fees, subject
to appropriation by the Legislature, available for the purposes of carrying out
the provisions of this bill, as specified.


4) Authorizes a developer to determine that a covered model qualifies for a limited

duty exemption if the covered model will have lower performance on all
benchmarks relevant per this bill and does not have a greater general capability
than a noncovered model that manifestly lacks hazardous capabilities or another
model that is the subject of a limited duty exemption, as specified.

5) Requires a developer, upon determining that a covered model qualifies for a

limited duty exemption, to submit to the Frontier Model Division a certification
under penalty of perjury that specifies the basis for that determination, as
specified.


6) Provides that a developer that makes a good faith error regarding a limited duty

exemption will be deemed to be in compliance with this bill if the developer
reports its error to the Frontier Model Division within 30 days of completing the
training of the covered model and ceases operation of the AI model until the
developer is otherwise in compliance, as specified.

7) Requires a developer, before initiating training of a covered model that is not a

derivative model and is not the subject of a limited duty exemption, to do all of
the following:


a. Implement administrative, technical, and physical cybersecurity protections,

as specified.

b. Implement the capability to promptly enact a full shutdown of the covered

model.

c. Implement all covered guidance.
d. Implement a written and separate safety and security protocol, as specified.
e. Ensure that the safety and security protocol is implemented as writing, as

specified.

f. Provide a copy of the safety and security protocol to the Frontier Model

Division.


-----

g. Conduct an annual review of the safety and security protocol to account for

any changes to the capabilities of the covered model and industry best
practices and, if necessary, make modifications to the policy.

h. If the safety and security protocol is modified, provide an update copy to the

Frontier Model Division within 10 business days.

i. Refrain from initiating training of a covered model if there remains an

unreasonable risk that an individual, or the covered model itself, may be able
to use the hazardous capabilities of the covered model, or a derivative model
based on it, to cause a critical harm.

j. Implement other measures that are reasonably necessary, including in light

of applicable guidance from the Frontier Model Division, NIST, and
standard-setting organizations, to prevent the development or exercise of
hazardous capabilities or to manage the risks arising from them.

8) Requires the developer of a non-derivative covered model that is not the subject

of a limited duty exemption, upon completion of the training of the covered
model, to perform capability testing sufficient to determine whether the
developer can make a positive safety determination with respect to the covered
model pursuant to its safety and security protocol.


9) Requires a developer, upon making a limited duty exemption with respect to a

covered model, to submit to the Frontier Model Division a certification under
penalty of perjury of compliance with the requirements of this bill within 90
days and no more than 30 days after initiating the commercial, public, or
widespread use of the covered model that includes the basis for the developer’s
determination that of a limited duty exemption and the specific methodology
and results of the capability testing undertaken pursuant to this bill.

10) Requires a developer, before initiating the commercial, public, or widespread

use of a covered model that is not subject to a limited duty exemption, to do all
of the following:


a. Implement reasonable safeguards and requirements, as specified.
b. Provide reasonable requirements to developers of derivative models to

prevent an individual from being able to use a derivative model to cause a
critical harm.

c. Refrain from initiating the commercial, public, or widespread use of a

covered model if there remains an unreasonable risk that an individual may
be able to use the hazardous capabilities of the model, or a derivative model
based on it, to cause a critical harm.

d. Implement other measures that are reasonably necessary, including in light

of applicable guidance from the Frontier Model Division, NIST, and


-----

standard-setting organizations, to prevent the development or exercise of
hazardous capabilities or to manage the risks arising from them.

11) Requires the developer of a covered model to periodically reevaluate the

procedures, policies, protections, capabilities, and safeguards implemented
pursuant to this section in light of the growing capabilities of covered models
and as is reasonably necessary to ensure that the covered model or its users
cannot remove or bypass those procedures, policies, protections, capabilities,
and safeguards.


12) Requires a developer of a non-derivative covered model that is not the subject

of a limited duty exemption to submit to the Frontier Model Division an annual
certification under penalty of perjury of compliance with the requirements of
this bill signed by the chief technology officer, or a more senior corporate
officer, as specified.

13) Requires the certification to specify or provide, at a minimum, the nature and

magnitude of hazardous capabilities, as specified, an assessment of the risk, as
specified, and other information useful to accomplishing the purposes of this
bill, as determined by the Frontier Model Division.


14) Requires a developer to report each AI safety incident affecting a covered

model to the Frontier Model Division, as specified.

15) Specifies that reliance on an unreasonable limited duty exemption does not

relieve a developer of its obligations under this bill, as specified.


16) Requires a person operating a computing cluster to implement appropriate

written policies and procedures to do all of the following when a customer
utilizes compute resources that would be sufficient to train a covered model:

a. Obtain a prospective customer’s basic identifying information and business

purpose for utilizing the computing cluster, as specified.

b. Assess whether a prospective customer intends to utilize the computing

cluster to deploy a covered model.

c. Annually validate the information collected pursuant to this bill and conduct

the assessment required.

d. Maintain for seven years and provide to the Frontier Model Division or the

Attorney General (AG), upon request, appropriate records of actions taken
under this bill, including policies and procedures put into effect.

e. Implement the capability to promptly enact a full shutdown in the event of

an emergency.


-----

17) Requires the developer of a covered model that provides commercial access to

that covered model and a person that operates a computing cluster to provide a
transparent, uniform, publicly available price schedule for the purchase of
access to that covered model at a given level of quality and quantity subject to
the developer’s terms of service and prohibits the developer from engaging in
unlawful discrimination or noncompetitive activity in determining price or
access, as specified.

18) Authorizes the AG, if the AG finds that a person is violating this bill, to bring a

civil action, as specified.


19) Authorizes the court, in a civil action under this bill, to award preventative

relief, including a permanent or temporary injunction, restraining order, or other
order as specified, or any other relief as the court deems appropriate, as
specified.

20) Requires a court to disregard corporate formalities and impose joint and several

liability on affiliated entities for purposes of effectuating the intent of this bill if
the court concludes that steps were taken in the development of the corporate
structure among affiliated entities to purposely and unreasonably limit or avoid
liability and the corporate structure of the developer or affiliated entities would
frustrate recovery of penalties or injunctive relief under this bill.


21) Prohibits a developer from preventing an employee from disclosing information

to the AG or from retaliating against the employee if the employee has
reasonable cause to believe that the information indicates that the developer is
out of compliance with the requirements of this bill, as specified.

22) Authorizes the AG to publicly release any complaint, or a summary of that

complaint, pursuant to this bill if the AG concludes that doing so will serve the
public interest.


23) Requires a developer to provide clear notice to all employees working on

covered models of their rights and responsibilities under this bill.

24) Requires developers to provide a reasonable internal process through which an

employee may anonymously disclose information to the developer if the
employee believes in good faith that the information indicates that the
developer is out of compliance with this bill or has made false or materially
misleading statements related to its safety and security protocol, as specified.


-----

25) Requires CDT to commission consultants, as specified, to create a public cloud

computing cluster, to be known as CalCompute, with the primary focus of
conducting research into the safe and secure deployment of large-scale AI
models and fostering equitable innovation that includes, but is not limited to, all
of the following:

a. A fully owned and hosted cloud platform.
b. Necessary human expertise to operate and maintain the platform.
c. Necessary human expertise to support, train, and facilitate use of

CalCompute.


26) Requires CDT, in order to meet the objective of establishing CalCompute, to

require consultants commissioned to work on this process to evaluate and
incorporate all of the following considerations into its plan:

a. An analysis of the public, private, and nonprofit cloud platform

infrastructure ecosystem, including, but not limited to, dominant cloud
providers, the relative compute power of each provider, the estimated cost of
supporting platforms as well as pricing models, and recommendations on the
scope of CalCompute.

b. The process to establish affiliate and other partnership relationships to

establish and maintain an advanced computing infrastructure.

c. A framework to determine the parameters for use of CalCompute, including,

but not limited to, a process for deciding which projects will be supported by
CalCompute and what resources and services will be provided to projects.

d. A process for evaluating appropriate uses of the public cloud resources and

their potential downstream impact, including mitigating downstream harms
in deployment.

e. An evaluation of the landscape of existing computing capability, resources,

data, and human expertise in California, as specified.

f. An analysis of the state’s investment in the training and development of the

technology workforce, including through degree programs at the University
of California (UC), the California State University (CSU), and the California
Community Colleges (CCC).

g. A process for evaluating the potential impact of CalCompute on retaining

technology professionals in the public workforce.


27) Requires CDT to submit an annual report to the Legislature from the

commissioned consultants to ensure progress in meeting the objectives listed
above, as specified.


-----

28) Authorizes CDT to receive private donations, grants, and local funds, in

addition to allocated funding in the annual budget, to effectuate this bill.

29) Specifies that the provisions of this bill are severable, as specified.

**Background**

_Author Statement. According to the author’s office, “large-scale AI has the_
potential to produce an incredible range of benefits for Californians and our
economy—from advances in medicine and climate science to improved wildfire
forecasting and clean power development. It also gives us an opportunity to apply
hard lessons learned over the last decade, as we’ve seen the consequences of
allowing the unchecked growth of new technology without evaluating,
understanding, or mitigating the risks. SB 1047 does just that, by developing
responsible, appropriate guardrails around development of the largest, most
powerful AI systems, to ensure they are used to improve Californians’ lives,
without compromising safety or security.”

Further, “SB 1047 will also promote the growth of the AI industry by establishing
CalCompute, a public AI research cluster that will allow startups, researchers, and
community groups to participate in the development of large-scale AI systems. By
providing a broad range of stakeholders with access to the AI development
process, CalCompute will help align large-scale AI systems with the values and
needs of California communities.”

_Defining AI. Reminiscent of the wave of legislation to regulate social media_
platforms, there are dozens of bills currently making their way through the
Legislature to regulate AI. Before potentially subjecting the industry and
regulators to a host of new laws, it seems imperative to first define exactly what it
is we are talking about and to harmonize the definitions being used just as was
done when the first comprehensive definition of “social media platform” was
codified a few years ago.

The task is not necessarily a straightforward one, as pointed out by the United
States Congressional Research Service:

Defining AI is not merely an academic exercise, particularly when drafting
legislation. AI research and applications are evolving rapidly. Thus,
congressional consideration of whether to include a definition for AI in a
bill, and if so how to define the term or related terms, necessarily include
attention to the scope of the legislation and the current and future
applicability of the definition. Considerations in crafting a definition for use


-----

in legislation include whether it is expansive enough not to hinder the future
applicability of a law as AI develops and evolves, while being narrow
enough to provide clarity on the entities the law affects. Some stakeholders,
recognizing the many challenges of defining AI, have attempted to define
principles that might help guide policymakers. Research suggests that
differences in definitions used to identify AI-related research may contribute
to significantly different analyses and outcomes regarding AI competition,
investments, technology transfer, and application forecasts.

There are several leading definitions of AI that can be relied on to craft a
reasonably clear yet appropriately broad definition for this advancing technology.
While there is no definition of AI in state law, the California Privacy Protection
Agency (CPPA) has published draft regulations that provide a definition for AI:

“Artificial intelligence” means a machine-based system that infers, from the
input it receives, how to generate outputs that can influence physical or
virtual environments. The artificial intelligence may do this to achieve
explicit or implicit objectives. Outputs can include predictions, content,
recommendations, or decisions. Different artificial intelligence varies in its
levels of autonomy and adaptiveness after deployment. For example,
artificial intelligence includes generative models, such as large language
models, that can learn from inputs and create new outputs, such as text,
images, audio, or video; and facial- or speech-recognition or -detection
technology.

At the federal level, the National Artificial Intelligence Act of 2020 provides the
following:

Artificial intelligence. The term “artificial intelligence” means a machinebased system that can, for a given set of human-defined objectives, make
predictions, recommendations or decisions influencing real or virtual
environments. AI systems use machine and human-based inputs to—

(A) perceive real and virtual environments;
(B) abstract such perceptions into models through analysis in an
automated manner; and
(C) use model inference to formulate options for information or
action.

Also at the federal level, NIST, in their Artificial Intelligence Risk Management
Framework (AI RMF 1.0), defined an AI system as “an engineered or machinebased system that can, for a given set of objectives, generate outputs such as


-----

predictions, recommendations, or decisions influencing real or virtual
environments. AI systems are designed to operate with varying levels of
autonomy.”

At the international level, the Organization for Economic Co-operation and
Development (OECD) published and then revised a definition for AI to serve as a
standard across jurisdictions:

An AI system is a machine-based system that, for explicit or implicit
objectives, infers, from the input it receives, how to generate outputs such as
predictions, content, recommendations, or decisions that can influence
physical or virtual environments. Different AI systems vary in their levels
of autonomy and adaptiveness after deployment.

With the revision, OECD made the case for harmonization:

Obtaining consensus on a definition for an AI system in any sector or group
of experts has proven to be a complicated task. However, if governments
are to legislate and regulate AI, they need a definition to act as a foundation.
Given the global nature of AI, if all governments can agree on the same
definition, it allows for interoperability across jurisdictions.

Most recently, the European Parliament signed the EU AI Act, which defines AI as
follows:

“AI system” means a machine-based system designed to operate with
varying levels of autonomy, that may exhibit adaptiveness after deployment
and that, for explicit or implicit objectives, infers, from the input it receives,
how to generate outputs such as predictions, content, recommendations, or
decisions that can influence physical or virtual environments.

In order to gain both the benefit of the expertise and compromise that went into
formulating these definitions and the efficiencies that come with harmonization,
the Senate Committee on Judiciary, the Senate Committee on Governmental
Organization, and a variety of stakeholders, including the author, have come up
with the following definition to begin this process and to include a definition of AI
in the bill, which incorporates elements of the NIST definition into the
internationally recognized formulations while eliminating unnecessary examples
from within it:

“Artificial intelligence” means an engineered or machine-based system that,
for explicit or implicit objectives, infers, from the input it receives, how to


-----

generate outputs that can influence physical or virtual environments and that
may operate with varying levels of autonomy.

_Frameworks for Responsible Development and Accountability in AI.  With recent_
dramatic advances in the capabilities of AI systems, the need for frameworks for
accountability and responsible development have become ever more urgent.

In January of 2017, AI researchers, economists, legal scholars, ethicists, and
philosophers met in Asilomar, California to discuss principles for managing the
responsible development of AI. The collaboration resulted in the Asilomar
Principles. Aspirational rather than prescriptive, these 23 principles were intended
to initiate and frame a dialogue by providing direction and guidance for
policymakers, researchers, and developers. Its endorsers include 1,200 leading
experts in the field of AI, including DeepMind founder Demis Hassabis and the
late Stephen Hawking.

As directed by the National AI Initiative Act of 2020, NIST developed the AI Risk
Management Framework to assist entities designing, developing, deploying, and
using AI systems to help manage the many risks of AI and promote trustworthy
and responsible development and use of AI systems. That framework highlights
the serious risks at play and the uniquely challenging nature of addressing them in
this context:

Artificial intelligence (AI) technologies have significant potential to
transform society and people’s lives – from commerce and health to
transportation and cybersecurity to the environment and our planet. AI
technologies can drive inclusive economic growth and support scientific
advancements that improve the conditions of our world. AI technologies,
however, also pose risks that can negatively impact individuals, groups,
organizations, communities, society, the environment, and the planet. Like
risks for other types of technology, AI risks can emerge in a variety of ways
and can be characterized as long- or short-term, high or low-probability,
systemic or localized, and high- or low-impact.

While there are myriad standards and best practices to help organizations
mitigate the risks of traditional software or information-based systems, the
risks posed by AI systems are in many ways unique. AI systems, for
example, may be trained on data that can change over time, sometimes
significantly and unexpectedly, affecting system functionality and
trustworthiness in ways that are hard to understand. AI systems and the
contexts in which they are deployed are frequently complex, making it
difficult to detect and respond to failures when they occur. AI systems are


-----

inherently socio-technical in nature, meaning they are influenced by societal
dynamics and human behavior. AI risks – and benefits – can emerge from
the interplay of technical aspects combined with societal factors related to
how a system is used, its interactions with other AI systems, who operates it,
and the social context in which it is deployed. These risks make AI a
uniquely challenging technology to deploy and utilize both for organizations
and within society. [. . .]

AI risk management is a key component of responsible development and use
of AI systems. Responsible AI practices can help align the decisions about
AI system design, development, and uses with intended aim and values.
Core concepts in responsible AI emphasize human centricity, social
responsibility, and sustainability. AI risk management can drive responsible
uses and practices by prompting organizations and their internal teams who
design, develop, and deploy AI to think more critically about context and
potential or unexpected negative and positive impacts. Understanding and
managing the risks of AI systems will help to enhance trustworthiness, and
in turn, cultivate public trust.

_The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the_
_American People. In October of 2022, the White House’s Office of Science and_
Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint)
identifying five principles that should guide the design, use, and deployment of
automated systems to protect the American public in the age of AI. The Blueprint
is intended to be a guide for a society to protect all people from AI-related threats.

The five identified principles include “Safe and Effective Systems.” The Blueprint
provides that automated systems should be developed with consultation from
diverse communities, stakeholders, and domain experts to identify concerns, risks,
and potential impacts of the system. Systems should undergo pre-deployment
testing, risk identification and mitigation, and ongoing monitoring that demonstrate
they are safe and effective based on their intended use, mitigation of unsafe
outcomes including those beyond the intended use, and adherence to domainspecific standards. Outcomes of these protective measures should include the
possibility of not deploying the system or removing a system from use.

Secondly, the Blueprint includes “Algorithmic Discrimination Protections.”
Algorithmic discrimination occurs when automated systems contribute to
unjustified different treatment or impacts disfavoring people based on their race,
color, ethnicity, sex (including pregnancy, childbirth, and related medical
conditions, gender identity, intersex status, and sexual orientation), religion, age,
national origin, disability, veteran status, genetic information, or any other


-----

classification protected by law. Designers, developers, and deployers of automated
systems should take proactive and continuous measures to protect individuals and
communities from algorithmic discrimination and to use and design systems in an
equitable way.

Additionally, the Blueprint discusses “Data Privacy,” stating that individuals
should be protected from violations of privacy through design choices that ensure
such protections are included by default, including ensuring that data collection
conforms to reasonable expectations and that only data strictly necessary for the
specific context is collected. Designers, developers, and deployers of automated
systems should seek your permission and respect your decisions regarding
collection, use, access, transfer, and deletion of your data in appropriate ways and
to the greatest extent possible; where not possible, alternative privacy by design
safeguards should be used.

Further, the Blueprint identifies “Notice and Explanation.” Designers, developers,
and deployers of automated systems should provide generally accessible plain
language documentation including clear descriptions of the overall system
functioning and the role automation plays, notice that such systems are in use, the
individual or organization responsible for the system, and explanations of
outcomes that are clear, timely, and accessible. Such notice should be kept up-todate and people impacted by the system should be notified of significant use case
or key functionality changes. You should know how and why an outcome
impacting you was determined by an automated system, including when the
automated system is not the sole input determining the outcome.

The final principle identified by the Blueprint is “Human Alternatives,
Consideration, and Fallback.” The Blueprint states that you should be able to opt
out from automated systems in favor of a human alternative, where appropriate.
Appropriateness should be determined based on reasonable expectations in a given
context and with a focus on ensuring broad accessibility and protecting the public
from especially harmful impacts. In some cases, a human or other alternative may
be required by law.

_California Privacy Protection Agency (CPPA) Actions. In March of this year, the_
CPPA voted to move forward with draft automated decision system (ADS)
technology and risk assessment regulations regarding how businesses use AI and
collect the personal information of consumers, workers, and students.

According to a March 13 article by CalMatters, “[t]he proposed rules seek to create
guidelines for the many areas in which AI and personal data can influence the lives
of Californians: job compensation, demotion, and opportunity; housing, insurance,


-----

health care, and student expulsion. For example, under the rules, if an employer
wanted to use AI to make predictions about a person’s emotional state or
personality during a job interview, a job candidate could opt out without fear of
discrimination for choosing to do so.”

Further, CalMatters states that under the proposed rules, “businesses must notify
people before using AI. If people opt out of interacting with an AI model,
businesses cannot discriminate against people for that choice. If people agree to
use an AI service or tool, businesses must respond to requests by individuals about
how they use their personal information to make predictions. The rules would also
require employers or third-party contractors to carry out risk assessments to
evaluate the performance of their technology.”

According to the CPPA, the proposed framework includes the following three key
components:

1) _Pre-use Notice Requirements. A business must provide information to the_

consumer about how it proposes to use the ADS so that the consumer can
decide whether to opt-out or proceed, and whether to exercise their access right.

2) _Opt-out Right Requirements. For opt-out requests submitted before the_

business initiates the processing, the business must not process consumer’s
personal information using that ADS. For opt-out requests submitted after the
business has initiated the processing, the business must cease processing the
consumer’s personal information using that ADS and notify relevant entities of
the opt-out and instruct them to comply.

3) _Access Right Requirements. This involves the purpose of using the ADS; the_

output with respect to the consumer; how the businesses used the output with
respect to the consumer; how the ADS worked with respect to the consumer;
and that the businesses cannot retaliate against consumers for exercising CPPA
rights.

The CPPA will spend the rest of the spring and summer performing a statewide
informational outreach tour before initiating formal rulemaking, which is expected
later this year.

_Safe and Secure Innovation for Frontier AI Systems Act. This bill takes a number_
of approaches in responding to concerns in AI regulation. The bill can generally
be broken up into the following elements:

1) Safety and security obligations imposed on developers of powerful AI models.
2) Know-your-customer obligations imposed on operators of computing clusters.
3) Price transparency and anti-discrimination provisions for models and clusters.


-----

4) Enforcement and whistleblower protections.
5) Establishment of a Frontier Model Division within CDT.
6) Creation of CalCompute.

The central focus of this bill is ensuring that developers of “covered models,”
essentially extremely powerful AI systems, are proceeding with caution given the
enormous potential for harm posed by them in the hands of malicious actors. At
the core of these obligations is the “limited duty exemption,” which is a
determination that a developer can reasonably exclude the possibility that a
covered model has a hazardous capability or may come close to possessing a
hazardous capability when accounting for a reasonable margin for safety and the
possibility of posttraining modifications.

It is important to note the magnitude of the computing power necessary to meet the
definition of covered model and, likewise, the magnitude of the hazardous
capabilities that the bill seeks to prevent.

First, “covered models” are artificial intelligence models trained using a quantity
of computing power greater than 10^26 integer of floating-point operations.
Floating-point operations, or FLOP, is a measure of the amount of compute used in
AI systems. This threshold of FLOP is currently out of reach for all but a handful
of entities. Therefore the bill focuses on the most highly capable AI models, often
referred to as “frontier models.” Some of today’s examples of frontier models are
GPT-4 (OpenAI), Claude 3 (Anthropic), and Gemini Ultra (Google).

The author explains this metric:

SB 1047 uses this threshold to have developers begin testing for hazardous
capabilities because it indicates that the model is larger than any model
trained today, and without testing it is impossible to rule out the chance that
the next generation of models will contain hazardous capabilities.

The average performance of AI models is predictably related to how much
computational power, measured in FLOP, was used to train them.
Performance on any individual task will tend to go up with FLOP used to
train, but with less predictability. The largest models trained so far are
believed to have used only about 10^25 FLOP. Safety testing from AI
developers and independent technical auditors suggests that, while current
models do not yet possess hazardous capabilities, they display warning signs
indicating that substantially larger models may possess such capabilities. As
computational training power moves beyond 10^26 FLOP, therefore, it


-----

becomes prudent and reasonable to test for these capabilities before
deploying them.

The goal of this is ensure the models powerful enough to cause critical harms
continue to be covered by the bill as metrics for determining the appropriate
thresholds advance. Writing in opposition, a coalition of industry associations led
by the California Chamber of Commerce raise issues with the definition:

There is little to no certainty as to what this translates to in practice and, in
any case, such thresholds will become obsolete within a year, requiring the
law to change yet again. Moreover, by equating model size to risk, the
definition of “covered models” is simultaneously overly broad and too
narrow as smaller and/or less performant models can present much greater
risks than large/higher performant ones. As a result, the bill both fails to
adequately address the very real risks posed by small but malicious models
and imposes significant costs on innovating performant but responsible ones.

With regard to the magnitude of harms being targeted, the level of potential
carnage required to constitute a “hazardous capability” is substantial. It must be
capable of being used to enable any of the following harms in a way that would be
significantly more difficult to cause without access to a covered model:

1) The creation or use of a chemical, biological, radiological, or nuclear (CBRN)

weapon in a manner that results in mass casualties.

2) At least $500 million of damage through cyberattacks on critical infrastructure.
3) At least $500 million of damage by a model that autonomously engages in

conduct that would be criminal if done by a human.


The definition also encompasses other comparably severe threats to public safety
and security and those capabilities made possible by fine tuning and post-training
modifications performed by third-party experts intending to demonstrate those
abilities.

A detailed safety and security protocol must be put into place that includes
specification of testing to be done and describes in detail how the developer will
meet the various security requirements of the bill. The bill requires developers to
ensure that these protocol are properly implemented as written, including by
designating specific senior personnel responsible for implementation and
conducting audits, as appropriate.

The bill also requires the developer to follow all “covered guidance,” which is
defined to include:


-----

1) Guidance issued by NIST and by the Frontier Model Division that is relevant to

the management of safety risks associated with AI models that may possess
hazardous capabilities.

2) Industry best practices, including safety practices, precautions, or testing

procedures undertaken by developers of comparable models that are relevant to
the management of safety risks associated with AI models that may possess
hazardous capabilities.

The bill does not prohibit initiating such widespread use of a model where a
limited duty exemption has not been made. Rather it requires the developer to
implement reasonable safeguards and other security measures to prevent an
individual from being able to use the hazardous capabilities of the model to cause a
critical harm or to use the model to create a derivative model to cause a critical
harm. The developer is required to provide reasonable requirements to developers
of derivative models to prevent an individual from being able to use a derivative
model to cause a critical harm. Developers must ensure, to the extent reasonably
possible, that the covered model’s actions and any resulting critical harms can be
accurately and reliably attributed to it and any user responsible for those actions.

However, the developer is prohibited from initiating the commercial, public, or
widespread use of a covered model if there remains an unreasonable risk that an
individual may be able to use the hazardous capabilities of the model, or a
derivative model based on it, to cause a critical harm.

For those models still not subject to a limited duty exemption, developers are
required to annually certify, under the penalty of perjury, compliance with the
safety and security requirements of the bill to Frontier Models Division.

_The Frontier Model Division. This bill establishes the Frontier Model Division,_
within CDT, and requires the division, among other things, to review annual
certification reports, advise the AG on potential violations, issue guidance,
standards, and best practices, publish anonymized AI safety incident reports, and
establish confidential fora that are structured and facilitated in a manner that allows
developers to share best risk management practices.

Additionally, this bill requires the Frontier Model Division to appoint and consult
with an advisory committee to advise the Governor on when it may be necessary to
proclaim a state of emergency relating to AI and advise the Governor what
responses may be appropriate in that event. Existing law, California’s Emergency
Services Act (ESA) gives the Governor the authority to proclaim a state of
emergency in an area affected or likely to be affected when: a) conditions of
disaster or extreme peril exist; b) the Governor is requested to do so upon request


-----

from a designated local government official; or c) the Governor finds that local
authority is inadequate to cope with the emergency. Local governments may also
issue local emergency proclamations, which is a prerequisite for requesting the
Governor’s Proclamation of a State of Emergency.

Specifically, the ESA authorizes the Governor to proclaim a state of emergency
when conditions of disaster or of extreme peril to the safety of persons and
property within the state caused by conditions such as cyberterrorism, sudden and
severe energy shortage, and electromagnetic pulse attack, or other conditions
which, by reason of their magnitude, are or are likely to be beyond the control of
the services, personnel, equipment, and facilities of any single county or city occur.

Further, this bill authorizes the Frontier Model Division to monitor relevant
developments relating to the safety risks associated with the development of AI
models and the functioning of markets for AI models and, on or before July 1,
2026, issue guidance regarding technical thresholds and benchmarks relevant to
determining whether an AI model is a covered model and technical thresholds and
benchmarks relevant to determining whether a covered model is subject to a
limited duty exemption.

_CalCompute. This bill directs CDT to commission consultants to create a public_
cloud computing cluster, to be known as CalCompute, with the primary focus of
conducting research into the safe and secure deployment of large-scale AI models
and fostering equitable innovation. Specifically, the consultants are required to be,
but not limited to, representatives of national laboratories, universities, and any
relevant professional associations or private sector stakeholders.

To meet the objective of establishing CalCompute, this bill requires CDT to ensure
that the consultants commissioned to work on this process evaluate and incorporate
all of the following considerations:

1) An analysis of the public, private, and nonprofit cloud platform infrastructure

ecosystem, as specified.

2) The process to establish affiliate and other partnership relationships to establish

and maintain an advanced computing infrastructure.

3) A framework to determine the parameters for use of CalCompute, as specified.
4) A process for evaluating appropriate uses of the public cloud resources and

their potential downstream impact, including mitigating downstream harms in
deployment.

5) An evaluation of the landscape of existing computing capability, resources,

data, and human expertise in California for the purposes of responding quickly
to a security, health, or natural disaster emergency.


-----

6) An analysis of the state’s investment in the training and development of the

technology workforce, including through degree programs at the UC, the CSU,
and the CCCs.

7) A process for evaluating the potential impact of CalCompute on retaining

technology professionals in the public workforce.

A cloud computing cluster is a group of connected servers that work together as a
single system, often over the internet, to perform computing tasks. This setup
allows for the pooling of resources, such as processing power and storage, from
multiple machines. This collective resource pool can be used more efficiently and
flexibly than single-server or single-computer solutions. Back in the day, the
machines used for high performance computing were known as “supercomputers,”
or big standalone machines with specialized hardware – very different from what
you find in home and office computers.

Nowadays, however, the majority of supercomputers are instead computer clusters.
These inter-connected computers are endowed with software to coordinate
programs on (or across) those computers, and they can therefore work together to
perform computationally intensive tasks.

Distributing computing tasks across several servers can significantly enhance
performance. Large computational tasks can be divided into smaller parts and
executed simultaneously across multiple machines, drastically reducing processing
time. Further, cloud clusters often utilize virtualization technologies, which allow
for the efficient use of hardware, reducing the cost associated with physical
hardware investments and maintenance.

There are currently several existing cloud computing clusters primarily operated by
major technology companies and organizations. These clusters serve a variety of
purposes from general cloud computing needs to more specialized applications like
AI research and big data analytics. A few prominent examples include: Google’s
Cloud Platform offering cloud computing services that include powerful compute
clusters; Amazon Web Services providing a comprehensive and widely used suite
of cloud computing services for large-scale computing tasks, including AI model
training and deployment; Microsoft Azure which supports large-scale computing
tasks similar to those intended for CalCompute; and the European Organization for
Nuclear Research (CERN) openlab, while not a commercial service provider,
CERN’s cloud computing resources are a notable example of public cloud
capabilities for scientific research. They provide a powerful infrastructure for
processing vast amounts of scientific data, particularly from particle physics
experiments.


-----

As proposed by this bill, CalCompute is intended to represent a significant step
towards integrating advanced cloud computing capabilities with public sector
innovation and research in AI. The establishment of CalCompute attempts to
harness the growing power of AI technologies while addressing critical aspects
such as safety, security, and equitable access and leans on the involvement of a
diverse range of stakeholders, from national labs to local universities and the
private sector.

This bill requires CDT to submit an annual report to the Legislature from the
commissioned consultants on CalCompute to ensure progress in meeting the
objectives laid out in the bill, and authorizes CDT to receive private donations,
grants, and local funds, in addition to allocated funding in the annual budget, to
effectuate CalCompute. This bill provides that CalCompute shall only become
operative upon an appropriation in a budget act, as specified.

The provisions of this bill are severable, that is, if any provision of the Safe and
Secure Innovation for Frontier AI Systems Act or its application are held invalid,
that invalidity shall not affect other provisions or applications that can be given
effect without the invalid provision or application.

**Prior/Related Legislation**

SB 313 (Dodd, 2023) would have established the Office of AI, as specified, and
require state agencies to disclose when they are using generative AI (GenAI) to
communicate with a person and to provide them an option to speak with a natural
person at the agency. (Held on the Senate Appropriations Committee Suspense
File)

SB 398 (Wahab, 2023) would have established the AI for California Research Act,
which requires CDT to develop and implement a comprehensive research plan to
study the feasibility of using advanced technology to improve state and local
government services. (Never heard in the Senate Committee on Governmental
Organization)

SB 892 (Padilla, 2024) requires CDT to establish safety, privacy, and
nondiscrimination standards relating to artificial intelligence services, as defined.
Commencing August 1, 2025, the bill prohibits a contract for AI services, as
defined, from being entered into by the state unless the provider meets those
standards. CDT is required to report to the Legislature regarding the standards it
establishes. (Pending in the Senate Committee on Appropriations)


-----

SB 893 (Padilla, 2024) requires GovOps, the Governor’s Office of Business and
Economic Development, and CDT to collaborate to establish the California AI
Research Hub in GovOps, as specified. (Pending in the Senate Committee on
Appropriations)

SB 896 (Dodd, 2024) the AI Accountability Act, establishes guidelines for the use
and regulation of AI and (GenAI) within state government operations.
Specifically, this bill requires the development and production of a Benefits and
Risk of Generative Artificial Intelligence Report; requires joint risk analyses of
potential threats posed by GenAI to California’s critical energy infrastructure, as
specified; requires the development of guidelines for the procurement of AI and
GenAI, as specified; requires the support and training of the state government
workforce for the next generation of skills needed to thrive in the GenAI economy,
as specified; and, requires a state agency to identify a person when that person is
communicating with a state agency directly through GenAI, as specified. (Pending
in the Senate Committee on Appropriations)

SB 942 (Becker, 2024) the AI Transparency Act, requires specified providers of
GenAI to create an AI detection tool available to the public to determine the extent
to which text, image, video, audio, or multimedia content was created by a GenAI
system, as specified; requires those providers to include a visible disclosure in AIgenerated content including a clear and conspicuous notice that identifies the
content as generated by AI; and creates the Generative AI Registry Fund and
requires those providers to register with CDT, as specified. (Pending in the Senate
Committee on Governmental Organization)

SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California
Legislature’s commitment to President Biden’s vision for a safe AI and the
principles outlined in the “Blueprint for an AI Bill of Rights” and expresses the
Legislature’s commitment to examining and implementing those principles in its
legislation and policies related to the use and deployment of automated systems.

AB 302 (Ward, Chapter 800, Statutes of 2023) requires CDT, in coordination with
other interagency bodies, to conduct a comprehensive inventory of all high-risk
automated decision systems used by state agencies on or before September 1,
2024, and report the findings to the Legislature by January 1, 2025, and annually
thereafter, as specified.

AB 331 (Bauer-Kahan, 2023) would have prohibited “algorithmic discrimination,”
that is, use of an automated decision tool to contribute to unjustified differential
treatment or outcomes that may have a significant effect on a person’s life, as
specified. (Held on the Senate Appropriations Committee Suspense File)


-----

AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as
defined, of an AI system or service to post on the developer’s website
documentation regarding the data used to train the AI system or service, as
specified. (Pending in the Assembly Committee on Privacy and Consumer
Protection)

AB 2885 (Bauer-Kahan, 2024) defines AI and inserts references to the provided
definition throughout existing code, as specified. (Pending in the Assembly
Committee on Appropriations)

AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer and a
developer of an automated decision tool to, on or before January 1, 2026, and
annually thereafter, perform an impact assessment for any automated decision tool
the deployer uses that includes, among other things, a statement of the purpose of
the automated decision tool and its intended benefits, uses, and deployment
contexts. The assessments must be provided to the Civil Rights Department within
7 days of a request. (Pending in the Assembly Committee on Judiciary)

SJR 6 (Chang, Res. Chapter 112, Statutes of 2019) urged the President and the
Congress of the United States to develop a comprehensive AI Advisory Committee
and to adopt a comprehensive AI policy.

ACR 215 (Kiley, Res. Chapter 206, Statutes of 2018) expressed the Legislature’s
support for a set of principles for the governance of AI known as the 23 Asilomar
AI Principles.

SB 532 (Dodd, Chapter 557, Statutes of 2017) added “cyberterrorism” to the list of
conditions constituting a state of emergency and a local emergency.

**FISCAL EFFECT:   Appropriation: No  Fiscal Com.:  Yes   Local:  Yes**

**SUPPORT:**

Center for AI Safety Action Fund (Co-source)
Economic Security California Action (Co-source)
Encode Justice (Co-source)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program


-----

Denizen
Depict.AI
District Council of Iron Workers for the State of California
Elicit
ENH Alpha LLC
FAR AI, Inc.
Fathom Radiant
Foresight Institute
ForHumanity
Future of Life Institute
General Agents
General Proximity
Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA: StateStrong
KIRA Center for AI Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Los Angeles Area Chamber of Commerce
Loveable Labs Incorporated
MIT AI Alignment
ML Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group

**OPPOSITION:**

Association of National Advertisers
California Chamber of Commerce
California Land title Association
California Manufacturers and Technology Association
Chamber of Progress


-----

Civil Justice Association of California
Computer and Communications Industry Association
Context Fund
Insights Association
Silicon Valley Leadership Group
Software and Information Industry Association
TechNet

**ARGUMENTS IN SUPPORT:  A coalition of supporters write that, “SB 1047**
requires developers to conduct self-assessments and take precautions to prevent
three major risks: cyberattacks causing over $500 million in damage, AI-enabled
crimes resulting in damages exceeding $500 million, and the creation of novel
weapons of mass destruction using AI. Developers must implement safety
measures before training and deploying risky models, monitor and report incidents,
and comply with safety protocols. The Attorney General of California is
empowered to take enforcement action against developers whose negligence poses
imminent threats to public safety or causes severe harm to Californian citizens.”

Further, “SB 1047 also aims to foster AI innovation and support smaller startups
and researchers. It establishes a public cloud-computing cluster for researching
safe and secure AI deployment, allowing diverse stakeholders to participate in the
development of large-scale AI systems. The bill creates an advisory council to
advocate for safe and secure open-source AI development and mandates
transparent pricing and non-discriminatory practices from cloud-computing
companies and frontier model developers to ensure equal opportunities for smaller
players in the AI ecosystem.”

**ARGUMENTS IN OPPOSITION:  Opponents of the bill write that, “SB 1047**
seeks to regulate frontier AI developers from innovating AI models that will result
in any kind of foreseeable harm—even harms that would not manifest from the
model itself. In doing so, the bill requires developers to comply with incredibly
vague, broad, impractical, if not impossible, standards when developing ‘covered
models’. For example, SB 1047 applies to AI models that either: (1) meet a size
threshold (a computing power greater than 10^26 integer or floating-point
operations in 2024), or (2) that perform similarly. What the latter category of
covered models looks like, however, is not entirely clear. (See Proposed Section
22602(f).) The bill merely states that they are models ‘trained using a quantity of
computing power sufficiently large that it could be reasonably expected to have
similar or greater performance as an AI model trained using a quantity of
computing power greater than 10^26 integer or floating-point operations in 2024 as
assessed using benchmarks commonly used to quantify the general performance of
state-of-the-art foundation models’. There is little to no certainty as to what this


-----

translates to in practice and, in any case, such thresholds will become obsolete
within a year, requiring the law to change yet again. Moreover, by equating model
size to risk, the definition of ‘covered models’ is simultaneously overly broad and
too narrow as smaller and/or less performant models can present much greater
risks than large/higher performant ones. As a result, the bill both fails to
adequately address the very real risks posed by small but malicious models and
imposes significant costs on innovating performant but responsible ones.”

Further, “[w]e cannot overemphasize the importance of ensuring consistency in the
AI regulatory landscape, nationally, and the need to follow federal guidance on
certain issues that transcend national borders. Relevant to this bill, in October
2023, the White House issued an Executive Order (EO) that requires companies
that are developing any foundation model that poses a serious risk to national
security, national economic security, or national public health and safety to notify
the federal government when training the model and share the results of all redteam safety tests to ensure that AI systems are safe, security and trustworthy before
companies make them public. While we appreciate that in some respects, SB 1047
appears in line with the goals of the federal government and the White House’s
EO, the [NIST] is already working with other agencies at the federal level to
establish testing and safety guidelines for large models. If enacted, SB 1047 would
likely result in confusion about the correct standards to apply and place additional
burdens on AI developers without commensurate gains in safety, especially as it
fails to align with regulations nationally and introduces novel concepts and
standards including around the assessment of what is a ‘hazardous capability’.
Indeed, given the definition of ‘covered models’ under this bill which also scopes
in any fine-tuning by downstream customers and users, SB 1047 is more farreaching than anything seen to date in voluntary commitments, federal guidance, or
the laws of any other countries.”

**DUAL REFERRAL: Senate Judiciary Committee (9-0) & Senate Governmental**
Organization Committee


-----

