When it comes to AI, as California goes, so goes the nation. The biggest state in the US by population is also the central hub of AI innovation for the entire globe, home to 32 of the world’s top 50 AI companies. That size and influence have given the Golden State the weight to become a regulatory trailblazer, setting the tone for the rest of the country on environmental, labor, and consumer protection regulations — and more recently, AI as well.

Now, following the dramatic defeat of a proposed federal moratorium on states regulating AI in July, California policymakers see a limited window of opportunity to set the stage for the rest of the country’s AI laws. In the early hours of Saturday morning, the California State Assembly voted in favor of SB 53, a bill that would require transparency reports from the developers of highly powerful, “frontier” AI models. The bill, which has passed both parts of the state legislature, now goes to Gov. Gavin Newsom to either be vetoed or signed into law.

The models targeted represent the cutting-edge of AI — extremely adept generative systems that require massive amounts of data and computing power, like OpenAI’s ChatGPT, Google’s Gemini, xAI’s Grok, and Anthropic’s Claude.

AI can offer tremendous benefits, but as the bill is meant to address, it’s not without risks. And while there is no shortage of existing risks from issues like job displacement and bias, SB 53 focuses on possible “catastrophic risks” from AI. Such risks include AI-enabled biological weapons attacks and rogue systems carrying out cyberattacks or other criminal activity that could conceivably bring down critical infrastructure. Such catastrophic risks represent widespread disasters that could plausibly threaten human civilization at local, national, and global levels. They represent risks of the kind of AI-driven disasters that have not yet occurred, rather than already-realized, more personal harms like AI deepfakes.

Exactly what constitutes a catastrophic risk is up for debate, but SB 53 defines it as a “foreseeable and material risk” of an event that causes more than 50 casualties or over $1 billion in damages that a frontier model plays a meaningful role in contributing to. How fault is determined in practice would be up to the courts to interpret. It’s hard to define catastrophic risk in law when the definition is far from settled, but doing so can help us protect against both near- and long-term consequences.

By itself, a single state bill focused on increased transparency will probably not be enough to prevent devastating cyberattacks and AI-enabled chemical, biological, radiological, and nuclear weapons. But the bill represents an effort to regulate this fast-moving technology before it outpaces our efforts at oversight.

SB 53 is the third state-level bill to try to specifically focus on regulating AI’s catastrophic risks, after California’s SB 1047, which passed the legislature only to be vetoed by the governor — and New York’s Responsible AI Safety and Education (RAISE) Act, which recently passed the New York legislature and is now awaiting Gov. Kathy Hochul’s approval.

SB 53, which was introduced by state Sen. Scott Wiener in February, requires frontier AI companies to develop safety frameworks that specifically detail how they approach catastrophic risk reduction. Before deploying their models, companies would have to publish safety and security reports. The bill also gives them 15 days to report “critical safety incidents” to the California Office of Emergency Services, and establishes whistleblower protections for employees who come forward about unsafe model deployment that contributes to catastrophic risk. SB 53 aims to hold companies publicly accountable for their AI safety commitments, with a financial penalty up to $1 million per violation.

In many ways, SB 53 is the spiritual successor to SB 1047, also introduced by Wiener.

Both cover large models that are trained at 10^26 FLOPS, a measurement of very significant computing power used in a variety of AI legislation as a threshold for significant risk, and both bills strengthen whistleblower protections. Where SB 53 departs from SB 1047 is its focus on transparency and prevention

While SB 1047 aimed to hold companies liable for catastrophic harms caused by their AI systems, SB 53 formalizes sharing safety frameworks, which many frontier AI companies, including Anthropic, already do voluntarily. It focuses squarely on the heavy-hitters, with its rules applying only to companies that generate $500 million or more in gross revenue.

“The science of how to make AI safe is rapidly evolving, and it’s currently difficult for policymakers to write prescriptive technical rules for how companies should manage safety,” said Thomas Woodside, the co-founder of Secure AI Project, an advocacy group that aims to reduce extreme risks from AI and is a sponsor of the bill, over email. “This light touch policy prevents backsliding on commitments and encourages a race to the top rather than a race to the bottom.”

Part of the logic of SB 53 is the ability to adapt the framework as AI progresses. The bill authorizes the California Attorney General to change the definition of a large developer after January 1, 2027, in response to AI advances.

Proponents of the bill were optimistic about its chances of being signed by the governor should it pass the legislature. On the same day that Gov. Newsom vetoed SB 1047, he commissioned a working group focusing solely on frontier models. The resulting report by the group provided the foundation for SB 53. “I would guess, with roughly 75 percent confidence, that SB 53 will be signed into law by the end of September,” said Dean Ball — former White House AI policy adviser, vocal SB 1047 critic, and SB 53 supporter — to Transformer.

But several industry organizations rallied in opposition, arguing that additional compliance regulation would be expensive, given that AI companies should already be incentivized to avoid catastrophic harms. OpenAI has lobbied against it, and technology trade group Chamber of Progress argues that the bill would require companies to file unnecessary paperwork and unnecessarily stifle innovation.

“Those compliance costs are merely the beginning,” Neil Chilson, head of AI policy at the Abundance Institute, told me over email. “The bill, if passed, would feed California regulators truckloads of company information that they will use to design a compliance industrial complex.”

By contrast, Anthropic enthusiastically endorsed the bill on Monday. “The question isn’t whether we need AI governance – it’s whether we develop it thoughtfully today or reactively tomorrow,” the company explained in a blog post. “SB 53 offers a solid path toward the former.” (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI, while Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic. Neither organization has editorial input into our content.)

The debate over SB 53 ties into broader disagreements about whether states or the federal government should drive AI safety regulation. But since the vast majority of these companies are based in California, and nearly all do business there, the state’s legislation matters for the entire country.

“A federally led transparency approach is far, far, far preferable to the multi-state alternative,” where a patchwork of state regulations can conflict with each other, said Cato Institute technology policy fellow Matthew Mittelsteadt in an email. But “I love that the bill has a provision that would allow companies to defer to a future alternative federal standard.”

“The natural question is whether a federal approach can even happen,” Mittelsteadt continued. “In my opinion, the jury is out on that but the possibility is far more likely that some suggest. It’s been less than 3 years since ChatGPT was released. That is hardly a lifetime in public policy.”

But in a time of federal gridlock, frontier AI advancements won’t wait for Washington.

The catastrophic risk divide

The bill’s focus on, and framing of, catastrophic risks is not without controversy.

The idea of catastrophic risk comes from the fields of philosophy and quantitative risk assessment. Catastrophic risks are downstream of existential risks, which threaten humanity’s actual survival or else permanently reduce our potential as a species. The hope is that if these doomsday scenarios are identified and prepared for, they can be prevented or at least mitigated.

But if existential risks are clear — the end of the world, or at least as we know it — what falls under the catastrophic risk umbrella, and the best way to prioritize those risks, depends on who you ask. There are longtermists, people focused primarily on humanity’s far future, who place a premium on things like multiplanetary expansion for human survival. They’re often chiefly concerned by risks from rogue AI or extremely lethal pandemics. Neartermists are more preoccupied with existing risks, like climate change, mosquito vector-borne disease, or algorithmic bias. These camps can blend into one another — neartermists would also like to avoid getting hit by asteroids that could wipe out a city, and longtermists don’t dismiss risks like climate change — and the best way to think of them is like two ends of a spectrum rather than a strict binary.

You can think of the AI ethics and AI safety frameworks as the near- and longtermism of AI risk, respectively. AI ethics is about the moral implications of the ways the technology is deployed, including things like algorithmic bias and human rights, in the present. AI safety focuses on catastrophic risks and potential existential threats. But, as Vox’s Julia Longoria reported in the Good Robot series for Unexplainable, there are inter-personal conflicts leading these two factions to work against each other, much of which has to do with emphasis. (AI ethics people argue that catastrophic risk concerns over-hype AI capabilities and ignores its impact on vulnerable people right now, while AI safety people worry that if we focus too much on the present, we won’t have ways to mitigate larger-scale problems down the line.)

But behind the question of near versus long-term risks lies another one: what, exactly, constitutes a catastrophic risk?

SB 53 initially set the standard for catastrophic risk at 100 rather than 50 casualties — similar to New York’s RAISE Act — before halving the threshold in an amendment to the bill. While the average person might consider, say, many people driven to suicide after interacting with AI chatbots to be catastrophic, such a risk is outside of the bill’s scope. (The California State Assembly just passed a separate bill to regulate AI companion chatbots by preventing them from participating in discussions about suicidal ideation or sexually explicit material.)

SB 53 focuses squarely on harms from “expert-level” frontier AI model assistance in developing or deploying chemical, biological, radiological, and nuclear weapons; committing crimes like cyberattacks or fraud; and “loss of control” scenarios where AIs go rogue, behaving deceptively to avoid being shut down and replicating themselves without human oversight. For example, an AI model could be used to guide the creation of a new deadly virus that infects millions and kneecaps the global economy.

“The 50 to 100 deaths or a billion dollars in property damage is just a proxy to capture really widespread and substantial impact,” said Scott Singer, lead author of the California Report for Frontier AI Policy, which helped inform the basis of the bill. “We do look at like AI-enabled or AI potentially [caused] or correlated suicide. I think that’s like a very serious set of issues that demands policymaker attention, but I don’t think it’s the core of what this bill is trying to address.”

Transparency is helpful in preventing such catastrophes because it can help raise the alarm before things get out of hand, allowing AI developers to correct course. And in the event that such efforts fail to prevent a mass casualty incident, enhanced safety transparency can help law enforcement and the courts figure out what went wrong. The challenge there is that it can be difficult to determine how much a model is accountable for a specific outcome, Irene Solaiman, the chief policy officer at Hugging Face, a collaboration platform for AI developers, told me over email.

“These risks are coming and we should be ready for them and have transparency into what the companies are doing,” said Adam Billen, the vice president of public policy at Encode, an organization that advocates for responsible AI leadership and safety. (Encode is another sponsor of SB 53.) “But we don’t know exactly what we’re going to need to do once the risks themselves appear. But right now, when those things aren’t happening at a large scale, it makes sense to be sort of focused on transparency.”

However, a transparency-focused bill like SB 53 is insufficient for addressing already-existing harms. When we already know something is a problem, the focus should be on mitigating it.

“Maybe four years ago, if we had passed some sort of transparency legislation like SB 53 but focused on those harms, we might have had some warning signs and been able to intervene before the widespread harms to kids started happening,” Billen said. “We’re trying to kind of correct that mistake on these problems and get some sort of forward-facing information about what’s happening before things get crazy, basically.”

SB 53 risks being both overly narrow and unclearly scoped. We have not yet faced these catastrophic harms from frontier AI models, and the most devastating risks might take us entirely by surprise. We don’t know what we don’t know.

It’s also certainly possible that models trained below 10^26 FLOPS, which aren’t covered by SB 53, have the potential to cause catastrophic harm under the bill’s definition. The EU AI Act sets the threshold for “systemic risk” at the smaller 10^25 FLOPS, and there’s disagreement about the utility of computational power as a regulatory standard at all, especially as models become more efficient.

As it stands right now, SB 53 occupies a different niche from bills focused on regulating AI use in mental healthcare or data privacy, reflecting its authors’ desire not to step on the toes of other legislation or bite off more than it can reasonably chew. But Chilson, the Abundance Institute’s head of AI policy, is part of a camp that sees SB 53’s focus on catastrophic harm as a “distraction” from the real near-term benefits and concerns, like AI’s potential to accelerate the pace of scientific research or create nonconsensual deepfake imagery, respectively.

That said, deepfakes could certainly cause catastrophic harm. For instance, imagine a hyper-realistic deepfake impersonating a bank employee to commit fraud at a multibillion-dollar scale, said Nathan Calvin, the vice president of state affairs and general counsel at Encode. “I do think some of the lines between these things in practice can be a bit blurry, and I think in some ways…that is not necessarily a bad thing,” he told me.

It could be that the ideological debate around what qualifies as catastrophic risks, and whether that’s worthy of our legislative attention, is just noise. The bill is intended to regulate AI before the proverbial horse is out of the barn. The average person isn’t going to worry about the likelihood of AI sparking nuclear warfare or biological weapons attacks, but they do think about how algorithmic bias might affect their lives in the present. But in trying to prevent the worst-case scenarios, perhaps we can also avoid the “smaller,” nearer harms. If they’re effective, forward-facing safety provisions designed to prevent mass casualty events will also make AI safer for individuals.

If Gov. Newsom signs SB 53 into law, it could inspire other state attempts at AI regulation through a similar framework, and eventually encourage federal AI safety legislation to move forward.

How we think about risk matters because it determines where we focus our efforts on prevention. I’m a firm believer in the value of defining your terms, in law and debate. If we’re not on the same page about what we mean when we talk about risk, we can’t have a real conversation.

Update, September 13, 2025, 11:55 am ET: This story was originally published on September 12 and has been updated to reflect the outcome of the California State Assembly vote.

Editor’s note, July 21, 2025, 2:15 pm ET: A recent Jubilee featuring journalist Mehdi Hasan, titled “1 Progressive vs 20 Far-Right Conservatives,” went viral on social media. In the video, some of the far-right conservatives Hasan debated refer to themselves as fascists, which has created controversy online. To learn more about Jubilee’s video strategy and the trend of combative political debate videos, check out the story below, originally published on October 10, 2024.

It seems as though the country has been engaged in one long screaming match since 2016. Go on YouTube or scroll through X and that feeling gets a face. Videos claiming that someone “silenced” or “destroyed” another party in a discussion about politics abound on social media. There are now nearly unavoidable clips of conservative personalities like Charlie Kirk and Ben Shapiro arguing with college students at liberal universities or leftist commentators on their social platforms. Meanwhile, videos of random folks with polar-opposite political views sitting in a dark room arguing over hot-button issues — and often saying wildly offensive or misinformed things — are on the rise.

At the end of September, a YouTube video titled, “Can 1 Woke Teen Survive 25 Trump Supporters” went viral, drawing attention for its absurd, Battle Royale-like premise. In two weeks, it had accumulated 9.6 million views. The video sees 19-year-old liberal TikTok pundit Dean Withers (a.k.a. the “woke teen”) thrown into a lion’s den of young, zealous Trumpers eager to prove him wrong. One by one, he argues with his opponents across a table about reproductive rights and Kamala Harris’s bona fides. One clip where he appears to stump a woman during a discussion about abortion and IUDs garnered millions of views on X.

This is just one of the contentious and extremely clicky scenarios explored by the media company Jubilee in its popular YouTube series “Surrounded.” The series’ setup looks like a satire of what debate has become in the age of Trump: extremely competitive, theatrical, and unbalanced (literally and emotionally) to boot. What should theoretically be an exchange of facts and logic has become the ultimate bloodsport for a certain type of “thought leader” often happy to traffic in opinions and distorted truths. These oral pugilists are more interested in some online-only version of “winning” than having meaningful discourse.

Across the political spectrum, there has proven to be an appetite for watching people shout at each other. These on-air clashes have been the bread and butter of cable news networks like CNN and Fox News. Still, these filmed debates mostly promote the pessimistic notion that the US is too polarized to be saved. They’re frequently a front-row seat to all the misinformation, conspiracy theories, and regressive attitudes polluting the political landscape and affecting people’s daily lives. So why can’t we stop watching them?

In the Trump era, liberal vs. conservative face-offs are everywhere

While this critique has certainly been amplified in the Trump era, the observation that public debate has become a circus is not exactly new. You can go back decades; in the 2000s, Jon Stewart (fairly) disparaged Crossfire; in the ’90s, Saturday Night Live parodied the unproductive and shouty nature of political panel show The McLaughlin Group and, later, The View. However, in the digital age, this kind of content has been mass-produced and even more degraded. You no longer have to watch CNN or programs like Real Time With Bill Maher to see opposing parties talk over each other and manipulate facts. Instead, you can go to the New York Post’s website to watch two random people shout about the legitimacy of the Black Lives Matter movement in a series called “Face Your Hater” or watch a group of strangers argue about traditional and modern masculinity on Vice’s YouTube channel.

Ryan Broderick, a freelance journalist who writes the newsletter Garbage Day, began noticing these viral confrontations ramping up after the Obama era, a period that saw a growing cultural backlash to progressive policies and rhetoric (i.e., the Tea Party movement) and eventually culminated in Trump’s election. This was a time when liberals and moderates were encouraging each other to “reach across the aisle” and talk about politics with their Trump-supporting relatives during holidays. He describes these filmed social experiments as an “impulse from extremely naive digital media companies.”

“That whole style of content got really popular because there was this impulse coming out of the Obama years that we could bypass all the unpleasantness of the last 10 years if we could just talk to each other,” said Broderick.

Some of these videos are at least designed as slightly more benevolent attempts to see if two supposedly opposing identities can find common ground or at least engage in a civil conversation. The YouTube channel Only Human has a series called “Eating With the Enemy” where two people from different backgrounds — like a drag queen and a Catholic priest, for example — share a meal while discussing political issues, like gay marriage.

Others, like Vice’s popular “Debate” series on YouTube, can get a little more dramatic and heated, like watching a daytime panel show or a scene from Real Housewives. Even with a moderator guiding the discussion, they aren’t exactly designed with the goal of finding middle ground or even having one side convince the other of their argument. Rather, they feel like useless surveys meant to convey our country’s deeply divided climate. For instance, one debate between a group of “anti and pro feminists” arguing over a slew of women’s and trans issues ends with some of the participants talking to the camera about their experiences. Ultimately, they leave more affirmed in their established beliefs than moved by other arguments.

Jubilee’s “Surrounded” series feels more like a MrBeast-inspired game show in its pure stuntiness. Even the way the channel highlights the number of people debating against one another resembles his excessive model. The prompts displayed in the top corner of the videos — like “trans women are women” or “Kamala Harris is a DEI candidate”— aren’t rigorous or challenging. They feel primed to become “rage bait” clips meant to get viewers excited or angry, to the tune of millions of clicks.

Still, this content is sort of genius in the way it attracts and satisfies a range of audiences because there’s typically someone you can agree with and believe made the better argument. For instance, someone can watch Jubilee’s video of Charlie Kirk being schooled by college students with more educated arguments and still, if they’re a fan of his, believe he won the debate. Broderick says that Jubilee, despite the pugnacious nature of their videos, inadvertently creates this sort of “feel-good centrist” content designed for everyone.

“I can’t fathom watching this and thinking that Charlie Kirk looks good,” says Broderick. “But from what I’ve seen of right-wingers watching this stuff, they’re like, ‘Oh yeah, he’s the one that’s making sense.’”

Online debates have become a successful way to self-brand

Conservative pundits, in particular, have taken online debate culture to competitive and self-serving extremes. The phrase “debate me, bro” has become largely associated with the very online and combative community of right-wing commentators, like Dinesh D’Souza and Steven Crowder — a.k.a. the guy in the “change my mind” meme — who are constantly challenging liberal politicians, women, or practically anyone who disagrees with them on the internet to verbally spar.

For personalities like Kirk, Ben Shapiro, and Jordan Peterson, these videos have become a promotional tool to prove their authority in the marketplace — or, more precisely, battlefield — of ideas. Given that many of them host debates or upload in-person confrontations on their media platforms, they’re able to edit or advertise themselves as outsmarting their opponents. For instance, the YouTube channel for Turning Point USA features videos of Kirk supposedly “destroying” “arrogant” and “naive” students on liberal college campuses on his speaking tours. These videos are not actually about producing an interesting dialogue but rather humiliating their opponents and highlighting their supposed stupidity.

Leftists, like YouTuber Destiny and livestreamer Hasan Piker, have also gained visibility and clicks via their eagerness to argue with conservatives. Journalist Max Read, who writes the newsletter Read Max, says that, when it comes to these chronic debaters, the line between “self-promotion and movement-building” can be very thin.

“I can understand the idea that you’re not just boosting your own profile; you’re boosting the profile of your politics and trying to bring more people into it,” says Read. “However, I’m inclined to be more generous to YouTubers who make explanatory response videos than join debates.”

Dean Withers, who’s participated in several Jubilee videos, hosts livestreams on TikTok where he debates with users about political subjects. He also posts solo responses to right-wing talking points. He says he understands people’s criticism around his debate content as clicky and unproductive. However, he says he uses these exchanges as opportunities to educate his audience.

“The main prerogative of my platform is to inform the people watching the debates that I have on what the issues are, why they matter, and why you should agree with me,” he says. “I know that getting my opponent to agree with me is more than likely to never occur.”

For someone, like Withers — who was in middle school when Trump was elected and whose political consciousness was developed in the social-media age — debating with strangers online may just seem like an obvious approach to activism. Research has found, though, that this phenomenon may create a more toxic picture of how humans engage in political discourse.

Political boxing matches might be entertaining, but they don’t reflect how we communicate in reality

A March study found that political debates on social media often give the impression of a climate that’s more combative and divided than it actually is. Specifically, research found that Americans are more likely to argue over political topics with people they know and trust, like family and friends, than strangers on the internet, and often leave these interactions with positive feelings.

University of California Berkeley professor Erica Bailey, who co-authored the study, says these intense, Jubilee-like debates “almost never happen in real life.”

“While these debates can seem ubiquitous because we’re constantly being fed them through our screens, my research has found that the typical American debates hot-button issues infrequently,” she says. “Of the most common topics, like vaccines, reproductive rights, and policing, only about half of Americans have debated these topics in the last year.”

On the rare occasion that you may be forced to defend a political stance, it can still be a pretty daunting task and cause feelings of anxiety. This seems to be one of the reasons we can’t stop watching these videos. On the whole, these exchanges seem generally unpleasant, but it can provide a sense of relief to watch an expert — or someone who claims to be an expert — confidently expressing their opinions.

“When you engage in debate, you often find out all the ways in which your knowledge and understanding is incomplete,” says Bailey. “Watching debate videos is cathartic because we get to cosplay as an excellent debater who can articulate our position with ease. It also helps that these clips are certainly edited to show us the most persuasive moment of the exchange.”

Humans also just tend to engage more with content that elicits a strong emotional response. It’s one of the reasons even the most obvious “rage bait” is hard to avoid on social media, whether you’re the type of person who would ordinarily click on it or not. This behavior, plus algorithms that boost this sort of controversial content, has created a cycle of doom content we can’t escape.

While content like Jubilee’s abounds, the staginess and over-produced structure of these videos underlie a comforting truth: This level of antagonism surrounding political discourse may be clicky but it is thankfully not natural.

“It might be surprising given the state of polarization,” says Bailey. “But humans are typically wired toward social cohesion. In the end, we really don’t want to fight; we want to belong.”