The science fiction author Isaac Asimov once came up with a set of laws that we humans should program into our robots. In addition to a first, second, and third law, he also introduced a “zeroth law,” which is so important that it precedes all the others: “A robot may not injure a human being or, through inaction, allow a human being to come to harm.”

This month, the computer scientist Yoshua Bengio — known as the “godfather of AI” because of his pioneering work in the field — launched a new organization called LawZero. As you can probably guess, its core mission is to make sure AI won’t harm humanity.

Even though he helped lay the foundation for today’s advanced AI, Bengio is increasingly worried about the technology over the past few years. In 2023, he signed an open letter urging AI companies to press pause on state-of-the-art AI development. Both because of AI’s present harms (like bias against marginalized groups) and AI’s future risks (like engineered bioweapons), there are very strong reasons to think that slowing down would have been a good thing.

But companies are companies. They did not slow down. In fact, they created autonomous AIs known as AI agents, which can view your computer screen, select buttons, and perform tasks — just like you can. Whereas ChatGPT needs to be prompted by a human every step of the way, an agent can accomplish multistep goals with very minimal prompting, similar to a personal assistant. Right now, those goals are simple — create a website, say — and the agents don’t work that well yet. But Bengio worries that giving AIs agency is an inherently risky move: Eventually, they could escape human control and go “rogue.”

So now, Bengio is pivoting to a backup plan. If he can’t get companies to stop trying to build AI that matches human smarts (artificial general intelligence, or AGI) or even surpasses human smarts (artificial superintelligence, or ASI), then he wants to build something that will block those AIs from harming humanity. He calls it “Scientist AI.”

Scientist AI won’t be like an AI agent — it’ll have no autonomy and no goals of its own. Instead, its main job will be to calculate the probability that some other AI’s action would cause harm — and, if the action is too risky, block it. AI companies could overlay Scientist AI onto their models to stop them from doing something dangerous, akin to how we put guardrails along highways to stop cars from veering off course.

I talked to Bengio about why he’s so disturbed by today’s AI systems, whether he regrets doing the research that led to their creation, and whether he thinks throwing yet more AI at the problem will be enough to solve it. A transcript of our unusually candid conversation, edited for length and clarity, follows.

When people express worry about AI, they often express it as a worry about artificial general intelligence or superintelligence. Do you think that’s the wrong thing to be worrying about? Should we only worry about AGI or ASI insofar as it includes agency?

Yes. You could have a superintelligent AI that doesn’t “want” anything, and it’s totally not dangerous because it doesn’t have its own goals. It’s just like a very smart encyclopedia.

Researchers have been warning for years about the risks of AI systems, especially systems with their own goals and general intelligence. Can you explain what’s making the situation increasingly scary to you now?

In the last six months, we’ve gotten evidence of AIs that are so misaligned that they would go against our moral instructions. They would plan and do these bad things — lying, cheating, trying to persuade us with deceptions, and — worst of all — trying to escape our control and not wanting to be shut down, and doing anything [to avoid shutdown], including blackmail. These are not an immediate danger because they’re all controlled experiments…but we don’t know how to really deal with this.

And these bad behaviors increase the more agency the AI system has?

Yes. The systems we had last year, before we got into reasoning models, were much less prone to this. It’s just getting worse and worse. That makes sense because we see that their planning ability is improving exponentially. And [the AIs] need good planning to strategize about things like “How am I going to convince these people to do what I want?” or “How do I escape their control?” So if we don’t fix these problems quickly, we may end up with, initially, funny accidents, and later, not-funny accidents.

That’s motivating what we’re trying to do at LawZero. We’re trying to think about how we design AI more precisely, so that, by construction, it’s not even going to have any incentive or reason to do such things. In fact, it’s not going to want anything.

Tell me about how Scientist AI could be used as a guardrail against the bad actions of an AI agent. I’m imagining Scientist AI as the babysitter of the agentic AI, double-checking what it’s doing.

So, in order to do the job of a guardrail, you don’t need to be an agent yourself. The only thing you need to do is make a good prediction. And the prediction is this: Is this action that my agent wants to do acceptable, morally speaking? Does it satisfy the safety specifications that humans have provided? Or is it going to harm somebody? And if the answer is yes, with some probability that’s not very small, then the guardrail says: No, this is a bad action. And the agent has to [try a different] action.

But even if we build Scientist AI, the domain of “What is moral or immoral?” is famously contentious. There’s just no consensus. So how would Scientist AI learn what to classify as a bad action?

It’s not for any kind of AI to decide what is right or wrong. We should establish that using democracy. Law should be about trying to be clear about what is acceptable or not.

Now, of course, there could be ambiguity in the law. Hence you can get a corporate lawyer who is able to find loopholes in the law. But there’s a way around this: Scientist AI is planned so that it will see the ambiguity. It will see that there are different interpretations, say, of a particular rule. And then it can be conservative about the interpretation — as in, if any of the plausible interpretations would judge this action as really bad, then the action is rejected.

I think a problem there would be that almost any moral choice arguably has ambiguity. We’ve got some of the most contentious moral issues — think about gun control or abortion in the US — where, even democratically, you might get a significant proportion of the population that says they’re opposed. How do you propose to deal with that?

I don’t. Except by having the strongest possible honesty and rationality in the answers, which, in my opinion, would already be a big gain compared to the sort of democratic discussions that are happening. One of the features of the Scientist AI, like a good human scientist, is that you can ask: Why are you saying this? And he would come up with — not “he,” sorry! — it would come up with a justification.

The AI would be involved in the dialogue to try to help us rationalize what are the pros and cons and so on. So I actually think that these sorts of machines could be turned into tools to help democratic debates. It’s a little bit more than fact-checking — it’s also like reasoning-checking.

This idea of developing Scientist AI stems from your disillusionment with the AI we’ve been developing so far. And your research was very foundational in laying the groundwork for that kind of AI. On a personal level, do you feel some sense of inner conflict or regret about having done the research that laid that groundwork?

I should have thought of this 10 years ago. In fact, I could have, because I read some of the early works in AI safety. But I think there are very strong psychological defenses that I had, and that most of the AI researchers have. You want to feel good about your work, and you want to feel like you’re the good guy, not doing something that could cause in the future lots of harm and death. So we kind of look the other way.

And for myself, I was thinking: This is so far into the future! Before we get to the science-fiction-sounding things, we’re going to have AI that can help us with medicine and climate and education, and it’s going to be great. So let’s worry about these things when we get there.

But that was before ChatGPT came. When ChatGPT came, I couldn’t continue living with this internal lie, because, well, we are getting very close to human-level.

The reason I ask this is because it struck me when reading your plan for Scientist AI that you say it’s modeled after the platonic idea of a scientist — a selfless, ideal person who’s just trying to understand the world. I thought: Are you in some way trying to build the ideal version of yourself, this “he” that you mentioned, the ideal scientist? Is it like what you wish you could have been?

You should do psychotherapy instead of journalism! Yeah, you’re pretty close to the mark. In a way, it’s an ideal that I have been looking toward for myself. I think that’s an ideal that scientists should be looking toward as a model. Because, for the most part in science, we need to step back from our emotions so that we avoid biases and preconceived ideas and ego.

A couple of years ago you were one of the signatories of the letter urging AI companies to pause cutting-edge work. Obviously, the pause did not happen. For me, one of the takeaways from that moment was that we’re at a point where this is not predominantly a technological problem. It’s political. It’s really about power and who gets the power to shape the incentive structure.

We know the incentives in the AI industry are horribly misaligned. There’s massive commercial pressure to build cutting-edge AI. To do that, you need a ton of compute so you need billions of dollars, so you’re practically forced to get in bed with a Microsoft or an Amazon. How do you propose to avoid that fate?

That’s why we’re doing this as a nonprofit. We want to avoid the market pressure that would force us into the capability race and, instead, focus on the scientific aspects of safety.

I think we could do a lot of good without having to train frontier models ourselves. If we come up with a methodology for training AI that is convincingly safer, at least on some aspects like loss of control, and we hand it over almost for free to companies that are building AI — well, no one in these companies actually wants to see a rogue AI. It’s just that they don’t have the incentive to do the work! So I think just knowing how to fix the problem would reduce the risks considerably.

I also think that governments will hopefully take these questions more and more seriously. I know right now it doesn’t look like it, but when we start seeing more evidence of the kind we’ve seen in the last six months, but stronger and more scary, public opinion might push sufficiently that we’ll see regulation or some way to incentivize companies to behave better. It might even happen just for market reasons — like, [AI companies] could be sued. So, at some point, they might reason that they should be willing to pay some money to reduce the risks of accidents.

I was happy to see that LawZero isn’t only talking about reducing the risks of accidents but is also talking about “protecting human joy and endeavor.” A lot of people fear that if AI gets better than them at things, well, what is the meaning of their life? How would you advise people to think about the meaning of their human life if we enter an era where machines have both agency and extreme intelligence?

I understand it would be easy to be discouraged and to feel powerless. But the decisions that human beings are going to make in the coming years as AI becomes more powerful — these decisions are incredibly consequential. So there’s a sense in which it’s hard to get more meaning than that! If you want to do something about it, be part of the thinking, be part of the democratic debate.

I would advise us all to remind ourselves that we have agency. And we have an amazing task in front of us: to shape the future.

There are two things you should know about President Donald Trump’s recently released budget proposal: First, it would significantly boost funding for Homeland Security and the Department of Defense while cutting social services that could hurt millions of people living in poverty. Second, his budget is just a proposal for Congress, and it almost certainly won’t become law.

But just because Congress is unlikely to pass Trump’s budget as is doesn’t mean that the proposal is entirely meaningless. It will likely influence what Republican lawmakers choose to focus on during negotiations in Congress, and, more importantly, it offers a window into his priorities.

Trump’s budget proposal looks to cut a total of $163 billion, slashing funds for education, housing, and health programs. Many of those cuts would come from programs that help lower-income Americans, from youth job training to Federal Work-Study. The cuts don’t include Social Security, Medicare, and Medicaid — some of the country’s biggest programs and the cause of the most heated debate over welfare reform. That’s because the proposal is limited to discretionary spending, and programs like Social Security and Medicare are mandatory spending.

Trump knows that gutting programs like Social Security and Medicare would likely have major political consequences. He has promised not to touch those programs other than by tackling waste, fraud, and abuse, though these are often a pretense to deliver benefits cuts anyway. The new budget proposal emphasizes that he hopes to overhaul the social safety net without igniting a fierce backlash. He’s betting, in other words, that people just don’t care enough about these less flashy parts of the social safety net or about how aid is delivered to people in need.

Trump wants to change how aid is distributed

One of the departments that would see the deepest cuts under Trump’s budget is Housing and Urban Development, which would lose $33.5 billion. Most of that, about $26 billion, would come from significantly reducing funding for rental assistance programs — including public housing, vouchers, and housing for the elderly — and combining them under a single program. The budget also proposes introducing a two-year limit on rental assistance for able-bodied adults.

The proposal would not only put millions of people’s benefits at risk, it would also upend how federal rental assistance works entirely.

Right now, money for housing vouchers, for example, goes toward directly subsidizing people’s rents. The White House wants that money to go to block grants instead. That means the funding would go into a pool of money that can be used to fund various state or local housing programs, giving states more flexibility in how they choose to spend it, even if that means they won’t go toward directly helping with rents. So hypothetically, money that is today intended for housing vouchers could be used to fund a program to give developers incentives to build more housing. (And while it’s good to build more to lower the cost of housing in the long term, that shouldn’t come at the cost of taking away direct rental assistance that helps keep people housed.)

There is precedent for this kind of switch. In 1996, Congress passed a law to create Temporary Assistance for Needy Families (TANF), which replaced the old, New Deal-era welfare system known as Aid to Families with Dependent Children (AFDC). While the latter provided direct federal payments to people who qualified, TANF created a system of block grants, where states could allocate welfare funds as they wished.

At the time, the argument for this funding structure was that states needed more freedom to spend welfare funds. But the block grants resulted in states diverting funds away from basic cash assistance. According to the Center on Budget and Policy Priorities, states spend just over one-fifth of their TANF funds on basic assistance, instead directing resources toward, in some cases, unrelated programs like funding tax cuts.

As Peter Germanis, who worked on welfare reform in the Reagan administration, put it: “When it comes to the TANF legislation, Congress got virtually every technical detail wrong,” Germanis wrote. “Congress gave states too much flexibility and they have used it to create a giant slush fund.”

By turning the federal government’s various forms of rental assistance into a single block grant program, the Trump administration might make it less likely for renters to receive the direct benefits they are entitled to, just as was the case with welfare reform in the 1990s.

That’s not to mention that the proposed budget cuts would be devastating to begin with. What renters need is actually the opposite of what Trump is proposing: more funding for rental assistance, not less. Federal rental assistance already helps lift millions of people out of poverty each year, but it doesn’t reach everyone who needs it, especially given the rise in housing costs.

Trump is targeting lesser-known programs

Another major cut that Trump is proposing is a program that helps families cover their home heating and cooling bills. His budget also includes a $4 billion cut to the Low-Income Home Energy Assistance Program (LIHEAP). Given that the LIHEAP budget is also around $4 billion, this essentially means that it’s seeking to eliminate the program altogether. The budget states that LIHEAP “is unnecessary” and alleges that the program is riddled with fraud.

LIHEAP, established in 1981, is one of those government programs that might not generate as much controversy as Social Security or Medicare, largely because it’s much smaller in reach and scale. But it’s a critical program that currently helps about 6 million families across the country. LIHEAP, along with the Weatherization Assistance Program, also helps cover the costs for home improvement projects, like wall insulation or furnace replacements, to make homes more energy efficient, especially in extreme weather conditions.

Eliminating this program could have catastrophic consequences for some families. “The stakes of this assistance can be life-and-death,” journalist Martine Powers wrote in the Washington Post last month. “Seniors are especially vulnerable to extreme temperatures. Getting electricity cut off for failure to pay bills can also be catastrophic for people with disabilities who depend on medication that needs to be refrigerated.”

Why this matters, even if it doesn’t become law

While LIHEAP has helped millions of families, it has also been underfunded, threatened by presidents of both parties, and doesn’t reach nearly as many people as it should. In fact, according to the National Low Income Housing Coalition, the program is only able to serve 20 percent of eligible households. Many eligible people also don’t know they qualify for the program and don’t apply for it.

Similarly, rental assistance programs help a lot of people stay housed, but they clearly aren’t reaching everyone they ought to, in large part because they’re underfunded. About half of renter households in the US are cost-burdened, which means they spend more than 30 percent of their income on housing.

Those are the kinds of issues that require real solutions, which would likely entail more funding, not less. But Trump’s budget underscores the problem that America’s social safety net constantly faces: Instead of looking for tangible fixes to improve programs like LIHEAP or housing vouchers, lawmakers often find ways to make them even harder to access, setting them up for failure. As I wrote in a previous issue of this newsletter, that was the case with public housing, which presidents and Congress routinely sabotaged before labeling it a failed experiment, even though they at times specifically designed it to fail.

And though Trump’s budget is not likely to become law — Republican Sen. Susan Collins, for example, said she has “serious objections” to Trump’s budget, including his targeting of LIHEAP — it shouldn’t be dismissed as a symbolic wish list. Even if Congress doesn’t deliver what Trump wants, these programs still won’t be safe. Just last month, for example, Trump abruptly fired the entire staff running LIHEAP, jeopardizing the delivery of heating and cooling assistance to families across the country.

So even if Trump’s budget never sees the light of day, here’s what you should keep in mind: It’s a blueprint for how his administration will hurt low-income families.

This story was featured in the Within Our Means newsletter. Sign up here.