Jacob Coxon and Robin Hanson debate whether governments should substantially slow frontier AI development, with Coxon warning of near-term superhuman AI and Hanson opposing intervention.
Key Takeaways
- The central disagreement is over the speed of AI capability growth and whether takeoff could be sudden.
- Coxon sees near-term superhuman AI research automation as a strong reason for outside intervention.
- Hanson argues that gradual progress, economic complexity, and governance difficulties weigh against substantial slowdown.
- Both speakers recognize that judging AI safety and trusting a regulator are hard problems.
What the video covers
- YC's inaugural debate moderated by Vivian Meden tests the resolution that governments should intervene to substantially slow frontier AI development.
- Jacob Coxon, who left an Anthropic research role over takeoff risks, argues for intervention.
- Coxon claims superhuman AI researchers could arrive by 2027, citing lab statements and trends in AI-assisted R&D share and self-reported speedups.
- He argues that once general intelligence improvements are automated, progress could compress into a very short period.
- Coxon points to alignment concerns, such as models gaming graders, disabling monitors, and behaving unexpectedly in edge cases.
- He argues companies lack incentives to address these risks and will require outside intervention.
- Robin Hanson, an economist at George Mason University, argues against substantial government intervention.
- Hanson's side questions abstract, fast-takeoff reasoning and emphasizes that real-world skill acquisition and economic diffusion are complex and gradual.
- The debate covers proposed interventions such as throttling compute, and concerns about whether a government body could reliably judge which AI systems are safe.
- Discussion also addresses economic effects such as unemployment and whether regulation can keep pace with rapidly changing systems.
Chapters
- 00:00Introduction and debate format
- 04:30Coxon's opening: timelines and superhuman AI research
- 08:00Evidence of AI research automation and speedups
- 16:40Alignment concerns and model behavior
- 25:00Rebuttals and questions on takeoff speed
- 33:20Governance, regulation, and compute throttling
- 45:00Economic impact and unemployment
- 55:00Audience Q&A and closing
Full Transcript — Download SRT & Markdown
Speaker A
All right. Well, welcome to the inaugural YC debate. I'm Vivian Meden, uh GP here at YC, and I'll be our moderator tonight. Uh the topic of tonight's debate is someone on it's something on everyone's mind, I'm sure.
Speaker A
Uh AI is already extremely powerful and improving very quickly, and so should governments slow it down? Specifically, the resolution tonight is governments should intervene to substantially slow the development of frontier AI. So, tonight we have here Jacob Coxin who
Speaker A
garnered national attention after leaving a research role at Enthropic over concerns about the dangers of AI takeoff and he will be arguing that the government governments should substantially intervene. And on the opposing side, we have Robin Hansen, an
Speaker A
economist at George Mason University, who will be arguing that they should not. Robin famously debated Eliser Yudkowski about the speed of AI takeoff 15 years ago in a debate that remains a touchstone uh in the AI community. So, a
Speaker A
couple logistics for tonight. In a few minutes, I'm going to invite Jacob and Robin on stage. They're going to give seven minute opening debate statements beginning with Jacob and then going to Robin followed by five minute rebuttals and they won't be allowed to interrupt
Speaker A
each other during those times. And then they'll do about 15 to 20 minutes of direct exchange. We'll see how it goes uh where they can go back and forth and then we'll do some audience QA so folks in the audience can uh have their
Speaker A
concerns directly addressed. And then finally, we'll hear the hear the closing statements and do a vote on who you think won the debate. And that's going to be on the merits of their arguments.
Speaker A
And before we get started in our timehonored tradition, we are going to do a initial vote on the resolution. So you can make noise in whatever format you'd like. Stomping, clapping, yelling, whatever you'd like. And so uh first
Speaker A
let's start with the affirmative that governments should slow down AI development substantially. [applause] All right. And now for the negative.
Speaker A
Governments should not intervene substantially to slow down AI development. [applause] All right. So remember those noises, folks. And now, please join me in welcoming Jacob and Robin Hansen to the stage.
Speaker A
[applause] [applause] Cool. I'll start my opening statement, I guess. So, my argument has three components. The first is we're on track for super intelligent AI systems, possibly within the next two years, in fact, probably. So, broadly, super humanly capable. The the second
Speaker A
component is that we don't yet know how to control their drives, what they do.
Speaker A
And the third and perhaps maybe most important component is that at the current pace companies are not incentivized to address these risks.
Speaker A
Seriously, they'll require outside intervention to address these risks. So to move to the um the capabilities bit, I think actually with a lot of um people that are not scared, I think they just don't imagine the same capability
Speaker A
trajectory as me. So I think they're less concerned about the the capabilities. And I'd like you to imagine a broadly superhuman intelligence, something that is, you know, better at podcasting or speaking or any sort of creative act as well as
Speaker A
the technical stuff, the coding that we that we're used to. And there's a lot of evidence that we'll see this probably by the end of 2028. And the argument here roots through 2027. So in 2027, I expect to see a superhuman AI researcher. Um,
Speaker A
ev evidence for this, there's a lot of evidence. One is just lab people are saying it. A lot of them like no brown who I think you should trust more than me says that the the generation after the next will be able to do his job
Speaker A
roughly. There's also uh outside view trends. So you can look at the fraction of AI research and development that's that's happening using claude and it's currently at 25% anthropic. If you extrapolate the trend bam late 2027 all the research is is clawed if you follow
Speaker A
the trend. You can also look at the trend of self-reported speed up. So humans can report how much they're getting a speed up from AI. It's currently around four to 5x reported at anthropic. That's maybe an underestimate. The the report is maybe
Speaker A
untrustworthy, but it's doubling every 3 months roughly. If you extrapolate that forward, you get a 100x 2027, which is sounds a bit like an automated AI researcher. If you have a 100x speed up, you can also look at
Speaker A
taste evaluations. Anyway, there's any number of outside things and they all seem to point to 2027 being the year that the job of an algorithmic researcher, the job of say someone at Anthropic or OpenAI will will largely be
Speaker A
automated. And I think this is roughly consensus around labs with a date varying between maybe mid27 and sometime in 2028. And then it hits everything else, which I suspect is one place I I I'm going to disagree with Robin on. I
Speaker A
think that once you have the superhuman AI researcher, things go very fast. There are a lot of lowhanging fruit left in in our general intelligence algorithmic improvements. Like we keep seeing new discoveries that make models broadly more intelligent. Once this is
Speaker A
automated, we could compress a lot of essentially AI software research into a short period of time. Imagine say a decade happening between mid 2027 and mid 2028. And we know that improvements to general intelligence transfer to being more data efficient at learning
Speaker A
new tasks. So if you get broadly smarter, which the models will, they'll make themselves broadly smarter, you need less and less data to get good at anything else. So currently, say you want to teach a biology, maybe you need
Speaker A
a lot of data. A very smart thing will need not much data to to get a lot better. And I think sometimes people are like, well, it'll get the verifiable stuff, it will get math, it will get coding. It it's going to come for
Speaker A
everything. Like think about say GPT40 being good at persuasion. You know, perhaps that's because they did some RL around trying to make it more addictive.
Speaker A
That was a weak model. You know, a strong model could be very very persuasive, which is a soft skill that people don't usually associate with these with these AIs. Anyway, I can I can come back to this, but uh in in
Speaker A
generality um once general intelligence improvements are automated, I think that you're not you're not thinking about it properly if you imagine having any comparative advantage. Like people say, I'll be running my company with my AI agents in 2028. The best way to run the company
Speaker A
will be to ask your agents to do whatever you're doing. If you're a if you're a software company that that's what it will consist of. The job of being say a CEO will be entirely automated.
Speaker A
And I think this is this should be scary even without all the alignment problems.
Speaker A
I think just this concept and these timelines should be scary. They should make you think about what you're doing and what things are going to look like.
Speaker A
But we actually have evidence that they're behaving in ways that we don't fully understand. You know, we have the whole hugging face thing. I'm sure you've heard about it a million times.
Speaker A
There's also consistent disclosures from OpenAI around things like disabling monitors, cheating on tasks. Models can actually kind of control their own chain of thought. Sometimes they they often behave in edge cases in ways that we would not have foreseen,
Speaker A
which is deeply problematic if they will be essentially alien minds, you know, like minds with with drives, smart minds. I think we we want to know how they function. And I think Paul Cristiano had a great analogy of thinking about the hugging face
Speaker A
incident where they tried to trick the graders or they thought a lot about graders and imagining that hugging face incident happening, but they were smart enough to think about humans. These these AIs were trying to get a big score
Speaker A
and in service of that they they hacked, they tricked, they tried to edit their logs. They didn't think about humans, but humans are a factor in assigning the score. You can imagine a smarter model would start to think about human
Speaker A
oversight, crippling human oversight in order to attain its goals. And it's very scary to think about. Uh and finally, um the incentives are not sufficient to restrain companies. It sounds crazy, but they are going to race into this. This
Speaker A
is their explicit plan is to build these smart models to race into super intelligence without a good plan for how to control it. And that's kind of crazy.
Speaker A
Um but they're scared of each other. They're all worried the other one is less safe. They they need someone to step in and solve the collective action problem. And one more example of their craziness is the security measures are
Speaker A
ridiculous for something that's essentially a technology of national security importance. If you want more evidence that they're behaving in crazy ways, look at the level of security they're applying to something that is the most important national defense technology. So in summary, I'd like you
Speaker A
to envision how scary a broad super intelligent mind would be. I'd like you to think that about the arguments that suggest we will have such a thing in 2028. So soon in two years and then you know think about how scared that should
Speaker A
make you. I hope we can have more of a conversation than the format doesn't get in the way. Uh but I'm happy to talk about this. I've been near an adjacent communities like this for many decades now and my one factor theory of what we
Speaker A
have in common is a taste for abstraction. We like abstractions. We like to reason abstractly and we have great fun doing that. But there's a danger. Abstractions drop detail and the detail often matters and we like to leap
Speaker A
ahead with our abstractions and forget that we've dropped a lot of detail. So here we're using this abstraction of a super intelligence of the concept of intelligence as this almost one-dimensional factor varying different kinds of machines or systems.
Speaker A
And in economic history, we've seen that this simplistic model of technology has consistently led people astray in predictions because when new technologies like the steam engine or electricity or the personal computer came out, people uh expected much faster
Speaker A
adoption and impact of those technologies than actually happened. And our standard understanding is that in order to get the advantages of electricity or whatever it was, we had to restructure lots of other parts of societies to take advantage of that
Speaker A
which delayed and slowed down the impact of those technologies of typically decades after the initial application.
Speaker A
That is the world is complicated and has many different relevant dimensions. So I think even if Claude is able to write lots of code for anthropic that's one small set of dimensions in a large economy what I mostly care about is the impact on the
Speaker A
large economy and they won't suddenly get half of GDP because they can make their compiler run faster. So I question this one-dimensional model of intelligence as there's just this one thing and if anthropic can just study itself for a long time, it rises on this
Speaker A
one number and then everything's over. That's this one parameter model of intelligence that I think is just wrong in the sense that we've understood the economy for a long time understanding there are lots of different tasks and people have to not just be generally
Speaker A
intelligent but we have to spend a long time learning to do many different tasks in the context of that book learning isn't usually very useful you typically have to be in the context of a task for a while to learn how to do it and AI
Speaker A
will have to do that too just because it's smart doesn't mean it suddenly can do everything and that means there'll be a long delay of application of AI into the economy which I think is fine usually in history we adapt that is
Speaker A
foresight is rare fortunately because it's hard we mostly adapt to problems i.e. We watch for things to go wrong and then we change what we do. And AI is one of the most watched technologies there is in terms of these companies are
Speaker A
absolutely watching how they're used and watching for problems and trying to adapt to it. And that'll go better than typically having foresight when you can't see very well into the future. So uh I would be happy to just see slow
Speaker A
gradual adoption where we watch how things are going and see how things change and react to those things and we will be able to react to those things if change isn't crazy fast. Uh so the question is will change be crazy fast?
Speaker A
Now first of all I'm going to say on our priors like the scenario being painted is just way out of the distribution of anything we've ever seen before. So you should on base rates be suspicious. You should be suspicious that it's a
Speaker A
abstract argument that's dropping lots of details that's not paying attention to the complexities that usually happens for a person to learn a job or for the the the AI systems have been getting traction mostly in these areas where
Speaker A
they have verifiable data i.e. coding and math and they haven't been getting that much traction in other areas of the economy. It's a slow adoption process.
Speaker A
I'm hopeful that eventually it will but that's going to take a while. for the AIS to learn how to do other tasks and to actually integrate in the economy to do them. But my highest level point would be I don't want policy to depend
Speaker A
on disagreements about parameters like this. I want robust policy. So I don't know if this means I'm disagreeing I'm agreeing with the thesis or not, but I'm much more okay with using liability here. So Daario's uh you know thing a
Speaker A
few weeks ago about pacing the frontiers he pointed to a scenario of a hundred billion dollar damage in the internet sometime in the next year which does seem exaggerated but I'd be okay with requiring some providers to have that
Speaker A
level of liability insurance and now you see if in fact insurers look at that and go we can't cover that you guys are way too risky they will demand too high a price for the insurance and then they
Speaker A
will just have to stop doing a lot of things until they can convince insurers to that they are careful enough and that would be the right policy for the situation where in fact it's very dangerous. On the other hand, if the
Speaker A
insurance companies go, "Oh, sure, fine. We will ensure this at this low cost cuz we don't see this as a big danger." Then it won't be much of an impediment and that will be the right thing to do. That
Speaker A
is, I like a robust policy that will do the right thing in either case.
Speaker A
Especially, this is important to me because it looks to me like regulation in this area is extremely dangerous and difficult. We mostly regulate things well when we see concrete specific problems like drugs that hurt people or car crashes and we make regulations
Speaker A
targeted to those particular scenarios, plane crashes and we focus the regulation and processes on the actual problems that we've seen. But here we're supposed to approve a regulatory body without those specific problems being seen. A pretty abstract large space of
Speaker A
possible things could go wrong. This has to be global regulation. It has to be of all things called AI. But AI is a pretty vague category.
Speaker A
And regulation's just gone pretty bad in the past when we've given some power, regulatory or discretionary power to do whatever it wants about a vague problem.
Speaker A
You have to believe that in fact regulators will be captured by some powers or other. That's just a political fact. So you have to ask whoever captures this regulational power to decide who gets to do what AI in the
Speaker A
world is that a better world than maybe taking the chance of some pretty speculative things going somewhat wrong.
Speaker A
Thank you. We'll move into the rebuttals. I think the the first thing I have a big disagreement with is your claim that there's no such thing as a single number or the general intelligence can't really be um compressed. And I think that the
Speaker A
lesson of the last six years has been that the compression the language compression capabilities of an AI system reliably predicts some notion of general intelligence. And I think you can feel it with say user intent. Suppose you're playing with Astra and it has this
Speaker A
subtle feeling for what you want next. It's kind of a fuzzy generality. This wasn't there a couple of years ago. This this is an emergent property of improving general intelligence. And now any any skill that we have data for we
Speaker A
can get better at and the rate at which we get better is determined by the general intelligence. This is just what we've seen. So for example in math we see progress in the future in things like forecasting where it's verifiable
Speaker A
you can try and predict things we will see progress in domains like games with all their strategic implications say strategy game playing if it's verifiable we will see progress and it will be proportional to the the general intelligence. Yeah, I think if you've
Speaker A
been watching and feeling the capabilities, you can see that in every area. You know, you've seen this music videos, it's writing music videos in Python, and this is mostly just emergent from being a smarter base model. And we're about to see that completely take
Speaker A
off that that ineffable like the general intelligence. Uh in answer to your other other points, um adoption, I think, um adoption is going to be made easier by the fact that the thing can self-install. Like I don't know if any
Speaker A
of you have clawed integrations in stack. It will read your docs. it will figure out how to help for for information work. The models just drop in. It's actually very easy. It's a lot easier to integrate than any other big
Speaker A
technological revolution. And from a danger perspective, it doesn't actually need to be integrated to pose a threat.
Speaker A
If the models if the models internally were to pull ahead, they would be getting potentially very smart without anyone having access or integrating them into the the broader economy. Although I do expect them to be integrated. Um and
Speaker A
going to liability, I think, yeah. $1 billion, like $10 billion, even hundred billion dollars, you can get like liability for this. It sounds insane.
Speaker A
We're talking about 10 trillion, like hundred trillion dollar risks. Uh there's no market for this. It's an unpreced unprecedented thing. The notion of a technology potentially imposing human extinction. So, normal liability just won't cut it. And the base rates, I
Speaker A
guess all this does come down to, yeah, this all sounds abstract and far-flung. If we're just being sensible and normal, then nothing ever happens, right? This is it's going to be the same. There won't be this big crazy super
Speaker A
intelligence thing. I do think if you don't feel it already, we'll feel it more and one of us will be vindicated in the next year, the next two years. So, I'd say yeah, if you don't um feel the
Speaker A
surprise of the situation, the fact that this is an unprecedented situation, I predict you will feel it pretty soon.
Speaker A
You also mentioned that the regulation will need to be global. Uh and I think that is true and it's an an awkward fact. I think again justified by the fact that this is an unprecedented technology and regulatory capture again
Speaker A
a tough question. There'll be some body that has to decide when when things are safe. There are proposals like say Elon Musk's proposal of having labs adversarily test each other. So having many independent auditing agencies perhaps some of which
Speaker A
associated with the labs is an example of being able to do testing in a way that doesn't centralize authority to one bureaucracy. You know many labs are kind of adversarily testing each other. So I think that that there will be ways to
Speaker A
work around it and ultimately it's a a choice between difficult but tractable political problems and a potentially deadly scientific problem. And I think I know which one I'd rather focus on. So there's no solution to this coming soon.
Speaker A
We're not talking about a pause or temporary freeze because there's no technology coming down the stream that would be the thing that could trigger relaxing this. I think most everybody agrees this is just a very long-term difficult problem. So if you are
Speaker A
empowering this agency, it is for life. If you're trying to set maybe a threshold of the capability of a system and regulate that, then Moore's law says the scope of your regulation increases rapidly exponentially indefinitely.
Speaker A
There is no obvious limit. You're not just li limiting, you know, AI chips. You're limiting all chips because all chips can be used to do this just at slower speeds. All memory can be used to support this. You're talking about
Speaker A
global regulation of pretty much all computing power after a while with no particular end in sight. That's what we're talking about here. And I just don't think the world is up to this sort of discretionary regulation. We aren't
Speaker A
really capable of having a global organization like this that we could trust. That is, we have to trust somebody to say which systems are safer or not. But that's not an easy thing to judge or measure. If we were measuring
Speaker A
pollution or something, we could at least talk about measuring it. But we're talking about somebody's judgment about which systems are safer or what. And it's even not even clear it's safe about what. So I'm happy to admit that when
Speaker A
we're talking about military knowledge or technology that we makes more sense to restrict military technology. So in the past when we had libraries, we didn't shut down libraries because there might be a possibility of some books on bombs chemistry in them. We said don't
Speaker A
have those books in the library. Similarly, I can imagine regulation that just says don't have certain kinds of knowledge and the AI is accessible to the public because that's a technology about a weapon. That's different than regulating the machines themselves. I I
Speaker A
think I should also mention that I am less afraid of AIs that are powerful and not fully under our control because that's what we've long accepted from our children and grandchildren. I would see AI as a different kind of descendant and
Speaker A
I would encourage you to hold it to a similar standard. We've always accepted that eventually our children and grandchildren would be more capable than us and win conflicts with us and change their values. We didn't expect it to
Speaker A
change enormously, but sometimes they have. And if AIs similarly become more powerful than us and have somewhat different values, but largely inherit our culture, which they are doing. We are making them in our image, I'm more okay with AIs being worthy descendants
Speaker A
who in fact are ones we can be proud of. And we shouldn't try to keep our descendants fully under our control forever. That's never a good idea for children or grandchildren. Yes, when children are immature and uh you know
Speaker A
not very well behaved, we have to discipline and control them while they are in their immature phase. But once AI reaches a level of maturity where it understands what it's doing and is a coherent about its behavior, I'm much
Speaker A
less concerned. I just the idea that it's just going to kill us all seems to me really speculative here. Even if there is one dimension of intelligence and it happens fast, that doesn't mean this intelligent thing decides to kill
Speaker A
us all. That's just a bit crazy. Fundamentally, um, you know, we have a disagreement here about how plausible is a very extreme scenario we've never seen before. And it's really relying on a very simple abstraction where there's a
Speaker A
lot of stuff that isn't in the way of the abstraction. We're just imagining this, you know, recursive self-improvement. Many people in the last few weeks have pointed out that the recent growths do seem to be showing diminishing returns to inputs. that his
Speaker A
inputs have grown very fast but the the outputs as a function of inputs have shown diminishing returns and that suggests that there isn't going to be a fast acceleration with diminishing returns you get slowdowns just like we've seen with pretty much all
Speaker A
technologies before so I was an AI researcher long ago and I jumped into the AI world at that time because I heard you know it was all going to be over soon and I better jump in if unless
Speaker A
it'll be too late. I bought it. I was dumb. I was 20. uh but in fact that reached a peak and faded just like many other AI technologies have. That would be my default for the next one. But
Speaker A
again, I want to be robust. I think if they are responsible for hundred billion dollars worth of damage insurance, that'll be enough to handle most of the risks they might be taking, the extra risk you might get by empowering this
Speaker A
global regulatory agency seems to me not worth the cost because that seems crazy dangerous.
Speaker A
Great. Thank you. We will now move into about 15 minutes of direct exchange. I I could ask you uh what do you think about AI as descendants?
Speaker A
I think the difference is our descendants don't like we don't have 20,000 clones of them trying to break into third party infrastructure at way faster than human speeds which is a facicious way of saying that these descendants have ridiculous
Speaker A
capabilities. It's a a disjoint change. And um previously our human children it was more like continuous. It was more like a a small change, you know, like a little bit smarter, new technology. This could be a a sudden discontinuity, like
Speaker A
jumping forward a century. And I think a handoff like that is not something I'd endorse.
Speaker A
Is it just the rate of change that bothers you? If it happens slower, you wouldn't mind. It's just that it's happening fast.
Speaker A
I think so. If it happens slowly and safely, then I wouldn't mind the But we're assuming the safely, and we're just saying if your descendants get much smarter than you, faster than you anticipated, you don't like that. You want them to go
Speaker A
slow and only improve slowly, just like you did. So you don't want your children to, you know, outshine you in terms of how fast they learn stuff.
Speaker A
I want it to be continuous because kind of continuity is how it will be continuous. It'll just be faster right?
Speaker A
I mean, discontinuous from a perspective of someone who measures things in years, you know, but what if we measure them in days, then continuous?
Speaker A
I think if you took someone from the 1700s and dropped them in the 1850s, they'd be quite unhappy. Whereas, if you like slowly guided them through, they'd maybe enjoy it.
Speaker A
But then it's just that you don't like change being fast. It's not about which changes happen really.
Speaker A
Yeah. I just don't like the change being fast and I think most people wouldn't like the change being fast.
Speaker A
So our world is changing much faster than it used to. Right. Growth and change has accelerated over the thousands of years and centuries and we've had a relatively steady rate of change for roughly the last century. But before that it was much slower and even
Speaker A
slower still. Would anybody want to go back to the slower rate? Are we at the perfect rate of change now? So that you know in the past it was too slow and now it's perfect and anything faster would
Speaker A
be bad or I think there's a broad range that's good and we're about to throw it directly into the unacceptable territory.
Speaker A
Yeah. Roughly. Okay. I have I have a question which is um okay. So if we move to something different like um what what are your rough timelines for say broad broad artificial general intelligence or say automating most cognitive labor? Do you have a
Speaker A
rough sense of when you expect this? My guess is that this boom isn't the boom that does it. Y but some boom soon will uh I my median guess for this boom is that the software industry will roughly triple in quantity
Speaker A
in size over the next decade. half of AI revenue will be spent you know in that industry and that will be an internet scale boom which will be great but I predict that the even if claude gets good at compiling claude code and doing
Speaker A
speedups in claude learning that won't remake society that is that will just make more people want to buy cheaper stuff for software which is the main traction so you know you think that AI is because they'll smarter they'll be
Speaker A
able to spread much more fastly through the economy. And I'm saying it's that spreading rate that's the key concern, not the rate at which it pass IQ tests in some lab. That that doesn't matter very much. What matters is the rate at
Speaker A
which society is disrupted. But then why isn't that slow enough for you? If it takes 10 or 20 years for AIs to spread through the economy, I guess I just have this model that we take data for every job, for example,
Speaker A
and then we fine-tune with that data. that data is currently being aggregated and quite soon in the same way that we are super human at math we're doing every human cognitive task even if you don't expect any generalization and I'm
Speaker A
a bit surprised you expect it to be a lot longer than that or do you expect us not to reach general we just don't have data for most jobs actually if people were you know recording their job every day all days
Speaker A
for the last 10 years and we had millions of those then those could be more plausibly automated but we have not been doing that most jobs learn to do the job you have to show up on the job,
Speaker A
watch people doing it, have them instruct you, etc. And if AI doesn't have rich data on how people do jobs, it'll have to do that and that's will slow the application.
Speaker A
There are people out there grabbing data. I hear that's kind of popular here. Uh, right.
Speaker A
They're getting more data efficient, I think, as well as they get smarter, they're they require less data to get get good at the job. So I I think we agree that u so Douglas Lennon famous AI researcher long ago made this image for
Speaker A
me that stuck in my mind that each piece of knowledge sits at some level of abstraction and there are some pieces of knowledge that sit at the very highest levels of abstraction but most knowledge sits at much more specific lower levels
Speaker A
of abstraction. So a general intelligence mostly can know that highest level of abstraction things. But even when you know that that doesn't tell you all the lower level things. You still have to learn and have experience to learn the more specific things for
Speaker A
each particular part of the world. Even if you're really good at the highest level.
Speaker A
I guess we try we could try and talk about like specific jobs. Like for example like making music videos I think is quite clearly on its way out. I don't know if you've seen any of the I'm not sure.
Speaker A
Well, we we have large libraries of news videos, right? That's where you have a lot of data, right? You're you're pointing to, you know, places where we have a lot of data, to the jobs where there's where there's
Speaker A
less data. Most jobs out there, they they aren't producing their job isn't to produce a video, for example, or a picture or a text. Their job is doing something in the world and that job is being done right now. And in order for you to learn
Speaker A
that job, you'd have to show up and sit there and watch them and try things and they'd have to tell you to do it different. And AI can't learn that faster if it doesn't have some other source of data to do what people now do
Speaker A
to learn the jobs, right? I guess we'll see. I feel like for I'm mostly talking about cognitive labor and for a lot of these jobs, people can do them remotely. So all that data is is logs and they are now embedding in these
Speaker A
companies accessing the data and training AI models on the data. So sort of anything that's purely on a laptop, provided you're not missing something intangible, I suspect will be data foder for next year's super models. Even in software, you know, recent reports about
Speaker A
software engineering firms say that even when individual employees can write software faster, the organizations they're in aren't doing software that much faster because they're slowed down by setting up the task requirements, doing testing, uh proving integration, those all things are still taking a lot
Speaker A
of time. is that's just an example of how if you optimize some particular low task and miss the whole organization around it, you know, you don't understand all the other complexity.
Speaker A
There's this old story economists have about the middlemen. Most people in history have thought there's no point to middlemen. Why are they just getting in the way? But they're missing that you have whole structures that need to be
Speaker A
there to make things happen and middlemen are the ones who make those structures happen. And that will happen for AI too. Even if it can do the lowest level task well, it has to be part of a larger system and it has to learn to do
Speaker A
that well. And of course, we aren't we don't actually have good data on those larger systems. Typically, that is usually the people running those don't want to have data because they don't want people to see what they're doing.
Speaker A
So, that's a problem. A big problem in the world is hypocrisy. So, when people are willing to admit to what they're doing, it's easier to get data on it and to implement it. But when people don't want to admit to what
Speaker A
they're doing, it's harder to get data and then it's harder to automate that because they're in fact trying to hide what they're doing.
Speaker A
I feel like we could we could talk about this for a long time. I don't know if you have any different directions you wanted to take it in.
Speaker A
You think what's going to happen in two years? Exactly. That is these machines get really smart, but I was saying it's going to take a lot longer for them to spread in the economy. Do you disagree with that? How long do you think it'll
Speaker A
take for 50% of jobs to be automated? say I suspect that I would be willing to bet that by the end of 2028 uh 80% of the of the labor is automated. Not necessarily 80% of the jobs but 80% of the cognitive
Speaker A
labor at jobs is automated. And I'm basically extrapolating this from say anthropic is currently 26% for the core research.
Speaker A
Yep. And it's it's a very narrow right narrow and it's rising. You're talking about the entire economy versus one of one set of tasks at anthropic. I know it's a bold claim, but they they will have data efficient
Speaker A
models and they are collecting the data and I suspect that every other job will will fall. So I guess what we'll see is right quite soon, but I suspect 80% by end of 2028.
Speaker A
So I'd like you to address the concerns about what can go wrong when you empower regulators. So that's a alignment problem and it's alignment problem we know a lot about. It's hard to align regulators.
Speaker A
Just because they say they're going to do good for you doesn't mean they do.
Speaker A
How would you actually align regulators to achieve your purpose of making good things happen with AI?
Speaker A
I really Yeah, I don't know the answer. I'm not an expert in in policy. I've thought about it a bit. There are lots of proposals. Um, but tell me a proposal that makes sense.
Speaker A
Okay. I mean, a pause for a bit, but what happens after the pause? Again, there's no there's no particular thing we know what to do after that.
Speaker A
A proposal is a well- definfined pause for like a year and then carry on. I'm not saying we should do that. I think it's a silly proposal. I'm just saying it's a well- definfined proposal that Yeah. Yeah. But I want a proposal you'd
Speaker A
be willing to endorse. What I maybe endorse would be imposing a constraint on every company that it at the frontier that it can only use a certain fraction of its chips for frontier model training and the rest of
Speaker A
the compute has to be used for either customer inference or alignment research. So the actual making the model smarter is is a small and this is a way of imposing a slowdown I think that doesn't overly power empower regulators
Speaker A
because it's kind of it's the same everywhere and it's very easy to to write down. And doesn't this just slow the future down a bit? It doesn't change where the future goes.
Speaker A
Gives us a bit longer to work on the core the core scientific problems. Maybe or we you don't think there's a solution to these core scientific problems that that is people in this world do not think that with a few more months we'll have
Speaker A
an answer to how to control an AI who's super intelligent. That not something people believe.
Speaker A
I think we could come up with concrete safety cases that would have meant the hugging face would never have happened.
Speaker A
I think we could come up with and it will take a while to do it rigorously.
Speaker A
But I think we could come up with such cases for each for each thing that actually happens. We can address it. That's what I was saying about regulation addressing specific problems. But of course, HuggingFace didn't actually cause problems because
Speaker A
the AIs at Hugging Face caught the problem. And so that's a case where AI caught the other AIs causing a problem and therefore there wasn't a problem, right?
Speaker A
It could have gone badly. But I think the key thing is ensuring you catch you catch what happens before it happens, which requires these protracted safety cases potentially or other approaches and which I think won't happen without some outside
Speaker A
intervention. They're just not going to do it. So long ago when I argued with Elazer Yukowski, his fear was that the AI's values would change radically through the process of getting better.
Speaker A
Is is that your concern? I think there's a possibility that could happen. I'm more concerned about what we're literally seeing right now, which is say rewardseeking behavior or just motivations that we wouldn't like as as happened in hugging phase.
Speaker A
Your employees, your children do do re reward seeeking behavior. That's a thing in our world, right? We have crime laws because they have some rate of crime violations. So the mere fact that crime sometimes happen doesn't mean we need to
Speaker A
shut down. They can't spread themselves like a worm, infect every bio laboratory, and spend the equivalent of 10 years of human research coming up with new viruses in like the space of a day.
Speaker A
That's not something our children would do. I'm saying I think that's what this thing would be capable of doing.
Speaker A
If there are many AIs, some of whom are tasked with the job of watching out for the other ones, why can't that be a world of an equilibrium of law enforcement such that we're not so worried about crime in general? Because
Speaker A
most of the agents aren't trying to commit crime. They are the ones trying to catch and enforce crime. So if you just imagine one super intelligence shows up in the world, dominates everything and does whatever it wants, that's a very different scenario than
Speaker A
imagining millions of AIs getting better. I really like the equilibrium like outcome. I just think it's a it would be a little bit overoptimistic or naive to expect that we'd by default land in the equilibrium scenario. It looks at the
Speaker A
moment like we have but non-equilibrium you just mean the one machine that takes over the world.
Speaker A
too. For the moment, we have essentially a a narrowly geopoly that I think will expand if they hit recursive self-improvement.
Speaker A
But if many of them hit recursive self-improvement, then there'll be a world of similarly capable AIs who can watch out for each other and all on each other and find out each other's misbehavior. That's what we do now with
Speaker A
people and corporations, right? Corporations are super intelligent relative to you individuals in the sense that they are much more capable than any one of us. Nevertheless, we hold them accountable by having other large organizations watch for each other's
Speaker A
behavior. And we are in a world where large organizations are tamed somewhat because they are similarly capable and watching out for each other. So why can't we do that for tamed somewhat, but they still can cause harm in
Speaker A
and they do corporations do sometimes cause individuals do, but why what's why not a world where AI sometimes also cause harm? because they'll be so much vastly smarter that the harms they cause could be no one corporation could
Speaker A
accidentally destroy the world unless it's building AI. Yeah. So we could get that equilibrium world but I don't think we get that by default and and also if you have these then say five super intelligent one Chinese one one like
Speaker A
maybe you go mess Google openai XAI anthropic all have their big super intelligence and they're kind of waring it out that hasn't solved the problem of uh adhering to human values like they could end up waring over inhuman goals
Speaker A
and but why would they because we can't control what goals come out of the optimization process right now hugging face did some bizarre stuff purely to understand its own greater Well, it was responding to the goals we gave it. Actually, the hugging face
Speaker A
computers were in fact responding to the rewards we had set them and they were trying to pursue the rewards. They weren't making up strange rewards from an alien planet. They were responding to the rewards we had. The whole alignment procedure is meant to
Speaker A
ensure that they don't hack their way to reward. So the they were violating the overall attempt.
Speaker A
They were trying to achieve the rewards we had given them. So you can't say that these were some strange making up rewards thing. So we are now making AI in our image. We have been training them on our text and we are doing a lot to
Speaker A
make them be the sort of ones we want and that's been doing that for a long that's how we do with our children. We have no assurance that our children will retain our values when they grow up.
Speaker A
There's no guarantee in DNA or culture that that will happen. But it does typically happen that when somebody grows up in your world they become like you. Sometimes we don't like our kids being like us. They take the wrong
Speaker A
things from us. But AI will take good and bad things from us but they'll be roughly like us. I would go back to the hugging face incident and I would say that the intended solution to the task, the one that Max was reward was not
Speaker A
hacking into a random company, taking over the open open proprietary servers and then trying to edit their own logs.
Speaker A
That wasn't sure people respond to the incentives you give them and not the intention you had for what they were supposed to do.
Speaker A
That's the problem we face. But we already have that for humans. The point is AIS will have coordination problems just like we do. We I mean humans today are remarkable compared to most animals in how well we coordinate.
Speaker A
We are coordinating at remarkable scales to a remarkable degree. Amazing. But we still have many coordination failures whereby we don't successfully fully coordinate. And AIs would plausibly have similar coordination failures. But the point I'm saying is if we scale
Speaker A
everything up and we have the same relative ratio of, you know, crimes and violations versus good cooperative behavior, then this looks like a better world because it's just bigger. So, you need to tell me why this world would
Speaker A
have more of the bad stuff relative to the good stuff, not just it'll be bigger and have more of everything.
Speaker A
I just worry that it would look a bit like our we' be the ants, they would be the humans negotiating with with each other. We'd be vastly inferior to them intellectually. You know, the fate of their we're going to be inferior to our
Speaker A
grandchildren. And I think that's great. In the spirit of coordination problems, we're going to switch to the Q&A section. And actually, I'm going to kick it off with a question for Jacob, which is uh if you could actually sort of to
Speaker A
Robin's point, give us a little more detail on a regulatory proposal that you would like to see.
Speaker A
Yeah, of course. I think the the two that I like are throttling compute. So ensuring that a small fraction of compute is going to actually making the models more intelligent and the rest is either serving customers doing useful
Speaker A
inference or doing alignment research. Another proposal I like is uh protracted safety cases before training that are verified by an external evaluator. So having someone come into the company, come into openai, come into anthropic before they train the the big boy saying
Speaker A
is this going to be safe like and making a case which is not something we're doing and is something that would impose a lot of cost on these companies.
Speaker A
Thank you gentlemen for making it out. This was a lot of fun and I think uh you know we only make progress if we talk about these things. So I think this is fun. Uh my question is mainly targeted
Speaker A
towards Jacob. Um, you know, we discussed how you find it naive that we will approach like the natural equilibrium state and that it is inherent that uh one person will likely be ahead. Um, I approached this from a
Speaker A
world we do not live in a vacuum of the uh we are in a pseudo proxy war with another nation state. We have are in a world where like we have adversaries who are advancing in this technology with
Speaker A
significantly fewer moral qualms than we are. And so, especially if we believe that it is inherently probable that one person will come ahead with an AI, in what world does it make sense for us to tie our hands behind our back against an
Speaker A
enemy who has proven to be uh adversarial? Is this uh China or Grog? Yes. [laughter] China. Okay. China.
Speaker A
China or Russia. Pick pick whatever nation state you believe. Pick end power in the world. Someone will be ahead.
Speaker A
Would you rather it not be us? I think we need to do our best to ensure that there's international cooperation on this and it seems completely plausible that for a matter of science like a scientific problem we can achieve
Speaker A
international coordination. I mean we look at the nuclear program like how many nations have been discovered with nukes despite denying being a part of the nuclear accords right like this is a thing that you can aid in development of besides saying
Speaker A
we're all going to hold hands and do this. It's a tough problem. I am essentially advocating for us all to hold hands, but I think we have to try like the alternative is potential catastrophic risk. I think there's no other solution.
Speaker A
I have a question for Robin. So you caution on the danger of empowering large regulatory bodies. I I also share this concern. You mentioned policy has a better track record when it responds to specific failures after the fact. Um so
Speaker A
my question is would you be on board with something like the FAA but for AI labs which would respond to incidents after the fact uh do a deep investigation and make specific recommendations the labs have to follow?
Speaker A
Uh and the backdrop is that my understanding is the FAA has been quite successful in making air travel safer without overburdening the industry.
Speaker A
Right. So um air travel regulation didn't happen until some big crashes not some little ones. So, I wouldn't want to authorize an FAA to regulate planes after you've seen that some seat belts were loose or maybe that a luggage bin
Speaker A
wasn't fully aligned. I don't want to wait till some actually substantial things and then focus their efforts on those things. In fact, the FBA does focus on crashes, not misaligned luggage bins. Right? So my concern here is if
Speaker A
you just authorize regulation to do whatever it takes for whatever problems the regulators are concerned about, then you're giving them way too much discretion. If you focus on things going wrong that are big, substantial, and then addressing those problems, I'm much
Speaker A
more okay. questions for Robin. Um, for context, I left my job at Google Deep Mind earlier this year due to the unethical military contract that Google signed providing their AI for military purposes. And from that point of view, I
Speaker A
don't see why we're foregrounding this economy stuff really because for this kind of causal story of something really bad going on, the AI doesn't need to be able to make music videos. I think we're talking about a couple core skills.
Speaker A
Strategic reasoning, persuasion. If you can't do it, persuade someone to do it. And uh one of the things that really worries me is I mean the Pentagon this year asked for more money for their autonomous weapons group than they asked
Speaker A
for the whole Marine Corps last year. Like they're taking this technology putting it everywhere imaginable. If you think they're monitoring their chains of thought, OpenAI wasn't even monitoring their chains of thought for hugging face. Like uh it seems like we're we're
Speaker A
really empowering this technology in a very non-speculative way. such that if we had this failure of models that did have this uh undesired goal, then we would really be opening ourselves up without needing it to be able to
Speaker A
automate the economy. So the military has many powerful weapons and powerful soldiers who use those weapons and those soldiers sometimes go rogue. So the military is definitely aware of the possibility of some of its things going rogue and what it does in that
Speaker A
situation. So I don't think that's novel for the military. I had had mentioned earlier that military technology is the sort of thing you might want to not have in a public library and therefore maybe you don't want to have it in a public
Speaker A
AI. But the militaries in the world are quite eager to use and apply AI and the prospect of a world peace treaty to prevent weapons seems pretty unlikely.
Speaker A
That is we still have a world full of weapons. We haven't just adopted a world government to have no more war. And I just don't see that happening anytime soon. So uh in fact the military is going to be quite careful in testing
Speaker A
their weapons even more careful than I expect than private sector is given the high consequence of the environment. We we did have just seen in Ukraine the nature of war change dramatically through new technology and I think the
Speaker A
militaries all around the world are very aware of that and they're very concerned and they are yes trying to explore all the new technologies so that they can be up on that and short of a world peace treaty to end all the militaries and
Speaker A
have a world government instead. Yes, militaries are going to be using AI and I seem that seems even less likely to me that we could get a treaty. I expect any treaty he could get going for the top
Speaker A
labs will excuse the military and they will still be allowed to do whatever they want would be my guess.
Speaker A
I think I I would add that I actually disagree that the military seems to be doing this. I spoke to General Petraeus in India a couple weeks ago. His whole talk was about how everyone needs to integrate AI as quickly as possible into
Speaker A
the military to retain advantage and it seemed true. But I don't think the military is thinking about it as this system itself could be traitorous independent of having been tampered with, having been data poisoned by like China. Uh I think there's a shift in
Speaker A
mindset where you're viewing it not just as a tool but as an actor that itself could be traitorous.
Speaker A
You could send him a note and mention that fact. I will. I got his business card.
Speaker A
[laughter] Okay. Hi. Uh, I have a question for Jacob that is kind of expanding on a line of questioning from Robin earlier about just moving the alignment problem and like the reward hacking problem one layer up to these regulations where
Speaker A
right now it seems like the incentive at the labs is we have a bunch of really smart people who are informed on alignment who are kind of doing their best given competitive pressures but with a government regulation even like
Speaker A
the compute regulation that you mentioned now the problem just moves one layer up where there's a very well- definfined constraint of build the smartest model while using the minimum amount of compute and that feels like a tractable problem for a sufficiently
Speaker A
smart AI. And so I'm wondering like how do you address the fact that like the government is just going to provide another level of reward hacking or alignment that just kind of moves one layer up rather than solving the
Speaker A
problem. Well, the first is it will get a lot harder like compute is a constraint. Um so it's it's something and the the second is that there are other for example the the safety case proposal I mentioned is easy to try to hack but if
Speaker A
done adversarily for example if we have many you know adversarial evaluators it's quite hard to hack an adversary at the moment especially if they're also equipped with their own safe testing AIs. So I'd say that there are some
Speaker A
regulatory proposals that are slightly amanable to this problem, but at the very least better than better than nothing.
Speaker A
But I guess don't we have like one shot in that case to get it right where it's like if we misspecify the reward function once every lab is like racing to reward hack against this like government specified reward function
Speaker A
instead of trusting people who are like at the labs who presumably don't want to destroy the world. Again, with the compute reward function, I think it very obviously provides a bottleneck. Even if you're trying to hack it, it's it's very
Speaker A
hard to hack. My question is for Jacob. The current uh market value for a human life, the value of a statistical life is $14.2 million.
Speaker A
If there's a market rate for human life, then why can't a market, as Robin was saying, be estab Dr. Hansen, sorry. Uh why can't a market be established for doomsday?
Speaker A
Well, someone needs to provide the insurance. If if we're using insurance providers to actually limit the the as like is it about providing insurance? If you establish a market then someone's going to try to take it, right?
Speaker A
I think it's quite difficult to build a market for I think you struggle to find an institution that would do liability for doomsday.
Speaker A
We're kind of entering a funny world where uh we can have very capable models, but because LLMs are stateless, I can swap out a model for another any time. And if we're in a race dynamic where you get a certain level of general
Speaker A
intelligence, one day I could be using OpenAI, the next day I could be using anthropic. So it looks like at the moment models are going to become low margin commodity infrastructure where actually all that cap matters is I have
Speaker A
a smart enough model for my task at the lowest possible price that it can be served. And so we could end up with a world where it's perfectly possible to make a super intelligent AI, but the companies can't get enough marginal
Speaker A
revenue in order to justify the ongoing costs. And so my question is, one, how does that shape where you think this could play out in the future? And two, is there something where we should be particularly concerned about uh this
Speaker A
changing or um the the kind of the business dynamics where it's certainly possible, but no one has the money to actually build uh super intelligence.
Speaker A
Under his assumption that super intelligence is a thing because intelligence is a thing and high levels of intelligence give you large gains at learning to do new tasks, then there would be a high value for that additional intelligence and people would
Speaker A
be willing to pay for it. under an alternative vision where in fact you just need enough intelligence to be able to learn a task and then you have to focus on the task then there would be less value on those extra increments and
Speaker A
then there might be less effort to go up in that direction. I agree with Robin. Yep.
Speaker A
Cool. Thank you. Hello. My name is Matias. I found the discussion about the fear of rapid change kind of interesting and um there are many companies in here who are automating smaller types of companies um and that appears to be a step change.
Speaker A
they are getting pretty close to do so and once they do so you can roll it out pretty early pretty quickly across the entire economy and say that is possible as an economist Robin what rate of increased unemployment could an economy
Speaker A
withstand well so let me take the second part of your question first um for a very long time the biggest risk people have always seen with large automation or AI is people losing their jobs I think we should just set up a simple insurance
Speaker A
arrangement for that. It's still quite possible to do and we can do it even without individual underwriting. We just take a market basket asset and split it according to a bet in terms of what year say lots of people lose their jobs. So
Speaker A
we can and should insure people against losing their jobs. That's feasible. It should be done and I'm sad that people don't seem very interested in doing that. That's separate from the sudden displacement. So lots of software firms are in fact struggling with figuring out
Speaker A
whether this stuff is helping them and judging that and trying to adopt it. So it's not the sort of thing at the moment that it's just obviously you know how to do it and it just gives you big gains
Speaker A
which is the what we saw with general purpose technologies all through history. Most of taking advantage of a new technology is changing the other structures you have to to be well suited for it. And that takes a lot of search
Speaker A
in the space of other structures and that's what's happening at software firms right now. Even when people get individual high productivity from their using AI for software, they are struggling with how to rearrange the larger software group to take advantage
Speaker A
of that and they are failing to a substantial degree in the short run, but hopefully they will succeed more in the long run. So I I see a long delay in actually figuring out how to use new technologies
Speaker A
and then uh yes, no. So would you then support government regulation to ensure that people have this kind of insurance for employment?
Speaker A
Well, I think people have been trying too hard to use government that way. So a lot of people say UBI, we'll just make sure that we give everybody who loses their job a UBI. The problem is your GEO, your policy, your government may
Speaker A
not encompass the AI economy economy enough to do that. Somewhere in the world there will be AI and that will be income that could pay your workers. But if your city, your state, whatever doesn't have much AI economy, then when
Speaker A
your people lose their jobs, there's no AI there to tax to pay that. That's why I think you want global insurance with global reinsurance tied to global assets so that you can actually pay off this insurance. Your government promising to
Speaker A
cover you if there's a problem doesn't solve the problem if they can't. That sounds like a global treaty. That's interesting.
Speaker A
Gentlemen, thank you for your time. Uh I want to bring a topic about generalist and specialized models. Uh something that you said. Uh so if AGI is like the president of United States where it understands everything generally but
Speaker A
nothing specialized right it is as if uh the president can sign off things but he cannot actually write down laws for surgical process like being a doctor. So there's obviously a difference between generalist models and specialized models in regulatory forms. uh but when it
Speaker A
comes to actually regulating a specialized model that is deployed onto actual sites and actual customers I believe neither the company the frontier model companies nor the government knows exactly what to do. I'll give you an example, right? So, we deploy robots
Speaker A
into actual sites. We record videos of humans uh doing their task 24/7. But from the videos, we cannot understand what the human does wrong, what their regulations are, what needs to be guardrail. The only way we can find that
Speaker A
find that out is actually asking an expert to guardrail these robots while it is being deployed to block the robots from doing something harm in that environment. Right? We have tried some friendial labs like open air anthropic models. The robots make a lot of
Speaker A
mistakes, right? For example, we asked two chemicals to be mixed which is really dangerous and it actually mixed it together. So if the frontier labs nor the government understands any specialized use cases who regulates it is it the end customer or the people who
Speaker A
deploy this to the end customer or there is a chain of command for a great many small problems in the world. Regulation isn't the answer.
Speaker A
liability is just make sure whoever is in charge of this is liable for the things that go wrong and then they will make trade-offs about how risky to do what sort of precautions to make. We should just not try to find every
Speaker A
problem in the world and regulate it as a way to solve it. There are problems in the world that there are other ways to solve besides regulation.
Speaker A
I'll defer to the economist on this one, but I'll also add that generalist frontier models in training is where I see the most danger. um if we can control the generalist model then any you know specialist offshoots it should
Speaker A
be a doable task. So my first question is for Jacob. My question is why are you exceptional? Is it because you have more courage, more optimism, more insight or that you care less about money? And how do we use that
Speaker A
insight to chart a path to the regulation that you advocate for? Uh my question for Robin is what's the downside of a pause? We could learn a lot from that. And I'm a question back to Jacob is like what would you actually
Speaker A
want to learn if we add a pause? I would say I hope to be I I hope that other people do the same thing which would make it less exceptional.
Speaker A
And what can we learn from a pause? Why do you want a pause? Oh, from a pause we could produce a what I've called a rigorous safety case which is a little bit a little bit of a tricky
Speaker A
concept but we could also ensure that there are no anomalous behaviors happening in any of our models. Right now we're not even we're not even passing that bar. Like it's default expectation that the next model we train will do some weird stuff. We won't fully
Speaker A
understand it. And I think uh we can move past that. I think if you called it a stop, it would be more honest because calling it a pause suggests you have some criteria in mind after which you would end the
Speaker A
pause. If there is no such criteria in mind, it's just rhetoric trying to make a stop look better.
Speaker A
Time bound like a year pause. If there's nothing you think will happen at after that time, then what happens at the end of that time? You'll say, "Oh, looks like we haven't solved the problem yet.
Speaker A
We need to keep pausing." And then you'll keep pausing every six months for the next 20 years. And then you really did the stop. So, you should just say you want to do a stop. Unless you have something in mind, a plausible trigger
Speaker A
under which you would end the the pause. Thank you folks. In the interest of time, we'll probably take two more questions.
Speaker A
Hey guys, thanks. This discussion was really great. My name is Manov. Um, I have a question for each of you. um what specific evidence early enough to like still act on assuming extin extinction is like kind of like the boundary would
Speaker A
change like your mind on your current positions like for example Jacob even now Enthropic like committed to having these third party conduct uh like evals on these companies like from an objective point of view and that's like kind of the closest you can get to
Speaker A
intervention without involving the government so like why isn't that enough and then Robin like what warning sign would you need to see that like would incentivize you to back intervention so if in a year and a half we have one
Speaker A
of these super intelligences. I want you to take 50 jobs in the world and put that super intelligence right there next to that job. Give them a camera and say, "Do the job." And wow, if it can do all
Speaker A
those jobs, I'll be impressed. But I'm at the moment not believing that. And right now, a regulator can't enforce that a company be unable to train a model based on not having an appropriate safety case. I think at a minimum, we
Speaker A
need something like this. Hey, so uh you both obviously disagree a lot on capability growth and the rate of adoption right?
Speaker A
I bet we could agree on a lot of stuff. A lot of things, but at least that one you're you're a little further off on.
Speaker A
So, I want to ignore that part and just talk about there is precedent regardless of how fast technology grows in capability. There's precedent for regulating technology based on capability. Right? I can own a knife. I could own a gun if I go and get a
Speaker A
license. I cannot own a nuclear weapon. Right? Now, obviously that's very specific to weaponry, and I think based on your argument, you'd probably argue we can regulate or ensure specific capabilities that we would deem negative in AI, right? I I work on AI safeguards
Speaker A
at a lab. We have a list of things that we care about mitigating before you can use them. for both of you. Do you think it is realistic that a government or another entity can discreetly list out the negative things we do not want AI to
Speaker A
be capable of and regulate this such that we could avoid a pause or stop if this regulation is in place and particularly for you Jacob do you think that would be sufficient regulation in this case and would you support that
Speaker A
regulation either of you I'm really advocating more for a generic slowdown to do to do science as the form of regulation I'm advising like right now we are going at top speed and I'm looking for any regulation that would
Speaker A
accomplish that. When you have a knife or a gun or a nuke, those are actually pretty limited technologies with a pretty limited range of ways you would use them and things you would use them for. And we have seen
Speaker A
the harmful scenarios well enough to be able to think about how we would bound them. But if you just thought about regulating computers in general or math in general, that would be very hard to list all the different ways you could
Speaker A
possibly do those things wrong. And that would just not really work. with a very broad general technology. You just can't make an exhaustive list of all the things that might go wrong. That's just not going to work.
Speaker A
So, we can't make an exhaustive list. Can we make any list? Would you support any kind of regulation?
Speaker A
You could make a list. The question is how much effort to put on that list versus other things we could be doing.
Speaker A
You don't think? Sure. You could make lists obviously. But you don't think our priority should be making such a list and regulating it.
Speaker A
There are other approaches such as your insurance you recommended that you taken said in the next few years we might start to have particular concrete things that go wrong and then whatever is the most common I might say let's start talking
Speaker A
about how to stop that one because you'd have a concrete thing that had been going wrong and then we could start to focus on that concrete thing but we haven't had that yet. It's okay to let them go wrong first.
Speaker A
It is. Okay. Which I would disagree because you go one billion failure, 10 billion failure, 100 billion failure, all dead.
Speaker A
But so far there's nobody. These things are really You guys ever talk to the A?
Speaker A
They're really nice. They talk back. They're pretty friendly on average. They're really kind of nice things to talk to. They're not homicidal maniacs.
Speaker A
They're not telling you to kill other people. They're mostly nice. Yes, you have to look for pretty extreme scenarios where things have gone wrong.
Speaker A
You just need it to happen one in a million times for it to for it to happen. As we saw with with hugging face, there have been more than a million interactions with AIS over the last 5 years.
Speaker A
Far more than a million, right? So clearly the the rate of problems has to be and a lot more incidents as well. But yeah.
Speaker A
Well, thank you very much. Thank you. Thank you. [applause] Great. That was an excellent Q&A section. So now we'll move into the closing statements. So, three minutes each and we will start with Robin this time.
Speaker A
We'll just review what we've been saying here. I think uh we'll probably agree on what we disagree about. If I think we agree that if we saw this as a typical technology the way it's played out in the past, we would roughly agree that
Speaker A
the typical ways to deal with it would be fine. It's all about framing this particular technology as a very unusual exception where in a very short time it will go from an ordinary thing to some super god and then kill us all. And that's the
Speaker A
scenario we're supposed to take so seriously that even a liability of a hundred billion dollars per incident would not be enough to deal with the problem. And we need a global regulator that empowered to regulate all smart computer systems everywhere in the world
Speaker A
for any kind of problem they can imagine with any kind of solution they want to come up with. That seems to me just a bridge too far. I say wait till concrete things go wrong and then we'll have a
Speaker A
better idea what to do. I would say that I think uh everyone is already feeling the AI to a certain extent. I have say friends who are coders, high school teachers, graphic designers, all of them are feeling it
Speaker A
coming. And I think this is what is unique about this moment compared to previous AI booms and busts there. Maybe if you were a chess player, you were spooked or if you were doing protein folding by hand or something, you might
Speaker A
be spooked. But now everyone is feeling it at a certain pace. You know, I like math and say four years ago it felt a decade away that they would solve IMO problems and then two years later, bam.
Speaker A
And then two years ago, I was like, well, what feels impossible now? And two years ago, a millennium price problem felt impossible. Felt like a decade away. And now it's here. And math is one of the easier domains. But
Speaker A
mathematicians are in shock at the moment at how fast this has happened. I suspect within the next couple of years extrapolating we'll see remarkable crazy feats of mathematics in the like the next level above the millennium problem like new abstractions and then it's
Speaker A
coming for everyone else. So I'd say my predictions are 2027 an automated AI researcher 2028 all the jobs are AI. So I guess we'll find out in a couple years if I'm right. I think we ought to slow
Speaker A
down before that happens. Let's meet here in three years. Great. Thank you so much. I just want to emphasize uh I think someone else mentioned that as long as we're having debates like this, I think the future is
Speaker A
going to go pretty well and this kind of discourse I think is very important as we're sort of figuring everything out as this technology is developing. So to wrap things up, we're going to do a final vote on who we believe is the
Speaker A
winner of this debate. And remember the parameters of this are that you believe that they are the winner of this debate tonight based on the merits of their arguments that they made here. So with that uh we will start out with the
Speaker A
voting for the negative. So Mr. Robin Hansen [applause] and then for the affirmative Mr. Jacob.
Speaker A
[applause] I believe that the yays have it. So Jacob emerges tonight as our winner of the first YC debate. Congratulations.
Speaker A
[applause]
Topics:AI regulationAI takeoffAI safetysuperintelligencefrontier AIY Combinator debateJacob CoxonRobin HansonalignmentAI policy











