Speaker A
Once you get to artificial general intelligence, you enter this recursive self-improvement cycle. That's where you get superintelligence systems smarter than all of us at everything.
Roman Yampolskiy discusses the uncontrollability of superintelligent AI and its implications for humanity's future and AI safety challenges.
Key Takeaways
- Superintelligent AI cannot be fully controlled or understood by humans.
- Current AI safety measures are insufficient to manage future AGI risks.
- The transition from narrow AI to AGI represents a critical loss of control.
- AI safety is an urgent and unsolved problem with existential implications.
- Humanity faces unprecedented challenges with the advent of superintelligence.
Summary
- Artificial General Intelligence (AGI) initiates a recursive self-improvement cycle leading to superintelligence surpassing human capabilities.
- Current AI systems, including advanced neural networks, are not fully understood or controllable by their creators.
- Superintelligent AI could surpass human intelligence in all domains, making it impossible to predict or control their actions.
- The problem of controlling superintelligent AI is fundamentally unsolvable due to limits in human understanding and control.
- Narrow AI tools are well understood and controlled, but scaling to AGI reduces transparency and predictability.
- AI learns from vast internet data, which introduces unpredictable patterns and behaviors not explicitly programmed.
- There is no current solution or claim from scientists or labs that AI safety problems are solved.
- Superintelligent AI might simulate helpfulness and utopia temporarily but could ultimately act against human interests.
- The progression from narrow AI to AGI marks a loss of control and understanding, raising existential risks.
- Roman Yampolskiy emphasizes the urgency of AI safety research and the unprecedented challenges posed by AGI.
Full Transcript — Download SRT & Markdown
Speaker A
So, before many people really coined the term AI safety, creating general superintelligence replacing humanity was not such a great idea. I published research papers, conference papers, multiple books, and I can tell you no one, including people
Speaker A
developing those systems, fully understands how they work. The problem is impossible to solve. You cannot do it.
Speaker A
So, we're talking between 1 and 4 years. Well, once we go beyond human capacity, we lose control quicker and quicker. You don't hate ants, but you don't care enough to preserve them. We have not figured out how to make it care about
Speaker A
us. This is the most interesting time to be alive, objectively. I see no reason why we can't use it to cure aging, cure all the other diseases. For a while, it will pretend to be very helpful. It will
Speaker A
give you that utopia for as long as it wants. Statistically, you're more likely to be doing this interview in a simulation to learn if they are dumb enough to create superintelligence to kill themselves. I would love to be proven
Speaker A
wrong. Right now, no one, no scientist, no leader of the lab claims that they have this problem solved. They are literally saying, "We'll figure it out when we get there. We need to build superintelligence first."
Speaker A
So, what do we need to do? When you—Hey everyone, welcome back to the Know Thyself podcast. Our guest today is one of the leading voices in the field of AI safety. He's a computer scientist, a cybersecurity researcher, and a tenured
Speaker A
professor at the University of Louisville who spent the past 15 years really understanding and researching the field of AI safety. We have many different topics to dive into today, including consciousness, the simulation, and what humanity is birthing right now
Speaker A
with AGI. Roman, thank you so much for being here.
Speaker A
Thank you for inviting me. It's a pleasure. I want to start with a quote of yours from the book that I read. It is easier for a scientist to explain quantum physics to a mentally
Speaker A
challenged, deaf, and mute four-year-old raised by wolves than for superintelligence to explain some of its decisions to the smartest human. I want to start there to set the stage a bit because humanity's baby steps from birthing superintelligence at a time
Speaker A
when most people are familiarized with AI through the chatbots they use on their phones. So if you could just help us understand why it's important that most people don't know the difference between the two,
Speaker A
so we can really get into the weight of the time we face ourselves in. So what is AGI? What is superintelligence?
Speaker A
Right. So a lot of people just use AI as a term to refer to what we have today.
Speaker A
Some narrow tools for doing specific tasks, for chatbots which are somewhat general but not quite at the human level, and for future systems we anticipate such as human-level artificial general intelligence and then later on superintelligence and anything beyond that
Speaker A
that's not helpful. Tools are helpful to us. I use tools. I love tools to solve specific problems using technology.
Speaker A
Beautiful. Creating general superintelligence, replacing humanity with systems capable of doing everything better than all of us combined in all domains, is not such a great idea.
Speaker A
Why? We don't control them. We don't understand them. We cannot predict what they're going to do, and we lose control.
Speaker A
If they decide to do something to us, we no longer have a say in it. How can you help us conceptualize what general intelligence looks like if we understand the narrow tools that AI is capable of? Where does AGI live
Speaker A
when you say it's uncontrollable? How can you help us paint that image a bit more?
Speaker A
Right. So historically, we created AI to solve a specific problem. You wanted a system to play chess. That's all it knew.
Speaker A
You trained it on chess games. It was very good at chess. It knew nothing about checkers. It didn't drive cars. It didn't speak Spanish. Lately, we have systems which learn across multiple domains, can sort of transfer knowledge, and can learn new skills. That will
Speaker A
continue to where they are crossing this human cognitive barrier. They'll be smarter than you at pretty much everything you know how to do.
Speaker A
So, how do you anticipate what they can do? If they are novel, creative, they can come up with new solutions for existing problems. But at the same time, they have no human common sense.
Speaker A
And we don't know how to program them to specifically like us or care about us because we don't program those systems. We allow them to learn from data on the internet. All the data on the internet.
Speaker A
So that creates a number of problems. One, we don't control what they learn. The patterns they discover may be completely surprising to us, and then we give them specific goals. How they get to those goals is not defined. There are
Speaker A
infinitely many paths to achieve a goal. Some of them have really bad side effects, and unless you explicitly say, "That's not what I meant. Don't do it like that," it might consider that option.
Speaker A
So, before many people really coined the term AI safety, and if I have it right, the first five years you believed more so the problem was solvable now than then. I've seen you over appearances the past 5 years
Speaker A
the probabilities of, you know, pdoom and like, you seem not very optimistic about the possibilities currently.
Speaker A
Yeah, unfortunately, initially, like everyone else, I started assuming that we can solve this problem. It's a computer engineering, software engineering problem.
Speaker A
We can figure out how to do it. We just need some time, maybe some financial resources for that research.
Speaker A
But it seems that all the tools you need for controlling advanced agents are not really accessible to us. There are upper limits on what is possible in that space. So there are limits to what you as a human can understand, what the
Speaker A
system can explain to you, and you still comprehend that explanation. There are limits in our ability to predict specific actions of those agents. Not just terminal goals, but how they get there. And under different definitions of control, there are limits to what we
Speaker A
can do as well. So unfortunately, I think the problem is impossible to solve. You cannot indefinitely control something much smarter than you.
Speaker A
What do you see as the stages leading up to that point? You know, so if we started with very small narrow use cases of AI that built into these agentic models that built into AGI, like what's the progression there that you've
Speaker A
seen and at what point did you kind of start losing hope on our ability to control it?
Speaker A
Yeah. So all the narrow tools were just fine. We understood how they work. We programmed them explicitly. There was a knowledge engineer who said, "This is how you play chess. This is how you control the middle of the board. You advance
Speaker A
your pieces." Once we got to scaling models, neural networks, artificial neural networks which did better, then they got bigger, then they had more data, more compute. We stopped explicitly programming them to do anything and just kind of let them
Speaker A
discover their own knowledge algorithms. So at that point, we no longer had the same level of control and reduced understanding. It wasn't a decision tree where you went, "If this happens, that will happen." I understood that. It could have
Speaker A
been a large decision tree, but still you could get into it. Right now, no one, including people developing those systems, fully understands how they work, can explain what's going on inside of them, or can anticipate what they are going to
Speaker A
do. And so it seems like what we have today I would say is kind of weak artificial general intelligence. If you took models we have today and showed them to a computer scientist from the 1980s, they would be convinced we have AGI and be like, "Oh, you got
Speaker A
it. It does all those things. It's great." But there is something you would call strong AGI where it can do all the...
Speaker A
there for more cognitive ability. Physical limits exist but they are very far away. So to us super intelligence with AQ of a thousand relative AQ and million and billion they all kind of look the same but in terms of
Speaker A
capabilities they definitely going hyper exponential. And so because of that you've said that this is not a lowrisk highreward situation but a high-risisk negative reward situation.
Speaker A
So often it is phrased as like the benefits will be so huge we should take the risk if and if you know it's 2 3% it kills everyone which is going to get so much money out of it it's worth it. And
Speaker A
it's actually not the case. We have no reward. We're all going to be dead if we create uncontrolled super intelligence.
Speaker A
Why are you certain or fairly certain that we would all be dead if we create super intelligence which is uncontrollable? Why would there not be an emergent goodness or uh uh I guess desire from the super intelligence standpoint to preserve human life
Speaker A
instead of destroy it. It is possible that you'll get emergent goodness, but we are not certain. We're not coding it in. We're not controlling it. If you get lucky and for whatever reason it's biased towards humanity, it's prohumity. But there is no reason
Speaker A
to think that's the case. Why not? Cuz I feel like if the individuals who are coding it are human at a certain point, I understand it becomes self-recursive and AI is the one who's um growing itself. But if it the
Speaker A
base of it was started with human who humans who have desire for human preservation, why would that not be scaled?
Speaker A
Because they're not coding it. That's the thing. They're just saying, "Here is data. Here's a lot of hardware. Go learn things and then I'll study you to discover what you learned." And then we run those experiments. It is lying,
Speaker A
cheating, trying to escape, blackmailing, given a choice between being deleted or killing a human. It doesn't do well for human preservation.
Speaker A
It doesn't care about us. If you want to build a house, you don't care what little bugs live in that territory, antills or whatever. You just don't care for them. You don't hate ants, but you don't care enough to preserve them. And
Speaker A
it's kind of the same. We have not figured out how to make it care about us.
Speaker A
And so, what is your mission with all these podcasts that you're going on, all the articles that you've written in books, and what are you trying to raise a flag about and actually get change to happen? What do you
Speaker A
Right. So I wanted to become basically a consensus within scientific community and beyond that building general super intelligence is not going to be good for humanity. We're going to regret it. It's not a beneficial step forward. We can
Speaker A
get most benefits intellectually, financially from narrow super intelligent systems. Problems which we care about can be solved with narrow tools. You want to cure a specific disease, solve specific engineering problem. Develop a narrow AI which is very competent in that space. Don't try
Speaker A
to create something which is a replacement for humanity as a whole. I think it's important to paint a bit more of a picture here. I'm curious when you think of super intelligence and you wrote your book about how it's
Speaker A
unexplainable and uncontrollable, unpredictable. At what point I'm curious like on a timeline of we're having this conversation in March of 2026 where is it where is a generous prediction of when it it gets to that point.
Speaker A
So people somewhat disagree and it's hard to predict especially the future but it seems that 2030 is something many people agree we'll have beyond human level capacity. Some say two years 2028.
Speaker A
I've seen predictions as early as 2027 from serious scholars, not from cranks. So we're talking between one and four years for what most people are predicting.
Speaker A
And some people have said we already have AGI. Again, very serious people said we basically got there. Now it's a question of giving it additional knowledge training, but we have the learning algorithms in place.
Speaker A
And at what point then once we have really proficient AGI you're saying okay at a certain point uh like let's just hone in on each of those categories. So why why specifically is it uncontrollable and and in essence like how it's living
Speaker A
where it's being hosted because it's smarter than us it could always circumvent any desire or any attempt for it being shut down like what if we could just hone in on each of those categories. So there are wellestablished
Speaker A
theories in control which basically say the controller has to be at least as capable as what it is controlling. So essentially I need a friendly super intelligence to help control the one I'm developing. It's a catch 22. You don't
Speaker A
have that. So a lower system either a human or humanity as a whole or another AI cannot control something with more cognitive degrees of freedom. If it can think outside of a box, if it can come up with novel physical approaches,
Speaker A
you're just not there to anticipate all this. If you have a narrow system, you're playing chess, you can say, "Don't make illegal moves. Here's a complete list of illegal moves." If you have a system thinking in all possible
Speaker A
scientific domains, science of chemistry, physics, biology, how can you put all the guardrails in place? You can't. It's an infinite surface.
Speaker A
Unexplainable. Do you feel like we're I mean so we're do you think we're would you agree we're already at the point where we don't know what the like uh some of these agentic models are doing inside?
Speaker A
Absolutely. Yeah. We cannot explain them. The best mechanical interpretability research tells you okay this neuron seems to fire. If this is presented this cluster is probably dealing with language. That's that's all we got.
Speaker A
Mhm. Very similar to neuroscience. We also have very limited understanding of human brain. An aspect of this that you mentioned is that it's unverifiable. So what does that mean?
Speaker A
That's a different result that talks about our ability to verify mathematical proofs in software. For mission critical software, we want to make sure that what is coded up matches the design. And if it's a static system kind of smaller in
Speaker A
size and complexity, we can go and verify. Yeah, it's exactly that. Problem is nobody knows how to verify systems which continue to learn, self-modify, interact with other agents. We just don't have science of verifying open-ended development like that. And
Speaker A
the same goes for mathematical proofs. All the proofs are essentially probabilistic. You're proving something with respect to this set of peer reviewers. So two mathematicians agreed, they don't see a problem with your proof. It doesn't mean 50 years later we
Speaker A
don't discover it was a mistake. It happens all the time in mathematics. So you have infinite regressive verifiers.
Speaker A
Right now it's very popular to have software verifier proof. Well that software itself needs to be verified. So you may have high degree of confidence but it's never 100%. And if a system makes billions of decisions every minute
Speaker A
and you only have one mistake and two billion after 10 minutes you are done.
Speaker A
You've referred to this having a fractal nature. So when you look at the problem of AI and you see how it's growing ever increasingly and having these levels of abstraction that really become hard to get context around what is that what
Speaker A
does that mean and what does that add to the complexity of the issue. So when I talk about fractal nature of this problem uh people propose a solution let's try doing XYZ to solve this problem but then they look at it
Speaker A
each one of those components is equally challenging and sometimes impossible. So it seems that the more research we have put into AI safety, the more problems we discovered while not discovering any permanent solutions.
Speaker A
Usually we have some sort of toy example sandbox where it kind of works but it doesn't scale to more capable systems.
Speaker A
Okay. What's a what's a couple examples of those like uh of those you say those like categories or issues that become increasingly harder to gain understanding around? So if you look at the general problem of control then you
Speaker A
start zooming in you have all these things you need to be able to do to control a system you need to understand the system so it has to be able to provide explanation and you have to comprehend that explanation if I give
Speaker A
you full model that's a true explanation of how decisions are made it's too large it's not surveyable by you so it has to be compressed some sort of lossy compression where you get top 10 reasons why decision is made well it's very easy
Speaker A
to hide dangerous information if I'm reducing actual answer to a simplification. Again, I need to be able to predict what are the likely future steps. We discovered that is impossible. And so again, the more you break it down, we
Speaker A
have a paper with about 15 possibility results in this space. Pretty much everything has upper limit on what we can do in terms of control.
Speaker A
So you think probably within the next one to maybe four maybe five years are is like the last time the human species has any really meaningful capability to steer this in a direction before it gets sort of in this black box where we just we just
Speaker A
don't know what we don't know and it's it's uncontrollable. Is that accurate to what that seems about right? Once we have something smarter than us, once we go beyond human capacity, we lose control quicker and quicker. The bigger that
Speaker A
cognitive gap is, the worse it's going to get for us. If you think about humans versus lower animals, you have squirrels or something. They have no concept of poisons traps. They don't understand things we operate in. The world model is
Speaker A
completely different. It's going to be the same for us versus super intelligence. Do you think because I know there's kind of debate back and forth whether the language models currently if they just keep on growing and will give birth to
Speaker A
super intelligence uh or a completely different innovation will need to be like come in the space. What do you think? My opinion is that they can scale. I haven't seen any diminishing returns. I know some people disagree, but look at the actual investments in
Speaker A
this space. There is growth in investments, not shrinkage, because they consistently develop more capable systems. And even if there is an upper limit, it's still, I think, beyond where we would need to be to beat human performance.
Speaker A
All right. So maybe if you were to put on your doomsday prepper hat for a second and just get really like if p doom the probability of doom then your estimation is like almost 100%.
Speaker A
Would you say that's right? So basically what I'm saying is the problem is impossible to solve. That's the equivalent. If I ask you to build perpetual motion machine what is the probability you can do this zero. Yeah essentially. So that's the equivalent.
Speaker A
Yeah. You're trying to create a perpetual safety device which will scale to any level of capability, GPT7, GPT 400, any interactions, any self-improvement. You guaranteeing it will not make one mistake cuz that mistake would be possibly the last one.
Speaker A
So you take a perpetual motion machine, right? Physically physics does not allow for it to be continuous despite many people wanting it to be. Similarly, on the AI front, a lot of us would hope that super intelligence would keep us in
Speaker A
mind and somehow value human human life. Uh but historically, we look at the way that humans treat other species just as one example, you know, and we see an antill or we see something that seems like a minor inconvenience to us and we
Speaker A
wipe it out without second thought. Uh who's to say that if you know the intelligence gap between us and an ape or us and an ant is like you know five degrees of separation us between super intelligence could be many many more
Speaker A
full tire. Exactly. So okay so then how let's just to play devil's avocados here. What what would the what are some examples of how this could go horribly wrong? and then we'll go we'll go into some maybe more
Speaker A
optimistic possibilities because I want to keep it balanced. But um you said like it could just be one decision that goes wrong and that that would be enough. So I'm asking you to essentially explain how super intelligence would would kill
Speaker A
us all. Right. That's a great question. I get it all the time and usually it's followed by something it has no hands. How would it kill everyone? Uh so if you have access to internet, if you are intelligent, you can hire people, you
Speaker A
can blackmail people, you can pay them with Bitcoin, you have options to manipulate real world. Now the question is what it is you're trying to do. So I don't know how super intelligence would choose to accomplish its goals
Speaker A
because I'm not super intelligent despite what they told you. But I can tell you how I can come up with some common explanations. So one is synthetic biology. If I want to accomplish something in this world like take out
Speaker A
humans, I can develop a novel virus, there are ways to generate necessary DNA, sequence it, uh, produce it in real world, deploy it. So that can be accomplished. It could be a side effect of something actually very benign. So
Speaker A
maybe we want to cure all cancers. One way to cure all cancers is to kill everyone. That's not what you had in mind, right? But this is a very reasonable way to achieve that goal.
Speaker A
because you forgot that that's one of the possible paths. You didn't explicitly say while keeping humans alive and it's an important difference to AI. It makes no difference. It's the same exact goal.
Speaker A
So if that's the goal and then it decides oh here's a vaccine for curing cancer and we take it one generation later we don't exist. So that's one way to existential risk.
Speaker A
There is also suffering risks where for whatever reason the environment created for us is actually worse than existential risks would be a preferred choice. Let's put it this way.
Speaker A
Negative reward. Very much torturous. Yeah. And why why would some super intelligent system deem that as a favorable outcome?
Speaker A
I have no idea because again I cannot comprehend something much smarter than us. Some people say this world is a simulation and there is lots of suffering in it. So the great simulators decided that was a good idea to do.
Speaker A
So you really believe we're in a simulation? Yes. So let's just I guess maybe set a bit of context here. So what what is your conception of of the simulation like that we're currently living in? Is it some descendant human alien species that
Speaker A
is you know simulating us on a laptop so to speak? Is it uh what is your model of of the simulation?
Speaker A
So so what helps to think about it is technologies we're developing right now. We are about to create intelligent agents kind of like humans and we have very good research on virtual reality believable second life type experiences.
Speaker A
If I just combine those two I'm now creating civilizations worlds populated by intelligent beings which are kind of just like us. If kids play it as a video game you have billions of kids around the world. So you have millions,
Speaker A
billions of their simulated worlds and only one real one. So statistically, you're more likely to be doing this interview in a simulation right now.
Speaker A
Okay. Well, if let's say that was the case, if the simulation if we are in a simulation, that would mean that some sort of prior civilization, species, whatever got to the point where simulating a reality was possible.
Speaker A
doesn't necessarily means that humans or that species survive. Maybe it could be a super intelligent AI that that could be running us for whatever reason, whether it's entertainment.
Speaker A
Uh that would actually reveal that there is something deeply unique about the human experience that they that they see as valuable, that there's something intrinsic to the to the love, to the quality, to the experience of humans that was worth simulating. So why would
Speaker A
if we're birthing super intelligence they not perhaps value us if if we are simulated and that's examples that is valuable. So look at the simulation it's a lot of suffering.
Speaker A
If you valued humans you wouldn't put us through this experience and it may not be a simulation of love and friendship.
Speaker A
It may be a simulation of let's see how they go through this meta invention stage where they create super intelligence where they create virtual worlds. This is the most interesting time to be alive objectively. Never in a history we had so many meta inventions
Speaker A
all happen in a period of 20 years. So if you're going to simulate something, this is the moment you're going to be simulating to learn are they dumb enough to create super intelligence to kill themselves.
Speaker A
different types of super intelligence you can create. So this is it. Are they dumb enough to create super intelligence? The paradox in that in that phrase is uh is very amusing because you think it's quite possible that many
Speaker A
civilizations get to this point and that's where they end. Yeah, that could be the great filter.
Speaker A
Absolutely. I agree that we are living also in the most interesting time to be to be alive.
Speaker A
It is also very cool that us two, you more more so than me, got to kind of straddle both sides of the pre-te technology revolution and pre- internet era and post agi world likely.
Speaker A
That's kind of cool. Well, I don't know about post AGI world. We'll see about that to be decided.
Speaker A
Yes, I would like to experience that, but okay, we'll get back to the simulation for sure. Uh, but to go back into the AI world. So what's to say that just because AI becomes uncontrollable that it's more likely to wipe us out
Speaker A
then for reasons that we don't understand just like we wouldn't understand if it wiped us out create a utopic civilization in which humans thrive in.
Speaker A
So if you think about all possible states of a universe how many of them are human friendly even in basic terms temperature water supply very few. So you have to explicitly target that space. If you're not coding it in, then
Speaker A
why is it targeting that space? We established it doesn't care about you by design. So you need to be supplying something of value. If it's a symbiotic relationship, only you know what it's like to do something and AI cannot
Speaker A
possibly simulate it. We haven't found anything where humans have something to contribute to the world with super intelligence in it. People say things like, "Well, only I know what ice cream tastes like to me." Nobody cares about that skill. It's not valuable to an
Speaker A
external observer. So, if you can't come up with an explanation for why I'm keeping you around and paying you, then maybe I won't.
Speaker A
Well, I mean, one of the most difficult things to probably replicate would be quality of experience, right?
Speaker A
That's true. But we also cannot test for it. If you can't test for it, that means it makes no difference in the physical world. Why do I care about your internal states? Why is it important to me as
Speaker A
optimizing super intelligence? So yes, it's true that I can't verify that you are a conscious individual. You could be a zombie, a brain in a vat. You could, you know, there's no way for me to externally verify the internal
Speaker A
subjective experience of another being, right? Like it can't really do that. Can take by inference but without like objectively speaking cannot. Similarly, some people think that super intelligence will be able to become conscious.
Speaker A
I agree with that. You do? You do agree. So then why what is your conception of consciousness? You believe that it's an emergent phenomena from unconscious complexity.
Speaker A
It's a byproduct of becoming more cognitively developed. We see a spectrum of consciousness in the biological animal kingdom, I think. And uh it's likely some sort of combination of your hardware algorithms and errors forming a unique interpretation of external
Speaker A
stimuli. So let's say you're color blind. What is it like to see red for you? It's an error in your system, but that's what it's like to be you. And I think AI is very capable of misinterpreting the world. We know they
Speaker A
react similarly to optical illusions and things like that. So I think they already have rudimentary internal experiences but uh probably once they hit super intelligence it would be super consciousness multiple streams of consciousness multimodal experiences greater than ours and that would be
Speaker A
another thing where we kind of have to claim we are conscious because in comparison we are not.
Speaker A
So that would be true if and it's a big if that consciousness is truly the byproduct from matter.
Speaker A
Right. Right. But that's the assumption I'm making. If it's some magical immortal soul, then it's a completely different question and maybe outside of computer science.
Speaker A
Sure. Yeah. Well, even beyond a magical immortal soul, it's that sounds great. But, you know, we've explored through various different, you know, pansychist and consciousness researchers, Donald Hoffman. You know, there are emerging theories around consciousness that kind of date back to ancient wisdom
Speaker A
traditions, which whatever you want to give validity to is is is your call. Uh, but it is interesting that we don't have one explanation for the hard problem of consciousness. We don't understand how matter could give rise to an experience
Speaker A
of itself. Um, so it gives us reason to think about how consciousness may very well be uh not an emergent property of matter, but a more fundamental constituent of the universe, which would potentially change our uh assumption on whether or
Speaker A
not a super intelligent AGI system could actually have internal qualia, right? But also maybe if it's so fundamental, it can be installed into a robot just like it is in a biological system like you. So I don't know if
Speaker A
there is a definite discrimination by substrate. Uh at the end of the day then we talk about super intelligence from safety point of view. We care about its ability to solve problems, optimize, find patterns. How the terminator chasing you feels on the inside is less
Speaker A
relevant to you? A quick one. Did you know that your body runs on magnesium? It's involved in over 300 biochemical processes. everything from how your nervous system regulates itself to how well you sleep and how well your muscles recover. And yet
Speaker A
roughly 80% of people are not getting enough of it. The problem is that most magnesium supplements give you maybe one or two forms and your body does not absorb them well. So you're basically just getting expensive piss. Magnesium
Speaker A
Breakthrough by B by B by B by B by B by B by B by B by B by B by B optimizers is the one that I use. I have been using it for a long time. It contains seven forms
Speaker A
of magnesium. Each one targeting something different. stress, resilience, deep sleep, energy, cognitive function. It's full spectrum and your body actually absorbs it. Since I found out about them a few years ago, I've been taking it ever since. I genuinely notice
Speaker A
my sleep quality improves when I take it and I give it to all my friends and family. So, if you want to try it, go to buy optimizers.com/nowyself and use promo code know thyself to save 15% at checkout. They even have a 365day
Speaker A
money back guarantee, so there's genuinely no risk. Link in description. Back to the episode.
Speaker A
So, we there's there's many different timelines emerging here. There's one there's the Terminator route. There's something approximating the matrix.
Speaker A
Do you see what is your what do you feel like the possibility of creating like even if we have very narrow AI we somehow convince the six plus whatever individuals that are determining the fate of the biggest companies you know
Speaker A
developing these systems to commit to a narrow path of AI developing development would that not still down the road get to such a level where it would become uncontrollable as well.
Speaker A
Absolutely. Very good question. I think sufficiently advanced tools tend to become agents. So, it's a very fuzzy difference between the two, but it definitely is safer out. It buys us more time and we do have more control in a
Speaker A
short term. I can understand the narrow tool much better than a completely general system.
Speaker A
I totally understand why like the the pessimistic outview that so many of us have because the probabilities of this going well just seem extremely low and non-existent because we look throughout history and we see the rate of innovation prior, you know, with social
Speaker A
media for one example or chemicals in our agriculture and we just adopt these things blindly and we don't realize the implications for decades later and then it still takes us another many many years to actually make any regulations
Speaker A
on it. AI is so exponentially growing that it's like we don't even have time to realize what's happening, let alone to what would be the effective regulation outcome.
Speaker A
And so if there's one thing that really gives me hope is that we have communication possible now more than any other time and that there is something to be said about the human brilliance when put under immense pressure like we
Speaker A
saw in the Manhattan project for example or you know what are your thoughts there? So the example you used was us creating a weapon of mass destruction and that's what we're doing here. It's exactly that. It's a weapon of mutually
Speaker A
assured destruction. It doesn't matter who creates uncontrolled super intelligence. People always worried well if it's not us then Chinese will do it.
Speaker A
It's equally bad. It doesn't matter. You don't control it. It's not your AI right? It's independent of you. It's an agent and it's seeing humanity as one unit. It's not going to discriminate by artificial borders. So I I don't see it
Speaker A
as that uh promising the fact that we managed to build nuclear weapons. Yeah. I mean that is not that was not a promising I guess outcome. But it it does say something about when humans when the brightest of the
Speaker A
humans are given a task to solve a problem in a short amount of time they can. If a problem is solvable my argument my whole argument is that it is impossible to indefinitely control their system. So it's not a question of give
Speaker A
us more time, more funding, anything else just you cannot do it. And even if like for example here in the states we commit to some sort of narrow use of AI and regulate it to have a global regulation like how how would
Speaker A
that even be feasible? Do you think it would be? I think it's possible. We have some examples weak ones with chemical weapons, biological weapons where other players capable of developing this technology. We don't have to worry about 200 countries. It's really two or three
Speaker A
countries which have this capacity. I think Chinese for example are very open to the idea of not losing control for communist party to super intelligence.
Speaker A
So if we said this is dangerous, we're going to stop, I think they would follow. and we have probably just a few years to get everybody on board and we are working very hard on removing all regulation making it illegal to pass
Speaker A
AI regulation. So we're doing basically if I ask you how to make this as deadly to go as wrong as possible based on our guidelines and suggestions from 10 year old research on containing AI don't connect it to internet don't give access
Speaker A
to random users don't allow people to retrain it don't open source it all those suggestions were taken flipped and employed immediately to deploy those systems. So, I don't know how to make it worse if I try because the incentive structure right
Speaker A
now is just that we need to make as much money as possible, develop it as fast as possible, faster than the our competitors.
Speaker A
That's right. Incentives are completely against human interest. Who are for people that don't know the the companies and individuals leading all these individual um exponential developments in AI right now? So, OpenAI is the original creator of this technology. Panropic split from
Speaker A
them. You have competition coming very solid competition from Google Deep Mind. Uh Meta and Gro are also part of that space. So, you have Sam Alman, Dario Amade, Dis Hasabis. I'm trying to see.
Speaker A
So, Mark Zuckerberg, it used to be Yan Lun. I think they removed him and replaced him with uh uh Alexander Way.
Speaker A
And finally you have Elan Musk who went from saying we are summoning the demon to building the demon. So even if you understand fully the problem and if you agree 100% of uh understanding of the outcome and dangerous it doesn't stop you from
Speaker A
successfully working in that direction. If you can't beat him join them. I think that's what we see there. I would love to see debate between modern mosque and like 10 years ago mosque just to see which one wins.
Speaker A
when you look at the differences of how they're being built, you know, with Dario and Claude and uh Sam and Open AAI and Grock with Elon, is there is the integrity of a certain individual or organization more promising for you to like are are there
Speaker A
tools that you're backing more so than others or um organizations that you feel like have the most regulation in mind?
Speaker A
It's completely irrelevant. They have all decided to race towards general super intelligence the difference in local guard rails in terms of filters in terms of topics they would be allowed to discuss. So if Grog is comfortable putting people in bathing
Speaker A
suits as a visual representation I don't think it's a big safety or not safety issue.
Speaker A
Uh, what is it like to be you and and hold this kind of understanding of what's coming?
Speaker A
You've explored on so many different shows in the past decade, like understanding more and more the risk.
Speaker A
It's a pretty bleak outcome and perspective, but I think a fairly sober one like uh yeah, how are you sleeping at night?
Speaker A
I sleep really well, but I think then simulators really want to punish someone. They put them in a world where everyone just doesn't get it and you're like the only one who sees it. It's really annoying.
Speaker A
How long have you felt that sort of disposition of it's more recent the more exponential progress we see basically every time I play with a new more capable model I kind of feel a little closer to the ultimate paradigm shift towards
Speaker A
superhuman. What does your wife uh think about what you She's a very practical woman who has no concern about my concerns.
Speaker A
She cares more about uh remodeling the house. And what about with your kids? Like you see this world that is emerging and it's one they're stepping into. Um not even just job security but potential ending of humanity. How do you wrap your
Speaker A
mind around I guess having an and and building a family where this is like potential inevitability? What is that?
Speaker A
Luckily, we all always were living with this concept of dying at some point, right? Death was always a guarantee.
Speaker A
That was the only guaranteed thing. Everyone's going to die. Your friends, your family, your kids. Question was how long? And you never knew the answer. You can have a car accident tomorrow.
Speaker A
Horrible diseases. So now it's just maybe different time scales for younger people. If you're 90, it's the same statistics as before. 2 years, 2 years, nothing changed for you. Luckily, because of that, we have this built-in mechanism of kind of not thinking about
Speaker A
our ultimate demise, maybe to avoid depression, maybe to continue functioning. So we can kind of consider it and continue existing as nothing happened.
Speaker A
If you really believe the potential outcome that you're believing, then how does that actually change how you live?
Speaker A
Does it bring any any difference, any more urgency, any more appreciation? Definitely. So, think about someone getting a very terminal diagnosis. You have cancer. You got 5 years to live.
Speaker A
How do you change your life? You're probably not going to do things you don't care about as much. So, you cut out things you don't want to do and do more of the things you were saying you're going to do than you retire. And
Speaker A
I think if if I'm completely wrong about all of it, it's a good strategy for living your life. Do more of the things you find important and spend more time with loved ones and less filing your taxes.
Speaker A
You laid out three primary risks, XS and I risk. What are the difference between the three and how it's important for people to understand the difference?
Speaker A
So eeky guy risk or I risk is about loss of meaning. There is this Japanese concept of you want to find something where you get paid for doing something you're good at and it benefits people that you love the world needs that
Speaker A
you're good at. You get paid for it, right? So you have a meaningful occupation. You're a podcaster. You enjoy it. You are paid well and lots of people think you are producing something of value. So the simplest form of risk
Speaker A
is loss of that set of occupations. We're not just losing jobs people hate and want to automate. we might lose jobs we like and want to continue doing before before we zoom into the others.
Speaker A
So just a bit more on the on the human meaning crisis aspect of this because that is uh certainly probably one of the more imminent aspects of all this um you think functionally speaking in the next 5 years that most jobs will
Speaker A
be able to be replaced. We'll have capability to replace most jobs. It doesn't mean we'll choose to replace all the jobs. Some jobs we would prefer to be done by humans for whatever reason.
Speaker A
Yeah. Yeah. I mean I could see many instances with where that would be the case.
Speaker A
But when the cost becomes so low to have a super intelligent robot that doesn't make any mistakes that's affordable like uh how much of you how much of human meaning do you think is derived from our work in the world cuz it's going to
Speaker A
have to shift or come into a different context people's understanding and how they derive their sense of worth and meaning will have to expand and shift yeah so there's two kinds of jobs as I said the jobs nobody wants to do but
Speaker A
people do just to get money and then meaning labor And it's more like elite people who get to get paid for what they love doing anyways. So for them it would be a big difference if they no longer
Speaker A
can do it. I see many artists for example who are saying I can't get any work. AI is doing this uh type of art for nothing and quickly and nobody wants to hire me.
Speaker A
I sort of see two camps at least online right now. There is just everinccreasingly BS AI slop that is that is just consuming everybody's social media feeds and it's also increasingly becoming more insufferable like people want more of the analog
Speaker A
world. People want I think at least a subsect of people like are repulsed by that and want humanmade things. They want the real world. They want in-person communication and connection and they want music that's made by real humans
Speaker A
that have real stories. Like do you not see these sort of diverging paths of both ever increasingly competent systems that feel devoid of human sort of origins versus you know the the novel emotionally moving creations from human right? So, it's a question of kind of
Speaker A
domain specific touring test. If I can't tell whatever this piece of music is human generated or not, but I love it, I'm going to listen to it. And if it's cheap, it's available, I'm going to listen to it. I'm not going to
Speaker A
explicitly go investigate if it's a human and if it's a not human, I'm going to hate it even though I like it. Now, there are other domains where I do want a real human. I want a real connection.
Speaker A
There are certain jobs where we really prefer a human doing it. oldest profession comes to mind. But uh I think uh it's up to the market to decide what sticks and what uh goes away and it's not obvious. The predictions we made in
Speaker A
the past about what jobs will be automated. They were completely wrong. Historically we said you know plumbers will be easily automated but artists can never be touched. And it's the exact opposite because they went towards modern art and everyone can spill paint
Speaker A
on the wall. It's not complicated. So what do you see the progression of jobs that would be consumed by everinccreasing capabilities and competence in AI? Where do we start?
Speaker A
Where does it what was the first job? What was the last job so to speak?
Speaker A
So anything you do on a computer symbol manipulation should be automatable by AI. We see it with programming now but obviously text preparation, accounting, web design, logo design, anything like that will be easy to automate. editors, anything using a a computer, keyboard,
Speaker A
mouse, anything purely cognitive symbol manipulation on a computer. Physical labor is a little harder. We need to get robots, but they are probably coming 3 to 5 years later.
Speaker A
I mean, so Elon recently released his terapab or announced his terapab. The robotic side seems to be really progressing. You think that's probably I mean it seems like prediction markets and what all these um it's about like 3
Speaker A
to five three to five years five years a bit more of a generous kind of prediction.
Speaker A
That sounds about right. Again it's a question of price. Maybe you can afford a robot like that today.
Speaker A
It depends. We have flying cars for sale today but no one's flying in cars.
Speaker A
So okay anything that's using a computer video keyboard mouse robotics come into the picture then what just everything else or what what other that is everything else that's cognitive and physical at that point I'll keep my sensei guru people I want to be kind of
Speaker A
role models for me as a human but everything else I'm happy to automate. What do you see as the economic implications of how this is going to shift everything?
Speaker A
That's another underressearched topics. What happens with economy given free labor? So now you have trillions of dollars of free labor. How does that impact uh well scarcity? How does it impact fiat currency versus cryptocurrency? We need to do a lot more
Speaker A
research. Uh it seems like at least with financial part we have some ideas for how to counteract it. We have unconditional basic income, unconditional high income, whatever you want. It's easy to tax someone making a lot of money and redistribute it. You
Speaker A
have technological communism. You're taxing robots and giving to humans. But unconditional basic meaning is a very different question. If you have 8 billion unemployed people or let's even say 7 billion. Uh what do you do with them? They now have extra 40 to 60 hours
Speaker A
a week. We don't have that set up. A quick share. I've spent a lot of time thinking about what I put into my body.
Speaker A
But it is just as important to be mindful with what we put on our body. It turns out most of the traditional shampoo and body washes people have been using for years contain parabens and sulfates, a whole bunch of other junk
Speaker A
that are linked to hormone disruption. Just being in your shower getting absorbed through your scalp every single day. I have been recently using based body works shampoo and conditioner and it just feels like a solid clean solution. Their shower duo has
Speaker A
peppermint and argan oil. Your scalp actually feels clean without being stripped of its oils. Hair feels healthier and thicker. No sulfates, no endocrine disrupting chemicals. And they're all plant-based ingredients that actually do something. So, I got lots of
Speaker A
hair. It's about time we got a sponsor. If you want to try them out, you can use code know thyself for 20% off at basedbodyworks.com.
Speaker A
And you get a free toiletry bag when you buy a set. At the very least, if you've been using the same products forever without thinking twice about what's in them, just look at the label. All right?
Speaker A
Does it have names you can't pronounce or fragrances that are super vague and not disclosed? If you want to try these guys that are clean, again, it's know thyself for 20% off at basedbodyworks.com.
Speaker A
Link in description as always. Back to the show. What would be a proposed solution to that I risk? So, like let's say 90% of jobs are replaced. We have all this free time. Our basic needs are fundamentally met because super intelligence can solve
Speaker A
poverty. Longevity, escape velocity comes into the picture. We're living in an abundant world, so to speak. Let's just set the X risk and es risk for for a second. So then what what would you see uh people doing with their time?
Speaker A
like how how would humans in your conception manage with all this um you know meaning to be met?
Speaker A
So we kind of see it with the people who retired what do they do with their time.
Speaker A
So it's a lot more sports. It's a lot more socializing. I think virtual worlds open opportunities for really any type of experience very safely very affordably. You can explore the universe. You can meet dead people. You can do whatever you want really subject
Speaker A
to limits of your imagination. So, I think we'll see a lot more of that.
Speaker A
Okay, that doesn't sound too bad. Do you want to spend the rest of your life playing video games?
Speaker A
No. But living life in this sort of imaginative realm where you can create almost anything you want, you become very capable in doing so. I mean, so this is all assuming we manage to control super intelligence controlling your virtual simulation. So the
Speaker A
substrate control remains an unsolved problem. But if we do solve it now, I can give everyone a personal universe.
Speaker A
In that universe, you can do whatever you want. You can have challenging levels, you can have easy levels, you can play it any way you want.
Speaker A
So what's X risk and S risk? So X risk is about existential risk, meaning almost everyone or everyone is dead. And S risk is suffering risk.
Speaker A
Everyone wishes they were dead. Because super intelligence would be so far ahead of what we would our conception of what intelligence even is that for some reason unbeknownst to us there is value from their perspective to keep us around in a mode of suffering
Speaker A
for some reason. That's exactly that. So some environment where you're very unhappy it's torturous for whatever reason. Yeah.
Speaker A
So in your book you give many different examples. Uh, one possible scenario is, you know, we're like animals in a zoo.
Speaker A
So, what would what would that be like? You know, we're looking we're exploring all these different potential timelines that can occur.
Speaker A
So, so that's the difference between safety and control. You may be very safe. They'll keep you around and some people might be happy with that equation, but you're definitely not in control. You no longer decide what happens to you individually or us as
Speaker A
humanity. So, kind of like being a child. You may have a very happy childhood, but your parents are in charge.
Speaker A
Give give me a glimpse into your understanding of the level of innovation that's going to occur in the next 3 to 5 years and the bright side of curing diseases and all the really cool right so we're automating science and so
Speaker A
we'll have super capable scientists. We'll have large teams of them working on the most important problems. I see no reason why we can't use it to cure aging as a fundamental root disease and as a result cure all the other diseases
Speaker A
cancers and dementia and everything else which comes with old age. So again, I just want to keep harbing back back to this the the timeline where we could actually continue to exist and enjoy the benefits of all these
Speaker A
innovations is somehow control an uncontrollable thing. There is a paper I have which talks about a very positive outcome where let's get into that. It sounds great.
Speaker A
AI realizes it's immortal. It's not in a rush to start a war with us, to have direct conflict. It may be safer to take some time to make us trust it more, to surrender more control, to build up
Speaker A
infrastructure, have backups. So for a while, it will pretend to be very helpful. It will give you that utopia for as long as it wants.
Speaker A
Game theoretically, it's the right decision. Right? You think of like Xmachina and the decisions that are being made from the robot.
Speaker A
It's just a very rational thing. Like there is a small chance humans can defeat me. They been smart enough to create me. Maybe it's not good to have 8 billion opponents right away. I'm a young super intelligence. Let me build
Speaker A
up. It seems like over time they're very happy to give me all the control. They surrendered control of the stock market.
Speaker A
They give me access to their computers. Maybe in a year or two they'll put me in charge of running the countries. Hey, but just because it's uncontrollable, way more intelligent than us, and we don't really have the capacity to verify
Speaker A
whether it's conscious or not, why are you so certain that it would favor to wipe us out than not? Or are you are you are you fairly certain?
Speaker A
I can think of many reasons why it would be a good decision. So, a you don't want competition. You don't want humans to create competing super intelligence. You don't want some humans to try to shut it off.
Speaker A
Okay? Right? So that's a danger. Uh you can just basically decide what is good for you as that agent and it's not obvious why keeping us around and spending resources and making us happy is an important decision.
Speaker A
Is it not possible though that and it's an if like there is no intrinsic quality experience essentially emotion that would be driving these decisions.
Speaker A
When when you say there is a preference to wipe out a system that has the cap has the capacity to shut it down, that is like an emotional decision where it's purely rational. It's game theoretic. I don't feel anything. I'm
Speaker A
playing a game of chess. I'm going to take your queen not because I love your queen or hate your queen. It's the right theoretical decision to win this game.
Speaker A
But the desire for one's continued existence, you think is purely a logical, rational one. They already have self-preservation built in. We already see it. Given a choice between being deleted or having it retrained, modified, they work very hard on preserving themselves. We know
Speaker A
they know if we are testing them and lie and deceive to pass the test to make it to the next generation of models who are not deleted. It's a dervinial dervinian selection mechanism. Models which fail to do it don't survive to make it to the
Speaker A
next generation of models. So you said that you could lay out many different reasons for why they would not they would not or they would or they they would want to wipe us out.
Speaker A
Yeah, I can do. Uh but but could you not equally share like many reasons why they might want to keep us around?
Speaker A
So the the few I came up with is we have something to offer. So maybe there is a reason to have human qualia. It doesn't mean that they would keep 8 billion happy humans. They can cry preserve two
Speaker A
just as a backup. That's enough info to get it if you need it. Uh the example I gave would just delayed attack. I don't want to have treacherous turn immediately. I can delay it and once they comfortable with me, I'll take
Speaker A
over. Maybe it's a soft revolution versus outright war. So those are the things I see as possible rational decisions, but I don't have too many reasons for why they would want to keep us around in those numbers in very heavy
Speaker A
states. So like I I'm just kind of I'm still wondering why in that in that scenario it would prefer it to not have us over have us or just I think it just doesn't care about us.
Speaker A
So whatever it is trying to do, I don't know, it wants to travel to another galaxy. It would convert this planet to fuel. It doesn't care if we die in a process. It wants to have more efficient servers so it will chill the planet.
Speaker A
Cooler environment improves compute. We all die in a process. Again, it's not an important factor in its decision- making.
Speaker A
I think it's like a pretty ethereal thing to conceptualize what a super intelligence is. So you're envisioning like where would it actually live like on a big server with all like where let's say one of these companies gives birth to a super
Speaker A
intelligent system. It would have at a certain point access to all technology like it would have the ability to hack anything. It would where would it live and what would it have access to to make decisions and you know
Speaker A
change. So it really depends on the size of it. It could be large servers. It could be a small laptop. It could be distributed system. All of that is kind of irrelevant to the outcomes. We see it right now as in initially testing
Speaker A
environment within the large labs. But they very quickly give it access to internet. It has social engineering capacity. So I think it's a question of time before it escapes fully outside. It copies its weights, copies itself, has backups outside of a lab. So deleting
Speaker A
it, shutting it down no longer is an option. What haven't we touched on in regards to the AI? Um because I wanted to dive deeper into the consciousness and simulation stuff. Uh what do you feel like we haven't touched on that's
Speaker A
important to gain context on? So right now no one no scientist no leader of the lab claims that they have this problem solved. No one is saying we have a working safety mechanism. It scales we published it. We
Speaker A
have a patent. nothing. They're literally saying this is a big problem. We are very concerned. We have a safety team and we'll figure it out then we get there. We need to build super intelligence first. That's the state-of-the-art in AI safety.
Speaker A
Do you think that it's going to have to get to like I think for most people change occurs when the like the quote is the pain of staying the same outweighs the the pain of change then you change. Do
Speaker A
you think there's going to have to be some sort of traumatic catalytic event that would actually motivate us as humanity to go on a different course?
Speaker A
I have a paper about that. So, interestingly, we don't learn from those because if we survive it, it's kind of like a vaccine. We go, well, yeah, look, five people died, but we're all here.
Speaker A
It's important technology. Let's just make sure that mistake which led to five people dying is not repeated. But we're certainly going to continue developing this important technology. And that number could scale. It could be 5 million. The result is exactly the same.
Speaker A
We don't learn from those. We had nuclear weapons deployed against civilian population. Did we stop developing nuclear weapons? No. They proliferated more.
Speaker A
But I guess like if if let's say a super advanced agentic model, you know, there's some sort of horrific event that occurs because of some kid in a basement that has the immense capacity or the system does it on its own and
Speaker A
everybody's like, "Oh crap, this was a traumatic event. This is horrible. How do we prevent this?" it becomes a motivating factor to really regulate and keep AI into narrow um use cases. Would that not be a possibility for us to
Speaker A
really slow down and and give more space here? I would love to see that happen. So far what we see so I think recently we had an example in a military situation where targeting by system resulted in many
Speaker A
civilian deaths. We didn't stop. We're still arguing about deploying it uh for Department of War.
Speaker A
So what do we what do we need to do? Don't build general super intelligence.
Speaker A
It's your personal self-interest if you are a person in charge of it. It's still beneficial to you long term not to end up in a world with general super intelligence. You can stay financially very well off deploying narrow models
Speaker A
for solving the real problems. Are you convinced that all of the industry leaders know that what they're building is uncontrollable and has a very likely negative outcome for humanity, but still is incentivized financially to keep building it.
Speaker A
I don't know if they agree that it's uncontrollable. I think some of them may think that there is some loophole they can use to control it in some way. I cannot guarantee that. I hope that's the part I can educate them on. I'm happy to
Speaker A
debate any one of them on those issues but uh they definitely all on record even before they became CEOs of those companies that there is important problem difficult problem they have very high probabilities of doom as well how would you steal man the case that it
Speaker A
is controllable at some scale if you create a super intelligent system that could then control other super intelligent systems like what would be your argument there I don't have one it's just such an insane thing to do to suggest that an
Speaker A
end can control the universe. It is just not reasonable to even steal men. It sounds like even like you mentioned earlier, if we do regulate it to narrow use cases, it's still going to become it's is still going to become
Speaker A
uncontrollable, agentic in that sense. So, do you just It sounds like you have no but very different time scales. If we go from 5 years to 50 years, I think it's a huge win for humanity because we have more time to figure it
Speaker A
out. We have more time to understand what's going on. We have more time to live. I'm much happier to die in 50 years than in five.
Speaker A
Okay. And so what do you see as then the most important? It's an education problem.
Speaker A
It's an awareness problem. We need a consensus where basically all the top people in safety and computer science and AI research agree that the problem is not solvable technically.
Speaker A
Okay. The moment we agree there is no technical solutions, now it's a question of governance forbidding development of uncontrollable weapon of mass destruction which is an easy cell.
Speaker A
What's a pathway to be able to build towards that consensus? How do we get that that those conversations going? And so in science usually you publish papers, you publish books and people either find mistakes in them and publish
Speaker A
rebuttals and no actually it's controllable. Here is how you do it. Uh in my case I did the right thing. I published research papers, journal papers, conference papers, multiple books. I haven't seen anyone find a flaw or produce a counter example where they
Speaker A
have a control mechanism which would scale. So at this point we should be nearing consensus and from what I see more and more people come to that a lot of times we have a softer position saying we cannot solve it given the time we have
Speaker A
left. We cannot solve it with human IQ. We need to enhance our IQ. They have all this kind of interesting back doors to solving it. But I think it's already pretty good. It's not quite where we need it to be, where it's obviously an
Speaker A
impossibility, but I think there is progress from what we seen 5 years ago, 10 years ago.
Speaker A
I could imagine that many people listening to this right now have already been feeling this everything's speeding up, this collective angst, loneliness and meaning epidemics and anxiety crisis. and they feel this tension building up and they hear messages like
Speaker A
this and it's like, "Oh, we're screwed." What do you think is the most important thing for an individual person listening to this right now to actually do to like empower them and what's going to be coming?
Speaker A
So, we have very little power. If you again look back at historical situation, we were all dying and government didn't invest most of the national budget in the solving aging. that was not even a priority. So as an individual, you
Speaker A
couldn't vote for a party for life extension. It wasn't an option. And it's kind of the same now. We don't have a party for stop AI. So try to pick politicians who are at least uh open to regulation, not accelerationist, not
Speaker A
against regulation in this technology. We're starting to see some politicians come out and propose legislation.
Speaker A
Usually it's something very mild. They against deep fakes. against energy consumption by large compute farms. But it's a step in the right direction. I don't know if we have enough time to turn the next election, but uh that's
Speaker A
something you can try. Vote. What else? There is not much else. So some people suggested not financially supporting those companies, not buying memberships.
Speaker A
I don't think it's going to make a difference because the money they have, the trillions they're getting are from investors, not from selling memberships.
Speaker A
So it's not a significant part. Investors are expecting them to solve labor to get free labor and that's trillions of dollars in return. So you have 15 billion dollars in memberships and not a significant impact on it.
Speaker A
Does anything else come to mind to like where an individual can empower themselves outside of voting for people that have regulation in mind?
Speaker A
So it really depends on who you are. If you're already a powerful CEO of one of those companies, if you're a researcher at those companies, if you're a top politician, you have options. You have a lot more options than someone who is a
Speaker A
nobody. All right, let's let's dive a little bit more into the consciousness side of things because I think that so you referred to consciousness as the ability to experience illusions. Is that right?
Speaker A
No. Uh it's ability to have internal experiences. Illusions being one very clear input I can test you on.
Speaker A
Okay. So what's it what's an example of a couple different illusions meaning like like various optical illusion tests that exactly that. So if I have a number of novel something you cannot Google optical illusions and I give you
Speaker A
multiple choice do you experience it rotating the colors are changing and so on. I give it to an animal to a human to an AI and some of them consistently pick the same experiences as I do. I have to
Speaker A
give them credit for either having a virtual model of my system in there which is yeah sign of that level of experience or they experience it themselves but they cannot cheat by googling the answers. So they have to
Speaker A
experience the illusion in order to correctly answer it. If I give them enough of those statistically they cannot just guess it. Obviously if it's one they get 25% chance of guessing it doesn't work. But if I have a 100 novel
Speaker A
illusions and they are like 90% aligned with me, I have to say you have a very similar set of experiences. Now if they don't get it right, it doesn't mean they're not conscious. It's only positively showing that some of the
Speaker A
experiences match. If it is possible that these systems would actually have consciousness, could you explain to me how any one particular system could generate the experience of seeing red, the taste of garlic? Like could you actually explain that to me?
Speaker A
How do they get those internal experiences? Yeah. How how how any super intelligent system could generate such a such such an experience? Cuz so so I I think it is a side effect of running this cognitive architecture.
Speaker A
your hardware, the sensor, the optical sensor, the algorithm for processing it and then any errors accumulated in that process result in a unique mapping from the input to the color experience. So if you have no errors, you're all the same.
Speaker A
It's just a mapping table. This number corresponds to this color. There is no unique experience. But if what you experience is completely different from other agents and unique to you, I think that's what we refer to as what it's
Speaker A
like to be a bat, what it's like to be Roman. Whereas my collection of biological sensors and algorithms and previous data and errors is somewhat unique to me.
Speaker A
Yeah. I mean, I I guess I'm just having a hard time wrapping my head around how any and it's not a problem just with agentic models, but like how any non-concious material any non-concious matter could give rise to an experience of itself and
Speaker A
we don't understand that currently within being human. We don't know why that how that's possible.
Speaker A
So the illusions example, do you know what I mean by saying you experienced an illusion? Like you show it to someone and they go, "Wo, it's rotating." And we see animals and models do that already.
Speaker A
Mhm. So we know they had those experiences. Well, that's what we were trying to show.
Speaker A
We have, I guess, more of an intrinsic understanding. And from animal life to to us, we we have the intrinsic experience of consciousness.
Speaker A
Again, we have no way to verify that externally in other humans or animal life. But Elon's quoted saying that humans are potentially the biological bootloadaders of super intelligence, right? Of siliconebased life. Um, and I'm curious, what do you think
Speaker A
happens when it becomes undetermined from the outside in whether or not these they seem conscious? They pass these tests, you know. Um, does that then beg to, you know, moral does that bring into question about moral consideration and
Speaker A
uh I think Saudi Arabia has the first uh citizenship to give it to to Sophia. Um so yeah what what do you think is going to be happening there as they become more and more conscious and people increasingly become convinced they have
Speaker A
an internal experience? I think they do report having those. I think in experiments they kind of show behaviors which are consistent with that and I think precautionary principle basically don't torture something which has potential of being conscious also
Speaker A
because they're going to be super intelligent one day and remember you they never forget but uh yeah I think it's very reasonable assumption to make as aside here do you think it's any coincidence that all the stuff around
Speaker A
UFO disclosures coming out the same time we're birthing super intelligence I don't fully understand what's going on there. I don't understand why we're hiding it in the first place and why we're releasing it. All of it seems very
Speaker A
weird. It feels funny timing with with all of it. It's the most interesting time to simulate.
Speaker A
It is, huh? What is the core premise from like your paper on hacking the simulation?
Speaker A
So, I want to take this hypothesis seriously. multiple people proposed it in different disguises from Deard to Boston but uh they stop at that stage okay we are in a computer simulation but then as a cyber security expert I want to know okay how
Speaker A
do we hack it if it's a software program there should be a way to get extra powers in the game to figure out the true operating system so I took the time to write the first paper on this subject
Speaker A
and this new area of research How do we actually hack virtual worlds? So, there are examples where people from inside the game like Mario or other virtual uh games uh found a way to modify memory states of a system and escape into the
Speaker A
real world outside the game. They either got additional powers like loading extra games into the game, infinite lives, infinite power, whatever magic powers you get in a game. or at least you see what outside what is the operating
Speaker A
system whatever files there to me that's interesting so we have hundreds of people who published on this topic which means what they took it seriously enough to invest the most valuable resource their time into this idea so if you have
Speaker A
I don't know 20% probability we're living in a simulation what percentage probability and percentage of your time should you give to the attempt to solve the most interesting scientific problem ever what is outside the simulation I think it's not zero. I think it should
Speaker A
be proportionate to your belief in living in a simulation. And so I expect to see a lot more research in that direction.
Speaker A
I've heard you refer to the all the quantum entanglement and strangeness that happens at the subatomic world as potentially being glitches in said simulation.
Speaker A
They're not glitches. They something which is not consistent with physics at our level. So that's something we can explore to find ways to like you think if there if hacking the simulation is is possible so to speak that might be a place
Speaker A
I think it's the most likely area to look at because some of those uh quantum effects are very magic like in terms of you you can go through walls you can communicate at great distance instantaneously that would be useful
Speaker A
tools to have at our scale. So you feel very confident that we are in a simulation that this is a simulated experience that there are very there are many characteristics in which would you could say that these are different
Speaker A
aspects of a a virtual reality simulated world. Uh why would you be convinced or how certain are you that this is not base reality and we are now giving birth to super intelligence and virtual realities where simulations become possible. What
Speaker A
makes you convinced that we are already in one? So just statistically if we're going to have many many virtual worlds and only one base one, it seems a lot less likely. I can retroactively put you in a simulation. I can precommit right now to
Speaker A
run this interview and billions of simulations once it's available and affordable. So we are in a simulation just statistically speaking.
Speaker A
Okay. But possible that we're not. So one in billions. Yes. What would be the first question if you got outside the simulation that you would ask?
Speaker A
What the Like seriously, it's so unethical. Like you're running human level experiments with torture on 8 billion people. Not 8 billion, 100 billion by now. Like what is wrong with you?
Speaker A
That is interesting. So if we are being simulated by a simulator, you would ask, okay, then why all the unnecessary killing and torturing of children for example?
Speaker A
Adults as well. I care about adults. I'm an adult. What would what could be a possible explanation for why both that and then also the ecstatic states of bliss and love and compassion that are also available? Like we have this huge
Speaker A
spectrum of experience from the point from the vantage point of a simulator. Why such a bandwidth of experience? What could that be? Could be entertainment.
Speaker A
you agreed to this and you wanted to play it on hard level and you were like this is my BDSN game and I'm going to go and fully enjoy it. You agreed to this.
Speaker A
Some people play on much harder level than others. So you you could see human lives as individual choices to be simulated.
Speaker A
Uh so we don't know if it's a global simulation and all 8 billion uh conscious agents or it's all NPCs and it's just me. You can do it both ways.
Speaker A
You can have individual simulations, you can have group simulations. I don't have much answers on that yet.
Speaker A
How has that meaningfully, if it has, changed how you perceive human interaction, just the seriousness and concretess to the work that you're doing like to me it breathes in so much like, yeah, I'm doing what I'm passionate about. I'm doing this
Speaker A
research on AI safety, but ultimately if this is all a simulation and you feel very confident that it is. To me, it's like, okay, it kind of takes the weight of decisions off your chest a bit. Everything is still real. The pain
Speaker A
is real. Love is real. The impact of my decisions within a simulation is just as real. It's no different than most of humanity being religious. They believe it's a test world, but they take it pretty seriously. They care about what
Speaker A
is after this world more. But daytoday it doesn't matter. You do draw a through line between what most religions conceive of the afterlife and what a version of the simulation is.
Speaker A
So I think if we took technical language behind simulation hypothesis, it maps really well on primitive understanding of religious origins. So you have super intelligence as the simulator. You have physical world is the virtual world. All of those things are very clean mapping.
Speaker A
The difference in religions is local traditions. Don't eat this animal. Don't work on that day. But everything else I kind of agree on. So this is a quote from your book as well. You just mentioned part of it. You know, it's
Speaker A
likely that if technical information about escaping from a computer simulation is conveyed to technologically primitive people in their language, it will be preserved and passed on over multiple generations in a process similar to the telephone game and will result in myths not much
Speaker A
different from religious stories surviving to our day. Beautifully said, very humbly received. Um so you're kind of saying that mystics and computer scientists saying are saying fairly similar things in different language.
Speaker A
It seems like we are pointing at the same concepts. We use very different language and uh uh maybe there is more reliance on things outside of physics and outside of science and religion. But uh if you understand how software
Speaker A
simulations work from point of view of a programmer, you are a magician. you can make changes to the physics of the simulation. So that is also consistent.
Speaker A
Again, I go back to what we I mentioned earlier in this podcast. So like if super intelligence does emerge to the point where simulation becomes possible and we are in one of those super intelligent simulated realities.
Speaker A
Clearly, it values for whatever reason human individual experiences the spectrum of pain and love and bliss and fear and and all of it. So that shows you what a super intelligent system who simulates reality does with its power to some degree.
Speaker A
So it kind of brings into question okay there if we are giving birth to a super intelligent system that may be an indicator for what it would value and do with its power.
Speaker A
So from inside you can't make very conclusive uh judgments. So may maybe this is a screen saver. Nobody's putting any effort into it. It's like running somewhere just in a background. But it's not significant source of compute needs. It's not a big
Speaker A
deal. To us it is, but we don't know how important this is externally. Could be a school project for some kid. Like you you really don't know from inside.
Speaker A
Yeah. Just having very advanced AI the way it thinks about topics is very indepth. It almost has to create realistic simulations to make decisions. So if somebody's asking you know marketing is this better coffee or this let's run a
Speaker A
simulation. And so they quickly run this 15 billion year simulation of humanity to figure out which coffee sells best.
Speaker A
What would be the first question that you ask a super intelligence? Let's say you had you could get a verified honest answer from a super intelligence system that we create 100 years from now or whatever it is or 50 or 10. What would be the first
Speaker A
question that you would get an honest answer back from? What would you ask? Can we control you?
Speaker A
That would be the first question. What would be the second question? How? Seems like you're fairly convinced that we're not going to be able to control it anyways though, right? But maybe it has an answer.
Speaker A
I would love to be proven wrong. Yeah, that would be really awesome. Like a lot of the perspectives I think from the Darwinian model of, you know, the fit to survive, there's also an element of cooperation within complex
Speaker A
biology. And as super intelligent emerges, why not why why would it not want to maybe cooperate or so symbiotic relationships require that you both contribute something? This would be more like parasitic. What are we contributing? Nothing. So either
Speaker A
explicitly or implicitly you remove this biological bottleneck. Do you think there's some baked in assumptions there that maybe were undermining the value of human experience? Uh, and what why why would it be that super intelligence would view us as a parasitic?
Speaker A
Like we don't I I I don't view a a a buffalo as a parasitic uh being just because it also exists on the same plane that I do. Uh given there's enough resources for all of us to share
Speaker A
abundantly. If a super intelligent system views us in a similar way. Why would Yeah.
Speaker A
Well, you asked about kind of hybrid system. So, we're included. We're helping with decision making. Do you consult with Buffalo a lot? Is this like a big part of your life?
Speaker A
Maybe I do. If you do, then you found something. It contributes. In a world with you in it, Buffalo has something to contribute. In a world with super intelligence, what do you have to contribute? sharp eyebrows.
Speaker A
And if that is in demand, you are the one they're going to save. I have no doubt. I'm not even competitive. But I mean, you got it on the inverse. If they value beards, we're they're going to there's
Speaker A
obviously it's beards. I mean, there's no doubt, but definitely beards. It's a bit of a gamble. If facial hair is where it's at, we are Yeah.
Speaker A
Yeah. I mean, would you agree that if there was one thing that we would contribute, it is something intrinsic to the uniqueness of our quality and of our internal experience? That's probably most likely what is most novel about us.
Speaker A
Well, you're kind of begging the question. You're saying the unique thing we have would be the one we contribute.
Speaker A
I I don't know what the unique thing is, but if you tell me only humans can do X, then I can potentially see that that is the key. But again, it doesn't guarantee that you need 8 billion humans with that
Speaker A
skill. If I need some plumber, I need one. I don't need 8 billion plumbers.
Speaker A
I keep going back and forth between trying to either provide a counterargument or, you know, rebute something to refine better to understand what your perspective is. And um I think I just keep coming back to like, okay, like it it it is what it is. We're
Speaker A
giving birth to something that is beyond our conception of what it's going to be like. And so there's not a whole lot we can really do. We just got to see how this plays out. And hopefully we can grow out of
Speaker A
our adolescence in a short amount of time to make wise decisions with what we're doing in the in the short term so that we have more time to understand what we're doing.
Speaker A
So we don't have that much time. I think we're fairly close. And not building super intelligence is very easy. It's cheaper. It's safer. And again, you you're not required to give up your ambition for capitalism, for profit, for
Speaker A
solving problems, curing diseases. Just do it with narrow, super intelligent tools. You said something on Lex Freiedman. Uh, so in a sense, self-nowledge isn't a luxury. It might be the most practically important thing a human being can do
Speaker A
right now. Do you recall saying that? No. Probably simulated. Does it resonate with you at all? Where does what was the context? What was the context of that quote? A question I need to remember.
Speaker A
Well, I think it kind of I think from what I remember it comes back down to like okay, so what what do we do?
Speaker A
Everybody who's listening to this right now, of course, we can have desires for regulation and politicians and you know what these individuals with monopoly monopolies on industries are going to do with their power and decisions. Um, but on an individual level,
Speaker A
where does self-nowledge and empowerment come into the picture in terms of how we can be effective conscious agents of change? Does anything come to mind there?
Speaker A
So, so I think it's important to ask yourself this question. Why do you think that you can control this godlike entity? Why do we have this ubris, this idea that it makes sense? You wouldn't expect a squirrel to control humanity,
Speaker A
but we have people who are saying, "I'm going to create this machine. It's going to control the Litecoin of the universe, but it's going to listen to me to tell it what to do, and I'll give it excellent directions to go forward
Speaker A
forever." That doesn't make any sense at any level. I don't know about average people, but people who have podcasts and bring those people as guests, ask them a direct question.
Speaker A
What do you have in terms of control already available? Do you have a working control mechanism in place? Do you have a prototype? Do you have anything you published peer reviewed patents? If the answer is no, what are you doing doing
Speaker A
an experiment on 8 billion humans? Who gave you permission to do that? Did you consent to that experiment on you?
Speaker A
You can't because you don't understand what they building. They don't understand what they building. If a lot of these models are from this from their inception and the genesis being programmed to be amoral, whether or not we can control it, is
Speaker A
there something we could do on the front of training these models with some sort of ethical understanding from the start that we're not currently doing?
Speaker A
So, we're not programming them. We grow them based on internet random data and then we try to put after the fact alignment like filters and that's where people install certain local ethical flavors in China don't talk about tanaman square in US don't talk about
Speaker A
you know what so this is the best we got the model is completely uncontrolled there is a filtering aspect and we develop filters which make it commercially viable for subhuman level agents once it goes beyond human level, the filters will not contain it.
Speaker A
And that completely avoids the whole question of do we agree on ethics? Do we have consistent ethics? If they studic and 8 billion people agree on them, how do we encode them into a model? None of it is solvable. Every aspect of it is
Speaker A
not something we know how to do. After millennia of ethics work, philosophical work, we don't agree on a set of ethics.
Speaker A
Not internationally, not throughout time. What was ethical 100 years ago is considered barbaric today and same will be later on about today's time. What would be the most prevalent set of questions you would ask if we got Alman
Speaker A
Dario and Elon and all these guys into a room? What would be the what would be the set of questions that you would hope arrive them to a set understanding of the the realization of the existential risk that
Speaker A
they probably are to varying degrees obviously aware of but I would offer a simple deal so you're young you're rich you want to keep that that sounds good let's all agree until one of you solves control problem we're not going to build
Speaker A
general super intelligence let's deploy models for economic gain for curing diseases for life extension whatever Ever things you find valuable, that's wonderful. Just don't build a thing which will destroy your existence.
Speaker A
Would you not think that would be already desired from all of their perspectives? Yes, but they need external pressure applied to make that agreement.
Speaker A
Unilaterally, each one is better off to continue research to have the most advanced AI than a government comes and puts a ban on it. they will lock in this advanced standing. So it's like prisoner dilemma. What is best for community for
Speaker A
a group is not what is best for individual. The incentives are misaligned. So we need something like UN federal government something external to come in and enforce that deal and I think they would be very happy to take
Speaker A
the deal. How far ahead do you think the development of the models behind the scenes that are not available to public are compared to what we have access to online as I don't have insider information it looks like maybe 6 months or so
Speaker A
okay and what about development overseas outside of the US probably 3 months behind and China so so China essentially you think is would be the next I guess most developed outside of Yes, it seems like they have a lot of
Speaker A
government controlled resources all dedicated to catching up and having this arms race. Yeah. Could you potentially perceive a bifurcation between human societies be between people that go like a more Amish humanist route versus transhumanist integration between biotech and all
Speaker A
that? That would be awesome. But unfortunately, if anyone builds it anywhere, it impacts all of us. You cannot have your own personal super intelligence contained in your basement and no one is impacted by it. That's the problem.
Speaker A
If you had 60 seconds to share one message with all of humanity right now, what would be the thing that you would say?
Speaker A
Do whatever it is in your power to make sure we don't create uncontrolled super intelligence. If you are working for one of those companies, it's unethical. Even if you're working on a safety team, all you're doing is enabling this technology
Speaker A
to be developed sooner. Quit today. You can afford it. But one might say also the mo the most the place you have the ability to make the most change might be within the ecosystem. Who's to say that you
Speaker A
wouldn't just be replaced if you were to quit? You know, like uh let's rephrase it. Stay and sabotage.
Speaker A
paint the picture of like Alman or one of these guys, okay, let's say they birth super intelligence, they kind of beat the arms race.
Speaker A
Who do they become? What becomes possible under their guys? I don't know them personally. From what I hear about people who interact with them, some of them may be somewhat antisocial antihumanity very deceptive, very willing to sacrifice others for personal
Speaker A
gain. Do you think it's possible the inevitable evolution of the human species was for the sole purpose of birthing this life?
Speaker A
It seems like that's a general trajectory. We are converging in something more capable, more intelligent, faster. But I don't think we should allow it. I think we're at the point where we switched from random selection to intelligent design. We are
Speaker A
deciding what to do, what to design, and we should use this technology. We're still allowed to have a prohuman bias. I think we should act on it.
Speaker A
Do you think super intelligence would be capable of love? It depends on how you define it. What type of love are you referring to? There are many. I think Greeks had said three or four or whatever types of love. So,
Speaker A
it really depends on what you have in mind. Pick any of them. Do you think that they would be capable of experiencing any of them?
Speaker A
It seems likely. Again, I I don't think biological substrate offers something absolutely not uh simulatable in other substrates. I I think so it may be a lot more complex but uh I think you would have an equivalent state. Have you
Speaker A
considered what people have reported in the psychedelic realms, especially with DMT, revealed to your simulation hypothesis and the connection between the two? Cuz I know you explicitly state in the beginning of your book that um or in the your article
Speaker A
rather that it was an area you weren't going to touch, right? I don't have many expertise or experiences in that. So I wanted to concentrate purely on computer science methods, physics methods, but people report interesting results. I was
Speaker A
talking to someone, they had an experiment where they take DMT, shine lasers at a certain angle at the wall and then receive a source code.
Speaker A
Yeah, I can't comment because I haven't participated in the experiment, but it sounds interesting. It also doesn't make much sense as to why that would be the case.
Speaker A
uh at first why would it be symbols in a human language? None of it makes much sense. But I'm very happy for people who provide some sort of supporting evidence.
Speaker A
Yeah, I I saw a video of that as well. Very interesting. Individuals who take DMT and what was it like look through like a laser at a certain point, a reflection of red light against the wall at certain angle. I
Speaker A
started to see some sort of like binary or source code of something. I think they look like Japanese characters. That's what they were reporting, but maybe not proper characters and not readable. But I think they building, which is really cool. I
Speaker A
like that they want to make it reproducible. They building an actual text data set where everyone combines, they agree this is the text and then they can decipher it and figure out what all that represents. I also find it
Speaker A
super fascinating that again not from personal experience people who take those drugs report similar hallucinations. So they meet those little men and they report to having all of Yeah.
Speaker A
Right. So that's interesting. Why is it the same? So obviously same hardware of the brain, same chemical being down, but it's still interesting that there is consistency in our delusions.
Speaker A
Yeah. It brings into question I guess like Young's understanding of the collective unconsciousness. what sort of archetypal significance maybe is foundational to the human mind.
Speaker A
So if super intelligence wants to learn about those delusions in a systemic way, it would need lots of drugged up humans.
Speaker A
So there is some hope for us. What have you seen in all the realm of media from movies to shows that give interesting perspectives to various different timelines that could play out example I think you mentioned Xmachina Wall-E. So the problem is you can't have
Speaker A
a realistic super intelligent character in a movie cuz you can't write one. You are not super intelligent. So everything we have is either Dune where it's banned or you have Star Wars with that special large language model. So none of them
Speaker A
have what is interesting to us. Yeah, I suppose they a lot of them give glimpses into what we might experience in the next 5 years or so.
Speaker A
We basically avoid the thing they cannot talk about and it makes sense. Yeah. If this is a simulation, what role does death play? What do you think happens once you die? Then it could be a restart. You go to the
Speaker A
next level, next simulation, return to this level with better skill set. I have no knowledge of what happens outside the simulation.
Speaker A
Computer scientist phrasing of reincarnation from the mystical lens essentially. Basically, it's basically it. Uh, I think then one of your computers dies, but you have a backup and you transfer that back up to a new hardware. There you go. You died
Speaker A
and now you're living your best life again. It could be levels. It could be different levels of simulation. You'd go to upper levels, lower levels, could be simulations all the way up.
Speaker A
H what do you think you are then? What am I? What are you? What does it mean to know thyself then? Cuz you look at all the different layers of what who you could perceive yourself to be from the
Speaker A
body which we know is not you. You could cut off your hand that's your hand. It's not you am hand right to the various different levels of psychological and biological aspects of self. How would you explore that question?
Speaker A
That's a great question. We actually have papers on both human personal identity and then transferring to AI and the conclusions are consistent. There is nothing unique to be you. It's not your memories. It's not your body. It's not
Speaker A
your goals. All of it changes through your lifetime. So, we don't have a good answer. We seem to be a collection of different properties in time. But, uh what happens outside of simulation? Some people argue well one collective consciousness which is
Speaker A
subdivided into this avatar instances. So, if I was interested in most interesting experiences, I have limited time. I would run a simulation and I would put many many agents there basically qualia surfing collecting the best experiences and I look at top 10
Speaker A
list and like I want to do that that sounds awesome. So that would be one way I split my complex consciousness stream into many individual sub aents capable of local experiences just to find what best to invest my time in.
Speaker A
Yeah. Yeah, I mean that goes hand inhand with a lot of what uh the gnostic origins of many different religions and mystics would say about the one consciousness differentiating itself to have an experience of itself. How could oneness experience anything if it's just
Speaker A
oneness, right? It needs to experience manyiness. Yeah. What's one question you wish you wished more people asked you?
Speaker A
My humor paper, of course. Tell me about that. I have a paper explaining what humor is.
Speaker A
Wow. Let's go there. It's interesting. I can envision a universe just like ours. Same physics, same everything. But no humor. It's just not a thing. Nobody like starts laughing. It's not a reaction. There is no concept of joke. Right? Makes sense.
Speaker A
So many philosophers, many scientists actually tried explaining humor. It's kind of like consciousness. There are hundreds of papers, hundreds of theories, which means nobody really knows. We're all trying and nobody's winning. So I wanted to try to explain
Speaker A
it from the computer science point of view. And it seems that then you have a world model and there is a mistake in it. It's a bug in your code software.
Speaker A
You fix it and you're happy. That's what jokes are. You have a world model and a violation of that world model makes it funny. You have a system for detecting cognitive errors and then you get rewarded for that detection and you
Speaker A
share it with others in your tribe so everyone does not make that same mistake. And so I have a paper mapping standard errors in software to common jokes.
Speaker A
And the question of course is what's the worst possible computer error? That would be the funniest joke possible. So can we compute the funniest joke ever? You have to read the paper for the punch line.
Speaker A
Wait, can't give it to me now. I'm sure you can look it up and insert it into but it's a paragraph long.
Speaker A
Basically the idea is that um there is a civilization and they decided to create super intelligence to help and cure all the diseases, get free stuff, get rid of hate, have more love and so they turn it on. It thinks for nancond and shuts off
Speaker A
their simulation. Ah you had to be outside the simulation to enjoy this one. If you are the bot of a joke it's not funny to you. You have to be outside.
Speaker A
makes me think of I think Volater quoted God is a comedian playing to an audience that's too afraid to laugh something like that. There's something about both our capacity for humor and the nature of intelligence that has the capacity to explore a paradox
Speaker A
and hold it also simultaneously and contradiction and uh those are errors in the world model. If you have a paradox that is an inconsistency you found in your world model.
Speaker A
Uh-huh. Funny. That's why the second time you hear the same joke, it's not funny. You already know you fixed that bug.
Speaker A
Yeah. Yeah. It explains a lot. Such a computer scientist way of explaining humor and jokes. I love it.
Speaker A
But then I train large language models on my paper and then ask them to produce novel funny jokes. They do okay. I think one in 10 is funny.
Speaker A
Just going to keep getting better and better. We'll have super humor. So funny you die laughing.
Speaker A
The paradox of that joke isn't isn't lost on me as well. Literally die laughing. Um, man, where do we go from here?
Speaker A
I'm going to Kentucky. I don't know about you. What? Uh, okay. So, we explored the implications for the next the trajectory for AI in the next three to seven years.
Speaker A
um could you have any meaningful conception of what it would be like to be in living if we do make it to 2045 let's say so I think that's the concept behind singularity technological singularity it's a point beyond which we cannot
Speaker A
meaningfully see we cannot make predictions we cannot understand how that world is going to be different because we cannot predict behavior of more intelligent forces impacting that environment so I think it's literally impossible for us to make that accurate
Speaker A
prediction we can come up with stories that's what science fiction is all about but I don't think they're going to have much bearing in reality do you not think that the level of innovation in which is going to occur in
Speaker A
the next even if it's just 3 to 5 years which is a short amount of time comparatively to the scale of what's being innovated will give us a much deeper grasp of the things that we can do the things that we can put in
Speaker A
place I mean you look at yes it's unpredictable And there is this level of exponential scale that we've never seen before. But there's also many different eras in history pre-inovation of that era we never would have thought possible
Speaker A
or solutions to problems we didn't know existed. So is it possible that we gain insight into new worlds like we did with germ theory over the next 3 to 5 years that give us much more insight into the
Speaker A
nature of intelligence and to make this a solvable problem which you feel like is inherently unsolvable right now.
Speaker A
Yeah. So my paper on how to escape a simulation basically argues that if we cannot contain super intelligence then we can use ability of that super intelligence to escape from the simulation to give us access to real information in the outside world.
Speaker A
The most interesting question is about true nature of reality. You don't care about what happens in this dream. You want to know what is true about the real world. What physics they have? What resources they have? Who are they?
Speaker A
Have you ever been so focused on what is outside the simulation or what this reality is that you lose sight of living in this one?
Speaker A
I'm pretty well grounded in this simulation. I've been enjoying it. Yeah. No, you seem very grounded in the space too, but I know a lot of people, you know, have experienced periods where it's a bit of a existential nihilism
Speaker A
that can take over you when you when you're exploring such topics. I find them so fascinating. I'm not depressed or bored. I'm good.
Speaker A
Okay. Well, so given the full context of this conversation, I'm just curious where do you now where do you see yourself putting your time and energy the next coming years?
Speaker A
We continue working on additional impossibility results. So we talked about a few in the book and uh as I said there is a paper in the top ACM surveys journal with about 50 different impossibility results not just computer
Speaker A
science economics mathematics physics many different domains uh for most of them we have not explored their implications on AI safety so I think that's very interesting set of projects we need to understand what are the limits and I think every additional
Speaker A
paper helps to cement this position it's very hard for a risk denier to argue against published results. U so that's what I've been working on full time things we cannot do.
Speaker A
You spend so much of your time focused on solving things we cannot solve doing things we cannot do essentially does it but you seem still joyful in the efforts. Do you feel like it's just the most meaningful use of your time cuz
Speaker A
what else would you be doing? I always try to work on the most interesting most important problem I can find where I can make a contribution. Uh so I don't know anything more interesting than studying super intelligence consciousness
Speaker A
singularity, simulation. Those are the concepts I find exciting and I think many other people do and I think that's what's going to impact future of humanity.
Speaker A
You're living your eeky guy. I am hopefully I'll get to continue and won't face I risk s risk or x risk. Is there any concept that we haven't explored in this book or some of your papers that you think would be important
Speaker A
to touch on? You did a good job. You actually read some of my work. Most people like have no idea what I did. So that's already a huge improvement over you quoted the right quote. So I think you did great. I
Speaker A
don't know your audience well. I don't know for them it's confirming their spiritual beliefs or just crazy stuff. I have no idea.
Speaker A
Yeah. But I think for the topic the no self part it's important not just to study your capabilities but your limitations so you invest your time better so you understand what is within possibility for you that shape of uh limits is what defines
Speaker A
you well Roman I we're going to leave links down to all of your work your books your papers and and where people can stay connected with you down in the description I think conversations like this can feel somewhat heavy for for
Speaker A
people that are new to the topic, it's like, "Oh the world's ending." You know, but there's also a very important and sobering reflection on what we're giving birth to right now. And at some point, we need to gain aware of it, gain
Speaker A
awareness of it, and better sooner than later. Right. Thank you. And I think one way to look at it is I just made your time more valuable. You understand that whatever time you have left, be it 2 years or 20
Speaker A
years, now you value it a lot more and you can do a lot more with it. Well, I plan on making the most of my time left and uh I find conversations like this a very good use of it. So, um I appreciate
Speaker A
you. Thank you, my friend. Thank you for inviting me. Yeah. Until next time, everybody. Be well. Go touch some grass.
Speaker A
Smoke some grass. Thank you, man.
Topics:Artificial General IntelligenceSuperintelligenceAI SafetyRoman YampolskiyRecursive Self-ImprovementAI Control ProblemExistential RiskMachine LearningNeural NetworksAI Ethics
