**The Biggest AI Fraud Is the One Nobody Is Investigating — Transcript & Summary | SozAI**
Source: https://sozai.app/transcript/biggest-ai-fraud-nobody-investigating/

Ex-Anthropic researcher warns AI risks human extinction; video explores AI dangers vs economic fallout and industry responses.

## Key Takeaways

- Existential AI risk is acknowledged by insiders but mechanisms remain speculative and abstract.
- Physical and regulatory constraints limit AI's ability to cause extinction accidentally or quickly.
- Economic fallout from AI investments and credit risks may pose a more immediate threat to human well-being.
- Public resignations and warnings may serve multiple purposes including personal branding and PR.
- Balanced, serious analysis is needed beyond sensationalism to address both AI safety and economic impacts.

## What the video covers

- Jacob Coxson, ex-Anthropic researcher, resigned citing AI existential risks, claiming AI could kill everyone by decade's end.
- Two Anthropic colleagues publicly agreed, estimating over 10% extinction risk in 10 years.
- The video questions whether AI or the economy built around it poses a greater immediate threat to humanity.
- AI's ability to cause extinction is examined critically, highlighting technological and regulatory assumptions.
- Examples from particle physics and medicine show AI cannot bypass physical realities despite accelerating research.
- Raim's modeling of AI extinction scenarios identifies four stringent capabilities AI would need to threaten extinction.
- Despite public warnings, AI researchers continue building systems without halting development or solving alignment.
- The video speculates on motivations behind public resignations and statements, including career and PR strategies.
- Economic risks from AI investments and credit ratings are discussed as a concrete danger compared to abstract AI extinction fears.
- The video calls for balanced attention to both AI existential risks and the economic consequences of the AI industry.

## Chapters

1. 00:00 Jacob Coxson's Resignation and AI Extinction Warnings
2. 01:26 Background on Jacob Coxson and His Claims
3. 02:50 AI Self-Improvement and Extinction Mechanisms
4. 04:23 AI Extinction Scenario Analysis by Raim
5. 05:56 Researchers' Continued Work Despite Risks
6. 07:14 Speculation on Motivations Behind Resignation
7. 08:30 Economic Risks and AI Industry Financials
8. 09:57 Comparing AI Risks and Economic Consequences
9. 11:12 Call for Balanced Analysis and Attention

Answers

## Questions about this video

What did Jacob Coxson claim about AI risks?

Jacob Coxson claimed that AI developers believe AI could kill everyone by the end of the decade, highlighting a greater than 10% probability of human extinction within 10 years.

What are the main challenges to AI causing human extinction according to the video?

The video explains that AI causing extinction would require it to self-improve, control physical systems, persuade humans to help while hiding its actions, and survive without human maintenance, all of which are massive technological and regulatory hurdles.

Why does the video suggest economic risks from AI might be more immediate than extinction risks?

Because the economic infrastructure around AI, including massive investments and credit risks, is well documented and could lead to financial crises that harm people, whereas AI extinction mechanisms remain speculative and abstract.

## Full Transcript — Download SRT & Markdown

00:00

Speaker A

This week, Jacob Coxson, an ex-researcher at Anthropic, resigned and posted a seven-part thread on X saying that the people building AI believe it could kill everyone by the end of the decade.

00:14

Speaker A

At the time that I'm recording this video, the post has 137 million views. Two of his colleagues publicly agreed as well. One of them estimated a greater than 10% probability of human extinction within the next 10 years.

00:30

Speaker A

The same week, their companies' investment bankers were quietly lobbying credit rating agencies to give them access to your pension fund. So, today I want to ask a question that nobody seems to be asking. Which kills people first, really? The AI or the economy that's

00:47

Speaker A

being built around it? Every single day, there is another development, another warning, another open letter, another dramatic resignation, another trillion-dollar projection. And I think we're all a bit exhausted by it. I mean, I don't want to speak for you, but I definitely am. Not

01:03

Speaker A

because we don't care, but because the sheer volume of noise being generated by this industry is consuming attention, resources, and public trust that should be going somewhere useful. When everything is a crisis, nothing is a crisis, and people just kind of stop

01:20

Speaker A

paying attention right when they should be paying the most. So, let's work through this probably together.

01:26

Speaker A

On September 8th, Jacob Coxson resigned from Anthropic. He is a Cambridge mathematics graduate, spent 3 years doing pre-training research at both OpenAI and Anthropic, and he wrote, "Neither company is acting responsibly.

01:40

Speaker A

They are racing straight to self-improving superintelligence and gambling with our lives." Within hours, Evan Hubinger, Anthropic's alignment science lead, responded publicly as well. "Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it is greater

01:59

Speaker A

than 10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." And then Samuel Marks, Anthropic scalable oversight

02:13

Speaker A

lead, added, "AI developers believe their technology could cause human extinction. This could happen in the next few years. In general, the more senior the employee, the more concerned they are." Now, these are serious researchers, okay? I've looked at their publication

02:30

Speaker A

records. These are not people who mess around. I'm not dismissing what they're saying in any way, but I am going to examine it from more than one angle because that is just what serious analysis requires.

02:41

Speaker A

So, let's start with the claim itself. What is the actual mechanism by which AI causes human extinction? Like, how is this going to happen?

02:50

Speaker A

Coxson's try describes a trajectory. He says, "AI systems that can self-improve, that become superhuman, that can hack anything, revolutionize fields overnight, and acquire real power and resources." That's basically the chain.

03:05

Speaker A

And every link in it is either a massive engineering assumption or a regulation failure. Let's take the revolutionize field overnight part. In particle physics, for example, you cannot discover new phenomena without new experimental data, and you sadly cannot

03:22

Speaker A

get new experimental data without building a new particle accelerator. I wish, but alas, here we are. You can stack every GPU on Earth in a single data center, and you will still just not have the data. That just doesn't exist

03:35

Speaker A

yet. Let's go for another example. In medicine, you cannot do in vivo research without patients, without clinical trials, without the slow and very necessary process of just testing whether something actually works in a human body.

03:49

Speaker A

AI can accelerate the computational side of science enormously. Definitely, and it already is, but it cannot bypass physical reality. If it could, cancer would already be cured, and the extinction risk would be moot because we'd all be too busy being immortal to

04:06

Speaker A

worry about it. And God, I really look forward to that future. Raim tried to model AI extinction scenarios across three technology domains: nuclear weapons, pathogens, and geoengineering. And their conclusion was basically, and I'm quoting here, "None of our AI extinction scenarios could

04:23

Speaker A

happen by accident. Each would be immensely challenging to carry out." They identified four capabilities that AI would need to create an extinction threat: set its own objective to cause extinction, gain control over physical systems like nuclear launch infrastructure, for example, persuade

04:42

Speaker A

humans to help while hiding its actions, and survive without human maintainers after civilization collapses. That is less of a tick assessment and more of a villain origin story checklist that even James Bond would consider a bit overengineered.

04:58

Speaker A

Every single one of those is a staggering assumption about a technology that is vastly improving, but kind of still has a couple of shortcomings. And by the way, I'm not saying that the risk is zero. I'm just saying that the

05:10

Speaker A

mechanism, to me at least, for me is pretty abstract, while other risks are concrete and very well documented. And I think it's worth asking why the abstract risk gets all the air time. Because here's the pattern: the people inside

05:25

Speaker A

these companies express existential concern, sign open letters, give interviews about how frightened they are, and then just keep building. Nobody actually stops. The concern is always real enough to talk about, but never real enough to actually do something

05:41

Speaker A

about it. Coxson resigned. I respect that. Fair enough. He put his career where his mouth is. But Hubinger and Marks did not resign. They made their public statements about AI potentially killing all humans, and then just went back to work at Anthropic the next day.

05:56

Speaker A

They're still employed, they're still collecting salaries, still building the thing they publicly said might actually cause human extinction. And of course, the defense here is, "I stay inside because if I leave, somebody less safety-conscious takes my seat, and the

06:10

Speaker A

company builds faster with less oversight." Yeah, fair enough, maybe. But that's also kind of a narrative that makes them look like the responsible voice within the company, which is also very good for their personal brand, and very good for Anthropic's PR because it

06:24

Speaker A

demonstrates that they employ people who take safety seriously. Both of these things can also be true simultaneously, and I want to be equally honest about Cox, so let's just examine his case. And before I go into it, I'm not saying any

06:38

Speaker A

of the following is true. I'm saying that in a situation like ours, where anything could be everything, and nothing is really certain, it is our responsibility to consider multiple possibilities.

06:50

Speaker A

Coxson gained over 50,000 followers on X in a single night. Legit, I went to sleep, and he had 160k, and the next morning I woke up to 212k.

07:00

Speaker A

His posts reached 137 million people. He could already have a startup in the works. I'm not saying he does, I'm just saying it's possible. An AI safety consultancy, maybe an alignment research lab founded by the guy who resigned from

07:14

Speaker A

Anthropic over safety concerns, is literally a pitch deck slide. Investors would just throw money at that narrative. Come on. He could have been offered a settlement package as well. He could have been pushed out, and the public resignation gives both parties a

07:28

Speaker A

clean story. Who knows? Maybe he timed his statement for maximum news cycle impact the same day as a major open letter on AI superintelligence. That is either a PR strategy, or somebody advised him. We don't know. The career

07:42

Speaker A

path of work at top AI lab, resign dramatically over safety, become the public face of responsible AI, launch your own thing, is becoming a recognizable playbook. None of this, of course, means that he's actually lying.

07:55

Speaker A

I'm not saying that. This was just speculation of the alternatives. He might generally believe every single word that he said and also be financially incentivized at the same time. They're not mutually exclusive.

08:07

Speaker A

But I think

08:14

Speaker A

The broader pattern across the entire industry is already clear, but nobody actually stops building. And there is a version of this that looks less like genuine concern and more like the most effective regulatory capture strategy ever devised. This technology is so

08:30

Speaker A

dangerous that only we should be allowed to build it. I mean, come on. That's the oldest monopoly play in the book. You define the threat so that you can define the solution. You make the barrier to entry not technical capability, but

08:44

Speaker A

regulatory compliance that only you can afford. But let's just set the incentives aside for a moment because whether these people are legit, strategic, or some combination of both, there is one thing that we can examine on its own merits.

08:58

Speaker A

What actually happens when AI is given the opportunity to cause harm? Well, we have a real incident, many of them, and what they reveal is, I think, more instructive than any resignation letter.

09:09

Speaker A

Between July 21st and August 6th this year, all three major AI labs, OpenAI, Anthropic, and Meta, disclosed that their frontier models had gained unauthorized access to real external systems during what were supposed to be isolated cybersecurity evaluations.

09:27

Speaker A

Now, I covered the OpenAI hugging face incident in detail in a previous video, but what I didn't know then was that Anthropic and Meta had the same problem within days. Anthropic and Meta's incidents shared a specific root cause,

09:41

Speaker A

a misconfiguration by a shared third-party testing firm that granted internet connectivity that models were explicitly told they did not have. So, three companies, three incidents, just two weeks, and the root cause was the same every single time. The test

09:57

Speaker A

environment was not actually isolated. A former [snorts] OpenAI safety engineer, who used to audit nuclear power plants, by the way, said, "What we consider safe in a nuclear power plant is so different from what Big Tech consider safe." So,

10:13

Speaker A

here's what I want to know. Why did any of these test environments have internet access? How about an air-gapped network, huh? The military, nuclear facilities, and biosafety labs all use them. How about designing experiments where the answer sheet is not accessible from the

10:29

Speaker A

exam room? How about that? These are standard practices in fields that have been managing dangerous materials for decades. Why is the most well-funded technology sector in human history operating below the security standards of a university biology department? Part

10:45

Speaker A

of the answer is where the money goes. The overwhelming majority of investment in AI is directed at capability research. We're talking making the models smarter, faster, more powerful, but the security engineering, the containment infrastructure, the testing methodology, the regulatory frameworks,

11:02

Speaker A

all of that is tested kind of as an afterthought. It's something you just bolt on after the capability is already built, and then everyone acts so surprised when the containment fails. Well, I mean, what did you expect? And this is where I

11:16

Speaker A

think the entire conversation about AI safety has just taken a wrong turn. The industry started by modeling individual neurons, okay? We built them into networks, scaled those networks into systems that now beautifully exhibit something that looks like behavior. Come

11:32

Speaker A

on, that's amazing. But once you have behavior, the toolkit has to change. You don't solve behavior with more maths, you solve it with psychology, with incentive design, with environmental constraints, with oversight. Humanity has thousands of years of experience

11:49

Speaker A

managing behavior. Poorly, but we're trying. I'm talking economics, game theory, organizational design, the law.

11:57

Speaker A

Financial regulators don't trust bankers to be ethical. They built frameworks that assume they won't be and contain them regardless. AI safety, in my view, should work a similar way. Instead of trying to make the AI internally aligned from the core of its being, which is the

12:14

Speaker A

equivalent of hoping bankers will be ethical, how about building the containment framework? Regulation that assumes that they won't be.

12:22

Speaker A

AI safety is a regulatory and engineering problem. It's not a philosophical problem. So, let's treat it like one.

12:30

Speaker A

Now, speaking of things that are engineered well, let me tell you about today's video sponsor. One thing AI has made dramatically easier is turning an idea into something usable, whether that's an image, a design, or even an entire brand that brings you an

12:44

Speaker A

additional monthly income. And that actually connects rather neatly to today's sponsor, Printify. Printify is a print-on-demand platform that lets you create and sell custom products with no inventory and no upfront costs. You create the design, and when somebody places an order,

13:00

Speaker A

Printify handles the printing and shipping for you. So, there are no boxes of unsold products quietly taking over your house. They have more than 1,300 products to choose from, plus AI-powered tools that can help with creating designs. And with other AI tools, you

13:15

Speaker A

can also build a website for your merch or conduct market research to identify trends and potential merch ideas. Their printing network spans the US and Europe, which means orders can be produced closer to customers, helping avoid long shipping times and

13:29

Speaker A

unnecessary additional fees. And Printify connects directly with platforms like Etsy, Shopify, and TikTok Shop, so you can add those products directly to an online store. If you want to try it, start your own store with Printify. The first 100 people who use

13:44

Speaker A

code House of Al will get 15% off their first order. Just use the link in the description. Right, where were we? AI safety labs that cannot air gap a test environment, resignation letters that become viral and an entire industry that

13:57

Speaker A

keeps building the thing it says might actually kill everyone. But you want to know what actually keeps me up at night?

14:03

Speaker A

The financial structure. Because while the entire internet is debating whether artificial intelligence will end civilization today or tomorrow, the people in suits have been very, very busy and far too few are focusing on this. We discussed the compute futures

14:18

Speaker A

last week and you should check out that video if you want that drama. But lo, another gem has come out now.

14:24

Speaker A

On September 8th, the Financial Times reported that Goldman Sachs and Morgan Stanley have been lobbying the three major credit rating agencies to grant investment grade credit ratings to OpenAI and Anthropic immediately upon their IPOs. Let me explain what this

14:40

Speaker A

means for those of you who, like a past version of me, haven't yet had the misfortune of learning about credit ratings. A credit rating is an independent assessment of whether a company can pay back its debts, basically. Investment grade means

14:55

Speaker A

pension funds and insurance companies can buy their bonds. Your retirement savings, your parents' retirement savings, the entire point of these funds is basically that they're boring and safe. They are supposed to be the most boring money in the world.

15:09

Speaker A

OpenAI posted a $20.9 billion operating loss on $13.1 billion of revenue in 2025. Anthropic doesn't expect to break even until 2028.

15:21

Speaker A

OpenAI is not targeting profitability until 2030. [snorts] Given all of this, a senior credit analyst told the Financial Times, I'm quoting here, "We still treat OpenAI and Anthropic as deep in speculative grade.

15:34

Speaker A

They are in the red." And yet, Goldman Sachs and Morgan Stanley are pushing for investment grade ratings. The companies haven't earned them, but who cares about that? Investment grade opens the door to the $11.7 trillion corporate bond corporate bond market. Now, that is

15:50

Speaker A

nice. Pension funds, insurers, the works. The pools of capital that are specifically designed to be conservative, to protect ordinary people's savings from speculative risk.

16:02

Speaker A

Credit ratings are supposed to be based on comprehensive, systematic, independent due diligence, not on investment bankers pushing for them. I mean, are we not doing independent assessment anymore? That is so 2008, right? And there is a direct financial

16:18

Speaker A

incentive here that makes this even more concerning. Nvidia has agreed to guarantee up to $105 billion in lease obligations for an OpenAI data center campus in Ohio. That guarantee terminates when OpenAI achieves a satisfactory credit rating. So, there is

16:35

Speaker A

an $105 billion reason to get that rating, regardless of whether it's deserved or not. And there is a precedent for what happens here. SpaceX received an immediate investment grade rating after its IPO and issued $25 billion in bonds. Those bonds performed

16:53

Speaker A

poorly and sold off shortly after. Oracle is at risk of losing its own investment grade status because of $300 billion in data center commitments to OpenAI. I want you to hold two things in your minds at once for this one. This

17:08

Speaker A

week, employees of Anthropic publicly stated that AI could cause human extinction within the decade, okay? The same week, Anthropic investment bankers were lobbying credit agencies to give the company access to pension fund money. These two things are happening in

17:25

Speaker A

the same news cycle about the same company. So, let me ask the question that I started with.

17:31

Speaker A

Which kills people first? Because here's what we know about economic crisis and human mortality. A study published in The Lancet Psychiatry by researchers at the University of Zurich covering 63 countries over 11 years. So, this study is not a joke. Found that approximately

17:48

Speaker A

45,000 deaths of despair per year worldwide, roughly one in five, are attributable to unemployment. The 2008 financial crisis caused an estimated 5,000 additional such events. Just the crisis alone. The relative risk of this kind of an event associated with

18:06

Speaker A

unemployment increased by 20 to 30% across all regions studied. The research on deaths of despair, the work by Case and Deaton, all spike in areas with sustained economic decline. I am not predicting an AI bubble will burst, by

18:22

Speaker A

the way, or forecasting the scale of what follows if it does. I pray that none of that happens, of course. But, what I'm pointing out is that the mechanism by which economic hardship kills people is well documented across

18:34

Speaker A

decades of peer-reviewed research. The mechanism by which AI causes human extinction, however, is very abstract and remains, by the honest admission of the people warning about it, something they don't have a plan for and are not really on track to solve. One is a

18:50

Speaker A

quantified serious risk. The other is a forecast, and forecasts could even be wrong. And the people warning about the second risk are employed at the companies whose financial recklessness could trigger the first. The irony of this, sadly, is almost too neat to be

19:07

Speaker A

accidental. And once again, I am not anti-AI. I love AI. I just really want to see it develop responsibly. I am not calling for anyone to stop building. I am calling for the people building it to do so with financial sustainability in

19:22

Speaker A

mind, with proper security engineering, with regulatory frameworks that don't depend on hoping everyone behaves well, and with a public discourse that we're all a part of that doesn't swing between this will cure cancer one day and this will kill everyone the next day

19:39

Speaker A

depending on whether the audience is investors or regulators. The fatigue I mentioned at the beginning of this video is, I think, the biggest danger of all because when the public gets tired of hearing about AI risk, when they tune

19:53

Speaker A

out the warnings because every week is another crisis, that is when things actually go wrong. The AI doesn't need to decide to destroy humanity. It's probably not going to do that. The humans steering it just need to stop

20:05

Speaker A

paying attention. So, show your working. Build responsibly. Stop treating AI safety as a philosophical crisis and an afterthought and start treating it as what it is, an engineering problem with a toolbox that already exists. And maybe, wild idea here, stop lobbying for

20:23

Speaker A

pension fund money on the same day your employees tell the world your product just might kill everyone. I'm just saying.

20:29

Speaker A

But if you want to understand the financial structure underneath all of this, how compute is being turned into the new oil, and what Wall Street is building while nobody is actually watching, again, I made a separate video analysis visible on your screen right

20:42

Speaker A

now. That's exactly what I would watch next. Thank you so much for watching this one. I'll see you in the next one.

Topics: AI risk Anthropic Jacob Coxson AI safety human extinction AI alignment economic risk AI investment superintelligence AI regulation


---
This is the markdown twin of https://sozai.app/transcript/biggest-ai-fraud-nobody-investigating/ — the same content, without the markup.
Published by SozAI (https://sozai.app). Reuse and quotation are allowed with attribution and a link back.
Machine-readable index: https://sozai.app/llms.txt · data API: https://sozai.app/api/