Speaker A
[music] Welcome, everybody. Sorry for everybody who was already here and missed the Coinbase guy. I have no idea where he went or why he didn't come. I was actually pretty excited to hear about his comments. But today,
Insights on how AI startups win enterprise contracts, focusing on efficacy, security, and buyer-seller dynamics in AI native enterprises.
Key Takeaways
- Only about 5% of AI startup demos convert into signed enterprise contracts due to multiple barriers.
- Enterprise readiness goes beyond product features to include security, compliance, and operational reliability.
- Shorter pilot windows demand faster proof of value and integration from AI vendors.
- Security requirements like ZDR and customer-managed keys are non-negotiable for enterprise adoption.
- Effective communication and listening to enterprise customer needs are crucial for startup success.
What the video covers
- The speaker discusses the challenges AI startups face in securing enterprise contracts, highlighting a low conversion rate from demos to signed contracts.
- Enterprise readiness is defined by efficacy, security, reliability, and legal compliance, with many startups failing in one or more areas.
- Efficacy requires the product to solve real problems, have clear pricing models, and demonstrate integrations from day one.
- Common issues include vaporware, upside-down pricing models, unfulfilled promises, and repitching declined features.
- Pilot timelines have drastically shortened from months to weeks, reflecting faster enterprise decision-making.
- Security is a critical barrier, with requirements such as zero data retention (ZDR), customer-managed encryption keys, and BYO gateway infrastructure.
- Role-based access control (RBAC) tied to enterprise identity systems is essential for managing entitlements and permissions.
- Startups often struggle with maintaining SLAs, providing reachable support, and offering transparent status pages and roadmaps.
- Enterprises require clean architecture, good integration, strong enablement, and change management to adopt AI solutions effectively.
- The speaker emphasizes the need for a new paradigm in entitlements and experimentation ecosystems within large enterprises.
Chapters
- 00:00Introduction and Audience Overview
- 01:11Legal Disclaimer and Speaker Background
- 02:15Enterprise AI Adoption Challenges
- 03:07Demo to Contract Conversion Rates
- 04:00Enterprise Readiness: Efficacy and Commercial Issues
- 04:55Common Pitfalls in AI Startup Sales
- 06:01Pilot Program Timelines and Trends
- 06:55Security Requirements and Challenges
- 08:04Support, SLAs, and Reliability Expectations
- 11:04Enterprise Architecture and Change Management
Full Transcript — Download SRT & Markdown
Speaker A
I'm going to talk about which AI startups actually win enterprise contracts. So, to begin, I thought this was going to be a different audience. I didn't realize this was going to be mostly people on the leadership track. I thought I was
Speaker A
going to be speaking to more AI engineers. So maybe just by show of hands, how many of you represent like the engineering or startup or like seller side?
Speaker A
And then how many of you represent maybe like the buyer side? Like you're in the enterprise, you're trying to get these tools in. Okay, so we got a good mix.
Speaker A
I'm going to try to balance that out today. I work on product stuff at Millennium, which is a hedge fund. We build a lot of stuff. I can't talk about any of it. So I'm going to talk about
Speaker A
stuff that we look at and evaluate. It's going to be pretty generic. I tried to make it as interesting as possible while still getting my compliance department to be okay with me doing this.
Speaker A
But also need to say legally that I'm speaking as an individual. I am not representing my company, and all opinions are my own.
Speaker A
So with that, we can dive in. So I ran through this with my parents last week. I don't—I grew up not that far from here. And my mom basically said, "Why are you spending your time teaching vendors how to sell to you?
Speaker A
Aren't you busy enough already?" And the real answer to that question is I really like how stuff works. I like seeing stuff come together. My bachelor's degree was in economics. And I really love seeing things just work well. So AI's been really interesting
Speaker A
because it's kind of a whole new paradigm of how businesses are doing work. That's the whole point of this track, this AI native enterprise track.
Speaker A
And so even though I don't need more people DMing me on LinkedIn, I'm actually really excited to talk about this.
Speaker A
So, my hypothesis in short is basically at current model intelligence, most of the value available is already being left on the table. This is not a hot take for most people, I think, who work in enterprise. You've probably seen this
Speaker A
problem. This little stat at the bottom is pretty heavily repeated for a lot of people who work inside of business circles, and they all kind of like laugh, and they're like, "Yeah, yeah, you know, all these AI tools, how
Speaker A
much are they actually doing?" And I want to talk about why. So, there's kind of two sides to this becoming gen AI native. You have models and products, which are one side, that's the seller side, and then you also have systems and
Speaker A
all of what's inside of the enterprise, that's the buyer side. So, that's the side that I deal with a lot.
Speaker A
So, we're going to talk first about the seller side, and then we're going to talk about the buyer side. So, per pain point, a lot of my job is kind of go around the company and figure out like
Speaker A
what are the pain points? What are we trying to solve for? Can we buy it?
Speaker A
Can we build it? So, let's say for a given pain point, maybe I identify 10 to 15 startups that look really interesting. Like, "Huh, maybe these guys can solve our problem for us, we don't have to build it."
Speaker A
Of those, after doing a little bit of due diligence on my own, I might schedule two to three demo calls.
Speaker A
Of those, we probably will land zero or one pilots. And of those, probably one in four of those longer term will actually end up with a contract.
Speaker A
So, what does this mean? This means about 5% of all of our demo calls actually end up in a signed contract. And this tracks with the industry. I had no idea that this was actually a benchmark, but it turns out that
Speaker A
there's quite a bit out there that indicates that this is really similar across the board.
Speaker A
So, I want to talk about what enterprise ready actually means from the inside, because we have a lot of startups that tell me what enterprise ready means, and we go through all of our requirements, and then we have a very
Speaker A
different idea of what enterprise ready actually means. So, we're going to talk about what breaks down and why.
Speaker A
So 40% of this is efficacy, so just value, commercial issues. Then there's a lot that dies in security. There's other things that die in reliability, and then there's some stuff that dies in legal.
Speaker A
So we're going to start with the requirements that we put forward and then some of the things that we've seen go wrong across various AI companies that we work with.
Speaker A
So our requirements for efficacy, maybe unsurprisingly, the product actually needs to solve the problem.
Speaker A
That seems pretty clear, but that's not always super clear. The next one is pricing models that need to reflect real value.
Speaker A
Clear demonstration of integrations on day one, not a hypothetical. And we define the success criteria, not the vendor.
Speaker A
So things we've seen go wrong, vaporware in short. We've had a lot of startups who come in, they pitch us an idea, and it's something that our platform team can rebuild in about six weeks. So this is not a knock,
Speaker A
this is actually just what's going on in the industry everywhere, on all sides of the equation. Sometimes it's actually better for us to build, and sometimes it is still better for us to buy even if we could rebuild.
Speaker A
Upside down pricing, so this one's crazy. We had a startup just recently tell us, "Hey, we know that all of the LLM traffic that we're using for our wrapper is passing through your LLM gateway, but we want you to report your gateway
Speaker A
telemetry to us so that we can then price a huge margin on top of that, even though none of it's running through our infrastructure." That did not work.
Speaker A
Another one is promises in demo calls, but no ETAs after two months. This is pretty common. Not a lot to say here.
Speaker A
And then repitching features we've already declined. So if you're a salesperson, my best advice to you is listen to your customers. It's not novel, but it still seems to be a struggle for some.
Speaker A
It's really just better to address the things that we've asked for. So, the other thing I want to point out at the very bottom of this slide is the pilot window collapsing. So, I've been at Millennium for a little
Speaker A
over two years, and when I started, a lot of these pilot timelines that people were used to were like, "Oh, maybe we'll run a pilot for six months." And then, not that long after that, it was like, "Oh,
Speaker A
maybe we only need it for three months." And anymore, it's like, "Maybe we can do this pilot for two weeks." Because it's just accelerated so rapidly.
Speaker A
So, then moving on to security. This is a huge one. I'm not a security expert, but I do run kind of frontline defense on talking to a lot of startups about security. And so, these are a lot
Speaker A
of the things that come up over and over. ZDR. So, this is a really hot topic.
Speaker A
Obviously, a lot going on with Fable, mandatory data retention requirements, and then a whole other battleground around customer-managed encryption keys.
Speaker A
So, ZDR is always best, of course. If that's not possible, customer-managed encryption keys and, with a big parentheses, that don't break the product. There are a lot of things that people are like, "Oh, yeah, it's fine. It'll work with
Speaker A
customer-managed encryption keys." And then, it breaks the product. So, that's a big product issue that we have to work through with people.
Speaker A
Other requirements, bring your own gateway. We prefer to route all of our own traffic through our own gateway and BYO infrastructure. We would prefer to host it in our own cloud infrastructure and have something that's deployable in our systems.
Speaker A
This is another really big one. SCIM-tied RBAC. So, for all of you who get that jargon, it's really important that we can tie our AD groups or other permission and entitlement groups to role-based access control. We want to make sure that w
Speaker A
don't just turn on features for everybody across the board. A lot of people don't think about this when they're designing their systems. They're like, "Oh, this is a great feature. We should just turn it on for everybody." Um when you work at a a
Speaker A
enterprise, that's not something that people want to do. Um there are usually different groups who should have different access at different times, and most of all we want it to be configurable via API.
Speaker A
Um for smaller companies, we want to see at least one real security hire. So, this is something that's really important. We know that security is not the first thing that people hire for. Um but in the age of AI, this is a very real
Speaker A
problem, and we need to make sure that the startups we're working with actually have somebody who can understand what's going on from the security standpoint.
Speaker A
Uh so, some of the things we've seen go wrong, um outright people just sending data to their vendors, uh cloud servers, and not following any of what we've asked for. Um this has been a problem in pilots. Uh thankfully,
Speaker A
all of our pilots run non-production data. Another one, like we kind of talked about, um read write all default scopes.
Speaker A
So, there's a lot of really cool tools out there, integrations, features. They're really flashy. You can click a button, and it'll integrate with everything. And then you get a little bit deeper and find out the only way that it'll work is if you literally give
Speaker A
it read write all to everything, uh which is a huge problem. Another one, uh kind of along the same lines, all or new beta features on by default with each release. So, if you're an enterprise, you don't want everything
Speaker A
just turned on with each release. Um so, being able to control that, and then the line that we hear a lot, which is we'll get you the security architecture diagram next week.
Speaker A
Uh we do weekly check-in calls during a pilot, and then we hear this over and over. Uh it's not usually a great sign.
Speaker A
Uh the question that we often have our CISO end up asking, which is um what are you going to do if there's a breach?
Speaker A
And we get this response, well, we haven't had a breach yet. Uh with the subtext of we don't know what we would do if we did.
Speaker A
Um okay, reliability, this is another one. So, a control plane that actually works. We want to see every admin setting available via API.
Speaker A
We want to see audit logs on config changes. So, if there are five different people who are given admin access and somebody accidentally changes something or does it because uh maybe it was really late at night and maybe they had
Speaker A
too many drinks. Uh we actually want to see what happened. Uh we want to be able to control the rollout on these changes.
Speaker A
Uh we want to see real SLAs and a reachable support engineer. That goes a very, very long way.
Speaker A
So, uh things we've seen go wrong, a lot of apps that are rapidly prototyping, they're shipping so quickly that they are maybe shipping updates multiple times a day and there's a really attractive little button that says relaunch to update and it happens
Speaker A
across 3,000 people. We have no way of tracking what's going wrong. Maybe then like SSL certificates break in one of the new releases and then we have no way of tracking because everybody's on a different version um and we have no way of being able to
Speaker A
deploy at scale. Um that's really challenging. No documentation versioning. So, support articles with new terms or risks that are not actually in the legal contract but show up in the website somewhere in a random support page and then we have no way of tracking what
Speaker A
they were before versus after and it just says updated yesterday. All of these are real examples, by the way. I am not naming and shaming. Um I'm just shaming. So, uh maybe if any of you are familiar, you
Speaker A
can put it together. Um core API's down for multiple hours during a busy trading day. Uh that is a really big problem for us because we run production systems. We are trading billions of dollars.
Speaker A
Um this is a really big issue for us. And then lastly, no SLA roadmap or status page. Um the status page is a big one.
Speaker A
Okay, last, legal issues. So, we don't want anybody training on our data regardless of what type of feature or product it is.
Speaker A
Uh we also want to see a lot of transparency in the sub processors. Um any fourth-party risk becomes our risk.
Speaker A
We want to see IP indemnification with reasonable liability caps. Uh we do not control the models, so if there's output that is IP infringing, we don't want to be held liable for it.
Speaker A
So, we have seen in pilots that people claim they have ZDR, they have it legally, but then they find out or we find out later that they actually retain some of our data because they say, "Hey, we were looking at something and we
Speaker A
noticed this thing." And we're like, "How did you notice that? You weren't supposed to have this data." And they're like, "Oh, yeah, you're right." Um so, that's not great. If you say ZDR, do ZDR. Um next, every feature that is conveniently
Speaker A
beta with permissive data retention clauses. So, we've seen some vendors who they will stop releasing new features in general availability. They will only make them beta, and then the beta comes with a secret little clause that says that they're allowed to retain our data,
Speaker A
which is a very sneaky way of trying to get our data. We don't like that. Um not great.
Speaker A
Another one kind of similar is fourth-party risk that's tucked away on a random website page that's not listed in the contract. Uh this is a really big problem for us managing risk.
Speaker A
So um it was the best of times, it was the worst of times. As a recap, the best startups have security architecture that actually works, support engineers who respond, an admin API from the beginning, a 90-day plan that deploys
Speaker A
into our infrastructure and cloud, and success criteria that we write. The worst AI startups don't have any security architecture diagrams, no path to a support engineer, no deployment control or audit logs, no ETAs, and salesmanship over solid product
Speaker A
building. Um this is really just kind of a recap of like what I have been through over the last 2 years. Um I actually don't think that any of this is novel, um but it is codifying a lot of what I
Speaker A
feel like is good and best practice. Um okay, so a new frontier model comes out on average every 11 days, but your architecture might be a decade or more old. So, you've got a bunch of cool new models, there's some amazing
Speaker A
capabilities is there, and then you have profitability on the other side of it. And what's in the middle? Maybe it's your legacy architecture, probably a lot of security and privacy issues, and a lot of change management. Um ChatGPT has
Speaker A
only been out for 43 months. There are a lot of companies who are still doing an ERP migration that might have been from 5 years ago. Um so, the timelines are very asymmetric. Uh and I think that sometimes we forget about that.
Speaker A
Um okay. So, my thesis again, half or more of getting to AI native is unsexy and has absolutely nothing to do with AI.
Speaker A
Um AI models and products today can't fix your legacy architecture. Although, if any of you are startup people, that's a great one to go for.
Speaker A
Um and it also can't run your change management. These are unscientific numbers that I'm putting up here, but I hypothesize that 40% of getting to AI native is AI models and products. The other 60% is all the other stuff that no
Speaker A
one really likes talking about anymore, uh which is like data hygiene, clean architecture, having good integration, strong enablement, and change management.
Speaker A
Um I really look at AI as a flashlight, not a band-aid. Um I really think that AI shines a light on a lot of what's already working or not working. It can accelerate what's working really well, and it breaks down very quickly when
Speaker A
things don't work well. Um I don't think that it's a band-aid, and I think that for everybody who's in tech leadership, it's really important to remember that if you have issues in your technology estate, those need to be
Speaker A
addressed before trying to plug in AI and just having everything rip. Um it's it's not going to work.
Speaker A
Um so, hehehe again, maybe an unpopular message, but I really believe that we all need to start with the boring 60%. I think that's where we all need to start to get to the other side of the road.
Speaker A
So, what did we learn as we shine the flashlight internally? Again, not revealing anything super proprietary, but I do think these are big picture lessons.
Speaker A
Number one, entitlements. Entitlements need a new paradigm. Uh there are a lot of people in a lot of large enterprises who are over entitled, under entitled. The entitlements model and how it works and how it's managed, all of that breaks down when you think
Speaker A
about agents and how quickly you want agents to work and what you want them to work on and their ability to exercise judgment. Um the entire paradigm just shifts.
Speaker A
Another one is cross-platform integration moved up the stack. So, AI is only as good as what it reaches and we want it everywhere. Uh so, having things that can integrate across platforms is really important uh even more than it already was.
Speaker A
Another one is centralized knowledge. So, this is something that um Emil brought up this morning in his keynote, which is that basically we need thinner agents and a smarter substrate. Um centralized knowledge is really key to that. So, all of your documentation, all
Speaker A
your support articles, everything that's going on inside of your company that's making it work, um all of that needs to be centralized and easily consumable.
Speaker A
Even better if AI can help write that in real time in a feedback loop. Uh that's something we've been talking about with some of our vendors.
Speaker A
Another one is a separate ecosystem for experimentation. Um some companies may need to get here.
Speaker A
That gap between your legacy architecture and where you want to go might be so vast that you actually just decide, "Hey, maybe we need a separate ecosystem to do a lot of this work, figure out what does work and what
Speaker A
doesn't and then kind of go from there." Um and that's something that we thought about as well.
Speaker A
So, to just put a finer point on the agents and the entitlement thing, uh agents inherit your foundations. So, I strongly recommend that everybody fix their entitlements if they are not working really well now um because this is something that if you
Speaker A
think about the problems that you run into when things go rogue, processes go rogue, people go rogue, agents are going to like 100X that problem. Um so, this is really something that's worth figuring out now.
Speaker A
So, to kind of recap uh as I wrap up here, the recipe, if you are one of the people in the first half who are raising your hand on like, "What do I need to do if I'm a startup and I want to work with
Speaker A
a really difficult large customer? Millennium's got like 8,000 people. We have very, very tight security, compliance, regulatory requirements.
Speaker A
Um this is the stuff that we care about. And we want to see more startups doing work that allows us to work with them.
Speaker A
Um I really view this as like one of the highest bars. We're probably not the highest, um although we're probably pretty close.
Speaker A
Um and I think if you can architect your startup to work with companies like this with this kind of architecture, um you're probably going to be able to satisfy basically everybody else.
Speaker A
Um on the other side for anybody who's buying, uh these are the things that I think again that kind of that boring 60% that really deserves a lot of work. Um off entitlements, governance, audit logging, etc. Um these
Speaker A
are the things that I think we need to have in terms of systems to get it working on the other side of the equation.
Speaker A
So, that's it. Um my only motivation here is to getting stuff working better and having better enterprise grade AI.
Speaker A
Uh that's a QR code to my LinkedIn and I appreciate all of your time. Thank you.
Speaker A
[applause] [music]
Topics:AI startupsenterprise contractsenterprise readinessAI native enterprisepilot programssecurity requirementszero data retentioncustomer-managed encryption keysrole-based access controlproduct efficacy
