Skip to content
Get App

Building a Debugging Agent Harness at Uber - Kriti Dangi

Uber engineer Kriti Dangi presents Debug Assist, an agentic AI system that root-causes production bugs in under 30 minutes and proposes validated fixes.

Key Takeaways

  • Agentic AI can compress root cause analysis of production bugs from hours or days to under 30 minutes when grounded in rich logs, metadata, and traces.
  • Mixing deterministic steps with LLM reasoning, plus turn limits and evidence-based outputs, keeps the agent reliable and reduces hallucination risk.
  • Scaling an agent in a large monorepo requires curating tools and skills and running it in standardized environments.
  • Automated fix generation paired with validation and human review can dramatically shorten time to resolution for customer-impacting bugs.

What the video covers

  • Uber's app quality teams built Debug Assist, an AI agent that automates debugging of production issues found via the Wisdom in-app bug reporter and the Healthline crash/performance analytics tool.
  • Debugging is hard at Uber: about 50,000 issues are assigned to engineers yearly, roughly 90% customer-impacting, generating around 12,000 pages, and resolution takes about 28 days on average.
  • The pipeline is end-to-end: discovery via Healthline and Wisdom, auto-triage for priority and ownership, deep root cause analysis, fixing and mitigation, and validation before delivering a fix.
  • Debug Assist is built on a LangGraph pipeline mixing deterministic nodes (e.g., data gathering) with LLM nodes, and connects to internal systems through MCP servers for logs, tracing, and incident data.
  • Guardrails include capping agent turns (e.g., around 20 turns) because longer runs usually indicate the agent is stuck in a wrong direction.
  • A live demo showed a battery-drain bug root-caused to a hot-looping background code path, with an evidence timeline supporting the conclusion and a fix ready before the developer reached the office.
  • A second example involved a P1 crash on rapid push-notification taps caused by a bad rollout; the agent pinpointed the code and produced a fix, which was merged and deployed the same day after being caught at a 5% rollout.
  • To scale across a monorepo with roughly 3,000 skill plugins, the system uses curated skill selection so the agent is not overloaded with context, and runs in pooled, pre-configured environments as a single-click operation.
  • Observability via LangChain integration tracks sub-agents and every tool call, and the team aims to raise accuracy toward 100% and further improve Jira resolution and mean time to resolve.
  • The speaker addressed audience questions on reproducing issues and on whether validation can rely on unit tests versus full app builds.

Answers

Questions about this video

What problem does Debug Assist solve at Uber?

It automates debugging of production issues surfaced by Wisdom and Healthline, reducing the long manual effort of finding root causes across scattered logs, code, and configs. The speaker notes engineers spend a large share of their time on debugging and that issues can take around 28 days to resolve on average.

How does the agent avoid going down wrong paths?

The team found through trial and error that if the agent takes more than about 20 turns it is usually stuck in a bad rabbit hole, so they cap turns as a guardrail. They also ground outputs in evidence timelines to reduce hallucinations.

How is the agent architected?

It is built on a LangGraph pipeline combining deterministic data-gathering nodes with LLM reasoning nodes. It connects to internal tools through MCP servers for logs, distributed tracing, and incident data, and LangChain tracing tracks sub-agents and tool calls.

Full Transcript — Download SRT & Markdown

00:00
Speaker A
Good morning everyone. I'm Kiti Dangi from India. Vatashiva Kriti Desu. Uh a little bit about me. I'm a senior software engineer at Uber. I've worked at Uber for almost 4 years now and I work in app quality teams as well as
00:17
Speaker A
production observability space and I'll be sharing some of that today with you. So this talk is about debug assist and AI agent for debugging that we built at Uber.
00:29
Speaker A
So today we'll talk about why debugging is actually hard in reality, how does debug assist help with it. We'll also have a demo and then we'll see a quick look at the internal architecture and lastly I'll uh share a couple of
00:44
Speaker A
interesting numbers with you. All right. So what you have in front of you is Uber's inapp bug reporter. So all the Uber internal employees as well as beta testers have access to report bugs on the Uber app itself via the beta
01:01
Speaker A
version and this is what gets captured in the system. So let's say I'm using my app, the screen goes blank or maybe I'm experiencing hangs or maybe there are battery issues. I can simply just click and report a bug. And for me it looks
01:17
Speaker A
like saying oh my screen is blank. That's it. This system captures all the information alongside it. For context, this product is called wisdom internally. So along with the natural language bug description, it captures the metadata around the issue. For
01:34
Speaker A
example, what was the app that was affected? What was the app version that was affected? What was the OS that was affected? Which city is the user experiencing the issue in? Along with this, we also capture a multitude of
01:47
Speaker A
logs like network logs, analytics logs, console logs, GraphQL logs, UI state logs, and so many more.
01:56
Speaker A
And as we can see, screenshots are also captured from the device. Um, there are no security concerns here. We only capture the Uber screenshots to debug the issue later on.
02:08
Speaker A
This is another product we have at Uber. It's called Healthline. So, Healthline is essentially a app quality analytics tool. Um, similar to Wisdom, it also captures issue. However, it does that automatically. So, every time your app crashes, you experience a hang, a jitter
02:27
Speaker A
or any kind of performance degradation, Healthline is a system that kicks in the back end. So, similar to Wisdom, it also captures the metadata. For example, what was the session ID? What was the analytics ID? what was the user doing
02:41
Speaker A
before this along with all the logs that we just listed and many more. However, unlike wisdom, it captures the stack trace or the whole whole crash dump as well. So, we can say which code was actually causing this problem. Now,
02:59
Speaker A
now between Healthline and Wisdom almost 50,000 issues are assigned to engineers every year. 90% of these are customer impacting and as a result we end up sending 12,000 pagers to our engineering teams everywhere.
03:15
Speaker A
So someone has to debug these right? Someone has to fix these. Before I move on to the next slide, a quick show of hands. How many of us do not like fixing production issues?
03:32
Speaker A
Lots of hands. I' I've touched enough. It is annoying when you receive that pager at 2:00 a.m. at night and you have to sit all night and debug it and you don't even know where to start because your brain is so foggy.
03:44
Speaker A
And that's how debugging in reality is. The errors are often unclear. Um we all have received that message something went wrong and then we have to figure out what went wrong. So we have to dig through unfamiliar code logic. The
03:59
Speaker A
context is also scattered. um you have to go through code, you have to go through logs, you have to go through configs and you have to go through data and to access that data there are so many tools internally.
04:11
Speaker A
Speaking of code bases, code bases at Hub at Uber are vast. So we have multiple monor repos. We operate in a monorreo architecture style which means that many many teams are contributing to a single repo. So to figure out who
04:28
Speaker A
caused the problem is another problem in itself and even when you do there are still cross team dependencies. The code might be owned by someone else. Even if it is owned in your team, you don't know all the code written by your team. Right?
04:44
Speaker A
This means that an engineer at Uber takes almost 28 days to resolve an issue on an average and by uh and as seen in a public report done by stripe developers spend 40% of time debugging an issue and
05:02
Speaker A
this is why we developed debug assist which aims to improve the app quality and enhance the debugging experience of developers so that we can do what we do best develop.
05:16
Speaker A
This is the overall pipeline that we generally follow. It is a end-to-end agentic resolution journey. It starts by discovery. Uh discovery happens through healthline and wisdom as we just saw.
05:27
Speaker A
Next, we move to the auto triad stage. This is the stage which figures out what is the priority of this issue, how severe it is and who actually owns it, who is the on call I should be tagging
05:39
Speaker A
here. then starts the debug assist actual pipeline. It does a deep root cause analysis in under 30 minutes.
05:47
Speaker A
We'll we'll see how it does that. We'll take a deep dive here. And lastly, we aim towards a resolution. So, we don't just want to provide a big dump of data to the developers. We want to make it
05:59
Speaker A
actionable. We want to resolve the issue end to end. Now comes the fun part. This is the demo part. I hope all of you are attentive.
06:11
Speaker A
So this is the wisdom screen that we just saw earlier. And this is a real life example by the way. One day one of our Uber developers woke up and he booked a cab to office. While waiting for the Uber, he realized that his phone
06:24
Speaker A
was burning up in his pocket. So he filed a wisdom bug report. All he said was, "Uber app is responsible for nearly 30% of my battery usage. My phone is heating up. That's it." Then he happily sat in the Uber, came to
06:39
Speaker A
office, opened his laptop to look at the issue. In those 20 minutes, Debug assist had already created a fix without him clicking a single button. All the developer had to do was review it and deploy it.
06:54
Speaker A
So how does this magic happen in the background? As you can see here, the wisdom has captured the natural language description. It has captured the screenshot of what the uh developer was seeing while booking his cab ride as
07:10
Speaker A
well as uh a screenshot uploaded by developer which says that oh my battery usage is being consumed by Uber and we can clearly see it's 29%.
07:21
Speaker A
There are metadata and logs associated it uh with it as well. And this is how debugging was actually assisted. So this is the output of debug assist.
07:34
Speaker A
Um on the left side we can see that debug assist has root caused the issue to a hot looping code path which means that there was certain code path that was supposed to run in the background but it was actually being treated as a
07:48
Speaker A
foreground loop path which led to high CPU cycle usage and high battery regression. Now along with this RCA it also tells us that oh by the way the app was backgrounded for 17 minutes. It was not in the foreground and all of this rooted
08:04
Speaker A
is rooted in evidence. So on the right side we can see there is a evidence timeline. So we know that the agent is not hallucinating and we are somewhat confident in our fixes.
08:16
Speaker A
And lastly we can see the developer was pretty happy. This is another example. So there was a crash that was occurring on the Uber app due to a bad rollout. Essentially whenever someone clicks on a push notification very very quickly the app
08:33
Speaker A
was crashing. This was a high severity crash. That is why it is marked as a P1 which is priority one and debug assist created a whole deep uh deep RCA for it.
08:44
Speaker A
It pinpointed the code which was causing this and it also created a fix for it.
08:49
Speaker A
The fix was sent to developer via a Slack message later on and in this case the fix was merged and deployed the same day. So we ended up catching the crash at a 5% roll out and fixing it at the
09:03
Speaker A
very same day. Now let's take a look at what debug assist actually does. How does it actually accomplish this? So as you can see the agent is built on a langraph pipeline. It is a mix of deterministic and LLM nodes. Why lang chain you might
09:23
Speaker A
ask? So agents generally have a planning phase and an execution phase. What lang graph or lang chain allows us to do is bypass that planning phase. We give the plan to the agent that these are the steps you have to follow every time. So
09:41
Speaker A
it removes certain levels of hallucination from the agent. We are not totally dependent on the LLM to come up with a good plan to fix the issue. We control the plan. So it starts with a context collector node. This node
09:56
Speaker A
essentially captures all the data that we just talked about from healthline, wisdom, all the logs etc. And this is a deterministic node. It's not an LLM node. Why not LLM in this world of AI?
10:08
Speaker A
Because everything doesn't have to be via agents. It's simply fetching data. um we don't want to overload the agent with a lot of context bloating. We don't want to add latencies to the agent of the APIs that we already know of. And of
10:22
Speaker A
course, LLM cost is a huge problem these days and we don't want to bloat that as well. Um also this allows us to parse and pre-process and prune those logs. So these logs that I'm talking about are huge. They are megabytes and megabytes
10:38
Speaker A
and gigabytes of logs. You cannot simply dump them to an agent. you have to pre-process them in some ways.
10:45
Speaker A
Next, we move to the classification or the RCA node. This is a LLM node. As you can see, it's highlighted in blue.
10:52
Speaker A
Now, this node looks at all the data we just fed it. It first tries to categorize the crash. Is this even happening because of an Uber issue or is this a third party library code? Is this an infrastructure issue? Is this a
11:07
Speaker A
network issue? Do I need to create a PR to fix this? Once the categorization is done, it comes up with a confidence score as we can see. And finally, if it feels I need to get more data, it goes and queries
11:23
Speaker A
Uber ecosystem. So, it does that via MCP connections like Jerger logging, um, Jerger distributed tracing, logging, uh, incident data, is there any ongoing incident that's generally impacting the users etc.
11:37
Speaker A
Once that is done, it performs the RCA via sub aents. So we fan out multiple sub aents in parallel and let them do a RCA and once we have that we consolidate the results. To give you example of some
11:51
Speaker A
of the sub aents um one sub agent is breadcrumb analysis. So we give all the session data to the agent and we ask it to recreate the timeline of the user.
12:01
Speaker A
This not only helps with a good RCA but it also helps us reproduce the issue.
12:07
Speaker A
Another agent um I can share with you is crash correlation agent. So whenever a crash is occurring can I correlate it to a particular version release? Can I confidently say that this version is experiencing the crash?
12:23
Speaker A
Similarly the next step is a fix node. Now before I move to the fixing part um you might see there are certain configs given in every node and RCA node actually runs on a lower model sonnet with limited number of turns. You might
12:38
Speaker A
think that this is a complex node but it's really not. If you think about it it's mostly looking at the data and gathering information and summarizing it. And that's why u it runs on sonnet with limited number of turns because uh
12:52
Speaker A
with trial and error we figured out that if it's taking more than 20 turns it's generally going in the wrong direction in a bad rabbit hole and that's how you also put guard rails around your agent.
13:03
Speaker A
Let's move to the next step which is the fixing node. Now this is the node which which needs the big brains. So it goes through the code base of Uber and Uber codebase is billions and billions of lines of code. Now it tries to figure
13:17
Speaker A
out which module might be affected, which code path would be affected and all the states are passed between each node. So it has the state of the previous node as well. It has not forgotten anything.
13:28
Speaker A
Before fixing, we first try to mitigate the issue too. So as you can see most of the Uber issues are behind a feature flag and this node connects to a feature flag MCP. It tries to correlate is this
13:42
Speaker A
feature flag causing this issue and um mind you in all of your phones that you're using Uber app there are at least tens of thousands of feature flag that are active in the background.
13:55
Speaker A
So just correlating that one crash to one of those feature flags is already a very complicated task.
14:03
Speaker A
If we have a very strong correlation, it simply goes and rolls the feature flag back and the issue is mitigated and then you're um free to fix it on your own time.
14:13
Speaker A
So we have the fixing and the mitigation step here which runs on opus with a higher number of max turns. Next we go to the validation stage. It's actually called basil test for legacy reasons. Um but it's actually a validation stage. So
14:27
Speaker A
in this case the agent tries to reproduce the issue either via a unit test or a mobile automated end toend test. Some flows at Uber have a automation test that runs on simulators or emulators directly especially for the
14:40
Speaker A
core flows like booking a ride. And this is the node that tries to validate its own fix. So you not only just give the PR to a developer and say test it, you also validate it and increase the
14:53
Speaker A
confidence of your own fix. And finally when you are happy with the fix we create a diff or a PR in this case. The PR is attached to the Jira to the issues on healthline wisdom. Um it is uh sent to the developer via slack
15:10
Speaker A
and all the information is updated internally as metadata. A few core things worth pointing out here is this is the basic skeleton of the agent. It never changes.
15:23
Speaker A
What changes is the skills and plugins. So each of these LLM nodes have their own skill. How to create a good PR at Uber? What are some good practices we follow for test planning? How to fix an issue in Android? Do I want to do lazy
15:37
Speaker A
loading always? Do I want to defer something to background etc. Now these skills and plugins are as the name suggests pluggable. So the skeleton remains the same and all of this core business logic is outside the skeleton which is fetched at runtime which means
15:55
Speaker A
tomorrow if I want to add a new agent it's a matter of crowdsourcing the skills which leads me to the next slide. How did we manage to productionize an agent at uber scale? Now running something on your local machine is one story but
16:11
Speaker A
running it at uber scale is another story altogether. So as I said we have provided the harness to the developers.
16:19
Speaker A
We have provided the steps to follow. We have provided the runtime environments, the dependencies etc. All the teams have to do is bring in their own skills, own specialized skills on top of what we already provide which also reduces the
16:33
Speaker A
load on us because now we don't have to maintain a multitude of skills and plugins and whatnot for everyone. You bring your own specialization and we provide the harness. So right now we have almost eight agents running at
16:48
Speaker A
scale across five monorreos and all of them share a single codebase. All of the code is inside a python monorreo and it is packaged as a python binary called pex. The pex is uploaded to a common artifactory and every time an agent
17:02
Speaker A
wants to run it it knows its type. So according to the agent type it fetches the correct binary. It fetches the correct docker runtime and uh docker images which provides all the dependencies and runtime environments.
17:13
Speaker A
And finally it fetches all of its pipelines which is scattered across different monorreos. Again um I would like to highlight that this is a harness and even the skills and plugins for every agent is fetched at runtime. So we don't do a dump of all
17:32
Speaker A
the marketplace plugins. Uber has a skills marketplace which is huge. There are almost 3,000 plugins right now and we can't dump the context of all the plugins to an agent. It goes crazy. So you need to limit the agent um in some
17:46
Speaker A
way and this is how we do it using the agent type parameter. Now we don't stop at PR creation and forget about it. We provide a interactive environment to the developers where they can talk to the agent and actually refine and validate
18:04
Speaker A
the output themselves. Firstly, there is a small diffixer that we have at Uber and what it essentially allows us to do is let's say someone wants to add um more unit tests, they want to refractor, they want to change a
18:18
Speaker A
variable name. So all of these small and incremental fixes can be taken care of by the diff fixer. The developer doesn't need to do anything. It's just a click of a button and a sing single line prompt.
18:30
Speaker A
Next up, we have the ask AI chat. So, the developer is free to talk to debug assist itself, ask about the RCA, the issue, and even provide corrections on top of it. Now, as soon as you provide corrections, you also build a feedback
18:46
Speaker A
loop to the agent, which helps improve its own prompts and skills. And lastly, we have open in your machine. Even after all of this, some people do like to open the issue in their own machines and debug it by hand.
19:00
Speaker A
Um, I would also encourage you to do so. Um, now that's the same feature that I'm talking about here. So, you can open the issue in your own machine or in a remote um remote machine that we have uh from a
19:13
Speaker A
pool of machines and the environment and everything regarding that issue is already set up for you. So there is no installation required. It's a singleclick operation. Again sharing some interesting impact numbers with you. Uh currently debug assist runs
19:33
Speaker A
on almost 9,000 issues every month. It provides 9,000 RCAs, deep RCAs every month. The RCA accuracy right now is 50% as compared to exact developer fixes.
19:46
Speaker A
And out of the remaining 50% almost 40% of those issues are still in the right direction. So debug assist maybe is going in the right module but not picking the right file or going to the right file or not picking the right
20:00
Speaker A
function but still it's in the same direction which still helps the developers fix the issue much much faster.
20:08
Speaker A
We have also noticed that the Jira resolution time has increased. it has almost um increased by twofold which is a huge improvement in in itself to clear the backlog of out of SLA issues. And lastly, the PR merge rate. Um back when
20:24
Speaker A
we started this one year ago, the PRs that we were creating um were let's just say not getting merged as frequently as you would like them to. So the merge rate was only 5%. And now that number has increased to almost 48%.
20:41
Speaker A
Having said that, we are not stopping here. The AI space is evolving continuously and we aim to push these numbers further. So we want to improve the coverage. We want to improve add more and more use cases to it. We want
20:54
Speaker A
to improve the accuracy and push it to ideally 100%. And as a result, we hope to increase the Jira resolution further as well as the meantime to resolve.
21:04
Speaker A
Thank you everyone and u uh if you have any questions, please the floor is open.
21:13
Speaker A
[applause] I see a question in the back. Could you maybe please come to the front?
21:27
Speaker A
There's a mic here. Perfect. Oh, yeah. Well, a wonderful presentation by the way. Um, I want to incorporate a lot of this into my uh my own workflow. So, fantastic. Thank you.
22:06
Speaker A
Um, I had a few questions. Um, one was for the testing. Um you were mentioning the the basil test that you do stage three.
22:19
Speaker A
Is that done via a like a model? And does it deploy for example multiple versions of Android and iOS as well to actually test like the new version of the app? Um that that was one question.
22:38
Speaker A
Uh yeah um I had another one but anyway yeah that was it. Okay thank you for that wonderful question. I'll just uh state the question for everyone again. So we have a question that uh that that's basically asking um how do we exactly test it
22:56
Speaker A
every time is a new version of the app built deployed for Android and iOS uh where we want to test. Um the answer is yes as well as no. So in cases where you are able to repro an issue using a unit
23:11
Speaker A
test use that use the shortest and the simplest path possible similar to how a developer would do. If you can repro the issue via a integration test or maybe a component test use that and finally if you are not able to repro an issue via
23:30
Speaker A
code at all you use the in um automation tests. So to give you an example, there was a particular issue that was occurring near an airport where the network uh strength was quite low and the issue was only occurring when the
23:47
Speaker A
network strength was low. So in this case, you can't really mock a network via code. It's really a question of the environment that the app is running in. So we basically spawned uh a simulator and an emulator and that is a
24:03
Speaker A
separate infrastructure at Uber. We have a separate team similar to uh if you know about browser stack or kobeton similar to that what they do is they create that environment. We already have all the data that we captured from the
24:17
Speaker A
user logs. So we know what was the network strength. We know what was the user battery. We know what the parameter flags or the feature flags the user was behind. We simply mock that information in that simulator or emulator and then
24:30
Speaker A
we try to build the APK or the IPA file and upload it and that goes in a iterative um feedback loop and we keep doing that till it passes. However, we again like have guardrails for it. There are fixed number of retries and then
24:46
Speaker A
finally the um fix is pushed to the developer. I hope that answers your question.
24:53
Speaker A
Any other questions? Hey. Hi. Can you please come forward? Oh, I have a question about the uh RCA tracking. So you mentioned driving sub aents to check for the cost and uh I suppose there are like tens or 20 sub
25:20
Speaker A
aent running to get to the truth of the RCA. So how do you control that? Do you so you mentioned about the um some agent tracing the session to identify the trace of user path and the other agent
25:35
Speaker A
trying to check the which versions this error is having statistical analysis. So how many strategy have you defined and how many agents are running in parallel to go to go to the um to track down the RCA?
25:52
Speaker A
Awesome. Uh thank you for that lovely question. I'll just state the question for everyone. The question was around we have these sub agents running in the RCA node. So how do we keep a track of those agents? How we how do we see how many
26:05
Speaker A
agents are running and how do we control what they are doing? Um now um we do have built-in tracing um in this agent.
26:14
Speaker A
I didn't talk about the observability of the agent itself. Even though I work in observability space, I didn't have enough time but we do have built-in tracing. Um if you are aware it's called arise tracing and that comes in really
26:27
Speaker A
really handy. So we have plugged it in with lankchain as well as each of the LLM calls that we make. Number one that helps us track not just sub agents but every tool call that we make. So all the
26:38
Speaker A
MCPS that we call every bash command that the agent runs everything an agent runs we have visibility into that.
26:46
Speaker A
Number one. Um number two how to control what each a sub agent itself does. Again I would say the biggest contributor there is the number of max turns. So if the agent is able to perform a certain action in maybe let's say 20 or 30 turns
27:03
Speaker A
it's good. If it's taking more than that, then probably something is is wrong. Um, and for example, there are specialized sub aents. Um, I I'll just give you one more example. There is a sub aent that tries to figure out who
27:16
Speaker A
which was the offending commit that led to this crash. Now that agent while it can access all the git history, do we want to do that? Probably not. So we supply that agent sub agent with some additional information. For example,
27:33
Speaker A
this was the clean version that I saw earlier and this is the bad version.
27:36
Speaker A
Something must have happened in between this. So you look between this. Then I want to look in maybe rider module, maybe driver module, maybe Uber Eats module. So let's just filter the history by that. And you keep pruning and
27:48
Speaker A
pruning and pruning. So you also control the input to those sub agents. Does that answer your question?
27:54
Speaker A
Oh yes, thank you. That's one more. So how many strategies do have you defined to for each agent for one strategy? So how many strategies have you defined or is it defined dynamically by leader agent or is it deterministic
28:10
Speaker A
right? Um so how many strategies have we defined for sub aents? Essentially it boils down to how many sub aents or unique sub aents do we end do we have at Uber. Right. Right. Um right now the number of sub aents is pretty limited. I
28:26
Speaker A
would say we have like um all in all 30 sub aents, 30 unique sub aents but we are building out something called um domain extensions so that all the different domain owners can plug in their own sub aents here and they have
28:42
Speaker A
to manage their own knowledge base and own domain because we are not the owners of domain we are the providers of the harness. So this number can uh by our projections is bound to shoot up to roughly 10,000 by the end of next half.
28:58
Speaker A
Great. Thank you. Thank you. All right. Um thanks everyone. I hope you had a good time.
29:08
Speaker A
[applause]
Topics:UberAI agentdebuggingroot cause analysisLangGraphproduction observabilityapp qualityMCPagentic AIcrash reporting

Get More with the SozAI App

Transcribe recordings, audio files, and YouTube videos — with AI summaries and speaker detection. 30 minutes free.

Or transcribe another YouTube video here →