Speaker A
[music] My name is Jay, and I'm here with Sonya. We are part of the computer vision team at Aruba. We're going to talk to you about a real-world production use.
Soumya Gupta & Jai Chopra from Uber discuss building scalable closed-loop evaluations for multimodal agents enhancing Uber Eats' visual content quality.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Visual content quality significantly impacts customer engagement and sales on Uber Eats.
- Closed-loop evaluation with continuous feedback enables scalable and safe agent deployment.
- Balancing creativity and control in agents is essential to preserve authenticity and user trust.
- Human labels remain the gold standard for aligning and validating AI-generated content.
- Comprehensive logging and structured data are critical for ongoing optimization and quality assurance.
What the video covers
- The video covers the design and implementation of closed-loop evaluation systems for multimodal agents at Uber Eats.
- Visual content quality, especially photos and videos, is critical for user engagement and conversion on Uber Eats.
- Challenges include maintaining authenticity, trust, and diversity while improving image quality at scale.
- The team balances deterministic rule-based approaches with creative agent autonomy under safety constraints.
- A multi-agent pipeline includes image understanding, routing, editing, QA, and post-processing with continuous feedback loops.
- Human-labeled data is used as the gold standard to align model outputs and handle drift over time.
- Logging and structured output generation are foundational for optimization and monitoring.
- The system operates globally across 10,000 cities with a 20% year-on-year growth in marketplace items.
- The approach addresses long-tail quality distribution and merchant-specific needs without cannibalizing any merchant.
- Safety, scalability, and cost-efficiency are key design goals for deploying these agents in production.
Chapters
- 00:00Introduction and Use Case Overview
- 01:36Importance of Visual Content in Uber Eats
- 02:32Merchant Challenges and Goals
- 04:53Balancing Authenticity and AI Creativity
- 05:48Agent Pipeline and Feedback Loops
- 06:49Logging and Data Optimization
- 09:00Routing and Image Understanding Agents
- 10:44Image Editing and QA Loop
- 11:49Human Alignment and Drift Handling
- 19:39Summary and Production Considerations
Full Transcript — Download SRT & Markdown
Speaker A
Oh, my son done. Okay. Try again. Okay. Don't worry. I'll I'll manage. You hear me now?
Speaker A
Oh, my son done. Okay. Try again. Okay. Don't worry. I'll manage. You hear me now?
Speaker A
into the agent design, we're going to talk about a little bit about the use case. So our delivery marketplace Uber Eats, we do about 90 billion run rate per year at the moment.
Speaker A
Okay, so we're going to talk to you today about a real-world production use case, and specifically, we're going to dive into how we design the e-bows and the e-bow loops. So, all right, cool. So just before we get
Speaker A
the mobility side on Uber today. Visual content actually plays a really important role for the user experience.
Speaker A
into the agent design, we're going to talk a little bit about the use case. So, our delivery marketplace Uber Eats, we do about a 90 billion run rate per year at the moment.
Speaker A
So a good photo can make the difference between someone scrolling through the feed and actually clicking on an item and adding to the cart. And more and more we're seeing different modalities on Uber Eats uh especially video content.
Speaker A
We are adding millions of items to the marketplace each and every year. Sorry, every month. We're growing at 20% year-on-year, and we operate in 10,000 cities globally. So, not many people actually know this, but our delivery marketplace is just as big as
Speaker A
And when we speak to our merchants, there are three themes that kind of emerge.
Speaker A
the mobility side on Uber today. Visual content actually plays a really important role for the user experience.
Speaker A
And this can be especially [snorts] problematic if the merchant is updating their menu over time.
Speaker A
So, a photo is quite often the first signal that a customer gets that gives them that initial impression about a merchant.
Speaker A
the Uber Eats app, the last thing that you want is to be scrolling through uh you know, food photography that looks like AI's lock.
Speaker A
So, a good photo can make the difference between someone scrolling through the feed and actually clicking on an item and adding it to the cart. And more and more, we're seeing different modalities on Uber Eats, especially video content.
Speaker A
editing, the diversity of the marketplace is going to collapse. We also, because we operate globally, we also have this long-tail distribution of different quality that we see it across the marketplace.
Speaker A
But this is a problem. So, our smaller independent merchants simply just don't have the level of quality for their photos that reflect what the eater is actually going to get.
Speaker A
So, what are our goals when we're designing these agents? When you think through these goals, you might actually be thinking through, you know, your own agents that you're building yourself.
Speaker A
And when we speak to our merchants, there are three themes that kind of emerge.
Speaker A
We want to optimize globally for for the entire marketplace. We don't cannibalize certain merchants. We want to ship safely, and this is going to be an important theme throughout the talk.
Speaker A
Lack of time, lack of know-how, and costs, because these professional photo shoots actually cost a lot of money.
Speaker A
So, agents are actually really well-suited to solve this problem. So, if you imagine a spectrum, on the one side, you've got something that's more deterministic. It's more rules-based. Um and uh you you have more control over it, but it's fairly it's a
Speaker A
And this can be especially problematic if the merchant is updating their menu over time.
Speaker A
Imagine the other side, you provide an agent with obviously a lot of creativity, it has a lot of agency. Um and that's actually what we want to lean into, but we can't leave that unconstrained, right? Cuz we have certain safety and
Speaker A
So, this problem is actually pretty challenging to solve for at scale, right? Because our consumers, they want authentic, real-looking photos, but a meaningful fraction of consumers actually distrust anything that is AI-generated. So, if you open up
Speaker A
Uh and that's kind of set the principle for the way that we think and design around agents and evals.
Speaker A
the Uber Eats app, the last thing that you want is to be scrolling through, you know, food photography that looks like AI's lock.
Speaker A
And we're going to go through each stage and how we eval it, and then talk through some continuous learning loops as well.
Speaker A
So, we're threading the needle here. We need to be able to stay faithful to the original image, preserve the brand of the merchant, and avoid everything looking the same. If we have the same prompt for every photo that we're
Speaker A
Um and then we we create a structured output from that, and we send it to a router.
Speaker A
editing, the diversity of the marketplace is going to collapse. We also, because we operate globally, also have this long-tail distribution of different quality that we see across the marketplace.
Speaker A
If we enhance it, we send it to our next agent, which is an image editing agent. And this can actually run in a loop. So, it gets feedback from a QA agent. Um it can edit uh in the in this loop and self-correct and
Speaker A
So, we've got some examples here. You might see food photography that, you know, has poor sharpness, poor composition, not centered, or poor colors as well. We also have a wide range of spectrum of user-generated content on the platform as well.
Speaker A
Then we actually send it to a final post-processing and QA step. If that's all good, we'll publish it to the menu.
Speaker A
So, what are our goals when we're designing these agents? When you think through these goals, you might actually be thinking through, you know, your own agents that you're building yourself.
Speaker A
Don't know if you can actually read the JSON here, but you might notice that all of the agents in this end-to-end orchestration is within one It's It's basically a flat structure in this JSON.
Speaker A
But for us, it's about one, preserving authenticity and trust. Two, improving the quality when we need to. So, we want to be able to improve quality selectively.
Speaker A
aggregates. Um and it's important to note here that, you know, we think this is important to start with. You want to start with your logging cuz if you don't start with it, you have nothing to optimize for, let
Speaker A
We want to optimize globally for the entire marketplace. We don't want to cannibalize certain merchants. We want to ship safely, and this is going to be an important theme throughout the talk.
Speaker A
Cool. We're going to dive um a bit deeper into the router. So, the router's actually pretty straightforward. If you remember, we you know, we have this multimodality input.
Speaker A
We want to learn continuously, and we want to operate at scale in a cost-efficient manner.
Speaker A
have these pass and fail criteria. The last step is we want to decide whether or not we should enhance or skip.
Speaker A
So, agents are actually really well-suited to solve this problem. So, if you imagine a spectrum, on one side, you've got something that's more deterministic. It's more rules-based. And you have more control over it, but it's fairly brittle. It's not actually going to be able to scale for the entire marketplace.
Speaker A
Um but we can look at things like the true positive cases, the false negative negative cases, and so on and so forth.
Speaker A
Imagine the other side, you provide an agent with obviously a lot of creativity, it has a lot of agency. And that's actually what we want to lean into, but we can't leave that unconstrained, right? Because we have certain safety and
Speaker A
user experience at the trade-off of quality. And if that's the case, instead of having a 2 by 2 matrix for your confusion matrix, you might actually have an n by n matrix.
Speaker A
certain guardrails in place that we need to adhere to. So, we want to find a balancing act.
Speaker A
So, I'm going to now hand over to Somya who's going to dive a little bit deeper into how we handle drift and human alignment.
Speaker A
And that's kind of set the principle for the way that we think and design around agents and evals.
Speaker A
For our use case, we consider human labels as the golden source of truth. And this is what we want to align our models to.
Speaker A
So, now we're going to actually dive a little bit deeper into a simplified but representative example of what we have in production.
Speaker A
This is to remove any subjective biases or any noise coming in from human labelers.
Speaker A
And we're going to go through each stage and how we eval it, and then talk through some continuous learning loops as well.
Speaker A
metrics, you go ahead and ship it. If not, then you go tune and you keep doing this until you meet your guardrail metrics.
Speaker A
So, first up, we have what we call image understanding and routing agents. So, this is where multimodality is pretty important. We actually ask the LLM to describe what it sees in the photo.
Speaker A
Here are some examples of the failures we've seen. Uh on your left you see a very good image of cheeseburger. Uh on the right you notice that the routing agent actually failed this. It said the technical is low ball and it will go
Speaker A
And then we create a structured output from that, and we send it to a router.
Speaker A
given it's already such a high quality image. And on the other end of the spectrum, you have a recall miss. So on your left you have an image with six chicken wings and on your right if you notice the dish
Speaker A
The router will then determine, do we enhance it, or do we skip it? If we skip it, we will keep the original.
Speaker A
going to hallucinate these two extra wings to to match the description. And that's also an that's a the cut we take at our faithfulness metric that Jay earlier showed us.
Speaker A
If we enhance it, we send it to our next agent, which is an image editing agent. And this can actually run in a loop. So, it gets feedback from a QA agent. It can edit in this loop and self-correct and
Speaker A
your prompts, agents, system itself is evolving over time. And that's what we've done uh for our system as well. And I'm talking more from the routing perspective, but every component in our system is able to tune itself uh for any
Speaker A
fix things as it goes. If it goes through a number of loops and it still fails, we don't publish it.
Speaker A
output we got from the labelers and see if there's a mismatch. If there's a mismatch, we have an umbrella diagnosis agent which takes in the feedback, localizes where this issue is happening, and and and triggers our auto-tuning pipeline.
Speaker A
Then we actually send it to a final post-processing and QA step. If that's all good, we'll publish it to the menu.
Speaker A
model. Uh if not, then you kind of keep iterating. And this happens on a regular basis on production data set.
Speaker A
And the last thing that's really critical is we log everything. Just a quick note about logging.
Speaker A
static model with the offline, but this is what is going to keep your system alive.
Speaker A
Don't know if you can actually read the JSON here, but you might notice that all of the agents in this end-to-end orchestration is within one. It's basically a flat structure in this JSON.
Speaker A
system. Uh so we start with a target agent, and we've already got these uh unseen eval samples from our humans.
Speaker A
And so, this is actually incredibly useful for the entire team because anyone, be it non-technical, technical folks on engineering or product, can actually dive in and look at specific cases to diagnose and also roll up things to look in
Speaker A
remove any noise, find any systemic issues that might be in your data set, and reflect on it and send that feedback to the synthesize agent. Now, the synthesize agent takes this feedback. It has your agent config. It goes and
Speaker A
aggregates. And it's important to note here that, you know, we think this is important to start with. You want to start with your logging because if you don't start with it, you have nothing to optimize for, let
Speaker A
agent. And this is a closed-loop system as I mentioned, no human in the loop. We definitely have observability on the guardrails, quick rollback built in in case of any issues with the system itself.
Speaker A
alone set up a self-learning loop. And at Uber, we use our eyes.
Speaker A
What we do is the first step, we generate a prompt specific to this image. We take in the description, we take in the directives we were getting from our routing agent, and we go ahead and generate a prompt for this image.
Speaker A
Cool. We're going to dive a bit deeper into the router. So, the router's actually pretty straightforward. If you remember, we have this multimodality input.
Speaker A
publish this. If it doesn't pass, you take the feedback back from the QA gate, push it back to your generate prompt along with the initial inputs you sent it, and go ahead and enhance it again.
Speaker A
We look at certain text description metadata, the image itself. We ask it to describe what it's seeing. We create structured output from that. With that structured output, we can then grade against a rubric. So, we
Speaker A
Here's an example. On your left, you see a bowl of sweet potato fries. We send it up for the first iteration and our QA agent rejects it because the portion size is incorrect, the plating is very unrealistic. We take that feedback in,
Speaker A
have these pass and fail criteria. The last step is we want to decide whether or not we should enhance or skip.
Speaker A
iterations, your pass rate will increase because you're getting more feedback in. Now, I'll pass it on back to Jay to cover the rest of this.
Speaker A
How do we actually eval this? This is, you could think of this as a more sort of traditional classifier. So, here we have a confusion matrix. Many of you are probably pretty familiar with this.
Speaker A
output image, and it's assessing whether or not it's better. But how do we actually find what's better? So, um we're not going to dive into too much of the details here cuz this is kind of like proprietary stuff, and so we'll
Speaker A
But we can look at things like the true positive cases, the false negative cases, and so on and so forth.
Speaker A
image on the platform into our Evals. Um so, examples here, is it faithful? Is it complete? Is it natural? Is it realistic? And there's a bunch of other things as well. The output of this is then uh a yes, no, or unsure.
Speaker A
Essentially, what we're doing is we're measuring the precision and recall. In practice, your routers might actually be much more sophisticated. So, for example, we might want to route an image to a lower latency smaller model to be able to save on cost and improve the
Speaker A
we shouldn't be. So, we failed faithfulness. This is where we go the other way.
Speaker A
user experience at the trade-off of quality. And if that's the case, instead of having a 2 by 2 matrix for your confusion matrix, you might actually have an n by n matrix.
Speaker A
So, we failed completeness. Here's actually a pretty interesting example where the agent actually attempted a more creative edit in the first iteration.
Speaker A
Where each grid is actually telling you whether or not you're correctly routing to that specific branch.
Speaker A
And it becomes overly conservative. Sort of falls back to this generic ceramic plate uh ceramic bowl, sorry.
Speaker A
So, I'm going to now hand over to Sonya, who's going to dive a little bit deeper into how we handle drift and human alignment.
Speaker A
pretty different. Here's another example where in the output the plate is covering the sauce.
Speaker A
So, now that we spoke about how we eval the routing, I want to talk about how do you get the first version of the model out.
Speaker A
Um and so, so object coherence and physics plausibility of the Evals that sometimes will coordinate with the frontier teams and and let them know about these problems and work together with them.
Speaker A
For our use case, we consider human labels as the golden source of truth. And this is what we want to align our models to.
Speaker A
So, we're not confident, actually. We're not sure. And so, this is an example where we would actually reject it in production and and it wouldn't it wouldn't go through.
Speaker A
The way we do about this is we go
Speaker A
Here, we do some policy checks. We also do some more quality checks. Um and you might be wondering, like, we've already done some QA. Like, why are we going to do another step of QA?
Speaker A
The reason is because we think of this like a Swiss cheese model. So, we want to try and optimize for reducing the chance of a failure getting into production. And so, there is some redundancy here or there.
Speaker A
And that's okay. Um and so, this QA gate is is a little bit more holistic. It captures more things. But, it also will will try and flag things that we should have caught upstream, as well.
Speaker A
All right. So, we've talked about a couple of uh feedback loops here. So, to summarize, we talked about predominantly this first one here, which is the model loop. And this is accounting for drifts and aligning with human labeled data set
Speaker A
that we have and we've established offline. But, we actually have more feedback loops. So, we we have that Uber what we we have is a is a great sort of dog dog feeding culture.
Speaker A
Um we will test apps before they go live. Um but, we also have when it goes live in production, how do we get that feedback back into our agent to be able to steer it appropriately?
Speaker A
So, as we're adding more of these feedback loops, we want to be able to generalize the system.
Speaker A
So, this is where we've actually created um a higher level of abstraction on top, which we call the diagnoser.
Speaker A
So, the diagnoser can take in any input from these different feedback loops that we're we're capturing. It can reflect on what actual agent within the overall system needs to be optimized, and it can route that agent to be able to fix that
Speaker A
configuration specifically. It could be one agent, it could be multiple agents. So, here's an example of internal dog fooding. You might see these in sort of different apps that you've got where you got the thumbs down and the thumbs up.
Speaker A
We also take some free form feedback as well. Uh and this is actually great cuz we'll get feedback from merchants directly.
Speaker A
We'll get feedback from, you know, design teams, other product teams uh at Uber. And we'll incorporate that feedback back into our diagnosis step and tune the system over time.
Speaker A
Again, similar sort of workflow pattern here. We'll replay the examples that we know are those ones that have been flagged, be it good examples, be it bad examples, uh and then we'll benchmark the metrics before we push the latest
Speaker A
config version. The last step is is actually getting this into production. And and this is where we're looking for a whole heap of different metrics we track for for the marketplace quality and health. Uh I've just called out one
Speaker A
here, which is conversion. So, we're looking for improvements in people adding to cart, converting, completing their orders.
Speaker A
Um I think this one's actually an interesting one to call out because now at I mean, at least at Uber, but especially in production um settings at scale, you have a wide um uh you have a lot of data that you can
Speaker A
actually slice and dice. So, in this area as opposed to the others, what we can do is sort of slice by geos, by device type, by dish type, etc. And we can look at where things are improving in different segments and
Speaker A
actually tune on certain segments as well. Cool, and that's it for our presentation. Appreciate it.
Speaker A
[applause] [music]
Topics:Uber Eatsmultimodal agentsclosed-loop evaluationcomputer visionimage qualityAI safetycontinuous learningagent designvisual contentproduction AI systems
