**Are Frontier AI Models Finally Good at Vision? | Visual Intelligence Summit 2026 — Transcript & Summary | SozAI**
Source: https://sozai.app/transcript/frontier-ai-models-vision-visual-intelligence-summit-2026/

Roboflow's Piotr shows how frontier vision-language models now handle OCR, object detection, segmentation, tracking and re-identification via single prompts, replacing months of custom pipeline work.

## Key Takeaways

- Frontier VLMs have become strong enough at OCR, detection and segmentation to replace many custom multi-model pipelines for some tasks.
- Visual example prompting (positive and negative examples) is a powerful alternative to text descriptions for hard-to-name objects.
- Benchmarks are still being designed to break these models, and top scores remain well below perfect on harder tasks.
- VLMs are currently most practical for complex, low-volume tasks rather than high-throughput real-time production.

## What the video covers

- Speaker Piotr, formerly responsible for open source at Roboflow and now evaluating frontier models on computer vision tasks, opens by framing VLMs as rapidly improving at vision.
- He recounts a basketball analytics demo that took about a month and six models (three fine-tuned) to build, which a single VLM prompt now approximates roughly a year later.
- VLMs take images, video frames or a whole video plus a text prompt and return text, enabling tasks like description, counting, detection, segmentation and re-identification.
- OCR: single-value extraction (serial numbers, expiry dates, license plates, meter readings) and multi-value structured extraction; the best model scores about 80% on his benchmark designed to break models.
- Text-prompted object detection has improved dramatically: a recent model reached 100% mAP on Lego piece detection with no training, while the top model scores 85% mAP on his harder dataset.
- Visual prompting with example boxes (positive and negative examples) lets the model infer the shared concept, such as detecting all puzzle pieces or pieces of one color.
- Segmentation via VLMs produces polygons; annotating a video takes about three minutes, and shape definitions can be given by example rather than language.
- Tracking works by sending sampled frames and prompting for coordinates, with interpolation between keyframes; re-identification of players who leave and re-enter frame is a harder, newly feasible case.
- The speaker concludes that increasingly complex computer vision tasks are one prompt away, though VLMs are likely best for complex or low-volume use cases rather than high-throughput production.
- He notes labs are actively working on segmentation and tracking, so better models should arrive soon.

## Chapters

1. 00:00 Introduction and speaker background
2. 01:37 The basketball AI demo and the old multi-model pipeline
3. 02:28 What VLMs are and what tasks they enable
4. 04:04 OCR: single and multi-value extraction and benchmarks
5. 06:02 Object detection with text and visual prompts
6. 10:48 Segmentation with VLMs
7. 14:23 Tracking and re-identification
8. 17:47 Conclusions and production outlook

Answers

## Questions about this video

How did the speaker's basketball AI pipeline compare to a single VLM prompt?

The original pipeline took about a month to build and used six models, three of them fine-tuned, covering detection, tracking, team association, jersey number reading and court keypoint homography. The speaker says a single VLM prompt now produces a result that is not perfect but close enough, raising the question of whether training such models is still worth it.

How well do VLMs perform on OCR according to the speaker?

On single-string OCR, the speaker says the best model scores around 80% on his deliberately difficult benchmark. He describes multi-value structured extraction as also working well, useful for things like counting containers in a yard.

What is visual prompting and why does the speaker find it useful?

Visual prompting means showing the model example objects, via highlighted boxes or coordinates, instead of describing them in text. The speaker finds it useful for shapes that are hard to express precisely in language, and negative examples can narrow the detection to objects of a specific color.

## Full Transcript — Download SRT & Markdown

00:20

Speaker A

Thank you very much. Okay. Hello everyone. My name is Piotr and I have one already.

00:27

Speaker A

I have one already. We have the backup plan. I'm Piotr and as you heard for the past three or so years I was responsible in Roboflow for open source. Recently my range of responsibilities got a little bit wider because now I'm also evaluating those big frontier models on computer vision

00:49

Speaker A

tasks. Tasks that you care about like OCR, like detection segmentation, tracking, re-identification. And I guess that's the reason for this presentation today. Those big models are getting a lot better at computer vision really fast.

01:06

Speaker A

I think it's worth to spend twenty minutes listening about it. So. One more thing that I was responsible in Roboflow over the years was also building demos, building tutorials to inspire people to use computer vision models, to show them how to use them. And this is one of the demos that I built just a

01:28

Speaker A

little bit over a year ago. So I called it basketball AI. I recorded the whole YouTube tutorial showing how to build something like this.

01:37

Speaker A

And to build it, I actually needed to create a whole computer vision pipeline. It took me like a month of my own to build it.

01:46

Speaker A

We have like six different models. Three of them are fine-tuned specifically for this task.

01:51

Speaker A

So we have a separate model for object detection to help us detect players and the numbers. We use some tool to track those players through the frames.

02:01

Speaker A

We have the whole separate team association pipeline that allows us to group players into teams and we have another pipeline to read their jersey numbers.

02:11

Speaker A

And as the cherry on top, we also fine-tune key point detection model to recognize characteristic points on a basketball court and then use it together with homography to map players' positions from the camera perspective the actual court.

02:28

Speaker A

But what if I tell you that you can actually one-shot prompt it with VLMs these days? So it was just a year ago.

02:37

Speaker A

I needed to build custom pipelines with six models. Three of them are fine tuned.

02:43

Speaker A

What you see right now is result of my one prompt yesterday. It's not perfect. But it's definitely close enough and it brings the question, is it still worth to spend a month of somebody's time training six models or is

02:58

Speaker A

it better to just do something like this with VLMs? So all of that is possible because of the magic of VLMs.

03:05

Speaker A

Of course those frontier models are more like multi-modal models. They can consume other modalities, not only vision and language.

03:13

Speaker A

But for the purpose of this conversation right here, right now, I guess we are all mostly interested in those two.

03:19

Speaker A

So obviously VLM is the type of model that can consume visual input in the form of single frame or set of frames or the whole video and a text prompt.

03:28

Speaker A

And it spits back the text. Turns out you can frame a lot of tasks this way.

03:36

Speaker A

The most obvious things like image description or counting objects on the image, but you can also go crazy and do object detection, segmentation, or re-identification, things like this.

03:51

Speaker A

So, before I will show you how I actually did what I did with that basketball video, let's go back a little bit and let's discuss other tasks that are right now possible to execute with VLMs. And we will start with OCR.

04:04

Speaker A

OCR is one of those tasks that was there from the very beginning, one of the most obvious things that you can do with VLMs.

04:12

Speaker A

In a simple, simple case of OCR as single string extraction. So those are things like this. Yeah there is a text somewhere on the image.

04:21

Speaker A

We have, I don't know, CCTV camera that is looking and we would like to extract that single value. That value can be the weight of the product, it can be expiration date of the product can be serial number,

04:36

Speaker A

can be license plate number, can be electric meter reading, things like this. And even though that is one of the oldest tasks, we still try to break those models.

04:49

Speaker A

So I'm responsible for building benchmarks that have one purpose to make those models fail. And I'm trying really hard, but the best model is actually scoring around 80% on my benchmark so far.

05:03

Speaker A

There is also A more complex version of this task, when instead of having one value that is on the image, we have actually multiple values.

05:11

Speaker A

And we would like to extract all of them at the same time, preferably in a structured manner, so that we can then ingest that into our database or run it through our already existing computer vision pipeline to know,

05:25

Speaker A

I don't know, for example, how many containers we have in our yard. And here once again, the models are doing great.

05:34

Speaker A

So this is text. Now let's talk a little bit about something a little bit more interesting, a little bit more controversial, let's say.

05:45

Speaker A

So doing object detection and other advanced computer vision tasks with VLMs. So VLMs could do object detection, say for two years now, but anybody who paid attention You probably know they were not really good at it.

06:02

Speaker A

There were tons of false positives, false negatives. So they were doing good detections from time to time, but on harder tasks, pretty much impossible. So this is the result on, I'm not sure how well you see this image, but this is Lego pieces.

06:17

Speaker A

Different colors, different shapes. And those are detections produced by Argon, one of the latest models released, I guess, like two weeks ago. And it's doing 100% mAP on that task with no training whatsoever. All I do is I prompt it with text, hey, I would like to detect Lego

06:34

Speaker A

pieces. Give me three by two, four by one, and four by two pieces. And it's doing that with perfect precision.

06:42

Speaker A

Here's another one example that I really like, pretty hard as well, because of the light conditions.

06:48

Speaker A

Those are chess pieces. Once again, no problem. Obviously, it can process documents. Obviously, it can process satellite images.

06:57

Speaker A

And our basketball game, even those small boxes, like for example, around numbers of players or the ball, that is actually against the stands, which anybody who tried to solve this task in the past with traditional computer vision, you know that when the ball is flying and there is a lot of faces,

07:18

Speaker A

there is definitely one bald guy in the stands and the ball is. No not where it's supposed to be.

07:25

Speaker A

I guess it's solved right now. And even though that I'm, like I said, trying really, really hard to break those models, the top model right now is scoring 85% mAP.

07:36

Speaker A

That's why I'm really motivated to create the second version of this data set and try to bring it back to like 60%.

07:43

Speaker A

Let's see if I can do it. And so that was prompting with text, something that sounds still intuitive.

07:51

Speaker A

So I'm just telling the model, hey, can you detect four by one bricks? And it's doing that. But you can also do prompting with boxes.

08:02

Speaker A

And that's a task mostly known from segment anything family. So it's about not telling model which objects you would like to detect, but showing model which objects you would like to detect.

08:16

Speaker A

And you can also do it with VLMs. So what you can do is you can highlight with images, or in other cases, just put in prompt the coordinates of the objects that you're interested in.

08:30

Speaker A

And just instruct the VLM to detect other objects of this kind. So in this case, we have a pretty complex puzzle, I would say, not a easy image to work with. And I would like to be able to detect all the pieces

08:44

Speaker A

that are visible on that image. So to do that, I'm highlighting four of them, different kinds, different shapes. The model is reasoning about it and coming down to the conclusion that the only common denominator between those four pieces is that they are just

09:01

Speaker A

puzzle pieces. So ultimately, it returns the boxes of all the other pieces that are visible on the image. It's all happening within a single prompt.

09:10

Speaker A

We can go even more creative. So instead of going for all of them, we could maybe target those with a single color.

09:17

Speaker A

And to do that, all you need to do is highlight two of them as a positive examples and provide another one of a different color as a negative example.

09:26

Speaker A

And once again, model is reasoning about it and coming down to the conclusion that this time there is one mon one more common denominator.

09:34

Speaker A

It's the color. We can target different color. We can also target different shapes. So if we would like to say, hey, I would like to only target this particular shape, doesn't matter which color it is, it's also possible.

09:47

Speaker A

This is also one of my favorite examples because if you would like to express that particular shape in language, I'm not sure if you would be able to do it so precisely. Showing an example of that object is a lot more easier.

10:01

Speaker A

And a cherry on top, you can also do it cross-image. So you can see on the previous examples that I showed you, though this was always the same image.

10:10

Speaker A

Now it's a different image. So I'm just providing a very similar prompt to the one that I had in the previous example when I was targeting blue pieces.

10:19

Speaker A

But I'm doing that for a different image as a reference image and a different image as my target image. And it's still working.

10:27

Speaker A

And all of that is possible with VLMs without them breaking a sweat. Now probably the most controversial example that I will show you today, just a side note, I'm not suggesting you should run VLMs for segmentation and production, but it's still incredible to see what's possible.

10:48

Speaker A

So the visualization that I had as my starting slide is actually a result of running a segmentation with Gemini 4 Argon.

10:55

Speaker A

All I did is instead of asking for bounding boxes, I asked about polygons. If you want to go super crazy, you can instruct the model that a single object can be actually multiple polygons.

11:07

Speaker A

And if you really, really want to try it out, you can also say that there can be some holes in some polygons. So you need to be able to build a structure that allows you to have polygons within polygons.

11:20

Speaker A

So obviously we have polygons for every object. There can be multiple, there can be holes.

11:26

Speaker A

Each polygon is multiple points. Each point is an xy coordinate. So you get what I'm saying. The JSON that you are getting as an output is getting really, really long. And it takes time for those models to generate that JSON.

11:41

Speaker A

So ultimately, it takes like three minutes to annotate a single video with those polygons. Still, I think it's incredibly impressive that we can do something like this with VLMs right now. So shapes does not matter anymore.

11:55

Speaker A

The fact that objects overlap each other does not matter anymore. The fact that the scene is crowded does not matter anymore.

12:03

Speaker A

You can go really crazy and actually solve pretty complex use cases this way. Here is a task that I'm specifically very excited about.

12:12

Speaker A

It's text localization and recognition. So to do good on this task, the model needs to possess two capabilities.

12:19

Speaker A

It to be able to be very good at detection and good at reading the text.

12:23

Speaker A

Which really narrows it down to only a handful of models right now, but I will show you the leaderboard in just a second.

12:31

Speaker A

So here's here's a silly example of Scrabble where I would like to be able to detect individual pieces and read the values.

12:40

Speaker A

But there are a lot more useful use cases for something like this. Old maps, technical drawings, PCB boards.

12:49

Speaker A

And my favorite OCR in the wild when people get really crazy with fonts and colors and shapes. And yeah, sometimes computer vision models still need to be able to read this one. As I told you, it's pretty demanding.

13:04

Speaker A

So far there is maybe like three, four models that are doing good on this task.

13:11

Speaker A

But because it's very useful I suspect those labs are working on it and we should see a lot more good models for this task pretty soon.

13:20

Speaker A

So all of the cases that I showed you so far, those were single image inference cases, where all the information that we wanted to extract was actually visible on just one frame. So we just prompted a single image.

13:34

Speaker A

But we can do a lot more with those models. For example, the tracking that I was showing to you in one of my first slides.

13:44

Speaker A

So the only change here is instead of sending one image, now we need to send multiple images.

13:51

Speaker A

So there is this. Information related to time that we need to capture. We need to be able to associate players in time and saying that this is the same player that it was five seconds ago.

14:02

Speaker A

So to do that, we need to send a series of frames. We probe them from the original video and we just send them.

14:08

Speaker A

Obviously, sometimes the video is too long and you won't be able to send all of those frames at the same time. Then you di just divide them into batches and you need to make sure that the batches overlap so that the information that was captured

14:20

Speaker A

at the end of the first batch is still accessible to the model at the beginning of the second batch so it can propagate through the video.

14:29

Speaker A

So at the end, you will get something like this, like a slideshow with keyframes separated by half a second, maybe a second.

14:37

Speaker A

And believe it or not, all you need to do to convert this into buttery smooth tracking is just linear interpolation.

14:44

Speaker A

So all I did is I took the slideshow that you saw in the very beginning, and I just linearly interpolated the values of the coordinates between those keyframes, and I'm getting tracking right now.

14:56

Speaker A

The beauty of this solution is that to now extract the teams that those players play for, all I change is that single sentence and that prompt.

15:07

Speaker A

So instead of asking only about the individual boxes and tracker IDs, I'm also asking about the teams, and it can propagate with no problem.

15:17

Speaker A

If I would like to extract the numbers on jerseys, I just appended into the prompt and I'm getting more information from that prompt.

15:26

Speaker A

I tokenize those frames anyway, so at this point I'm only paying mostly for the output tokens, so I can extract a lot more information.

15:36

Speaker A

Okay. What's interesting, you can also run re-identification this way. You need to make your prompt a little bit more complex, because what you need to do is you need to be able to reference players that may no longer be in the current

15:53

Speaker A

batch. How you do it is you preserve the crops, showing those players that were visible at some point in time. You just save them in your memory bank and you attach it to your prompt with every batch.

16:06

Speaker A

For basketball game is not a problem. In regular rotation of a single team there's probably like ten players that will play in the game, so collectively twenty players.

16:17

Speaker A

So attaching those twenty small crops in every request it's it's not a big problem at this point. And what you can get is something that I couldn't get a year ago, at least on my own in one month, which is re-identification of players across

16:34

Speaker A

different plays and being able to still assign the proper ID to the player that was sitting on a bench for some time and now is back on the play on the court.

16:49

Speaker A

And now, I think kind of like a cherry on top, something that I tried for the first time last week, and I succeeded the first try.

16:58

Speaker A

I was blown away. So I told you that last year we also built a custom key point model to detect characteristic points of the court, to then use homography to map players' positions from one perspective to another.

17:13

Speaker A

It took us weeks to annotate the data because key point data is very time consuming to create. And in my pipeline, all I did is I changed the prompt to now return x and y coordinates in feet. And I specify that the court is from 0 to 94 feet

17:32

Speaker A

and in other direction, 0 to 50 feet and it returned me the correct coordinates, and this is what we got.

17:40

Speaker A

It's not perfect. My pipeline last year was probably a little bit more accurate, but it's still super impressive.

17:47

Speaker A

But that's the first model that actually succeeded on that task. So I was blown away. So, if there is one information that I would like you guys to stay with after this presentation, is that more and more, complex, advanced, and valuable computer vision tasks are now one prompt away.

18:08

Speaker A

And with Roboflow, all of those models are also accessible with one key. So we have products like workflows, with products like playgrounds, where you can try all of those models, usually the day of their release, with already optimized prompts for computer vision tasks like detection and

18:25

Speaker A

segmentation, because a lot of those models have certain preferences when it comes to coordinate systems and processing of them.

18:34

Speaker A

The only problem with those models is obviously the efficiency. They are slow, in some cases ridiculously slow, and they are expensive.

18:43

Speaker A

For reference, just a single six-second video that I showed you of a single play cost me one dollar to process. So obviously, this cost limits the amount of use cases that you can really try to those that are Either the most expensive or the

19:00

Speaker A

most complex, those use cases that otherwise would not be possible. That's probably the only way you run VLM on computer vision tasks directly in prod.

19:11

Speaker A

But you can still benefit from those models. You can use those models for auto-annotation.

19:17

Speaker A

You can use those models for building your own. Custom weights model that you can deploy on your hardware that can run real time, that can run offline. So that's it.

19:28

Speaker A

I'm very excited about the future. VLMs are getting really, really good at computer vision.

19:34

Speaker A

I think we are finally in this time, in this moment, where it's very hard to predict what will happen next year. But I'm super happy to talk with each of you about it after this presentation.

19:46

Speaker A

So Have a great rest of the conference and see you guys around.

Topics: vision-language models VLM computer vision OCR object detection segmentation object tracking re-identification visual prompting Roboflow

Study this video

- [Flashcards from this video](https://sozai.app/tools/flashcard-maker/?from=transcript&id=93761)
- [Quiz on this video](https://sozai.app/tools/quiz-generator/?from=transcript&id=93761)
- [Crossword from this video](https://sozai.app/tools/crossword-maker/?from=transcript&id=93761)
- [Word search from this video](https://sozai.app/tools/word-search-maker/?from=transcript&id=93761)
- [Study guide from this video](https://sozai.app/tools/study-guide-maker/?from=transcript&id=93761)

Free, made by AI from this transcript. No signup.


---
This is the markdown twin of https://sozai.app/transcript/frontier-ai-models-vision-visual-intelligence-summit-2026/ — the same content, without the markup.
Published by SozAI (https://sozai.app). Reuse and quotation are allowed with attribution and a link back.
Machine-readable index: https://sozai.app/llms.txt · data API: https://sozai.app/api/