Speaker A
This is my data set, and you can see that I'm detecting here four things in the guitar.
Learn how Felipe Tomino built an AI guitar teacher using Roboflow's platform and serverless video streaming API for real-time guitar scale detection.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Roboflow’s platform enables quick dataset creation, auto-labeling, and model training for custom object detection.
- Serverless video streaming API provides fast, real-time predictions suitable for live applications.
- Custom AI models can be used to enhance musical instrument learning by visualizing scales on the instrument.
- Even a small dataset (48 images) can be effective with the right model and auto-labeling tools.
- Open sourcing projects can help the community build similar AI-powered educational tools.
What the video covers
- Felipe Tomino, a Roboflow employee and guitar player, created an AI guitar teacher to help study scales more easily.
- The project detects four guitar parts: sound hole, nut, fret wires, and fretboard using a custom dataset of 48 images.
- He trained models using Roboflow's RF-DETR medium and nano architectures, leveraging auto-labeling to speed up annotation.
- The application uses Roboflow’s serverless video streaming API to process live webcam video for real-time predictions.
- The AI overlays detected guitar parts and scale notes on a canvas in the front-end application.
- Users can select different scales and root notes to visualize on their guitar fretboard.
- The project demonstrates fast live video performance compared to traditional API calls.
- Felipe plans to release the code publicly on GitHub after the webinar for others to build upon.
- The workflow includes uploading videos, extracting frames, training models, and deploying with serverless API.
- The tool aims to make practicing guitar scales less boring and more interactive.
Chapters
- 00:00Introduction to Dataset and Guitar Part Detection
- 00:40About Felipe Tomino and Project Motivation
- 01:34Challenges of Studying Guitar Scales
- 02:31Uploading and Processing Videos in Roboflow
- 03:09Model Training with RF-DETR and Auto Labeling
- 04:36Using Roboflow Serverless Video Streaming API
- 05:46Codebase Overview and Upcoming GitHub Release
- 08:35Live Demo and Application Features
- 10:51Using Different Scales and Final Thoughts
Full Transcript — Download SRT & Markdown
Speaker A
The sound hole, the nut, the fret wires, and the fretboard itself. I opened the WebRTC stream with Roboflow serverless API, and then Roboflow returns to me the predictions of the image I'm sending them from my webcam,
Speaker A
and I print this on a canvas overlay in my application, in my front-end application. Voila. You have the scale right here. So, now I can [music] just start studying its shape and see how it goes.
Speaker A
[music] [music] So, hello everyone. I am Tomino, Felipe Tomino, however you want to call me. I work here at Roboflow for about 3 months now. I built this project you just saw, a quick preview
Speaker A
in order to get, you know, Roboflow application a little better when I started working here. So, I tried to use as much of Roboflow platform as I could to understand where I was going to work on, and to
Speaker A
make things better for Roboflow users. So, I wanted to do that, and I thought about this project.
Speaker A
I play the guitar for almost 20 years now, but I play it kind of well, but I'm still lazy since I started working on it, and I never stopped to study the scales because it's
Speaker A
if some of you play an instrument, you know how boring it is to just look at some spreadsheets and try to imagine where the notes are on your guitar, on your keyboard, and
Speaker A
yeah. So, to sum up, I'm lazy, and I built this to help me study in an easier way. So, that's pretty much it. And how did I do that? Before I started to explain those things to you, I will do some
Speaker A
upload here in Roboflow to show you how it works. I don't want to spend too much time on this, but when you upload a video, for example, you can ask for how many frames you want to
Speaker A
extract from the video, and it's already processed, so it's fast, way faster than I expected.
Speaker A
So, let's dive into how I built this. So, this is my data set. You can see actually that it has only 48 images, and that was like a magic number in my head, but it should work with even fewer
Speaker A
images because I am using RF-DETR. I tested two versions of it, the RF-DETR medium and the RF-DETR nano. If you wonder why the RF at the beginning, it's because this is a Roboflow model. So, it's our RF-DETR model. I used it to
Speaker A
build this whole application. And yeah, this is my data set, and I did upload those images here, so you can see this. Let's try auto label to see how it goes.
Speaker A
Oh, it's using Sentry. Oh, no, it's using my model, actually. So, once you have your model trained, and I had labeled just a few images, I could auto label probably like 40 of my 48 images,
Speaker A
and it gives me this. So, then I just had to kind of approve the auto label stuff. So, it was pretty easy actually to label those. It could also get boring and take a long time to do this manually, but Roboflow auto label works pretty good and worked pretty good for me, too.
Speaker A
And you can see that I'm detecting here four things in the guitar. The sound hole, the nut, the fret wires, which can also be called fret, but fret wire when you use Sentry gives way better results, and the fretboard
Speaker A
itself, which is the whole fretboard. Here in Brazil, we call it in Portuguese, of course, but we call it the guitar's arm.
Speaker A
So, I used these four objects to plot the notes on the screen. So, I created my model using RF-DETR nano. I trained both models, I guess, in the same, yeah, both with the
Speaker A
48 images data set. And after they were trained, we can take a look here after, but after they were trained, I built a simple workflow in Roboflow.
Speaker A
It doesn't have much here. It's just the model. It makes the predictions and returns them in a JSON format, and I did this workflow so I could use Roboflow serverless API. More specifically,
Speaker A
serverless video streaming API. With the video streaming API serverless, you can use your webcam, RTSP cameras, our WebRTC videos, and just get the results back. You could even add
Speaker A
a block to visualize the results in the image you received. But I didn't want to do that. I just wanted the predictions themselves.
Speaker A
So, yeah. I am using Roboflow serverless video streaming API for that. And I'm using this because the live video performance of this is pretty good.
Speaker A
Way better than using the usual API, which can take a little longer. But when we use the streaming API, it makes things really fast for when you need those live real-time predictions.
Speaker A
So, that's pretty much it. Here I have the auto label. We can check some of the images. Let's check just these four, and they recognize everything I need to. The frets it doesn't see, are actually hidden behind my hand. So, we can see how that works. And just doing this, I could just trigger auto label, and it would label my 12 images itself. So, then I could use it
Speaker A
to train again and improve my model. And yeah, that's pretty much what I did to build this.
Speaker A
I don't want to spend too much time on this, so just to give you a peek of the code base.
Speaker A
It's a simple code base with an HTML file, some JavaScript that is running on the client side.
Speaker A
And the most important part here is that I am using the Roboflow serverless streaming API. Here I just have to pass through my workspace name and the workflow I built that I just showed you. And here I
Speaker A
have some other boring stuff about code. If anyone is interested in code specifically, it's not public yet, but I will make the repository public on GitHub right after the webinar, and you will be able
Speaker A
to see the whole code base, and you can just use it for your own purposes as a reference to build things with Roboflow.
Speaker A
But yeah, what I did was I built this. Let's go back to the diagram. So, I have my webcam. I open it in my browser in the application you were seeing at the beginning.
Speaker A
I opened the WebRTC stream with Roboflow serverless API, and I have my own server just to do this handshake to open the connection with Roboflow serverless, and then Roboflow returns to me the predictions of the image I'm sending
Speaker A
them from my webcam, and I print this on a canvas overlay in my application, in my front-end application. And we have the final result, which I will show to you right now.
Speaker A
Do a quick demo. So, it's a pretty simple application. You just choose your webcam or whatever camera you want to use here.
Speaker A
You can choose which scale you want to see, which root note you want. So, you can see every single one of these scales in whatever root notes you want. The default patterns and let's do a quick
Speaker A
demo on it. So, the only thing you need to do is actually show the fretboard and see, as I told you, I was also using the nut and the sound hole.
Speaker A
So, to get better coordinates on where to plot the fretboard, I will hide it just for a while so you can see the start of this process. So, here I'm just connecting to the Roboflow
Speaker A
streaming serverless API and now it's running and now it will start to try to detect my guitar. So, you can see here that I have the fret zero, which would be the nut, and it's already recognizing my fret here.
Speaker A
This one is flickering a little bit. Of course, I'm hiding it a little with my finger, and so it asks you to just show the whole guitar and voila, you have the scale right here and you can study pretty much
Speaker A
any scale you want. Of course, there are many more scales that I haven't put here in the application yet. When I make it public, feel free to write some more here.
Speaker A
And let's say let's change to a weird scale. I wouldn’t...
Speaker A
uh any scale you want. Of course, there are many more scales that I haven't put here in the application yet. When I make it public, public, feel free to to write some more here.
Speaker A
And let's say let's change like for a weird scale. I wouldn't know this uh from the top of my head. So, I just get here. Here in red, you can see the root note and then you just start like
Speaker A
uh running through the scale starting here and then I here I have the full shape uh of course, because I know this shape only and let's get a shape that I don't know. Let's say this one.
Speaker A
Um [music] Yeah. So, you see that I really don't know this scale from the top of my head, the shape. But, the idea is this uh is to study the scales in a much easier way like really as it was a mirror and
Speaker A
you're looking at a mirror and you can just see the notes magically drawn into uh your fretboard. Before we end this, uh actually, my main motivation uh to build this is because I usually play uh at my church and I use uh this other
Speaker A
guitar here where I use the DADGAD uh tuning, which I like so much. And why do I like it?
Speaker A
Because you don't have to do anything for it to sound good. Let's put this in D. It's kind of buggy right now because I changed the guitars, but when that happens, you just reset it and make things uh work for your new guitar. So,
Speaker A
it's pretty fast and I was saying why do I like this tuning? Uh again, because I'm lazy and if I just play it uh open, it already sounds [music] beautiful and it's not much difficult like I could just
Speaker A
use those [music] notes from the top here and start like doing stuff and [music] it will sound kind of beautiful.
Speaker A
Yeah, let's say [music] like this. And [singing] yeah, here I can study the scales on uh a different tuning that I like, but again, I'm lazy, so I never stop it to really study the scales on this different tuning. And so, you can move
Speaker A
your guitar. It will try to keep up with you. Yeah, seems right here. And you can just like study the shapes in this new Oop yeah.
Speaker A
Uh and that's it. This is the the application.
Topics:AI guitar teacherRoboflowRF-DETRserverless APIlive video streamingguitar scale detectionauto labelingmachine learningcomputer visionmusic education
