Laya 진짜 추천합니다. Jev를 오픈소스로 쓰는법

Explore training and testing of open-source AI model Laya for Korean CS tasks, comparing it with Jeb for accuracy and cost efficiency.

Key Takeaways

Laya can be effectively trained on consumer hardware to achieve high accuracy in Korean CS classification.
Routing user queries to different AI models optimizes cost and response time without sacrificing quality.
Jeb outperforms Laya in accuracy but at higher cost and possibly slower response times.
Incremental training data significantly improves Laya's performance, with diminishing returns after 300 cases.
Locally hosted open-source models like Laya offer a customizable and cost-effective alternative for specific tasks.

What the video covers

Introduces Laya, an open-source AI model trained on a MacBook for Korean customer support classification.
Training Laya on 1,000 cases increased accuracy from 33% to 86% in 48 minutes.
Explains the importance of routing queries to appropriate AI models to balance cost, speed, and accuracy.
Compares Laya's performance with Jeb, a high-performing model achieving 100% accuracy on the same test.
Demonstrates incremental training of Laya with 100, 300, 1,000, and 10,000 cases, showing accuracy improvements.
Training Laya on 10,000 cases took nearly 8 hours and reached 99.7% accuracy, close to Jeb's performance.
Highlights the need for fast, inexpensive models for simple queries and powerful models for complex ones.
Discusses practical CS examples like delivery inquiries, returns, and disputes to illustrate routing benefits.
Mentions speed testing, with Jeb API response times around 0.5 seconds, emphasizing trade-offs between speed and accuracy.
Concludes that Laya is a promising customizable alternative to Jeb, especially for Korean language tasks and local hosting.

Chapters

What hardware was used to train the Laya model?

The Laya model was trained on a standard M4 Pro MacBook with 24GB of RAM, demonstrating that high-performance AI training can be done on consumer-grade hardware.

How does Laya's accuracy compare to Jeb's?

After training on 10,000 cases, Laya achieved 99.7% accuracy, which is very close to Jeb's perfect 100% accuracy on the same Korean customer support classification task.

Why is routing important in AI-powered customer support?

Routing helps direct user queries to the appropriate AI model based on complexity, which optimizes response quality, reduces costs, and minimizes waiting times by avoiding overuse of expensive large models.

Full Transcript — Download SRT & Markdown

Speaker A

So, the hottest model these days is Jeb. I heard there's an open-source AI model called Laya that can replace Jeb, so I trained it myself and put it to the test. I trained Laya on my completely ordinary, everyday M4 Pro MacBook with only 24GB of RAM, and the accuracy on my pre-made Korean customer support CS test jumped from 33% to 86%.

Speaker A

It only took 48 minutes to train. That was for training on 1,000 entries. And here, I’ve even pushed it up to 10,000 entries. Since I covered Jeb recently, many of you are likely using it; Laya shows immense potential if you

Speaker A

want a model that can be optimized for your specific tasks rather than a general-purpose one like Jeb, and it’s free to use because it’s locally hosted. In today’s video, I’ll show you the training process beyond 1,000 entries up to 10,000, the

Speaker A

testing, and a comparison with Jeb. So easy. Now, this is something I already explained in the Jeb video, but let’s go over it once more. Why is routing necessary? When building an AI service, implementation is very easy if we can

Speaker A

send all user requests to the highest-performing LLM. However, if we route every simple query through the best model, the API costs and response waiting times will skyrocket.

Speaker A

Conversely, if we assign everything to cheap and fast models, the quality of answers for complex requests might be disappointing. That’s why we look at the request and categorize it first. We determine if the inquiry can be handled simply, if a more powerful model is

Speaker A

needed, if a human needs to check it, or if no reply is needed at all. To put it briefly, routing is essentially deciding where a request should go for these specific purposes. Therefore, you can use Jeb, which I introduced in the

Speaker A

last video, for this exact purpose of routing. By placing Jeb in front of the LLM that will write the actual response and giving it the request and the options, Jeb chooses the path on its own. Then, the program sends the

Speaker A

request to the categorized model based on Jeb's result. This allows us to configure the system to use large models only for the requests that truly need them. This means you can save a significant amount of time and money.

Speaker A

Well, I’ve already tested all of this, so you can just refer to my video.

Speaker A

Now, I mentioned CS as an example last time, and the reason CS is so well-suited for routing is that we collect data through various consultations, and the tasks required for each inquiry are very different.

Speaker A

Also, because such a massive amount of data comes in, if we’re not careful, we could end up wasting a lot of money.

Speaker A

This means we need a very fast and inexpensive model at the front end that can perform accurate classification and judgment. For example, if we receive a question asking where a delivery is, we can simply look up the order

Speaker A

information and provide the answer. And if a customer says, "I want to return this," we can just route them to the process of checking if a return is possible. If it’s a dispute that seems difficult to resolve based on

Speaker A

regulations alone, we can configure it to hand it off directly to a human agent. The theory is simple, but it gets a bit confusing if we try to judge things just by looking at the words.

Speaker A

That’s why we have no choice but to use AI; for example, "Shipping is late" and "Shipping is late, so please cancel my order" both mention shipping, but...

Speaker A

The second customer definitely wants to cancel their order. If we route this incorrectly, no matter how well the subsequent backend handles the response, the customer might not get the outcome they want. Or, if we send it to

Speaker A

the wrong pipeline, it could result in a bizarre response. It’s already been proven that Llama is better at English, but our goal is to focus on Korean-based tasks, so I had Llama read Korean inquiries and choose from six

Speaker A

categories, such as delivery, cancellation, exchange, or return. I have already set the classification criteria and the correct answers in advance. We’re going to compare how much the number of incorrectly routed inquiries has decreased before and after training. So, my initial

Speaker A

prediction was that if I taught it about 1,000 cases, it might reach a similar level to me. That’s why I created synthetic data by pairing up Korean inquiries and their correct answers that would actually occur in an online shopping mall. So, I set aside

Speaker A

250 test data points separately from the training set for a final frozen test, and I compared the performance at each learning stage using this same test. So, I tested how accurately Raya classifies inquiries and compared it directly with Zeb. First, I checked the

Speaker A

baseline to see how many classification answers Raya got right before any training. Well, 82 out of 250. So, the accuracy rate was only 32.8%. This means we can judge that Raya's basic classification performance without training isn't that great. So, based on

Speaker A

the classification criteria I set, it only got about one out of three right. And 32.8% is obviously a figure we cannot use right away. So, we trained it in stages, comparing how the accuracy improved by training with 100, 300, and 1,000 cases. When we train it,

Speaker A

we provide the actual CS inquiry along with the correct classification. We are giving it the correct answer. Then, we compare the model's prediction with the actual answer. You can think of it as a method of adjusting weights little by

Speaker A

little to make decisions correctly. By saving the model we trained this way and having it retake the 250-question test we set aside, we can judge how much the performance has improved. So, it took 4 minutes to train with 100

Speaker A

cases. The accuracy rate went from 32.8% up to a whopping. It increased. So, it achieved this result in just 4 minutes.

Speaker A

On the same 250-question test, it originally got 82 questions right. But this increased significantly to 172. It means that with just 100 training data points, it got 90 more questions right on the test. So, to further improve the

Speaker A

performance, we tried training it with 300 cases. It took 15 minutes to train with 300 cases. The accuracy became 83.2%. And when I tried training it with 1,000 cases, it took 48 minutes, and the accuracy rate rose to 86.4%.

Speaker A

When conducting this experiment, I didn't keep adding data to the model that was initially trained with 100 cases. Each one started from a different base model baseline. We just provided different amounts of training data. 100 cases, 300 cases, and 1,000

Speaker A

cases like that. The actual frozen test questions used for evaluation were kept the same. So, while the model couldn't get a single one right at first, we can now achieve results where it gets 216 out of 250 correct at once. And the

Speaker A

fact is, I easily created a model that fits our classification criteria much better in under an hour on my standard MacBook. Now, although we can predict this to some extent by looking at the numbers, the jump in accuracy was

Speaker A

smaller when increasing from 300 to 1,000 cases than from 100 to 300. Even though we added more data, the score only rose slightly from 83.2% to 86.4%.

Speaker A

So, if we give this same problem to 'Je', how well will it perform? Well, 'Je' got 250 out of 250, getting every single one right. So the accuracy was 100%. Personally, I thought it would do well, but I didn't expect it to get

Speaker A

every single question right. But it did an amazing job. It understood the intent well; I initially thought it might get close with 1,000 training examples, but when I showed it the test, there was definitely a clear performance gap. I really felt that it

Speaker A

is definitely a very good model. Now, we have only trained it on 1,000 cases.

Speaker A

In 'La's' case, we can continue to train it infinitely. So, if we compare them, 'La' got 216 correct with 1,000 training examples, while 'Je' got 250.

Speaker A

So, I wondered what would happen if we trained it further, and I thought, why not try training it on an extreme 10,000 cases instead of 1,000? So this time, I trained it on the same MacBook for 7 hours and 52 minutes. I fed it

Speaker A

10,000 items of data. Before, it finished in un

Speaker A

10,000 items of data. Before, it finished in under an hour, but this time it ran for nearly 8 hours. But actually, 8 hours isn't really that much time. Especially when we consider that we are training it on such a small

Speaker A

computer. Next, we had it solve 600 validation problems to verify the results, and the accuracy of 'La' without any prior training on these was 35.8%. So, it's very similar to before.

Speaker A

But after training it on 10,000 cases, the accuracy went up to 99.7%. Meaning it got 598 out of 600 correct. So, this means we are getting very close to 100% of the capability that Zephyr displayed . So, just by training it for nearly 8

Speaker A

hours, we were able to raise our accuracy rate in this one area we tested to be very close to Zephyr's performance. Since Zephyr is a general model, it will perform very well in other areas without training, so I

Speaker A

can't say we've completely caught up to Zephyr with Llama, but we have sufficiently proven that if we train Llama on a specific domain we want to use it for, we can achieve a very high accuracy rate depending on the volume

Speaker A

and quality of data. Now, if this were a real-world scenario, we could even push the accuracy to an extreme level by having the system re-train on the problems that Llama keeps getting wrong . We can understand this more

Speaker A

concretely by looking at the actual problems it got wrong; for example, there was an inquiry asking, "Where do I find the return request form for the mug?" The information that a return had already been requested was included, and by our standards, it should have

Speaker A

been sent to "Exchanges/Returns," but Llama classified this as "Shipping." Then there was another inquiry: "I want to compare keyboard color options." This was a situation where the user was comparing colors before purchasing, but it was routed to "Exchanges/Returns"

Speaker A

instead of "Product Inquiry." The first case is something where it's understandable how it could get it wrong, but the second case was a completely absurd mistake. So, in these cases, we wouldn't just have the CST team correct the answers; we could

Speaker A

gather together these potentially confusing scenarios. We could place these cases side-by-side—the case of asking about colors before purchase versus the case of exchanging a received product for a different color —and guide the model to learn what differences lead to the different

Speaker A

classifications. Then, we can re-train it with the augmented data and perform another round of testing on new inquiries that weren't included in the training to see if it classifies them well. So, I believe that because we can use Llama by tailoring it directly to

Speaker A

our business standards and correcting it as we go, it is definitely worth considering if you need a router for your front-end operations. And there's the advantage of being able to run it locally. If there are specific areas where zephyr might struggle, we can

Speaker A

train Llama directly, which gives us the benefit of being able to keep improving our results. Now, routing is a step we basically have to go through before we even start generating an answer, right? Right? If we have high

Speaker A

latency here, the time it takes for the LLM to start writing the answer on the backend will be delayed by that much.

Speaker A

Obviously, performance aside, the speed also needs to be extremely fast. So, I recorded the response times, and for the zephyr API, the median time from sending the request to getting the result was about 0.5 seconds. Now, I used zephyr through an AI gateway. An

Speaker A

AI gateway. And for the 1,000 cases trained on Llama, it only took 0.11 seconds to classify a single case. Of course, this doesn't mean the zephyr model itself is slow. Since zephyr requires going through the internet, the time includes server processing and

Speaker A

round-trip travel, which inevitably leads to higher latency, whereas Llama is the time taken while the model is running locally. So, if we run Llama on the same computer or network, we can handle the classification work without having to send data back and forth

Speaker A

externally. This means we can configure customer inquiries to be classified internally instead of sending them to an external model server, which makes this a much more meaningful option if you're dealing with sensitive company data. Of course, in this process, even

Speaker A

when I trained Llama with 10,000 cases, zephyr was still generally more accurate. So, you'll need to consider accuracy, response time, and whether you actually need to train it yourself when making a choice, especially for tasks where mistakes are not an option;

Speaker A

you need to verify how well they handle those cases. However, as we were able to see in this experiment, a huge advantage was being able to train and run our own model based on our classification criteria even on just an

Speaker A

M4 Pro MacBook with 24GB of RAM. And there wasn't even a special GPU environment involved. It only took 48 minutes to train on 1,000 cases, and during the test, the accuracy rate jumped from 32.8%all the way up to 86.4

Speaker A

%. So, if you want to build your own router that perfectly fits your workflow, I think Raya is definitely an option worth considering, and personally, I’ve been really enjoying experimenting with it over the past few days for this video. It was a lot of

Speaker A

fun, and I hope this helps you out; if it did, I’d really appreciate it if you could subscribe, like, turn on notifications, and even say hi! Alright , I'll see you in the next video with even better content. Thank you. Soji.

Speaker A

Yeah.

Topics:Laya AIJeb AIopen source AIcustomer support AIKorean language AIAI model trainingAI routingLLM comparisonAI cost optimizationlocal AI hosting

Get More with the SozAI App

Transcribe recordings, audio files, and YouTube videos — with AI summaries and speaker detection. 30 minutes free.

App Store Google Play

Or transcribe another YouTube video here →

Free tools: TXT to SRT · SRT Validator · Merge SRT · Subtitle to Text · All tools