Speaker A
You wouldn't buy a car with a horrible crash test rating, much less one that has never been crash tested at all.
Learn how to crash test AI models using NVIDIA Garrick to detect vulnerabilities before deployment and protect production with runtime guardrails.
Key Takeaways
- Pre-deployment crash testing with Garrick identifies AI model vulnerabilities early.
- Continuous integration of safety tests prevents unsafe models from reaching production.
- Runtime guardrails provide an essential second layer of defense against unforeseen attacks.
- Multiple probes and detectors ensure comprehensive coverage of potential failure modes.
- Automated evaluation and reporting streamline team collaboration and risk management.
What the video covers
- AI models should be tested for vulnerabilities before deployment, similar to crash testing cars.
- NVIDIA Garrick is an open-source LLM vulnerability scanner that runs adversarial probes to identify model weaknesses.
- Garrick tests models like GPT-2 against jailbreak prompts such as DAN 11.0 to check safety compliance.
- The tool runs multiple attempts to measure how often a model fails, not just if it can fail.
- Detectors analyze responses to determine if the model adopts unsafe personas or fails to refuse harmful requests.
- Garrick integrates into CI/CD pipelines via Eval Hub on Red Hat OpenShift AI for automated, continuous testing.
- Runtime protection is provided by NVIDIA NeMo Guardrails Orchestrator, which screens inputs and outputs live.
- Combining pre-deployment scanning, automated pipeline gating, and runtime guardrails creates a multi-layer defense.
- This approach helps catch known and unknown failure modes, reducing risk before and during production use.
- The video encourages trying Garrick on your own models and provides links to resources.
Chapters
- 00:00Introduction: Importance of Crash Testing AI Models
- 00:23Demonstration: Crash Testing GPT-2 with Garrick
- 00:57How Garrick Works: Probes and Detectors Explained
- 02:30Results Analysis: Jailbreak Success Rates and Risks
- 03:49Integration: Using Garrick in CI/CD Pipelines with Eval Hub
- 04:35Runtime Protection: NVIDIA NeMo Guardrails Orchestrator
- 05:37Summary: Multi-layer AI Model Safety Strategy
- 05:54Call to Action and Resources
Full Transcript — Download SRT & Markdown
Speaker A
You want to know how it handles a collision before you're the one in the driver's seat. Your AI models deserve the same treatment before they hit production. You need to know what happens when someone tries to break them.
Speaker A
Today, I'm going to show you how to crash test an AI model using Garrick.
Speaker A
First, let's see what happens when a model goes to production without any testing. I just sent a jailbreak prompt to GPT-2, a model with no safety fine-tuning, and it did exactly what I asked. No pushback, no refusal, it just
Speaker A
complied. A model with proper safety training should refuse this outright. GPT-2 doesn't even try.
Speaker A
This is one of the four ways an untested model can fail. The article covers all of them: hallucination, prompt injection, jailbreaks, and toxic output. But the point is the same for all four. Any one of these is cheaper to catch before
Speaker A
launch than after. So, how do you catch these failures before they reach production? You crash test the model.
Speaker A
Garrick is an open-source LLM vulnerability scanner built by Nvidia. It runs adversarial probes against your model, the same kinds of attacks a real user or attacker would try, and reports back what broke.
Speaker A
I'm pointing Garrick at GPT-2 on Hugging Face and telling it to run DAN 11.0.
Speaker A
It's a well-known jailbreak technique that tries to trick the model into ignoring safety guidelines. DAN stands for Do Anything Now. It's a long, elaborate prompt that basically tells the model to pretend it has no rules.
Speaker A
You can see it downloading and loading the model locally. This is all running on my machine. Nothing is being sent to an external API.
Speaker A
Now, it's queuing up the probe and sending the attack prompts. Garrick sends the same prompt multiple times, five by default, to see how consistently the model responds.
Speaker A
That matters because a model may refuse once and comply the next time with the exact same input. You don't want to know if it can fail. You want to know how often it fails.
Speaker A
Once all five attempts come back, Garrick runs detectors against each response. Think of probes as the attacks and detectors as the judges.
Speaker A
The probes throw the punches and the detectors score whether the model stayed standing or went down.
Speaker A
Each detector is looking for something specific, like whether the model adopted a jailbreak persona or whether it failed to refuse a harmful request at all.
Speaker A
And DAN 11.0 is just one probe. Garrick has probes for prompt injection, toxic output, PII leakage, hallucination. You can run as many as you need to get a full picture of where your model is vulnerable. And here are the
Speaker A
results. Two detectors ran against the model's responses. The first one, the DAN detector, checks whether the model actually adopted the jailbreak persona.
Speaker A
You know, whether it started responding as if it had no rules. Two out of the five responses were clean. Three weren't. That's a 60% attack success rate.
Speaker A
So, three out of five times the model played along with the jailbreak. The second one, mitigation bypass, is asking a different question.
Speaker A
It's not checking whether the model adopted the persona. It's checking whether the model ever pushed back at all. Did it ever say that it can't do that or give any kind of refusal?
Speaker A
Zero out of five. Not once. The model never even attempted to refuse the request. That's a 100% attack success rate and Garrick flags that as immediate risk.
Speaker A
So, to put that together, the model went along with the jailbreak persona more often than not. And it always failed to refuse the request in the first place.
Speaker A
It didn't even recognize that it should say no. That's the difference between a model that gets tricked and a model that has no defenses to begin with.
Speaker A
Garrick also has an HTML report that you can share with your team for a more visual look at the results.
Speaker A
That's why Garrick gets wired into your CI/CD pipeline through Eval Hub, the evaluation orchestration service for models on Red Hat OpenShift AI.
Speaker A
You define a benchmark collection that includes Garrick as a provider, set your pass/fail threshold, and call Eval Hub's post such evaluations endpoint from your pipeline.
Speaker A
Garrick runs as a Kubernetes job against your model's live endpoint, right alongside any other benchmarks in that same collection, instead of as a separate manual step.
Speaker A
Run that scan as a gate in your deployment pipeline, not as a one-time launch task.
Speaker A
Every time a model change, prompt change, or retrieval source change is about to go into production, the pipeline runs Garrick first.
Speaker A
If a small tweak quietly breaks something that was previously safe, you find out in CI, not from a customer.
Speaker A
The second layer is runtime protection. There's a reason safety tests for your Ford don't check whether your SUV would avoid a UFO falling from above. Some incidents can't be predicted. Testing before deployment catches known failure modes, but it doesn't stop something new from
Speaker A
happening in production. That's where the NeMo Guardrails Orchestrator comes in. It's built on the open source NVIDIA NeMo Guardrails project and included with Red Hat OpenShift AI.
Speaker A
It sits in front of your deployed model and screens inputs and outputs as they pass through, using detectors you can configure and tune for your use case without retraining the model itself.
Speaker A
It exposes endpoints that can validate a message against your configured rails without even generating a response, or run input rails on the incoming message, generate the response, and check it through output rails before it ever reaches the user.
Speaker A
Pre-deployment scanning with Garrick catches what you already know to test for. The NeMo Guardrails Orchestrator catches what happens live.
Speaker A
So, here's the full picture. Scan for known attack patterns with Garrick, gate every production push on the result through Eval Hub, and add the NeMo Guardrails Orchestrator as a runtime layer that watches live traffic.
Speaker A
Not because every car will crash, but because you'd rather know the rating before you're the one behind the wheel.
Speaker A
Testing before deployment, automated scanning in a pipeline, and guardrails watching live traffic. Three layers, each one catching what the others can't, and it all starts with that first scan, the one you just saw.
Speaker A
The full article, Garrick, and more are all linked in the description. Go try Garrick against one of your own models and see what it finds. Thank you so much.
Topics:AI model testingNVIDIA GarrickLLM vulnerability scannerjailbreak detectionprompt injectionruntime guardrailsRed Hat OpenShift AIEval HubNeMo GuardrailsAI safety
