Shopify's Continual Learning Loop with PyTorch and vLLM - Cody Mazza-Anthony - PyTorchCon NA Keynote — Transcript & Summary

Shopify engineer Cody Mazza-Anthony presents the 'flywheel,' a continual learning loop built on PyTorch and vLLM that lets task-specific models beat frontier models at lower cost and latency.

Key Takeaways

A high-quality, expert-validated ground-truth dataset and a judge calibrated to it are the foundation of the whole loop.
Continuous production failure detection, repair, and retraining lets a specialized model outperform frontier models over time while reducing cost.
Prompt compression via learned special tokens and tuned vLLM parameters can significantly reduce latency and increase throughput without changing model weights.

What the video covers

The flywheel is Shopify's continual learning loop built on PyTorch and vLLM, designed to outperform frontier models on specific tasks at a fraction of the cost while improving latency and throughput.
The loop begins with building a reliable, trustworthy ground-truth dataset aligned with what 'good' means for the product, described as the most important and most often done wrong step.
Teams should launch with a frontier model to start collecting data, then use the flywheel after shipping to production.
An LLM judge is built from a rubric of criteria with integer scores (0, 1, 2) and detailed explanations; the more detailed the explanations, the better the judge calibrates.
Two senior product experts blind-label about 50 conversations each, and agreement is measured with Cohen's kappa; disagreements prompt revisiting product understanding before continuing.
The judge is calibrated to human labels using prompt optimizers such as GEPA and ACE, with human agreement treated as the ceiling rather than perfection.
Production failures are detected with the trusted judge, repaired by a council of high-thinking models that replay conversations or regenerate outputs with injected hints, then rescored.
High-scoring repaired examples are added back into the training data, and a new candidate checkpoint is trained.
Using the buyer profile model as a case study, the training set grew from about 29,000 examples and the model quickly outperformed a frontier model on that task.
Inference is optimized with prompt compression (distillation of a long system prompt into learned special tokens via KL divergence against a teacher), which shrinks prefill and improves latency and throughput while keeping model weights frozen.

Chapters

What is the flywheel Shopify presents in this talk?

The flywheel is Shopify's continual learning loop built on PyTorch and vLLM. It cycles through collecting ground truth, judging production outputs, repairing failures, adding them to training data, training new checkpoints, and optimizing inference. The goal is to outperform frontier models on specific tasks at lower cost.

How does the speaker suggest calibrating an LLM judge?

He recommends writing a rubric with criteria, integer scores, and detailed explanations, then having two senior experts blind-label about 50 conversations each and measuring agreement with Cohen's kappa. The judge is then tuned with prompt optimizers like GEPA and ACE until it matches human judgment, with human agreement as the ceiling.

What is prompt compression in this context and how does it help?

Prompt compression, implemented through distillation, trains a student model to replace a long system prompt with a short sequence of learned special tokens, using KL divergence against a teacher with the full prompt. This shrinks prefill, which improves latency and throughput, and the model weights remain frozen so deployment is straightforward.