What is Distillation?
TL;DR
A technique that transfers knowledge from large AI models to smaller ones, creating lightweight yet high-performing models.
Distillation: Definition & Explanation
Distillation (Knowledge Distillation) is a technique that transfers the knowledge of a large 'teacher model' to a smaller 'student model.' By training the student model to mimic the teacher model's outputs (probability distributions), it can achieve performance close to the teacher model with far fewer parameters. For example, GPT-4-level knowledge can be distilled into a small model that runs on a smartphone. OpenAI's GPT-4o mini, Google's Gemma, and Meta's Llama 3.2 are examples of lightweight models developed using distillation. This technology contributes significantly to deploying AI on edge devices and reducing API usage costs.