hAIgopala › AI terms › Rapid Fire (3 terms per short) › Day 25

Distillation

Model CustomisationChapter 7: Speaking in the School's VoiceTerm 124 of 253

Training a smaller 'student' model to mimic a larger 'teacher' model.

At Sunrise School

Gopal asks

The big model is slow and costly for 600 chats.

Bit answers

Distillation: a small student model learns from the big teacher's answers, keeping most of the quality at a fraction of the cost.

In the real world

Expert teaches a junior to handle 90% of cases.

Watch the short

Short coming soonSeason 5 · Day 25Follow @haigopala

Taught together with

DPO (Direct Preference Optimisation) · GRPO

Practise it

Try this term and all 253 others in the free Model Training Arena.

Open the Arena →

Follow Bit daily: YouTube English · తెలుగు · हिंदी  |  Instagram English · తెలుగు · हिंदी