hAIgopala › AI terms › Rapid Fire (3 terms per short) › Day 46

KV Cache

Infrastructure & HardwareChapter 13: The Engine RoomTerm 227 of 253

Stores previous attention calculations to speed up generating each new token.

At Sunrise School

Gopal asks

Long chats seem to slow down.

Bit answers

The KV cache stores attention work for earlier tokens, so I don't reprocess the whole chat for every new word.

In the real world

Not re-reading the whole book for each word.

Watch the short

Short coming soonSeason 5 · Day 46Follow @haigopala

Taught together with

Compute / FLOPs · Distributed Training

Practise it

Try this term and all 253 others in the free Model Training Arena.

Open the Arena →

Follow Bit daily: YouTube English · తెలుగు · हिंदी  |  Instagram English · తెలుగు · हिंदी