KV Cache
Infrastructure & HardwareChapter 13: The Engine RoomTerm 227 of 253
Stores previous attention calculations to speed up generating each new token.
At Sunrise School
Gopal asksLong chats seem to slow down.
Bit answersThe KV cache stores attention work for earlier tokens, so I don't reprocess the whole chat for every new word.
In the real worldNot re-reading the whole book for each word.
Watch the short
Short coming soonSeason 5 · Day 46Follow @haigopala
Short coming soonSeason 5 · Day 46Follow @haigopala.telugu
Short coming soonSeason 5 · Day 46Follow @haigopala.hindi
Taught together with
Practise it
Try this term and all 253 others in the free Model Training Arena.
Open the Arena →Follow Bit daily: YouTube English · తెలుగు · हिंदी | Instagram English · తెలుగు · हिंदी