VRAM
Infrastructure & HardwareChapter 13: The Engine RoomTerm 222 of 253
GPU memory; limits the size of model that can be loaded.
At Sunrise School
Gopal asksWhy can't the school PC run a 70B model?
Bit answersVRAM, the GPU's memory, must hold the whole model. A 70B model at 16-bit needs about 140 GB; the school PC has 8 GB.
In the real world70B model needs ~140 GB at 16-bit.
Watch the short
Short coming soonSeason 5 · Day 45Follow @haigopala
Short coming soonSeason 5 · Day 45Follow @haigopala.telugu
Short coming soonSeason 5 · Day 45Follow @haigopala.hindi
Quick check
Quick check. Can a big model run on a GPU with too little VRAM? Yes or no?
No! The whole model has to fit in memory.
Taught together with
Practise it
Try this term and all 253 others in the free Model Training Arena.
Open the Arena →Follow Bit daily: YouTube English · తెలుగు · हिंदी | Instagram English · తెలుగు · हिंदी