hAIgopala › AI terms › Rapid Fire (3 terms per short) › Day 45

VRAM

Infrastructure & HardwareChapter 13: The Engine RoomTerm 222 of 253

GPU memory; limits the size of model that can be loaded.

At Sunrise School

Gopal asks

Why can't the school PC run a 70B model?

Bit answers

VRAM, the GPU's memory, must hold the whole model. A 70B model at 16-bit needs about 140 GB; the school PC has 8 GB.

In the real world

70B model needs ~140 GB at 16-bit.

Watch the short

Short coming soonSeason 5 · Day 45Follow @haigopala

Quick check

Quick check. Can a big model run on a GPU with too little VRAM? Yes or no?

Taught together with

CUDA · TPU / NPU

Practise it

Try this term and all 253 others in the free Model Training Arena.

Open the Arena →

Follow Bit daily: YouTube English · తెలుగు · हिंदी  |  Instagram English · తెలుగు · हिंदी