hAIgopala › AI terms › Rapid Fire (3 terms per short) › Day 13

Tokenization

Generative AI & LLMsChapter 4: Choosing the LLMTerm 62 of 253

Splitting text into tokens before feeding it to the model.

At Sunrise School

Gopal asks

How is a message turned into tokens?

Bit answers

Tokenization splits text into known pieces. “Kondapur” becomes “Kond” + “apur” because it's a rare word.

In the real world

Chopping a sentence into Lego pieces.

Watch the short

Short coming soonSeason 5 · Day 13Follow @haigopala

Taught together with

GAN (Generative Adversarial Network) · Small Language Model (SLM)

Practise it

Try this term and all 253 others in the free Model Training Arena.

Open the Arena →

Follow Bit daily: YouTube English · తెలుగు · हिंदी  |  Instagram English · తెలుగు · हिंदी