Tokenization
Generative AI & LLMsChapter 4: Choosing the LLMTerm 62 of 253
Splitting text into tokens before feeding it to the model.
At Sunrise School
Gopal asksHow is a message turned into tokens?
Bit answersTokenization splits text into known pieces. “Kondapur” becomes “Kond” + “apur” because it's a rare word.
In the real worldChopping a sentence into Lego pieces.
Watch the short
Short coming soonSeason 5 · Day 13Follow @haigopala
Short coming soonSeason 5 · Day 13Follow @haigopala.telugu
Short coming soonSeason 5 · Day 13Follow @haigopala.hindi
Taught together with
GAN (Generative Adversarial Network) · Small Language Model (SLM)
Practise it
Try this term and all 253 others in the free Model Training Arena.
Open the Arena →Follow Bit daily: YouTube English · తెలుగు · हिंदी | Instagram English · తెలుగు · हिंदी