Attention / Self-Attention
Generative AI & LLMsChapter 4: Choosing the LLMTerm 69 of 253
Lets the model weigh how relevant each word is to every other word in the input.
At Sunrise School
Gopal asks“Can I change my son's bus, he is moving to Kondapur.” How does Bit link “he” to “son”?
Bit answersAttention lets every word look at every other word. “he” attends strongly to “son”, and “bus” to “Kondapur”.
In the real world'Bank' relates to 'river' in 'river bank'.
Watch the short
Short coming soonSeason 2 · Day 15Follow @haigopala
Short coming soonSeason 2 · Day 15Follow @haigopala.telugu
Short coming soonSeason 2 · Day 15Follow @haigopala.hindi
Quick check
Quick check. Does attention look only at the word right before? Yes or no?
No! Every word can look at every other word.
Practise it
Try this term and all 253 others in the free Model Training Arena.
Open the Arena →Follow Bit daily: YouTube English · తెలుగు · हिंदी | Instagram English · తెలుగు · हिंदी