HackerTransformers, the tech behind LLMs | Deep Learning Chapter 5
3Blue1BrownYouTube ↗
Further reading 1
Caption notes
Follow tokens through embeddings, attention, feedforward layers and sampling. The weights learned during training differ from the representations computed on each run.
Original passages · 5
The attention block is what's responsible for figuring out which words in context are relevant to updating the meanings of which other words,The weights are the actual brains, they are the things learned during training,A higher temperature gives it a chance to choose less likely words, but it comes with a risk.The weights are the actual brains, they are the things learneddirections in the space have a kind of semantic meaning