Let's build GPT: from scratch, in code, spelled out.
Andrej KarpathyYouTube ↗
Further reading 5
From predicting a token to using a language model →Building a predictor makes its training target concrete. Using a trained model shows how context and instructions put that predictor to work.What does dropout do inside a GPT training loop? →The subnetworks interpretation explains why units are omitted during training. The implementation shows where the random masking happens in an actual model.What does attention add to layered activations? →A feedforward network propagates activations through its layers. Causal attention lets each position combine information from earlier tokens.How can many subnetworks share one set of weights? →Dropping units during training can be viewed as training different subnetworks with shared parameters. The test-time prediction is not necessarily their exact arithmetic mean.When does a text predictor become an assistant? →Next-token prediction teaches continuation. Compare how later training shapes responses to a user; the distinction does not prescribe a single post-training algorithm.
Caption notes
Starting with character prediction, build causal attention and Transformer blocks to predict the next token. The resulting toy GPT illustrates a mechanism, not the whole ChatGPT system.
Original passages · 6
trade off the code book size and the sequence lengthsnodes from the future never talk to the Past because they would give away the answertraining an ensemble of sub networks and then at test time everything is fully enabledit takes the model from being a document completer to a question answerergive it the start of a sequence and it completes the sequenceyou can do weighted aggregations of your past Elements