HackerThe spelled-out intro to neural networks and backpropagation: building micrograd
Andrej KarpathyYouTube ↗
Further reading 2
Why do tests pass when the program is wrong? →Gary Bernhardt’s isolated tests and Andrej Karpathy’s neural network both deliver good news when the code is wrong.How far is next-token prediction from answering a person? →Separate three questions: how an input becomes a representation, how training changes a network and how a capable text predictor becomes an assistant. An explanation at one level does not settle the others.This video helps answer · How does a network change after a wrong answer?
Caption notes
Build automatic differentiation from scalar operations, apply the chain rule backward and train a small classifier. Reusing accumulated gradients reveals why training loops clear them.
Original passages · 6
back propagation is is just a recursive application of chain rule backwards through the computation graphwe have to accumulate these gradients these gradients addthese gradients just add up and they never get reset to zeroyou may have lots of bugs in the code and uh your network might actually workefficiently evaluate the gradient of some kind of a loss functionchanging this variable is not changing the loss function