Brains, Sex, and Machine Learning
Google TechTalksYouTube ↗
Further reading 2
What does dropout do inside a GPT training loop? →The subnetworks interpretation explains why units are omitted during training. The implementation shows where the random masking happens in an actual model.How can many subnetworks share one set of weights? →Dropping units during training can be viewed as training different subnetworks with shared parameters. The test-time prediction is not necessarily their exact arithmetic mean.
Caption notes
Randomly dropping units during training discourages co-adaptation. The lecture relates dropout to averaging many subnetworks that share weights.
Original passages · 5
all of these architectures are sharing the same weights, which is regularizing them a lot.that's not the same as running all possible nets through an averaging them, taking the geometric mean.do something that's generally helpful. That will cause you to do something different from what they're doing, but not relyingpick a task at random, and this willThat means these nets ought to be very good in genetic algorithms