HackerIlya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
Dwarkesh PatelYouTube ↗
Further reading 2
How far is next-token prediction from answering a person? →Separate three questions: how an input becomes a representation, how training changes a network and how a capable text predictor becomes an assistant. An explanation at one level does not settle the others.This video helps answer · What is the next unit, and how far back can prediction look?How can you tell a correct answer from understanding? →Ask two people to explain their reasoning, then change the presentation and ask them to reconstruct the answer. Peer instruction and geometric intuition offer different ways to expose gaps.This video helps answer · Filling in a pronoun and explaining its referent
Caption notes
Sutskever proposes that sufficiently good next-token prediction may require learning about the world that generated the text. It is a hypothesis about how a simple training task could yield complex capabilities.
Public machine transcript (may contain transcription errors) ↗Original passages · 2
The underlying reality that led to the creation of that tokenWhat is it about people that creates their behaviors