When does input become an instruction?

A typo can still be valid code. A data format can turn out to express computation. Follow an input inward and identify who decides how to interpret it.

Connected readings

Read the essay ↓

Questions in this reading · 4

When is a typo a valid program?
Should a parser repair invalid input?
When does an expressive format become an attack surface?
Where should untrusted input stop?

When is a typo a valid program?

Features can interact so that an unintended expression still parses. The language accepts it, leaving the programmer to discover the changed meaning.

Compare original passages and timestamps · 2

all these things tend to work together to mask errors.

JavaScript: The Good Parts · ▶ 33:55–34:22

features that individually seem to make sense but brought together might be terrible

Modern Dictionaries by Raymond Hettinger · ▶ 63:47–64:01

Should a parser repair invalid input?

Correcting an input can make a system convenient, but another component may interpret the correction differently. Rejection makes that boundary explicit.

Compare original passages and timestamps · 2

correcting what they see back to the nearest valid neighbor

But what are Hamming codes? The origin of error correction · ▶ 3:28–3:37

if you haven't recognized it don't goddamn process it

28c3: The Science of Insecurity · ▶ 25:04–25:15

When does an expressive format become an attack surface?

A format can admit computation its designer did not anticipate. That changes what it means to process input supplied by someone else.

Compare original passages and timestamps · 2

essentially imitate any transition function of any other touring machine

On The Turing Completeness of PowerPoint (SIGBOVIK) · ▶ 4:07–4:17

treat input handling computational power as privilege and reduce that privilege

28c3: The Science of Insecurity · ▶ 34:49–34:59

Where should untrusted input stop?

Compare an instruction embedded in model input with data entering a parser. Both examples ask who gets to decide what is treated as an instruction.

Compare original passages and timestamps · 2

prompt injection is about hijacking the large language model giving it what looks like new instructions

[1hr Talk] Intro to Large Language Models · ▶ 52:11–52:21

if you haven't recognized it don't goddamn process it

28c3: The Science of Insecurity · ▶ 25:04–25:15

Try it on your own example

Choose a parser boundary in your system. Does it reject, repair or execute ambiguous input?