From results to evidence

Testing & evaluation

A passing test, a top score and a successful experiment leave a question open: what does the result actually establish? Follow the code, the task and the measurement back to their assumptions.

12readings · three paths

NEW HERE? START HERE

Why do tests pass when the program is wrong?

Start with a familiar green test. Separate matching an expectation from getting that expectation right.

Start reading ↗

It passed. What could still be wrong?

Ask what was checked, then ask how the checker was checked.

01Why do tests pass when the program is wrong?Gary Bernhardt’s isolated tests and Andrej Karpathy’s neural network both deliver good news when the code is wrong.6 sources · traceable passages 02Why does testing a business rule need a database?Separate who qualifies for a reminder from whether the reminder can be delivered.2 sources · traceable passages 03Can today’s record explain yesterday’s decision?Keep the value a decision actually used so that later changes do not erase its explanation.9 sources · traceable passages 04The machine says it passed. Who can check?Follow a ballot into a voting machine, then follow a read request across a motherboard. At each step ask what was examined, who delivered the answer and whether someone else can inspect the same evidence.17 sources · traceable passages 05How do you turn a one-off failure into a replayable test?Suppose an acknowledged transfer disappears during a leader change. Six engineering talks separate reproducible inputs, shrinkable operations and invariants, then ask whether the fix actually exercises the original failure.6 sources · traceable passages

Does a better score mean a better outcome?

Compare what a task covers with what happens when a system learns to chase its score.

01Will the leaderboard winner solve your problem?A public score can shortlist models. A new version of your own task can reveal what the score misses.10 sources · traceable passages 02What if the best-scoring move is to pause the game?Optimizing a score can reveal what its designer forgot to ask for.16 sources · traceable passages 03How much does a successful demonstration establish?A program gives the expected output. Between “it did” and “it will” lie the test distribution, formal assumptions and deployment context. Those boundaries are easy to miss under pressure to ship.41 sources · traceable passages

Does the conclusion survive a different measurement?

Revisit a good result through experimental conditions, measurement boundaries and use cases.

01Did the experiment measure what the everyday claim promises?Start with a morning walk. Give each observation a precise verb: moved the eyes, reported a letter, measured an enzyme. When the verb changes, the research may be answering a different question.28 sources · traceable passages 02Why does moving the measurement change the verdict on a machine?Open a dishwasher after its first drain, then remove the bread from a toaster. Two small experiments expose where a measurement begins, where it ends and what the instrument actually senses.14 sources · traceable passages 03What use case does a good rating actually measure?Start with an incandescent bulb’s lifetime, then examine detergent, a heat pump, a winter cycle route and a food label. Recover the working conditions behind each advertised measure.12 sources · traceable passages 04How far is one working chip from a viable production line?Work backward from a die that passed test. How many others passed on that wafer? Who qualified the machinery? Was the line under financial pressure before an interruption?8 sources · traceable passages

BEYOND THIS FIELD

Take a question into another field.

These links come from existing readings. They do not imply that two fields share the same standards of proof.

Representation & understanding →

An editorial connection

Include the instrument in the question

The AGC restoration checks logic through schematics and internal signals. Image analysis includes light paths and image processing in its explanation. Both suggest looking for intermediate stages that can be checked separately instead of equating an output with the object being measured.

Is the shape in an image the shape of the object? ↗The machine says it passed. Who can check? ↗

Check the passages on both sides

glare orientation is relative to the camera

Gimbal UFO - A New Analysis · 10:22 ↗

we can get right in and check the signals on the chip

Apollo Guidance Computer Part 1: Restoring the computer that put man on the Moon · 8:51 ↗

See these connections on the map →

Software & systems →

An editorial connection

What must survive a change?

A snapshot gives a reader an earlier value to consult. Braid makes preservation playable: restore the frame and see whether play can resume from it. The two examples use different storage techniques.

Can today’s record explain yesterday’s decision? ↗I already threw the punch. Which frame can a late input change? ↗

Check the passages on both sides

the next version of the value shares a lot of structure with the prior version

Rich Hickey · Are We There Yet? · 40:03 ↗

record the full state of the world, because you know that's robust. You know you can reproduce that.

The Implementation of Rewind in Braid · 5:06 ↗

See these connections on the map →

Cross-domain questions →

An editorial connection

Reading a winning score

Turning district votes into seat winners can discard margins. A benchmark also combines cases into one score, prompting a check on the variation hidden by the result.

Will the leaderboard winner solve your problem? ↗Who wins when the same voters are counted differently? ↗

Check the passages on both sides

these metrics are not particularly well aligned with real life usage

Jeremy Howard · A Hackers’ Guide to Language Models · 59:39 ↗

80% -- wanted someone else as king

Why Your Vote Doesn't Count · 1:21 ↗

See these connections on the map →