HackerHow do you turn a one-off failure into a replayable test?
Suppose an acknowledged transfer disappears during a leader change. Six engineering talks separate reproducible inputs, shrinkable operations and invariants, then ask whether the fix actually exercises the original failure.
Questions in this reading · 4
Turn the incident into an input
Suppose a transfer disappears during a leader change. Save the request and send it again tomorrow: it may work every time. Wilson’s simulator also takes control of random choices, keeping the seed as an input. That lets the implementation encounter the same event order again. The team even compares the next random outputs from repeat runs to detect divergence inside the simulator. Replay is only as faithful as the external events represented in the simulation.
Passages and timestamps · 2
the seed of that random number generator becomes part of the input of the programif those don't match exactly we know there was a determinism failureAfter replaying a thousand steps, which ones must stay?
Handing a developer the entire failing history still leaves a hard explanation problem. Hughes models the effect of each API call, generates sequences, and removes calls while the failure persists. Halloway’s generative specifications add a separate question: what distinguishes a bad generated example? If only the debit landed, a sum-of-balances check fails. If neither side changed, the same check passes. To catch the missing transfer, use its acknowledged ID to check for the expected debit and credit. Keep shrinking while the particular check still fails.
Passages and timestamps · 3
we generate a minimal failing examplefor each API call we define a state transition functionwhen you spec something, Clojure can generate example data for youWhat does a counterexample refute?
Wayne writes a no-overdraft invariant and gets an explicit interleaving from TLC. Parlar’s Alloy examples expose another possibility: the surprising instance exists because the designer omitted a constraint. Walk the trace twice. Does the intended rule allow each step? Does the running implementation allow it? The first answer may repair the model; the second can turn the trace into an implementation-level test. A bounded counterexample alone does not identify a particular production incident.
Passages and timestamps · 2
find one that violated our invariant it also provided an error tracewe haven't been precise enough in our specificationAfter the fix, make two explanations disagree
The old seed now passes. Perhaps the bug was fixed, or perhaps the run no longer reaches the triggering path. Halloway suggests asking what observation a hypothesis forbids. Keep the schedule, assert both the transfer record and the account total, and deliberately restore the suspected defect. A test that stays green may never have exercised the failure. The experiment turns “I haven’t seen it again” into a claim another engineer can try to disprove.
Passages and timestamps · 2
you say I have a hypothesis it implies there's something I should not seethe hypothesis um is supported by the experiment but maybe you still don't have an answerTry it on your own example
Pick an intermittent bug. Name the inputs to retain, one step a reduced trace cannot drop, and a property that might fail despite a successful request. Restore the old defect: which check should now fail?