HackerTube 中文?
Software boundaries and change →

When one component fails, who can keep working?

One process fails. Must unrelated requests wait? Separate failure propagation from throughput limits, then trace what the operating system and service manager can actually isolate.

Questions in this reading · 7
  1. Can a service survive a local failure?
  2. Can healthy work continue when one component fails?
  3. Who manages a service after startup?
  4. Where does the kernel's job end?
  5. How far can one bottleneck spread?
  6. What must happen before work can run in parallel?
  7. What does a repeated microbenchmark leave out?

Can a service survive a local failure?

Compare containing an exceptional condition with isolating a failing process. In each case ask which healthy work can still proceed.

Compare original passages and timestamps · 2
it might throttle because the cpu is overheating but it shouldn't crash
Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein (Germany) · ▶ 18:47–19:18
when a process crashes everything else progresses right so most of the system still works
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:12–14:23

Can healthy work continue when one component fails?

Process isolation and decoupling services address failure at different scales. Both limit propagation, but shared resources can still carry failure across a boundary.

Compare original passages and timestamps · 2
when a process crashes everything else progresses right so most of the system still works
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:15–14:23
the ability to isolate bottlenecks so that they don't spread through the system
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 9:40–9:49

Who manages a service after startup?

A service manager responds to change across a system. Isolated processes contain failures inside a running application. Each boundary protects different work.

Compare original passages and timestamps · 2
maintaining services needs to listen to hardware and software changes
The Tragedy of systemd · ▶ 12:43–12:55
separating failures by separating things into different processes
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:37–14:43

Where does the kernel's job end?

Compare the kernel's responsibilities with the work of keeping services running as hardware and software change. Starting a process is just one part of managing its life.

Compare original passages and timestamps · 2
the operating system for this machine includes linux as a component but it also has to include all the rest
USENIX ATC '21/OSDI '21 Joint Keynote Address-It's Time for Operating Systems to Rediscover Hardware · ▶ 17:53–18:07
systemd as a project ends up complementing the linux kernel by providing all of this user space system layer
The Tragedy of systemd · ▶ 15:32–15:45

How far can one bottleneck spread?

Process isolation contains certain runtime failures. Decoupling services can contain a capacity problem. Identify what is still shared before relying on either boundary.

Compare original passages and timestamps · 2
separating failures by separating things into different processes
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:37–14:43
ability to isolate bottlenecks so that they don't spread
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 9:40–9:48

What must happen before work can run in parallel?

Follow the point where workers have to wait. Adding workers helps only after the serial activity that feeds them can keep up.

Compare original passages and timestamps · 2
increasing the chances of pullaways and parallels, unlike the more orderly methods
The Fastest Way to Board a Plane Feels Like the Slowest · ▶ 3:30–3:47
the dispatch process itself became a bottleneck and not necessarily just
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 12:53–13:02

What does a repeated microbenchmark leave out?

Compare the costs visible in an isolated hash-table operation with the effect of workload and cache behavior. Microbenchmarks help answer a narrower question than end-to-end measurements.

Compare original passages and timestamps · 2
But without running the data once, how do you know what dimensions are inconsequential?
CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step” · ▶ 20:19–20:25
when you go to time it you're timing the feature of interest which puts it in cash anyway
Modern Dictionaries by Raymond Hettinger · ▶ 48:33–48:41

Try it on your own example

Draw the waiting chain and the failure chain for the same incident. Where do they coincide?