HackerTube English?
软件边界与变化 →

坏掉一个部件,谁还能继续工作?

一个进程出了错,其余请求是否必须等待?把故障传播和性能瓶颈分开看,再追踪操作系统与服务管理者各自能控制的边界。

本文的问题 · 7
  1. 遇到局部异常时保住剩余服务
  2. 隔离故障的目标,是让一处失败不必拖垮仍然健康的工作
  3. 集中管理生命周期,同时限制故障传播范围
  4. 系统管理职责超出 Linux 内核边界
  5. 进程隔离如何变成瓶颈隔离
  6. 并行工作前,先拆掉串行入口
  7. 只测一个反复运行的小例子,可能看不到真实负载的性能成本

遇到局部异常时保住剩余服务

Hetzner高温测试要求机器可以降频但不崩溃;BEAM演示单个计算进程崩溃后其余请求继续执行。两者都把异常时保持可用的行为当作设计目标,分别在硬件热控制与进程隔离层实现。

对照原文与时间戳 · 2
it might throttle because the cpu is overheating but it shouldn't crash
Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein (Germany) · ▶ 18:47–19:18
when a process crashes everything else progresses right so most of the system still works
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:12–14:23

硬件降频不是进程故障隔离;对照不意味着两套机制互相替代或具相同保证。

隔离故障的目标,是让一处失败不必拖垮仍然健康的工作

进程隔离与服务间解耦是两个尺度的例子;共享资源仍可能引入级联故障。

对照原文与时间戳 · 2
when a process crashes everything else progresses right so most of the system still works
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:15–14:23
the ability to isolate bottlenecks so that they don't spread through the system
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 9:40–9:49

集中管理生命周期,同时限制故障传播范围

systemd 演讲解释服务管理需要响应硬件和软件变化,已超出一次性启动命令;Erlang/Elixir 演讲展示进程隔离如何让局部失败不拖垮其他活动。并读可以区分系统层生命周期管理与进程层故障遏制这两种互补的可靠性职责。

对照原文与时间戳 · 2
maintaining services needs to listen to hardware and software changes
The Tragedy of systemd · ▶ 12:43–12:55
separating failures by separating things into different processes
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:37–14:43

systemd 与 BEAM 处在不同系统层次;材料没有主张服务管理器本身采用 Erlang 式进程模型。

系统管理职责超出 Linux 内核边界

硬件演讲说明整机的资源与保护职责横跨 Linux 和外围固件;systemd 演讲则把用户态动态服务纳入系统层。两者都要求从实际管理职责出发画系统边界,避免把一个内核当成全部系统。

对照原文与时间戳 · 2
the operating system for this machine includes linux as a component but it also has to include all the rest
USENIX ATC '21/OSDI '21 Joint Keynote Address-It's Time for Operating Systems to Rediscover Hardware · ▶ 17:53–18:07
systemd as a project ends up complementing the linux kernel by providing all of this user space system layer
The Tragedy of systemd · ▶ 15:32–15:45

外围固件与 systemd 不是同一层,也不互相替代;共同点仅是系统职责边界大于 Linux 内核。

进程隔离如何变成瓶颈隔离

Erlang/Elixir 讲解将不同进程作为隔离失败的边界;WhatsApp 的规模化实践则把解耦目标表述为隔离瓶颈,避免局部问题扩散。把两者并读,可以从运行时故障隔离延伸到生产系统的容量与故障域设计。

对照原文与时间戳 · 2
separating failures by separating things into different processes
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:37–14:43
ability to isolate bottlenecks so that they don't spread
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 9:40–9:48

WhatsApp 的讲解不证明所有瓶颈隔离都由 BEAM 进程隔离实现,也描述了特定系统架构中的取舍。

并行工作前,先拆掉串行入口

后排优先登机使旅客堵在同一放行李位置;WhatsApp 即使有许多工作进程,单个分发进程仍成瓶颈。两者都需把待处理工作分散到多个可同时服务的位置。

对照原文与时间戳 · 2
increasing the chances of pullaways and parallels, unlike the more orderly methods
The Fastest Way to Board a Plane Feels Like the Slowest · ▶ 3:30–3:47
the dispatch process itself became a bottleneck and not necessarily just
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 12:53–13:02

登机是物理队列,WhatsApp 是进程调度;这里只迁移串行入口限制吞吐的机制,不主张直接套用登机算法。

只测一个反复运行的小例子,可能看不到真实负载的性能成本

两者具体讨论哈希表与缓存。结论是要覆盖使用情境,不是微基准没有用。

对照原文与时间戳 · 2
But without running the data once, how do you know what dimensions are inconsequential?
CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step” · ▶ 20:19–20:25
when you go to time it you're timing the feature of interest which puts it in cash anyway
Modern Dictionaries by Raymond Hettinger · ▶ 48:33–48:41

读完再想一步

对同一次事故,分别画出等待链和故障传播链。两条线在哪里重合?