Stefan AgnerandClaude Fable 5 4b71253ed9 Force filesystem sync when unpacking container image layers (#4994) ...

containerd fsyncs its metadata database when committing a snapshot, but
never syncs the unpacked layer data itself. On an unclean shutdown
within the writeback window after an image pull this leaves layers with
zero-byte files (correct names/modes/mtimes, no data) while the
snapshot stays durably recorded as valid. Because existing snapshots
are never re-verified or re-unpacked, such corruption even survives
deleting and re-pulling the image, and with shared base layers it can
only be recovered by wiping the Docker storage entirely.

This is the failure signature behind a growing number of reports of
broken plugins/add-ons/Core after power loss (e.g. 0-byte coredns in
home-assistant/plugin-dns#207, supervisor#6476/#6835, #4913).

Enable the containerd diff-service sync_fs option (containerd >= 2.0,
containerd/containerd#10284): the daemon then issues one syncfs(2) per
applied layer, making unpacked data as durable as the metadata that
references it. It applies to all clients of the daemon, including
dockerd with the containerd image store.

Verified in QEMU on haos 18.3.dev (Docker 29.6.2, containerd 2.2.6):

* With completely stock settings, a power cut <1s after `docker pull`
  of the DNS plugin image corrupts the image 2 out of 2 runs: one run
  zeroed /usr/bin/coredns entirely, the other zeroed the s6 scripts
  and left coredns truncated (20021248 of 25919650 bytes) - matching
  the two damage profiles reported in plugin-dns#207.
* Deterministic A/B with kernel background writeback suppressed, so
  survival can only come from the stack syncing explicitly: without
  this option 1215 zero-byte files including a 0-byte coredns failing
  with "exec format error"; with sync_fs the image survives intact
  (152 kB dirty at the power cut vs 76 MB).
* strace on containerd confirms 0 syncfs calls per image load before,
  one per layer after - which also confirms dockerd pulls route
  through the containerd daemon's diff service where this option
  takes effect.
* Cost of loading+unpacking the 2 GB Core image: 41.6s -> 42.2s on a
  fast disk, 226s -> 230s (+1.9%) on a 30 MB/s / 250 IOPS throttled
  disk. Note QEMU throttling does not model SD-card sync latency, so
  the real-world cost on the slowest cards may be somewhat higher, but
  it stays bounded at one syncfs per layer.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

2026-09-02 17:17:22 +02:00

..

config.toml

Force filesystem sync when unpacking container image layers (#4994)

2026-09-02 17:17:22 +02:00