Serving /nix/store at Scale with CVMFS and mergerfs
Managing a large Nix deployment across many machines means shipping a lot of store paths which the hosts will most likely never touch and creating pressure on local storage. We replaced bulk pre-population with a lazy-loading approach: CVMFS exposes the Nix store as a read-only FUSE filesystem, fetching file content on demand from HTTP-backed object storage. Mergerfs sits in front, presenting a single unified /nix/store that layers a local read-write volume over one or more read-only CVMFS mounts. The Nix daemon sees a normal store; nothing needs patching. This lets us distribute the full closure of our internal packages across campuses without paying the bandwidth or local storage cost upfront. We'll cover how the solution interacts, what CVMFS brings that a plain binary cache doesn't, the failure modes we hit Applicable to anyone running Nix across a large number of servers where store size and bandwidth matter.
Software engineer at Morgan Stanley