Consuming Nix artifacts at scale has been a challenge to many. When there is a large swarm of ephemeral compute (CI jobs, spot instances, cloud developer environments, etc.) relying on Nix artifacts, traditional S3-based binary caches fall short due to the need for each instance to re-download entire NAR closures.

Off-the-shelf shared filesystems like NFS are not optimized for Nix: Program startups trigger stat storms against the single-point-of-failure, and this is exacerbated by how Nix organizes lookup paths.

We present Lazy Nix Store, a dynamic file system that provides the Nix store backed by an S3 bucket containing globally deduplicated, content-addressed chunks. We leverage the immutability of Nix store paths to minimize network round trips by requesting metadata for a whole NAR in one round trip and serving subsequent VFS requests from memory.

In this talk, we elaborate on the design decisions we made along the way, the tradeoffs we considered, and some preliminary benchmarking results at scale.