Why warm npm installs are slow: materializing 1M files, not downloading

A developer working in a large JavaScript monorepo noticed that installing dependencies in a fresh git worktree took over six minutes even when the package cache was already warm — despite nothing needing to be downloaded. Breaking the install process into its three phases (resolve, fetch, materialize) revealed that materializing the node_modules tree — roughly one million mkdir/open/write/symlink syscalls — is the actual bottleneck, not fetching packages. A warm cache eliminates network and disk I/O for bytes, but every directory entry still has to be created from scratch, so filesystem metadata and journaling overhead dominates.

This also explains why hardlink-based install strategies, like pnpm's or yarn's hardlinks-global mode, don't help as much as expected: they reduce bytes written but not the number of filesystem entries created. In one benchmark, yarn's hardlink mode was actually slower than a standard install (467s vs 389s).

The fix that worked on macOS/APFS was to skip re-materialization entirely using clonefile(2), which copy-on-write clones an entire directory tree in a single syscall. By cloning an already-built node_modules tree from another worktree and only reinstalling the delta when lockfiles differ, install time dropped from 389s to roughly 70-83s — about 5x faster — with near-zero extra disk usage. Per-file copy approaches (cp -cR, cp -R) performed far worse. The approach has been packaged as an open-source tool (wt-deps) for macOS/APFS, compatible with yarn, pnpm, and npm, with a lockfile-diff safety check to avoid silently using a stale tree.