NixCon 2026

This was my first time in Poland, and I really loved Kraków. Visiting it swept away any preconceptions I had about Poland. I arrived on Thursday 24 September, two days before NixCon, and left on Monday morning, the day after it ended. We had time to explore on foot, try local restaurants and enjoy the city’s atmosphere.

During NixCon, I spent most of my time talking to as many people as possible, joining parties organised by various actors whose business revolves around Nix, and making some very interesting new friends.

But one piece of work took up a lot of my time, and it had started weeks before the conference.

Nix in a nutshell

#

If you have never used Nix, here are the few ideas you need to follow this fairy tale story:

• Derivation: a recipe describing how to build something, with all its inputs (e.g.: sources, dependencies, build instructions)
• Nix store: where derivations and their results are kept, in /nix/store/<hash>-<name>-<version>. The hash is computed from all the inputs of the recipe: the same inputs always give the same path, and changing a single input gives a different one
• Binary cache (or substituter): before building something, Nix asks the caches it knows whether the result for this exact hash already exists, and downloads it if so. The official one is cache.nixos.org
• Build-time dependency: a tool needed to build something (e.g., a compiler), which does not necessarily end up referenced by the result
• Runtime dependency: a tool or library needed to run something, which is referenced by the result
• Closure: a store path together with everything it references. An installable’s runtime closure contains its runtime dependencies. Build-time dependencies are separate inputs of its derivation. Copying a path with nix copy copies its runtime closure

Build pipelines going offline

#

A few weeks before NixCon, we ran into a technical problem at work. We have a project hosted on GitLab, on our own self-managed corporate instance, that builds artifacts (e.g.: tools and container images) with Nix. Downstream applications then use those images as the basis for their own builds. We do not build the applications themselves with Nix… yet. That would require a greater commitment from the development teams, especially time to learn Nix. I have not given up on that idea, but it is not on the agenda yet.

One of the dependencies required to build the images (a flake input) was hosted on a remote server that was often unavailable, which already made our builds flaky. Then one day it stayed down for several days and the container project stopped working altogether. That dependency was central to our GitLab build system: a Nix library that let me generate a dynamic pipeline.

Our main GitLab CI/CD file was essentially a bootstrap file. It ran a Nix command (nix run .#gitlab-ci > dynamic.yml) that generated a YAML file, which GitLab then used to create the pipeline. This was very convenient: instead of writing YAML by hand, I used Nix to generate the different parts of the configuration. As a user of flake-parts, I could split the pipeline into several modules, each one describing a part of the build. The pipeline was dynamic too: it adapted itself to the structure of the project and to the files present in the repository. It was easy to maintain as well, since changing the Nix code was enough for the generated YAML to follow. I liked that approach a lot: it was a good example of how Nix can manage complex build systems.

The dynamic pipeline also had a cost. A runner first had to start and generate the CI configuration, then GitLab would schedule the resulting jobs, which needed runners of their own. That extra stage made the pipeline slower.

The flakiness was already getting on our nerves, so my first reaction was to clone its repository to our own forge, keep it synchronised with upstream and use that clone in place of the original. It did not solve anything: the library had transitive dependencies of its own, downloaded from the same unreliable servers. The workaround lasted only a few hours, and I could not see myself maintaining it forever.

When the server went down completely, I contacted the project author, who confirmed that it was overwhelmed by bot requests and had no immediate solution. I was stuck, so I made the call: remove the dependency from all our Nix projects and go back to a good old static .gitlab-ci.yml file. Goodbye, dynamic pipelines.

This incident raised some questions in the team. Out of frustration, someone suggested that we check whether we could avoid downloading build dependencies on every commit. I was not convinced: avoiding external servers entirely seemed technically impossible for us at that point. But I took the suggestion as a challenge. After all, solving this would be good for our infrastructure and would speed up our builds as well.

Making our builds more resilient to network failures was not my primary goal, more of a bonus. I was not worried about implementing it, because I was mostly focused on the correctness of the builds. All the artefacts that we build are reproducible, and thanks to that property, adding a caching layer is usually not an issue, quite the opposite.

Caching Nix build inputs

#

I then looked for a way to make our Nix builds less dependent on “the internet”. The first thing I did was to set up an HTTP caching proxy for cache.nixos.org (the official Nix binary cache) whose storage is hosted on Amazon (the partnership has continued for another year). The proxy (named in this context: internal-binary-cache) keeps copies of requested cache data on one of our own servers, so later requests for the same data can be served locally. I hoped this would make builds more resilient, reduce internet traffic and speed them up.

I tried a few solutions, first Caddy and then Nginx, and ended up using the latter. Then, for each project that uses Nix, I had to update the nix.conf configuration and add our proxy server as a substituter. I gave it a priority value lower than 40 (the one of cache.nixos.org and the lower the value, the higher the priority) so that Nix would query our own server first. Falling back to cache.nixos.org should theoretically never happen, since the proxy forwards every request it cannot serve from its own storage.

It worked well at first and considerably reduced the amount of data fetched over the internet from cache.nixos.org. I was pleased with the result.

To be fair, speed was never the main argument. cache.nixos.org is served through a CDN and is very fast, probably as fast as our own cache server. The real motivation is elsewhere:

1. Ecology: fetching the same build dependencies over the internet on every single run is a lot of needless network traffic, and serving them from our own infrastructure removes most of it.
2. Digital sovereignty: depending on third-party servers for every build means that our ability to build and ship our software is in someone else’s hands (as we painfully learned it !). Hosting our own copy of the build dependencies is a small step towards controlling our own supply chain.

However, there was one catch. The proxy cached data fetched from the upstream binary cache but it did not publish the outputs built by our own internal projects! Those outputs were still rebuilt on every commit, even when nothing relevant had changed.

So the next challenge was: how could we cache the builds of our own projects ?

Caching our own builds

#

Several dedicated tools are commonly recommended: nix-serve and nix-serve-ng seemed popular. There was also Attic, which appeared to be unmaintained, and Celler, a fork of Attic that was not yet available in Nixpkgs (a pull request to add it had been opened and then closed). I tried several solutions, but none convinced me. Setting up another server just to store our build outputs was not an exciting prospect.

I started looking at how we could use GitLab itself to cache our build outputs. That is when I found nix-gitlab-ci, which contains scripts for using different kinds of caches. During NixCon, I set up this approach for our project.

One constraint shapes everything that follows: we strive to use stateless runners, for obvious reasons of security, maintenance and transparency. Every job starts from a clean environment, with nothing left over from a previous one: no forgotten secrets or files, no long-lived machine to patch or debug by hand, and a build that depends only on what is declared in the repository. The flip side is that the Nix store of a runner is empty at the start of each job, so everything has to be fetched again every time. This is exactly why the cache must be restored explicitly.

Before each build, GitLab restores the .nix-cache directory from its job cache, when a matching cache is available. Nix is configured to use that directory as a local substituter, pointing directly to the folder (file://<absolute-path>/.nix-cache). After the build, a script copies the selected output paths from the local Nix store into the cache .nix-cache directory using nix copy. GitLab then uploads the updated directory as a cache, ready for a later job to restore. In our setup, the before_script and after_script hooks handled the preparation and publication and GitLab handled the directory transfer automatically.

The first question was: how could I tell Nix which store paths to put in the cache ?

One option would be to copy every path in the local Nix store into the cache (nix copy --all). That would be wasteful, though, and our GitLab cache was limited to 2GB.

Creating cache nix-cache-protected...
.nix-cache/: found 3360 matching artifact files and directories
Using presigned URL for cache upload
Uploading cache.zip to https://<redacted>/nix-cache-protected
FATAL: received: 400 Bad Request. Request failed with code: EntityTooLarge, message: Your proposed upload exceeds the maximum allowed size
Failed to create cache
Cleaning up project directory and file based variables 00:00
Job succeeded

Instead, I needed to record the output paths produced by the checks. The --print-out-paths option prints those paths, so I could save them to a file and use that list after the checks completed.

Unfortunately, nix flake check gained support for this option in Nix 2.35. That meant upgrading Nix in our CI image before I could use this approach (Nix 2.35 release notes).

So I built a custom Nix base image with Nix 2.35 and added scripts to prepare the GitLab cache before and after the job.

I replaced each nix flake check call with nix flake check --print-out-paths > nix-out-paths. In after_script, another script reads that file and passes the paths to nix copy, targeting the file store at the absolute path of .nix-cache. GitLab runs after_script before uploading the cache, although a failure in after_script does not automatically change a successful job’s exit status (GitLab documentation).

Concretely, here is what a job does from start to finish:

1. 1

   Restore the GitLab cache

   GitLab restores .nix-cache when a matching cache is available.
2. 2

   Configure the local substituter

   before_script makes .nix-cache available to Nix.
3. 3

   Run the checks

   Run nix flake check --print-out-paths.
4. 4

   Copy outputs into the cache

   after_script copies the selected outputs into .nix-cache.
5. 5

   Upload the cache

   GitLab uploads the updated cache for later jobs.

All of this is packaged in a hidden job that every Nix job extends:

 1.nix-job:
 2  variables:
 3    NIX_CACHE_DIR: ".nix-cache"
 4  image: $CI_REGISTRY/<group>/<project>/nix-image:<tag>@sha256:<digest>
 5  artifacts:
 6    expire_in: 1 day
 7  cache:
 8    key: "nix-$CI_COMMIT_REF_SLUG"
 9    fallback_keys:
10      - "nix-$CI_DEFAULT_BRANCH"
11    paths:
12      - "$NIX_CACHE_DIR/"
13  before_script:
14    - gitlab-ci-nix-cache before_script "$CI_PROJECT_DIR/$NIX_CACHE_DIR"
15  after_script:
16    - gitlab-ci-nix-cache after_script "$CI_PROJECT_DIR/$NIX_CACHE_DIR"

• image is our custom Nix image, pinned by digest, which contains Nix 2.35 and the gitlab-ci-nix-cache script.
• cache.paths tells GitLab to restore and upload the .nix-cache directory around the job.
• before_script prepares the directory before the job runs. after_script runs nix copy with the paths listed in nix-out-paths.

The most interesting part is the cache key. With key: "nix-$CI_COMMIT_REF_SLUG", every branch gets its own cache, so branches do not overwrite each other. The downside is that a new branch starts without any cache, which means a cold cache and the worst case described below. This is where fallback_keys is really handy: when no cache exists for the current key, GitLab tries the keys of the list in order. In our case, a new branch restores the cache of the default branch, starts warm, and then uploads its own cache under its own key at the end of the job. Only the first pipeline of a branch benefits from the fallback, the following ones find their own cache.

To make sure all the projects using Nix were up-to-date and to avoid duplicated jobs and efforts, I also built a GitLab CI component:

1include:
2  - component: $CI_SERVER_FQDN/<redacted>/nix-flake-check@1.0.0

It encapsulates the setup and runs the nix flake check command in a consistent way everywhere.

But the story does not end there…

NixCon contributions

#

Even after all that, as I made commits, tested changes and scrutinised the build output, I noticed that the same dependencies were still being downloaded on every run:

copying path '/nix/store/sqg55j6303c9x9jgjvf8rpi3q79x9ngv-zstd-1.5.7-dev' from 'http://internal-binary-cache'...
copying path '/nix/store/qnbgx63w8yya5330xcxyixja3ah6gfzm-uvwasi-0.0.23' from 'http://internal-binary-cache'...
copying path '/nix/store/zf8c839ahp1fn736lih7ai2y14rrx2gn-nbytes-0.1.4' from 'http://internal-binary-cache'...
copying path '/nix/store/dw6xlar7m4qfrhbk63zhcc5pmixa1ga1-simdutf-9.1.0' from 'http://internal-binary-cache'...
...
copying path '/nix/store/kpp9w8fia6pksmjpxdz3v83clgck338g-nodejs-slim-24.20.0-corepack' from 'http://internal-binary-cache'...
copying path '/nix/store/1spdvj1bpqmbw00146adbg1zpaq22inx-nodejs-slim-24.20.0-npm' from 'http://internal-binary-cache'...
copying path '/nix/store/v6wsd9nyglmwh32yn7w8z11dq9cksg4m-nodejs-24.20.0' from 'http://internal-binary-cache'...
copying path '/nix/store/lc479s2c4bbwaryyn8mfr8xb0w0nzs6z-prettier-3.9.6' from 'http://internal-binary-cache'...
building '/nix/store/nypq9453vrjph831dbj78n8psi45p69m-check-flake-file.drv'...

All our Nix projects use denful/flake-file, a library that generates the flake.nix from Nix module options. Instead of writing a static flake.nix, you define your inputs (and outputs) with the real Nix language, using the module system and typed schemas. It also provides a check, check-flake-file, which makes sure that the committed flake.nix is up to date with those options. I noticed that every time check-flake-file was built, many of its build-time dependencies were downloaded again, even when only non-Nix files had changed. I mentioned this to Matt Sturgeon during Eelco’s talk. He suggested checking whether the check used runCommandLocal, which disables substituting the derivation’s output. I promptly opened an issue in denful/flake-file to investigate, which was properly resolved a few hours later by Jason Bowman in a pull request.

The discussion with Jason Bowman uncovered a more important detail: check-flake-file depended on the whole project source, so even a commit that changed only the README produced a new derivation. The check’s output was empty, and the tools needed to run it were build-time dependencies, not part of the output closure being copied into our cache. Changing runCommandLocal alone would not solve that broader problem. The issue led to a change that narrowed the files read by the check, so unrelated source changes would no longer invalidate it.

I would not have made sense of this on my own. Gabriel Nützi, Guillaume Maudoux, Jörg Thalheim and Matt Sturgeon helped me investigate the caching behaviour and work through what was happening. I really appreciated all four of them taking the time to discuss it with me at NixCon, thank you guys ♥.

Following the denful/flake-file issue, I opened a pull request in Nixpkgs to fix a similar issue with treefmt. Both contributions grew out of the same frustrating observation: even after setting up the GitLab cache, checks kept fetching tools and dependencies from our caching proxy (and upstream cache.nixos.org) on every run.

The pull request proposed replacing runCommandLocal with runCommand for treefmt’s check derivation, while setting preferLocalBuild = true. This would still prefer a local build when one was needed, but allow Nix to substitute the check’s output if that exact derivation was already available from a cache. The discussion clarified the trade-off: this may avoid rebuilding an unchanged check, but it does not cache the formatter tools needed when the project changes and the check has to run. I had focused on making the check derivation substitutable, while the larger cost in my CI was repeatedly fetching its build-time dependencies.

Why some dependencies are never cached ?

#

The goal is to stop downloading the same dependencies on every run by relying on GitLab’s own cache, which is already designed for this kind of use. The difficulty is deciding precisely what to put in that cache.

--print-out-paths is very useful here: it gives me the output paths of the checks, which I can copy into the GitLab cache with nix copy. On the next pipeline, Nix finds the output in the local substituter and skips the build. But this only works if the derivation is identical between two runs, because the output path is derived from the hash of the derivation, and that hash includes all its inputs.

The treefmt check takes the project sources as an input. Every commit changes the sources, so the derivation hash changes, so the output path changes, and the previous cached output is useless. The check is rebuilt on every commit. That is expected and even desirable: a formatting check must run again when the code changes.

flowchart TB
A["New commit:
sources change"] --> B["Derivation hash changes"]
B --> C["New output path"]
C --> D{"Output in a cache?"}
D -- "No (always)" --> E["Build the check"]
E --> F["Fetch build-time dependencies
treefmt, nodejs, prettier..."]
D -- "Yes" --> G["Download the output,
skip the build"]

The real problem is what happens next. To rebuild the check, Nix must first realise all of its build-time dependencies: the treefmt wrapper, its formatters (prettier, nodejs, ruff, …) and their closures, as well as tools such as gitMinimal and stdenvNoCC. The wrapper is itself an installable output, but we were only listing the check outputs in nix-out-paths. The check refers to the wrapper as a build-time input, so it is not part of the check’s runtime closure. These paths were therefore never stored in our GitLab cache, because:

• --print-out-paths only prints the outputs of the checks, and the output of a check is an empty file (e.g.: treefmt-nix)
• a build-time dependency is only part of the derivation’s inputs. The output does not reference it, so it is not part of the output’s closure, and nix copy of the output does not carry it along

We know that buildInputs are not needed for the output itself to work. That is, after all, why they are called “build"-inputs. But this is a particular case where they are useful: the check’s output is rebuilt whenever the sources change, and its build-time dependencies must be available to do so.

So the cache contains results that are invalidated on every commit, and none of the tools that are needed to rebuild them. Hence the repeated copying path ... from 'http://internal-binary-cache'.

Caching the Treefmt build-time dependencies

#

The output I added was checks.treefmt-dependencies: a linkFarm containing each formatter and the check’s buildInputs (which include the treefmt wrapper):

 1# cache-treefmt.nix
 2{
 3  perSystem =
 4    {
 5      pkgs,
 6      config,
 7      lib,
 8      ...
 9    }:
10    {
11      checks.treefmt-dependencies = pkgs.linkFarm "treefmt-dependencies" (
12        # Link the Treefmt wrapper, and its formatters
13        [
14          {
15            name = "treefmt-wrapper";
16            path = config.treefmt.build.wrapper;
17          }
18        ]
19        # Link the Treefmt check's build inputs
20        ++ (lib.map (drv: {
21          name = "buildInputs/${drv.name}";
22          path = drv.outPath;
23        }) ((config.treefmt.build.check null).buildInputs ++ [ pkgs.stdenvNoCC ]))
24      );
25    };
26}

The difference between the 2 derivations looks like this:

flowchart LR
subgraph checkSub["packages.treefmt-check"]
SRC["project sources"] --> CHK["treefmt-check
(empty output)"]
TOOLS1["treefmt + formatters
(build-time inputs only)"] -.-> CHK
end
CHK == "nix copy (empty closure)" ==> CACHE[".nix-cache"]

flowchart LR
subgraph depsSub["checks.treefmt-dependencies"]
DEPS["linkFarm"] -- "references" --> TOOLS2["treefmt + formatters + check build inputs"]
end
DEPS == "nix copy (whole closure)" ==> CACHE[".nix-cache"]

A symlink to a store path is a reference. Nix scans the output for store paths and records them as runtime references, so the listed tools and check build inputs are part of the closure of treefmt-dependencies. Since this derivation does not read the repository sources, its hash is stable from one commit to the next, as long as its inputs do not change (e.g., after a flake.lock update). The result:

1. nix flake check --print-out-paths also prints the path of treefmt-dependencies
2. nix copy copies it together with its whole closure into .nix-cache, so the formatters and their dependencies are included
3. GitLab uploads the directory as a cache, and the next job restores it
4. when the real treefmt check has to be rebuilt, its build inputs are already available from the local substituter instead of being fetched from the network

In short, the trick turns build-time dependencies into runtime dependencies of a stable derivation, which is exactly what nix copy knows how to cache.

Results and limits

#

On a subsequent pipeline, the job log now shows the dependencies being copied from the local .nix-cache directory instead of the network. Only the derivations that really depend on the sources are rebuilt:

Initialized empty Git repository in /builds/<project>/.git/
Created fresh repository.
Checking out e717a503 as detached HEAD (ref is main)...
Skipping Git submodules setup
Restoring cache 01:46
Checking cache for nix-push-tpsswzwqsymk-protected...
Using presigned URL for cache download
Selecting primary cache URL https://<redacted>/nix-push-tpsswzwqsymk-protected
WARNING: file does not exist
Failed to extract cache
Checking cache for nix-main-protected...
Using presigned URL for cache download
Selecting primary cache URL https://<redacted>/nix-main-protected
Downloading cache from https://<redacted>/nix-main-protected
Successfully extracted cache
...
$ gitlab-ci-nix-cache before_script "$CI_PROJECT_DIR/$NIX_CACHE_DIR"
Configure local Nix binary cache
Configuring the local Nix binary cache at /builds/<project>/.nix-cache with priority 10
Nix will use /builds/<project>/.nix-cache as an additional substituter with priority 10
...
$ nix flake check --print-out-paths > nix-out-paths
evaluating flake...
...
copying path '/nix/store/a010sq4pc8bq1dhm7a3haaddzwxqq69q-plantuml-1.2026.8' from 'file:///builds/<project>/.nix-cache'...
copying path '/nix/store/2cv1b5y6h029cvg07c944pq89fszwbya-openjdk-21.0.12.1+1' from 'file:///builds/<project>/.nix-cache'...
copying path '/nix/store/1fl4qsghsryd5f764m5v64vlxdz9d8qx-gtk+3-3.24.52' from 'file:///builds/<project>/.nix-cache'...
copying path '/nix/store/ls0kgq2mxbggk4gblqysncnvry4qzkki-treefmt.toml' from 'file:///builds/<project>/.nix-cache'...
...
building '/nix/store/z7xrmrzrhnqbx1z7q2rvmzmmc2y9ji6v-treefmt.drv'...
building '/nix/store/qr7x0hjmf5r8w4qj0ci7ry28qfack5jh-treefmt-check.drv'...
building '/nix/store/g65y5lcsqb4qrh4723g9sa4pa6ab7qi9-treefmt-dependencies.drv'...
...
all checks passed!
warning: The check omitted these incompatible systems: aarch64-darwin, aarch64-linux, armv6l-linux, armv7l-linux, i686-linux, powerpc64le-linux, riscv64-linux, x86_64-freebsd
Use '--all-systems' to check all.
Running after script...
$ gitlab-ci-nix-cache after_script "$CI_PROJECT_DIR/$NIX_CACHE_DIR"
Publishing build output closures to the local Nix cache at /builds/<project>/.nix-cacheCopy to Gitlab CI cache
copying 346 paths...
...
copying path '/nix/store/msh4dkzrdvx3zjxdqc370lskwgk8ylr8-flake-edit.toml' to 'file:///builds/<project>/.nix-cache'...
copying path '/nix/store/fqmf9qkqmdzjw4vvqc56grhbinkzvbkj-statix-config' to 'file:///builds/<project>/.nix-cache'...
copying path '/nix/store/2dq7qmq8v7bda181g50a44j9r83xrnfh-treefmt-check' to 'file:///builds/<project>/.nix-cache'...
copying path '/nix/store/zd187i16ckmkmhc3x432nw0vjiqhagjv-check-flake-file' to 'file:///builds/<project>/.nix-cache'...

On this project, the pipeline went from about 20 minutes to about 5 minutes.

There is also a way to look at it in terms of complexity, with a caveat. Before, every pipeline fetched all the dependencies from the outside, so the amount of data pulled from cache.nixos.org grew linearly with the number of pipelines $N$: $O(N \cdot D)$, where $D$ is the size of the dependencies. Now they are fetched from upstream once, then reused through the cache: $O(D)$ upstream traffic in total, plus the small cost of the dependencies that actually changed. The cost is paid once and amortised over the following pipelines.

The chart below is an idealised illustration of cumulative upstream traffic, normalised to the size of the dependency set $D$. It assumes a warm cache and excludes dependencies that change between pipelines (it does not represent the total cost of each job).

In the vocabulary of algorithm analysis, the worst case is a cold cache (e.g.: first pipeline, new cache key, or a cache lost because it exceeded the size limit): everything is fetched from upstream, which costs $O(D)$, exactly like before. The typical case is a warm cache, where only the changed dependencies are fetched. Averaged over many pipelines, this is an amortised analysis: the rare expensive pipeline pays for the many cheap ones.

The proposed solution has limits, though:

• GitLab’s cache is currently limited to 2GB on our instance. This limit is set by the administrators of our corporate instance. Beyond that, the upload fails with EntityTooLarge (see the log above) and the whole cache is lost, while the job still reports success.
• The cache only grows: every update of a tool adds new store paths, and nothing removes the old ones. It needs to be rotated, for instance by changing the cache key regularly or by adjusting the cache expiry policy.
• It requires Nix 2.35 or later, for nix flake check --print-out-paths.
• Each project must expose its own *-dependencies check. Nothing is cached automatically.

A possible improvement is mic92/niks3, an S3-backed Nix binary cache with garbage collection, which also supports GitLab CI authentication through OIDC. It would lift the 2GB limit and take care of the ever-growing cache.

Wrapping up

#

NixCon was simply great. The community you meet there is quite different from the one you can glimpse online, where discussions are often sometimes tense. In Kraków, everyone was open, curious and generous with their time and knowledge. Going there to chat with these people is always a pleasure, and I am already looking forward to the next one.

Looking back, the real lesson was not about treefmt at all. These conversations helped me better understand Nix caching, what our cache was actually saving, what Nix considers an input to a check, and why caching the output of a check is not the same as caching the tools needed to run it. Of everything I brought back from Kraków, that is probably the most useful.

One question is still open: could those tool dependencies reach GitLab’s cache without an extra checks.treefmt-dependencies ?

I do not have the answer yet, but I will keep digging.