Systems models are typically arborescent, or tree-like. There is a root source of truth, and everything else extends from there. Kubernetes has its API server. Ansible has whichever laptop you ran the playbook from. Terraform has a state file with a lock. Even my own deploy-rs setup had a root: one machine with the keys, pushing closures outward to nodes that could only receive. That’s a fine shape when an ops team is the root. For one person with a few computers, it means one of them is special, and the whole system is only as available as that one.

A rhizome, like ginger or crabgrass, is a network of connected points. In A Thousand Plateaus, Deleuze and Guattari lay out some of the principles of rhizomatic thought. A rhizome connects different types of things together, with no central pivot. A rhizome may be broken at any point, and it starts up again along its old lines or along new ones. Finally, they insist on the map over the tracing: a tracing copies a structure that was decided in advance, while a map is drawn from whatever actually connected and stays open to redrawing.

But this isn’t a philosophy post. It’s about how the endgame of my fleet configuration ended up taking a rhizomatic shape due to a handful of decisions.

The Fleet Config Monorepo

If you install NixOS on a single computer, you end up with a /etc/nixos/configuration.nix which is the source code for the computer, usually importing an auto-generated hardware-configuration.nix. If you install NixOS on several computers, you end up with several of those. They’ll tend to have some overlap, on everything from timezones to your preferred dev tooling. If you want to upgrade something, you’ll repeat the same process on each computer. If you want to add cross-cutting functionality, well, you get the idea.

The move then is to create a single flake.nix whose schema provides a top-level nixosConfigurations. Your fleet becomes an attrset. Forgiving some pseudo-Nix for the rest of the post:

{
  desktop = { ... };
  laptop = { ... };
}

This of course allows you to abstract out shared segments of their definitions:

let devTools = [ firefox git ]; in
{
  desktop = {
    packages = devTools ++ [ ... ];
  };
  laptop = {
    packages = devTools ++ [ ... ];
  };
}

It’s a pretty neat way to parameterize any number of machines under a single source of truth!

One of the first steps in Rhizome-hood is to introduce a forge.

{
  vps = {
    services.forgejo.enable = true;
  };
}

Its role is to

1. Host the config repo (among others)
2. Run CI; in particular, a check.yml that builds all the hosts on every PR. Breaking changes don’t make it in.

I want the forge highly available, so I host it on my low-resource VPS, and my home laptop, which has the RAM for it, serves as the CI runner.

Notice what that does to the shape. The config is no longer administered from outside the fleet by one special machine with the keys. It is hosted, built, and checked by members of the fleet it describes.

Any Node Reaches Any Node

Nebula is a nifty tool from Slack that lets you define a network overlay with certificate-based identities. One well-known node (the “lighthouse”) acts as a rendezvous point, not a relay. Nodes usually talk directly once they find each other, even from behind NAT.

The VPS is the obvious choice for the lighthouse since it has a stable public IP. Making use of the rec keyword for a recursively evaluated attr-set, we can co-configure the nodes like this:

rec {
  vps = {
    services.nebula.networks.rhizome = {
      enable = true;
      isLighthouse = true;
    };
  };
  laptop = {
    services.nebula.networks.rhizome = {
      enable = true;
      lighthouses = [ vps ];  # really the lighthouse's overlay IP
    };
  };
}

Now every node has a stable address on a flat private subnet, no matter where it physically is. The payoff is that the low-resource VPS never has to build anything. It delegates every Nix build to the laptop over the overlay:

{
  vps = {
    nix.settings.max-jobs = 0;  # never build locally
    nix.buildMachines = [{ hostName = laptop; sshUser = "nixremote"; }];
  };
}

The same overlay is how I SSH into the laptop from anywhere, with its sshd closed to everything but the Nebula interface. Other tricks follow, like an nginx reverse proxy on the VPS serving websites that actually run at home.

Automate the Updates

Here’s where we start to close the loop, via 3 different update modes.

1. Push: Updates to the VPS trigger a CI job wrapping the deploy-rs step.
2. Pull: Laptops get a systemd timer that checks the forge for main’s tip, then builds and switches to that commit.
3. Self-Bump: Apps running in the fleet are flake inputs. A green merge triggers a version bump in the config repo.

All the friction in Decoupling My Deployment Model just becomes automated CI steps.

Push

The push to the VPS is the deploy-rs command from the earlier posts, run by CI instead of me. It doesn’t need to happen on every merge, though, so the workflow only fires when the push asks for it. A commit trailer turns out to be the cheapest “deploy this” flag there is:

git commit -m "Tweak nginx" -m "Deploy: vps"

The workflow reads it with git interpret-trailers and runs nix run .#deploy-rs -- .#vps. No second system, no deploy button, just a line in the commit message.

Pull

Laptops don’t accept inbound connections, so instead of being pushed to they poll. A systemd timer asks the forge for the tip of main, and if it isn’t what’s running, switches to exactly that commit:

{
  laptop = {
    systemd.timers.fleet-switch.timerConfig.OnCalendar = "*:0/5";
    systemd.services.fleet-switch.script = ''
      rev=$(git ls-remote $FORGE/fleet.git main | cut -f1)
      [ "$rev" = "$(nixos-version --configuration-revision)" ] && exit 0
      nixos-rebuild switch --flake "git+$FORGE/fleet.git?rev=$rev#laptop"
    '';
  };
}

The comparison works because the flake stamps every build with the commit it came from (system.configurationRevision = self.rev). Pinning the exact revision it observed means a laptop that was asleep for a week catches up to the same commit as everyone else, not to whatever main happens to be mid-build.

Self-Bump

Apps that run on the fleet are flake inputs. Each one’s own CI, after a green merge, runs a single step:

nix run $FORGE/fleet.git#bump -- myapp=$sha

That re-pins the input, runs nix flake check with the new lock (a bump the fleet can’t build never lands), and pushes to main. A small map in the fleet repo says which hosts a moved pin should deploy:

rolling = {
  myapp = [ "laptop" ];
  website = [ "vps" ];
};

That one line and the step above are all the app and the fleet know about each other. The “tab back to the server repo, nix flake update, deploy” loop is gone.

Tool Bumps

The same trick works pointed at upstream instead of my own repos. Claude Code and Codex are pinned independently of nixpkgs, because the model gate is server-side: a CLI older than a new model locks every agent on the fleet out of it the day it ships. So a daily workflow compares the pins with what Anthropic and npm publish, rewrites them, checks, and pushes with a Deploy: vps trailer:

on:
  schedule:
    - cron: "17 6 * * *"
jobs:
  bump:
    runs-on: native
    steps:
      - run: nix run .#bump-tools

These show up in main’s history:

6f84965 bump tools: claude-code 2.1.287 -> 2.1.288
5ddc48f bump tools: claude-code 2.1.286 -> 2.1.287, codex 0.159.3 -> 0.160.0

YOLO Mode: Fleet-Wide Features

With these pieces in place, one PR in one repo can orchestrate cross-cutting features. Every host lives in the same repo, CI builds all of them on every PR, and the merge deploys itself. That makes it a great place to let an agent loose, since a fleet-wide change is a single reviewable diff with a build gate in front of it.

As an example, I asked for an observability system using ntfy. The topic name is the whole credential, so it lives in a secret, and the app on my phone subscribes to it.

{
  vps.fleet.notify.watch.laptop = { host = laptop; port = 22; };
  laptop.fleet.notify.watch.vps = { host = vps; port = 22; };
}

The hosts probe each other, and report each transition, from reachable to silent and back again. Any service unit can also wire in:

systemd.services.backup.onFailure = [ "notify-failed@%n.service" ];

This way, if there’s a rupture in the flows, I’ll know about it quickly.

The Knots

A rhizome can still tie itself in a loop, and that’s where some actual problem-solving went.

A cross-cutting change triggers a re-deploy of the VPS and the laptop. That means the CI runner deploys itself. A switch that restarts the runner kills every job inside it, including the deploy-rs activation of the VPS. Two rules untangle it. The laptop’s pull updater drains the runner before it switches, so no new jobs start mid-deploy. And the updater and CI’s VPS deploy share a single flock, so replacing the runner has to wait for any in-flight deploy to finish, and vice versa.

Secrets are the other knot, since they’re the one thing that can’t be declarative. sops-nix gets most of the way: each file is encrypted for my admin key plus the hosts that consume it, so “move a service to another host” is add a recipient, re-key, commit.

creation_rules:
  - path_regex: secrets/myapp\.yaml$
    key_groups: [{ age: [ *admin, *laptop ] }]

But the Nebula certificates are still signed by hand from a CA directory and copied out of band. Maybe the rhizome has a root after all. Oops!

Summary

In my rhizomatic system,

• the config is a member of the fleet it describes
• every node can reach every other
• main is converged on from three directions
• every edge reports its own failures

Heterogeneous things connected with no pivot, rupturable anywhere, and a map rather than a tracing. Together they mean I’m basically out of the loop, and the commit log on main reads like the fleet maintaining itself.

As usual, Deleuze and Guattari (illustrated by Marc Ngui and Magda Wojtyra) should help clarify matters.

1. Different components (websites) are treated as flake inputs.
2. The nginx config refers to the packaged outputs.
3. The entire server only builds if all the components build.
4. Deployments are atomic and minimal; the exact system is copied up.

This idea becomes particularly powerful once you have multiple nodes and can write integration tests for your entire system then deploy across multiple machines with a single command.

But for my personal use case I started to notice some slight friction. Say I’m updating this website with a new post. I’m working in its repo. Once I’m done, I tab back to the server config repo, run a nix flake update to pull in and re-pin the website, then deploy. It makes logical sense but doesn’t feel as ergonomic as it could.

The system also breaks down with more granular deployments. Say I want to give an untrusted tenant the ability to update paradigmatic.systems without interfering with the other services running on the same server. It’s not an unusual use case, and deploy-rs is designed with it in mind. Now I’ve finally taken the time to migrate from node-based to profile-based deployment.

Attempting to Keep Using deploy-rs

The deploy-rs API is built for this, and it should be a simple change!

In the Server Repo

I removed paradigmatic-systems from the flake inputs and instead defined an unprivileged deploy user:

users.users.paradigmatic-deploy = {
  isSystemUser = true;
  group = "paradigmatic-deploy";
  createHome = true;
  home = "/var/lib/paradigmatic-deploy";
  shell = pkgs.bashInteractive;
  openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAA..."
  ];
};
users.groups.paradigmatic-deploy = { };

nix.settings.trusted-users = [ "paradigmatic-deploy" ];

# ensure the profile directory exists before the first deploy
systemd.tmpfiles.rules = [
  "d /nix/var/nix/profiles/per-user/paradigmatic-deploy 0755 paradigmatic-deploy paradigmatic-deploy -"
];

Yes, I’ve contradicted myself by making paradigmatic-deploy a trusted user. I’ll come back to that later. But for now let’s say “we don’t think they’re going to upload malware, they just need guardrails from the other services”.

Then I proxy the website to the actual user profile:

services.nginx = {
  enable = true;
  virtualHosts."paradigmatic.systems" = {
    root = "/nix/var/nix/profiles/per-user/paradigmatic-deploy/website";
  };
};

In NixOS, a profile is a stable and mutable pointer into the (immutable) Nix store. It’s a symlink managed as a series of numbered generations. For a desktop environment it might wrap your entire system closure, including your kernel, services, and every installed package. In this case it’s just a single website!

In the Website Repo

We add deploy-rs to the website flake inputs, and configure it to set the website profile:

deploy.nodes.paradigmatic-systems = {
  hostname = "my-node";
  profiles.website = {
    sshUser = "paradigmatic-deploy";
    user = "paradigmatic-deploy";
    path = deploy-rs.lib.x86_64-linux.activate.profile
      self.packages.x86_64-linux.website;
  };
};

profiles.website is the rendezvous point for the two flakes. The path attribute is what actually gets copied up. The activate.profile wrapper is an activation hook that deploy-rs uses to verify the deployment. It’s also the villain of this story…

The Snag

The above code worked beautifully from my NixOS machine. I was excited to demonstrate the system to my “untrusted tenant” and went to run the deployment from her macOS machine. My eyes darted across deploy-rs.lib.x86_64-linux.activate.profile and briefly questioned things before seeing a blaring System Mismatch error.

The activation wrapper and the host system require different architectures. To build it correctly I’d need to pull in an entire Linux toolchain. Cross-compilation is pretty robust, but how much do I want to beef up my flake for a static website that won’t differ from one architecture to the next!?

An Easy Escape Hatch

One quick option offered by deploy-rs is the remoteBuild = true flag which pushes the derivation up and has the server build it on that end. Yes, it’s that simple. However, my cheap-o server is sized for the services it runs; not for pulling in the whole build-time closure.

Switching it Out to nix-copy

In the interest of moving forward with my life I opted to go under the hood to the same core Nix tools used by deploy-rs. The server flake remained basically the same! In the website flake I defined a simple deploy app:

  deploy-website = pkgs.writeShellApplication {
    name = "deploy-website";
    text = ''
      target="''${DEPLOY_TARGET:-paradigmatic-deploy@my-node}"
      site="${self.packages.${system}.website}"

      nix copy --no-check-sigs --to "ssh://$target" "$site"

      echo "==> Activating"
      ssh "$target" nix-env --profile /nix/var/nix/profiles/per-user/paradigmatic-deploy/website --set "$site"

      echo "==> Deployed https://paradigmatic.systems"
    '';
  };

All we’re doing is copying up the package and making the profile switch. The --no-check-sigs flag is the counterpart of being in the trusted-users list. Let’s talk about that briefly.

The Threat Model

I’ve given a fairly powerful capability to what I called an “untrusted” user, so I should probably address that a bit. Realistically, anyone who can write to the shared /nix/store is within my trust boundary.

Signing keys improve the situation with respect to accidents and shallow compromises. The trust is revocable and auditable, and composes into real CI architecture. But the tenant can still inject arbitrary store paths.

For handling a true scumbag tenant the design changes shape completely. They never get store write access whatsoever. They push up source and the server builds on its own terms. That, or they get a dedicated container or VM with its own store.

Conclusion

I needed more granularity in my deployment, so I broke out a piece of my server config. Instead of being baked in, it’s configured with a symlink that’s switchable by a specific user.

To route around a mild inconvenience I dropped deploy-rs in favor of Nix primitives. I’m using the same underlying manipulation of the nix store, so I still get atomicity, history, and garbage collection.

The main thing I lost is the “magic rollback” feature of deploy-rs where the system auto-reverts if SSH becomes unreachable. However, for this class of deployment (a static site swap) it’s not a big risk.

deploy-rs is a powerful tool for multi-node or multi-profile orchestration with interactive confirmation, and it’s on my roadmap to work on some more interesting examples later!

The integration test framework lets you define virtualized nodes with pass/fail conditions on the behavior of the components therein. Furthermore the nodes can be embedded within an arbitrary network topology to test the overall system-of-systems behavior. With the tests defined, they’re wrapped into the very notion of validity for your configuration(s).

This system is obviously the future of infrastructure. NixOS is doing things that are deep on the long-term roadmap for other system-of-system solutions. You could burn serious man-hours approximating this capability in docker-compose and end up nowhere near the raw reproducible bare-metal power.

However, in my own experiments, the virtualization layer quickly made my $300 dev laptop smoke at the ears for anything beyond 1 node. That’s why I’m excited that Nixcademy has announced a new feature. An unnamed corporate patron has supported development of namespace-based containerization backed by systemd-nspawn.

Here’s an abbreviated example illustrating how I could bake this very post into my VPS deployment!

{
  description = "NixOS configuration with test";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };
  outputs = { self, nixpkgs }: 
  {
    checks.x86_64-linux.test = let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
    in pkgs.testers.runNixOSTest {
      
      name = "configuration-test";
      
      nodes.server = { config, pkgs, paradigmatic, ... }: {
        services.nginx = {
          enable = true;
          virtualHosts."_" = {
            root = "${paradigmatic.packages.${system}.website}";
          }
        };
      };
      
      nodes.client = { config, pkgs, ... } : {};

      testScript = ''
        server.start()
        client.start()
        server.wait_for_unit("nginx.service")
        client.wait_for_unit("multi-user.target")
        client.succeed("curl http://server/posts/integration-containers 
        | grep 'Hello from this very text block!'")
      '';
    };
  };
}

Nice!

Want to chat about NixOS adoption? Email me

The Tool and the Language

Let’s bootstrap ourselves with some circular facts.

Nix is a tool for building (or “packaging”) software components using instructions expressed in the Nix language. The Nix language is even able to package the Nix tool (which is mostly written in C++), thus closing the epistemological loop.

The language features first-class treatment of filesystem paths and URLs (to source code and tarballs, for example). Nix expressions are evaluated in a hermetic environment with no path variables and no files. All outside references get copied into content-addressed folders in the store, and marked as build-time dependencies in any derivation that gets defined. Retained runtime dependencies are calculated by scanning the actual binary for store paths.

The Build

The component is the result of realizing the derivation. That means grabbing all the store path contents and combining them in the prescribed way. Imagine you have 3 components that are build-time dependencies:

1. A compiler
2. Source code
3. A bash script that uses the compiler to build the source

The derivation describes where to find these, and the (obvious) way that they fit together. The result is content-addressed with a hash based on all its dependencies, and placed in the store.

nixpkgs

It turns out this scheme is highly flexible and repeatable. nixpkgs is a repo full of Nix expressions for over 120,000 packages. You can pull any one of them into an ephemeral shell to try it out, or install it system-wide if you’re using:

NixOS

Finally, notice that “software components” could have a very broad definition. So broad, in fact, to include an entire OS with all of the installed programs and services. That’s what NixOS is: A Linux distro defined entirely in Nix and built around the /nix/store path model instead of the more usual FHS. I’ve written a bit about how the Nix module system helps encapsulate complex system recipes into simple one-liners.

All my homies use NixOS. To paraphrase from NixOS is the endgame of distrohopping :

No more configuration drift - solved ✅

“Works on my machine!” - solved ✅

Dependency Hell - solved ✅

Virtual and development environments - solved ✅

package management - solved ✅

Ready to go deeper?

To get started I highly recommend working through the tutorials at nix.dev!

• social problems caused by scaling organizations to thousands of engineers
• existing complexity
• laziness or desire to throw money at a problem rather than solve it

While they do solve real technical problems, I’ll argue here that there’s superior tooling for most use cases outside of big tech and legacy systems.

Containerize or Modularize?

We need deployment consistency and environment isolation, and containers solve those!

The 2016 essay Docker Considered Harmful explains how Linux primitives solve the same problems in a more robust way. One valid criticism of this is that Docker can get a junior developer productive more quickly without needing to learn about arcane system settings.

Fine, so we need an abstraction to work with.

Does the abstraction need to expose us to a bloated ecosystem prone to drift? What if it could evaluate to the precise, minimal, and correct system without introducing overhead? What if it’s a better Docker image builder than Docker’s image builder?

Enter Nix with its module system. Nix approaches the “dependency hell” problem from first principles. It requires a radically different mental model, which is itself a social organization problem. This model, by the way, is laid out best in Eelco Dolstra’s 2006 PHD Thesis (PDF).

When you think about

• polyglot system with multiple language ecosystems working together
• co-existing packages with conflicting dependencies (i.e dfferent versions of python)

do you get a headache thinking about all the setup? Well then you should probably swallow some Nix pills. No other tool can encapsulate cross-ecosystem practices and Linux expertise so cleanly. A simple configuration change like

services.postgres.enable = true;

is the tip of a carefully crafted reproducible iceberg. This line of code hides a host of system-level changes including

• User and groups setup with correct isolation and permissions
• Directory structure with ownership and permissions
• Initialization, default template databases
• SystemD service with correct dependencies, start order, lifecycle and recovery and much more!

It’s a massive setup process that would normally be done manually, maybe captured in brittle shell scripts, or containerized. But here it’s a one-liner, and it’s exposed for use as a declarative model that allows completely deterministic and atomic system upgrades!

So while containers provide language-agnostic deployment, Nix achieves the same universality at the package level while maintaining better reproducibility and lower overhead. You can even spit out a Docker image as an afterthought if that’s what you really need.

Cluster or Cluster*@!#?

We need service discovery, load balancing, and observability! Kubernetes provides these with a vast ecosystem of battle-tested tools, and its nice declarative operator pattern for nearly every infrastructure need.

The infrastructure problem is solved by creating different problems: cognitive overhead, ecosystem lock-in, long development cycles, and significant cost markup. There are compelling reasons to look for a better alternative.

Well, the largest package repo in the world (Nixpkgs) has nice modules defined for over 20,000 world-class open-source tools. With a 3-line change to your system configuration, you’re now building a robust set of tools:

services = {
  nginx.enable = true;      #Load balancing
  consul.enable = true;     #Service discovery
  prometheus.enable = true; #Observability
  #You'll want to configure these, of course.

But that’s not it. The Nix flake output schema specifies nixosConfigurations as a key. Note the plural. So you might do something like:

nixosConfigurations = {
  web = mkNode "web" {
    services.nginx.enable = true;
    services.nginx.virtualHosts."web" = {
      locations."/api/" = {
      proxyPass = "http://${cluster.db.ip}:5000/";
    };
    app.db_address = cluster.db.ip
  };

  db = mkNode "db" {
    services.postgresql.enable = true;
    services.postgresql.enableTCPIP = true;
    services.postgresql.authentication = ''
      host all all ${cluster.web.ip}/32 trust
      '';
    api.enable = true
  };
};

where the 2 nodes configurations make them aware of each other.

The same flake can even include

• integration tests using the insanely powerful integration test driver
• deployment metadata with deploy-rs to deploy the whole cluster with 1 command

So the question then becomes, do you have a nest of messy legacy code and need to throw money at scaling it up? Or do you have the luxury of using the tools directly and scaling piecewise as needs arise?

Summary

It’s never going to be easy to retrofit legacy enterprise systems. But for designing new systems, the NixOS advantage is massive. Your entire system configuration, from kernel to application dependencies to infrastructure services, is a single, reproducible, version-controlled artifact. It allows for

• One deployment mechanism
• Deterministic deployments
• Exact parity between development and production

and most importantly, a level of autonomy and sovereignty that you yield when your system is built on unstable abstractions.

1. All the dependencies, pinned in the flake.lock and updatable with nix flake update.
2. A dev environment accessible with nix develop.
3. A package derivation that encapsulates the build process in a nix build command.
4. A module that exposes the app as a configurable system-level service that can be leveraged in integration checks with nix flake check.

With these things in place, activities that traditionally create overhead for developers (environment and deployment) are abstracted and standardized. And even though other tools solve similar problems, none of them achieve the universality of the Nix model. That’s because the most natural and complete description of software packaging is as pure functions over store paths. But in this post I’m focusing on convenience. The correctness of Nix, and the fact that correctness leads to convenience, are topics for a different post.

I’ve released some Nix infrastructure for Phoenix in a repo called PhoeNIX. It’s constructed, as is tradition, by copy-pasting from someone smarter (shoutout Zeke Dou and jurraca) and then extending to meet my needs. I haven’t seen any other flake templates for Phoenix that hit all 4 of my holy grail elements, so I hope this repo is useful for someone.

If you’ve got a deploy-rs setup for pushing system closures, then you can achieve a workflow that makes other DevOps, Cloud, and Containerization solutions look somewhat inadequate.