In my previous introduction to deploy-rs I shared a particular model of NixOS server deployment that I found very interesting:

  1. Different components (websites) are treated as flake inputs.
  2. The nginx config refers to the packaged outputs.
  3. The entire server only builds if all the components build.
  4. Deployments are atomic and minimal; the exact system is copied up.

This idea becomes particularly powerful once you have multiple nodes and can write integration tests for your entire system then deploy across multiple machines with a single command.

But for my personal use case I started to notice some slight friction. Say I’m updating this website with a new post. I’m working in its repo. Once I’m done, I tab back to the server config repo, run a nix flake update to pull in and re-pin the website, then deploy. It makes logical sense but doesn’t feel as ergonomic as it could.

The system also breaks down with more granular deployments. Say I want to give an untrusted tenant the ability to update paradigmatic.systems without interfering with the other services running on the same server. It’s not an unusual use case, and deploy-rs is designed with it in mind. Now I’ve finally taken the time to migrate from node-based to profile-based deployment.

Attempting to Keep Using deploy-rs

The deploy-rs API is built for this, and it should be a simple change!

In the Server Repo

I removed paradigmatic-systems from the flake inputs and instead defined an unprivileged deploy user:

users.users.paradigmatic-deploy = {
  isSystemUser = true;
  group = "paradigmatic-deploy";
  createHome = true;
  home = "/var/lib/paradigmatic-deploy";
  shell = pkgs.bashInteractive;
  openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAA..."
  ];
};
users.groups.paradigmatic-deploy = { };

nix.settings.trusted-users = [ "paradigmatic-deploy" ];

# ensure the profile directory exists before the first deploy
systemd.tmpfiles.rules = [
  "d /nix/var/nix/profiles/per-user/paradigmatic-deploy 0755 paradigmatic-deploy paradigmatic-deploy -"
];

Yes, I’ve contradicted myself by making paradigmatic-deploy a trusted user. I’ll come back to that later. But for now let’s say “we don’t think they’re going to upload malware, they just need guardrails from the other services”.

Then I proxy the website to the actual user profile:

services.nginx = {
  enable = true;
  virtualHosts."paradigmatic.systems" = {
    root = "/nix/var/nix/profiles/per-user/paradigmatic-deploy/website";
  };
};

In NixOS, a profile is a stable and mutable pointer into the (immutable) Nix store. It’s a symlink managed as a series of numbered generations. For a desktop environment it might wrap your entire system closure, including your kernel, services, and every installed package. In this case it’s just a single website!

In the Website Repo

We add deploy-rs to the website flake inputs, and configure it to set the website profile:

deploy.nodes.paradigmatic-systems = {
  hostname = "my-node";
  profiles.website = {
    sshUser = "paradigmatic-deploy";
    user = "paradigmatic-deploy";
    path = deploy-rs.lib.x86_64-linux.activate.profile
      self.packages.x86_64-linux.website;
  };
};

profiles.website is the rendezvous point for the two flakes. The path attribute is what actually gets copied up. The activate.profile wrapper is an activation hook that deploy-rs uses to verify the deployment. It’s also the villain of this story…

The Snag

The above code worked beautifully from my NixOS machine. I was excited to demonstrate the system to my “untrusted tenant” and went to run the deployment from her macOS machine. My eyes darted across deploy-rs.lib.x86_64-linux.activate.profile and briefly questioned things before seeing a blaring System Mismatch error.

The activation wrapper and the host system require different architectures. To build it correctly I’d need to pull in an entire Linux toolchain. Cross-compilation is pretty robust, but how much do I want to beef up my flake for a static website that won’t differ from one architecture to the next!?

An Easy Escape Hatch

One quick option offered by deploy-rs is the remoteBuild = true flag which pushes the derivation up and has the server build it on that end. Yes, it’s that simple. However, my cheap-o server is sized for the services it runs; not for pulling in the whole build-time closure.

Switching it Out to nix-copy

In the interest of moving forward with my life I opted to go under the hood to the same core Nix tools used by deploy-rs. The server flake remained basically the same! In the website flake I defined a simple deploy app:

  deploy-website = pkgs.writeShellApplication {
    name = "deploy-website";
    text = ''
      target="''${DEPLOY_TARGET:-paradigmatic-deploy@my-node}"
      site="${self.packages.${system}.website}"

      nix copy --no-check-sigs --to "ssh://$target" "$site"

      echo "==> Activating"
      ssh "$target" nix-env --profile /nix/var/nix/profiles/per-user/paradigmatic-deploy/website --set "$site"

      echo "==> Deployed https://paradigmatic.systems"
    '';
  };

All we’re doing is copying up the package and making the profile switch. The --no-check-sigs flag is the counterpart of being in the trusted-users list. Let’s talk about that briefly.

The Threat Model

I’ve given a fairly powerful capability to what I called an “untrusted” user, so I should probably address that a bit. Realistically, anyone who can write to the shared /nix/store is within my trust boundary.

Signing keys improve the situation with respect to accidents and shallow compromises. The trust is revocable and auditable, and composes into real CI architecture. But the tenant can still inject arbitrary store paths.

For handling a true scumbag tenant the design changes shape completely. They never get store write access whatsoever. They push up source and the server builds on its own terms. That, or they get a dedicated container or VM with its own store.

Conclusion

I needed more granularity in my deployment, so I broke out a piece of my server config. Instead of being baked in, it’s configured with a symlink that’s switchable by a specific user.

To route around a mild inconvenience I dropped deploy-rs in favor of Nix primitives. I’m using the same underlying manipulation of the nix store, so I still get atomicity, history, and garbage collection.

The main thing I lost is the “magic rollback” feature of deploy-rs where the system auto-reverts if SSH becomes unreachable. However, for this class of deployment (a static site swap) it’s not a big risk.

deploy-rs is a powerful tool for multi-node or multi-profile orchestration with interactive confirmation, and it’s on my roadmap to work on some more interesting examples later!