STAFFA SOFTWARE SOLUTIONS
[EN]
A cute little robot toy behind bars

How To Put Your Coding Agent Into Jail - With microvm.nix

Sebastian Staffa

Alternative Sprache verfügbar

Diesen Blog-Beitrag gibt es auch in einer Sprache, die Ihren Einstellungen entspricht. Klicken Sie hier, um zur übersetzten Version zu gelangen.

All LLM-driven coding agents like Claude, Codex, or OpenCode share a common problem: the underlying technology is unpredictable. If, for example, a coding agent stumbles upon a prompt injection in a software project, there is a good chance it will execute it. In the best case, the agent only deletes its own project files; in the worst case, credentials are extracted or a backdoor is installed on the host system.

That is why it is important to lock agents away as securely as possible. In the best case, the agent can only see the code of the project currently being worked on and only has the credentials it actually needs to perform its task. Reality usually looks different: Coding agents in particular run directly on a developer’s machine. Far more projects are visible to the agent than just the one currently being worked on. In addition, credentials are frequently available that grant full access to the cloud resources of one or more applications.

But how can you lock up an agent so securely that it has no way of breaking out of its prison? The answer is a technology that has been used in security research for a long time: Virtual Machines (or VMs for short). Over the past few months, an entire ecosystem of tools has formed around this idea. This blog will be about the tools I evaluated for this task, as well as the setup I ended up with.

Docker Sandbox

My first contact with the topic of agent isolation came through a social media post in one of my timelines, in which a user spoke about their experiences with Docker sandboxes. I quickly decided: “I need something like this too” and began to take a closer look at the tool.

Docker itself is not a security tool. Docker containers do not provide strict isolation between the resources in the container and those on the host. The Docker developers know this too, which is why docker sandbox relies on small VMs, so-called microVMs.

Docker Sandboxes run AI coding agents in isolated microVM sandboxes

— 

“Get started with Docker Sandboxes”, Docker Sandbox documentation

Technically, Docker sandboxes work in a fundamentally different way compared to standard docker containers. Even so, they are able to run existing Docker images. In addition, they offer a number of useful features beyond merely providing a simplified slim VM: Docker sandboxes have, for example, their own concept for sharing credentials with the agents in the container, as well as the ability to define granular network policies.

But if the tooling is so well thought out, why didn’t I use it? The answer to this question can be found in the setup guide of the tool (archive link). For Linux, it starts with:

 curl -fsSL https://get.docker.com | sudo REPO_ONLY=1 sh
 sudo apt-get install docker-sbx
 sbx login

I have a problem with the last step. Unlike in the rest of the Docker ecosystem, a login is mandatory for the tool. From my point of view there is no technical reason for this. In the FAQ (archive link) of the tool, as of today, three reasons are given for this requirement:

  • Tie sandboxes to a real person.

  • Enable team features.

  • Authenticate against Docker infrastructure.

— 

Docker Sandbox FAQ

As someone whose stack is entirely built on open-source and, where possible, community-driven tools, this sounds like the beginning of a vendor lock-in. Even though the idea of wanting to tie agents to a real person sounds noble at first, the solution created here can easily be circumvented by simply reinstalling the agent locally. Team features are not relevant to me. And so far, no login has been necessary to perform “normal” interactions with the Docker infrastructure, such as downloading images.

microsandbox

While searching for a login-free, easy-to-configure alternative, I came across microsandbox. With microsandbox, Docker containers (OCI images) or images of existing virtual machines can be run. As with Docker Sandbox, there are ready-made tools for sharing directories and secrets with the VM. Configuration is done either directly via the CLI or programmatically via one of the available bindings.

For users who do not use NixOS, or for teams that want to share an agent configuration, the tool is my clear recommendation.

However, the way a microsandbox is configured has, from my point of view, three weaknesses that I can eliminate by leveraging my existing NixOS:

As with all projects that use a container environment to encapsulate dependencies, the containers used can degrade. This can happen either because the local tooling and the tooling in the container drift apart, or because the upstream resources needed to build the container (e.g. apt repositories) are no longer available.

Secondly, depending on the stack, the dependencies for a project can become very large. Keeping a copy on the host (after all, I still want to be able to run and build the software myself) as well as in the container takes up unnecessary storage space in times of steadily rising SSD costs. Since the /nix/store is already read-only, it should be no problem to use binary dependencies both on the host and inside the VM to produce compact images.

Thirdly, I have already configured my NixOS with a number of tools that I want an agent to always have access to: ripgrep, jq, and of course my existing OpenCode configuration with all skills and agents.

microvm.nix

These problems are solved by microvm.nix.

microvm.nix makes it possible to create VMs with a footprint of only a few megabytes by using the /nix/store of the host system inside the VM. VMs can be configured either ad-hoc or as a systemd service on the host. Eight different adapters are available for the hypervisor. Since the entire VM is configured declaratively with nix, I can reuse the existing nix dependency declarations from my projects, thereby avoiding “Container Rot”.

For the question of how to set it up, I refer to Michael Stapelberg’s excellent tutorial (archive link), though I do have a few notes and additions. My complete, adapted configuration can be found at the end of this chapter.

Wi-Fi Adapter

Both the tutorial linked above and the official documentation on the network setup use a LAN port of the host system to create the bridge adapter.

If the connection to the internet from the host is established via Wi-Fi, both tutorials cannot be applied one-to-one. Since the created VM has its own MAC address, packets from two different MAC addresses now pass through the same host adapter. This is a problem for Wi-Fi connections, because Wi-Fi routers suspect foul play in this case and cut the connection. For Wi-Fi connections to be forwarded over the bridge adapter, an additional NAT is required on the host, which terminates the connection from the VM.

Which is why my final network configuration looks like this:

  systemd.network.netdevs."20-microbr".netdevConfig = {
    Kind = "bridge";
    Name = "microbr";
  };

  systemd.network.networks."20-microbr" = {
    matchConfig.Name = "microbr";
    address = ["192.168.83.1/24"];
    networkConfig.ConfigureWithoutCarrier = true;
    linkConfig.RequiredForOnline = false;
  };

  systemd.network.networks."21-microvm-tap" = {
    matchConfig.Name = "vm-*";
    networkConfig.Bridge = "microbr";
    linkConfig.RequiredForOnline = false;
  };

  networking.nat = {
    enable = true;
    externalInterface = "wlp2s0";
    internalInterfaces = ["microbr"];
  };

OpenCode Configuration

In his blog post, Michael Stapelberg works with Claude and creates a shared directory for the configuration files for all VMs, which is also used to store the provider credentials. In the case of OpenCode, all sessions also live in this directory, which in turn contains project-specific information.

I therefore decided to create a separate configuration directory for each VM. This way I can prevent project information from leaking through the session history. I hard-link the provider credentials into the VM configuration directory so that I can continue to maintain them centrally.

Additionally, I wanted to replicate Claude’s “yolo” mode with OpenCode. There is no dedicated option for this. However, the OpenCode permissions can be overridden via an env variable.

home.sessionVariables = { OPENCODE_CONFIG_CONTENT = ''{"permission":{"*":"allow","external_directory":"allow"}}'';};

Passing the nix-shell

In most of my projects there is a dedicated shell.nix file with the dependencies of the respective project. Until now, the dependencies of the shell had to be downloaded after entering the VM. Depending on the project, this could take anywhere from a few seconds to several minutes. At the same time, the dependencies of the projects are already present on my host system, so I could share them with the VM. To implement this behavior, and thus save time and storage space, I pass the project’s nix-shell into the VM builder:

{
    shellNix ? null
    # ... other params ...
}:
let
    shellDrv =
      if shellNix != null
      then import shellNix {}
      else null;
in
    # ... inside vm definitions
    system.extraDependencies =
      pkgs.lib.optional (shellDrv != null) shellDrv;

extraDependencies inserts the dependencies into the VM without making them available to the user. nix-shell still takes care of that. This way, existing shell hooks, e.g. for setting environment variables, also work without the dependencies having to be downloaded.

As a further simplification, I added a small script to the zsh init of the VM that sets the working directory to the project directory and, if present, directly drops into the nix-shell.

programs.zsh.initContent = ''
  if [[ -t 1 ]]; then
    cd "${workspace}" 2>/dev/null
    ${pkgs.lib.optionalString (shellNix != null) ''
    if [[ -z "''${IN_NIX_SHELL:-}" ]]; then
      exec nix-shell "${shellNix}"
    fi
  ''}
  fi
'';

Final Configuration

As Michael Stapelberg did, I split my configuration into two parts: mkVm.nix creates a single VM, while microvm.nix defines the necessary system configuration as well as the list of all VMs.

mkVm.nix closely follows the template mentioned above, with two differences:

# mkVm.nix
{
  home-manager,
  pkgs,
  hostUserName,
  openCodeConfigStubMountSrcBasePath,
  ...
}:
{
  projectName,
  ip,
  mac,
  workspace,
  shellNix ? null,
  vcpu ? 8,
  mem ? 4096,
}: let
  vmName = "vm-${projectName}";
  hostname = "ag-${projectName}";
  tapId = vmName;
  openCodeConfigStubMountSrcPath = "${openCodeConfigStubMountSrcBasePath}/${projectName}";
  shellDrv =
    if shellNix != null
    then import shellNix {}
    else null;
in {
  inherit vmName ip;
  sshConfig = ''
    Host ${vmName}
      HostName ${ip}
      # We disable strict host checking because the host
      # key regenerate on every vm start and
      # pregenerating + mounting them is a pain in the but
      # if we ever switch machines
      StrictHostKeyChecking no
      UserKnownHostsFile /dev/null
  '';
  vm = {
    # do not start any of the vms on host boot
    autostart = false;
    config = {
      microvm = {
        inherit vcpu mem;

        # Enable writable nix store overlay so nix-daemon works.
        # This is required for home-manager activation.
        writableStoreOverlay = "/nix/.rw-store";

        volumes = [
          {
            mountPoint = "/var";
            image = "var.img";
            size = 4096; # MB
          }
          {
            mountPoint = "/nix/.rw-store";
            image = "rw-store.img";
            size = 8192; # MB
          }
        ];

        shares = [
          {
            source = "/nix/store";
            mountPoint = "/nix/.ro-store";
            tag = "ro-store";
            proto = "virtiofs";
          }
          {
            proto = "virtiofs";
            tag = "opencode-auth";
            source = "${openCodeConfigStubMountSrcPath}/config";
            mountPoint = "/home/${hostUserName}/.local/share/opencode";
          }
          {
            proto = "virtiofs";
            tag = "workspace";
            source = workspace;
            mountPoint = workspace;
          }
        ];

        interfaces = [
          {
            type = "tap";
            id = tapId;
            inherit mac;
          }
        ];
      };

      #-----------------------
      # system networking
      networking.hostName = hostname;

      systemd.network.enable = true;
      systemd.network.networks."20-lan" = {
        matchConfig.Type = "ether";
        networkConfig = {
          Address = ["${ip}/24"];
          Gateway = "192.168.83.1"; # the host's microbr address
          DNS = ["192.168.83.1"]; # or e.g. 1.1.1.1
          DHCP = "no";
        };
      };

      networking.nameservers = [
        "8.8.8.8"
        "1.1.1.1"
      ];

      networking.firewall.enable = false;

      #---------------------------
      # ssh config
      services.openssh = {
        enable = true;
        settings = {
          PasswordAuthentication = false;
          KbdInteractiveAuthentication = false;
        };
      };

      #-----------------------
      # home manager and home config
      imports = [
        (import "${home-manager}/nixos")
      ];

      home-manager.useGlobalPkgs = true;
      home-manager.useUserPackages = true;

      users.users.${hostUserName} = {
        isNormalUser = true;
        extraGroups = ["wheel"];
        openssh.authorizedKeys.keyFiles = [
          /home/${hostUserName}/.ssh/id_ed25519.pub
        ];

        # this declares zsh as the default shell for this user
        # which allows it to launch immediately when logging in via ssh
        shell = pkgs.zsh;
        # zsh is configured via the included cli module.
        # without this flag, nix complains that zsh is not configured
        # and may not work
        ignoreShellProgramCheck = true;
      };

      home-manager.users.${hostUserName} = {
        home.stateVersion = "26.05";
        # import cli tooling that is also present on my host machine
        imports = [
          ./../home/cli.nix
        ];

        # enable yolo mode inside the vm
        home.sessionVariables = { OPENCODE_CONFIG_CONTENT = ''{"permission":{"*":"allow","external_directory":"allow"}}'';};

        programs.zsh.initContent = ''
          if [[ -t 1 ]]; then
            cd "${workspace}" 2>/dev/null
            ${pkgs.lib.optionalString (shellNix != null) ''
            if [[ -z "''${IN_NIX_SHELL:-}" ]]; then
              exec nix-shell "${shellNix}"
            fi
          ''}
          fi
        '';
      };
      #--------------------------
      #system tweaks

      # Force every store path the workspace's shell.nix needs into this VM's
      # system closure (the toplevel derivation's "extraDependencies"). Since
      # the host shares /nix/store with the VM as a read-only virtiofs mount,
      # those paths are then already present the first time the shell is
      # entered - no fetching or building in the VM.
      system.extraDependencies =
        pkgs.lib.optional (shellDrv != null) shellDrv;

      system.stateVersion = "26.05";
      systemd.settings.Manager = {
        # fast shutdowns/reboots! https://mas.to/@zekjur/113109742103219075
        DefaultTimeoutStopSec = "5s";
      };

      # Fix for microvm shutdown hang (issue #170):
      # Without this, systemd tries to unmount /nix/store during shutdown,
      # but umount lives in /nix/store, causing a deadlock.
      systemd.mounts = [
        {
          what = "store";
          where = "/nix/store";
          overrideStrategy = "asDropin";
          unitConfig.DefaultDependencies = false;
        }
      ];

      # precreate the path where the opencode config is mounted.
      # without this, the paths leading up to .../share/opencode
      # would be owned by root as they are created when mounting the
      # config dir
      systemd.tmpfiles.rules = [
        "d /home/${hostUserName}                             0755 ${hostUserName} users -"
        "d /home/${hostUserName}/.local                      0755 ${hostUserName} users -"
        "d /home/${hostUserName}/.local/share                0755 ${hostUserName} users -"
        "d /home/${hostUserName}/.local/share/opencode/repos 0755 ${hostUserName} users -"
      ];
    };
  };
}

My microvm.nix contains the network configuration for the host system described above, as well as the definition of the individual VMs:

# microvm.nix
{
  microvm,
  home-manager,
  pkgs,
  hostUserName,
  openCodeConfigStubMountSrcBasePath,
  ...
}: let
  mkVm = import ./mkVm.nix {
    inherit home-manager pkgs hostUserName openCodeConfigStubMountSrcBasePath;
  };

  vmSpecs = [
    {
      projectName = "project-a";
      ip = "192.168.83.2";
      mac = "02:00:00:00:00:01";
      workspace = "/path/to/project/a/";
      shellNix = "/path/to/project/a/shell.nix";
    }
    {
      projectName = "project-b";
      ip = "192.168.83.3";
      mac = "02:00:00:00:00:02";
      workspace = "/path/to/project/b";
    }
    #...
  ];

  builtVms = map mkVm vmSpecs;

  vmAttrs = pkgs.lib.listToAttrs (map (v: pkgs.lib.nameValuePair v.vmName v.vm) builtVms);

  sshExtraConfig = pkgs.lib.concatStringsSep "\n" (map (v: v.sshConfig) builtVms);
in {
  imports = [microvm.nixosModules.host];
  microvm.vms = vmAttrs;

  #----------------------------
  # Host options: network

  # This option enables systemd to manage
  # networking. We use NetworkManager
  # to manage most of the connections,
  # but to define them declaratively we need
  # to enable systemd network managing as well.
  systemd.network.enable = true;

  # By default, systemd waits for all configured
  # networks to come online through the
  # `systemd-networkd-wait-online.service`.
  # All microvms do not automatically start on boot,
  # so the networks do not come online and the wait-online
  # services fails.
  # We can disable waiting for singular interfaces
  # through the use of `linkConfig.RequiredForOnline = false`
  # (see below), but as all other networks are
  # managed through NetworkManager. This leaves
  # systemd with an empty set of networks
  # to wait for - which causes the wait-online service
  # to fail with a timeout when switching to
  # new configurations.
  # Should we ever configure additional networks on the host
  # using system, this option should be re-enabled.
  systemd.network.wait-online.enable = false;

  # the following options are propagated to the host
  # system to provide the needed adapters for
  # the network bridge to work
  systemd.network.netdevs."20-microbr".netdevConfig = {
    Kind = "bridge";
    Name = "microbr";
  };

  systemd.network.networks."20-microbr" = {
    matchConfig.Name = "microbr";
    address = ["192.168.83.1/24"];
    networkConfig.ConfigureWithoutCarrier = true;
    # this option is only here to futureproof the config.
    # see comment on systemd.network.wait-online.enable above
    linkConfig.RequiredForOnline = false;
  };

  systemd.network.networks."21-microvm-tap" = {
    matchConfig.Name = "vm-*";
    networkConfig.Bridge = "microbr";
    # this option is only here to futureproof the config.
    # see comment on systemd.network.wait-online.enable above
    linkConfig.RequiredForOnline = false;
  };

  networking.nat = {
    enable = true;
    externalInterface = "wlp2s0";
    internalInterfaces = ["microbr"];
  };

  #----------------------------
  # Host options: ssh
  programs.ssh = {
    extraConfig = sshExtraConfig;
  };
}

To start and connect to a VM, the following is commands are all that is needed:

systemctl start microvm@vm-project-a.service
ssh vm-project-a

Future Improvements

In the future, there are two improvements I still want to make to my setup. Both concern the problem of portability between systems: For one, the VMs depend on the manually created configuration directories in which I keep, among other things, the LLM credentials. The central problem here is that the LLM provider credentials do not exist in the nix store (and should not). The second problem concerns networking, in whose configuration my adapter names are currently hard-coded.

In the long run, both problems could be solved by wrapping the microvm.nix VM start service in its own script that creates the necessary adapters as well as the directories ad-hoc.