Building disposable Proxmox VMs from mutable base VMs with ZFS

When I moved my lab from VMware Workstation to Proxmox VE, one of the first things I missed was linked clones.

In VMware Workstation, I could take a snapshot of a normal VM and create a linked clone from it. The clone was available almost immediately and used very little additional storage at first. It was ideal for disposable test and development environments.

Proxmox VE also supports linked clones, but their source must be a template. That did not fit how I wanted to maintain my base machines. A template cannot be started like a normal VM, so updating it requires a separate workflow.

With several base systems, I would end up maintaining a collection of templates whose only purpose was to create clones. I wanted something simpler: keep each base as a normal VM, boot it when I needed to install updates or change its configuration, shut it down, and create new linked clones from that state.

PVE Forge implements that workflow using Proxmox VE and ZFS snapshots.

The basic idea

I keep two separate things:

• a base VM, which is a normal, bootable VM;
• a golden snapshot, which is the fixed point used to create clones.

The workflow is straightforward:

base VM
    │
    ├── boot, update and configure
    │
    ├── shut down and seal
    │
    ▼
golden ZFS snapshot
    │
    ├── linked clone
    └── full copy
            │
            ▼
       new Proxmox VM

The base VM can change over time. Existing clones keep using the snapshot from which they were created. New clones use the latest one.

This lets me update a base without breaking machines created from an older version.

Sealing a base VM

Before creating a golden snapshot, PVE Forge shuts down the base VM. If it does not stop, the command aborts.

I do not use QEMU Guest Agent in these images. Without it, a snapshot of a running VM would not include a guest-assisted filesystem freeze. Requiring the VM to be stopped avoids that problem.

Once it is stopped, PVE Forge runs virt-sysprep against the ZFS volume. It removes SSH host keys, users’ .ssh directories—including manually managed authorized_keys—the machine ID, logs, temporary files, and old cloud-init state. SSH access that is not provisioned again through cloud-init will therefore be lost when the base is sealed. It then creates a snapshot named like this:

golden-YYYYMMDD-HHMMSS

If that name already exists, for example because two seals finish in the same second, PVE Forge appends a numeric suffix such as -2.

Removing the old cloud-init state makes the next boot a new instance. The clone can then receive its own user, SSH keys, address, gateway, and DNS configuration from Proxmox.

Linked clones and full copies

The default mode creates a linked clone directly with ZFS:

zfs clone source/volume@golden-snapshot target/volume

The new volume is writable but initially shares its blocks with the snapshot. Creating it is practically instant and the initial storage cost is very small. It starts consuming more space as the guest writes new data.

This is the mode I use most often.

PVE Forge can also create an independent copy:

zfs send source/volume@golden-snapshot | zfs receive target/volume

After receiving the volume, it removes the copied snapshot from the target. The new VM starts from the same sealed state but no longer depends on it.

Both modes use the sealed snapshot. They do not copy the current disk state of a base VM that may be in the middle of an update.

Creating the Proxmox VM

Cloning the ZFS volume is only the first step. The new VM also needs a Proxmox configuration.

PVE Forge reads the configuration saved in the golden snapshot and copies the settings that should be inherited. It creates the rest again for the new VM:

• VMID and name;
• MAC address;
• SMBIOS UUID and VM generation ID;
• cloud-init volume;
• VLAN and static IP address;
• PVE Forge ownership and origin tags.

The tool currently expects every base to have one ZFS-backed scsi0 disk, one cloud-init drive, and one network interface. If it finds another disk or NIC, an EFI disk, or TPM state, it stops before cloning.

Those cases could be supported later. For now, I prefer a clear error to a clone that is missing part of its base VM.

Network configuration

Each VLAN in my lab has a static range reserved for these VMs. There is no DHCP in those VLANs, so PVE Forge only needs to avoid addresses already assigned to a Proxmox VM or container.

It reads their current configurations, skips the gateway, and chooses the first unused address in the configured range.

The first version of my script also read the snapshot sections stored inside Proxmox configuration files. As a result, an old IP could remain marked as used even when it only existed in a historical snapshot. PVE Forge now reads only the current section.

It does not use ping, ARP, DHCP, or an external IPAM service. The configured range must be reserved for the tool. That matches my network, so adding network probes would make the result more complicated without improving it.

Isolation and PCI passthrough

Some of my test VMs should be reachable over SSH but should not be able to start connections to the rest of the network. The --isolate option creates Proxmox firewall rules before the VM starts.

The rules drop inbound and outbound traffic, allow SSH from the management network, and can allow extra TCP ports for a specific configured base. Because the rules are applied by Proxmox, the guest cannot remove them.

PVE Forge checks that the tested pve-firewall backend is running before it creates an isolated VM. If the firewall is disabled, the command stops instead of creating a VM that only appears to be isolated.

For hardware tests, I can also pass one PCI device to a VM. I mainly use this to assign a complete USB controller. The tool adds the hostpci0 setting, but the host still needs a correct IOMMU and VFIO configuration.

Keeping old snapshots safely

Linked clones depend on their source snapshot. That snapshot cannot be deleted while the clone exists.

PVE Forge keeps a configurable number of recent golden snapshots. When there are older ones, it checks the OpenZFS clones property before removing them.

The rule is simple:

Never delete a snapshot that is still the origin of a live clone.

If an old snapshot still has clones, it stays. Once those VMs are gone, a later seal can remove it. If cleanup fails for another reason, PVE Forge also leaves the snapshot in place and tries again the next time.

Full copies do not have this dependency.

Rolling back older snapshots on ZFS

Snapshot retention and snapshot rollback are separate operations. Proxmox’s ZFS storage backend refuses to roll a volume back when snapshots newer than the selected one still exist.

For a disposable VM, PVE Forge therefore refuses an older rollback by default and lists the snapshots that would have to be removed. The destructive path must be requested explicitly:

pve-forge snapshot restore VM SNAPSHOT --force

Before deleting anything, the tool checks whether one of those newer snapshots still has a dependent ZFS clone. It then asks for confirmation, removes the newer snapshots from newest to oldest, and only then runs the rollback. The VM is left stopped so that starting the restored state remains a separate, explicit operation.

The deletion sequence is not transactional. If removing one snapshot fails, PVE Forge does not attempt the rollback, but any newer snapshots already removed remain deleted. The confirmation shows the complete deletion set before this process starts.

The same limitation can affect a base VM when someone creates a manual snapshot after the latest golden-* snapshot. In that case base recover refuses the operation unless it is repeated with --force; the newer snapshots are handled with the same dependency checks and deletion order.

Handling failures

Creating a VM changes several things: ZFS volumes, Proxmox configuration, cloud-init storage, and sometimes firewall and PCI settings. A command can fail halfway through.

Until the VM is fully configured, PVE Forge keeps track of its VMID and removes anything created by a failed attempt. Once configuration is complete, it stops doing automatic cleanup. If QEMU then fails to start, the VM remains available for inspection.

There are a few other checks:

• flock prevents two PVE Forge commands from allocating the same VMID or IP;
• destructive commands only accept VMs inside the configured range and carrying the pve-forge tag;
• a base VM cannot be destroyed while a linked clone still depends on one of its snapshots;
• ambiguous VM names are rejected;
• destructive operations ask for confirmation;
• the tool stops if it detects a Proxmox cluster.

Cluster support is outside the current scope. VMID allocation, storage, locking, and networking would all need to work across nodes. PVE Forge currently targets the standalone Proxmox host in my lab.

Current limits

PVE Forge currently supports:

• one standalone Proxmox VE node;
• ZFS-backed QEMU VMs;
• one disk and one NIC per base VM;
• reserved IPv4 /24 ranges;
• offline sealing and snapshots;
• the tested pve-firewall backend;
• one optional PCI passthrough device.

It does not create the base operating systems or install software inside them. I create each base VM once, configure it as Debian, Ubuntu, or any other system I need, and then let PVE Forge manage its snapshots and clones.

That is enough for my current workflow. A new disposable VM is one command away:

pve-forge create BASE NEW_VM

Here, BASE is the configured base name and NEW_VM is the name to give the new disposable VM.

The repository contains the code, installation instructions, an example configuration, tests, and a checklist for validating it on a real Proxmox host: PVE Forge.