Immuarch

Summary

Immuarch is a minimal toolset for immutable arch-linux, with atomic, transactionnal updates and integrated backup solution and security features:

In opposition to other projects of this kind, this is not an arch-based distribution, and do not intend to be, our purpose is to integrate in in the arch-ecosystem and preserve the liberty and principles Arch provides to users.

The principle is to mount both / and /home as writable volatibe in-RAM overlays on read-only BTRFS subvolumes. Nothing is wrote on disk without the explicit user control. Using overlays allow the user to tinker the system as he see fit and commit changes without compromising recoverability, a characteristic Archlinux sometimes lacks.

The « immuarch-evolve » tool provides what is necessary to make changes to the lower read-only layer using BTRFS snapshots of the previous home & root iterations and mounting those in a evolution runtime. You can also import an OCI from podman (optional dependency) for clean, reproducible system build.

Immuarch provides the user with a GRUB menu where he can choose to boot to the current or the choosen number of previous iteration. Support for full-disk encrypted setup (with grub setup on EFI) is provided, as well to boot any kernel in /boot for each iteration.

Users can also opt-in to security features provided by Immuarch, to harden their system to a full boot chain-of-trust, from the UEFI to userpace. More info in the FAQ at the bottom of the README

Disclaimer

This is done in the Archlinux spirit and embraces KISS philosophy, you will have to edit and maintain some textfiles to make this work and most management is done in CLI with the evolve script.

And just in case, this isn't declarative like NixOS, it's managed and updated like good old Archlinux with some additional wizardry involved. However, you can have declarative-ish build by building an OCI image the way you see fit, and import it with immuarch-evolve as a root iteration.

Dependencies

See dependencies in the PKGBUILD or AUR package.

How to install

You have two possibility for installing Immuarch :

• Install from scratch (safer and cleaner, but longer)
• Install inplace after a normal Arch install (easier)

Install from scratch (manual installation from archiso) / for imaging

Refer to arch-wiki for more info on things relative to a normal arch install

1. Format your disk

• The minimal layout is one EFI partition and one BTRFS partition
• Instead of the BTRFS partition, you can have a LUKS2 partition which will containe the BTRFS system (recommended)

2. Set up the efi filesystem and the second filesystem

3. Mount the BTRFS filesystem in /mnt

4. Run mkdir /mnt/immuarch-subvolumes && chmod 700 /mnt/immuarch-subvolumes

5. Create the BTRFS subvolumes

   • Run export TS="$(date +%F_%s)"
   • Run btrfs subvolume create /mnt/immuarch-subvolume/ROOT_$TS
   • Run btrfs subvolume create /mnt/immuarch-subvolume/HOME_$TS
   • Run btrfs subvolume create /mnt/immuarch-subvolume/VARHOME (if you need/want it)
   • Run btrfs subvolume create /mnt/immuarch-subvolume/SWAP (if you need it)

6. Mount the subvolumes and /efi

   • Run mount --mkdir --bind /mnt/immuarch-subvolume/HOME_$TS /mnt/immuarch-subvolume/ROOT_$TS/home
   • Run mount --mkdir --bind /mnt /mnt/immuarch-subvolume/ROOT_$TS/immuarch
   • Run mount --mkdir /dev/DISK_PART_1 /mnt/immuarch-subvolume/ROOT_$TS/efi (DISK_PART_1 : sda1,nvmeon1p1, etc…)
   • If you created VARHOME : mount --mkdir --bind /mnt/immuarch-subvolume/VARHOME /mnt/immuarch-subvolume/ROOT_$TS/varhome

7. Run pacstrap -K /mnt/immuarch-subvolume/ROOT_$TS base base-devel git linux efibootmgr (and other stuff you want, you'll need a text editor as well)

8. Chroot in the root subvolume :

   • Run arch-chroot /mnt/immuarch-subvolume/ROOT_$TS
   • Comment CheckSpace in /etc/pacman.conf

9. Follow the archlinux install guide until the bootloader section. DO NOT REBOOT.

10. Run mkdir /efi/EFI/arch (you can replace arch by whatever name you wish, immuarch,grub,etc...)

11. If you haven't done yet, create a user. It will be used to run makepkg, as running makepkg as root is a security disaster and should never be done.

12. Run cd tmp && git clone https://aur.archlinux.org/immuarch-git.git && chown -R YOURUSER:YOURUSER immuarch-git && cd immuarch-git && sudo -u YOURUSER makepkg

13. Install the packages :

• pacman -U immuarch-{core,utils}-git*
• If you want/must, you can install verity as well with pacman -U immuarch-verity-git*. Beware : verity has a dependency on aide, so you shall install it before. Besides, aide need to manually import a GPG key before installing.

14. Setup the immuarch config /immuarch/immuarch-etc/env.conf with your favorite text editor. If this is of help, in vim, you can paste the result of a command with :!r CMD

• If you want as swap, setup the swapfile according to Archwiki, you can put the fstab entries in /immuarch/immuarch-etc/add-fstab. You can also add there other mountpoints thall will not be managed by Immuarch (for instance, an external storage disk).
• If you do not want VARHOME, remove or comment the config line in /immuarch/immuarch-etc/subvolumes.conf

16. Run touch /immuarch/immuarch-etc/.INSTALLED

17. Run btrfs subvolume set-default /immuarch/immuarch-subvolume/ROOT_$TS

18. Add hook sd-immuarch in /etc/mkinitcpio.conf in the HOOKS array.

19. If everything is alright, you should be able to run IMMUARCH_MANUAL_INSTALL=y immuarch-evolve -Jp and see one entry. Otherwise, the guard script wil complain and give you infos and what's wrong.

20. If you use full-disk encryption, don't forget to properly setup the initrd-time unlock, using the sd-encrypt hook (refer to arch doc, again)

21. run IMMUARCH_MANUAL_INSTALL=y immuarch-evolve -Ry

22. Check the reboot output log, if it's good, go to next step.

23. Register grub to UEFI, for instance, if your disk is /dev/sda and your efi app is called arch (adapt this to your config) : efibootmgr --create --disk /dev/sda --part 1 --loader '\EFI\arch\grubx64,efi' --label 'Immuarch GRUB bootloader' --unicode.

24. If the last command passed, you're done, congrats, you can reboot !

In-place installation

To simplify things for users, you can install Immuarch in-place in a previously installed Arch system. I recommend to do it on a clean, minimal and up-to-date system.

If you want to install Immuarch on a old or important Arch system, please do backup before and be sure you have a way to rollback.

1. First step is doing a normal Archlinux install with the following requirements. You'll need :

• An efi partition, with the GRUB bootloader installed on it
• A btrfs partition
  You can either do the classic manual install or use arch-install (if you know what you're doing obviously).
  You can setup other partitions like swap/storage/etc as you see fit.
  Optionally, you can follow the Archwiki documentation for full disk encryption with dm-crypt/cryptsetup/luks2/sd-encrypt, or stuff like dm-verity/dm-integry for further fiability.

2. Make sure you use the systemd mkinitcpio hook and not the udev one in etc/mkinitcpio.conf

3. install the immuarch-utils-git and immuarch-core-git package, either through AUR of by building the package yourself from this repo (the PKGBUILD in pkgs/immuarch). The core package is for files outside the Filesystem Hierarchy Standard (/immuarch) while the utils package is for files inside it.

4. edit the configuration file /immuarch/immuarch-etc/env.conf and file the needed fields, eventually, you may want to checkout /immuarch/immuarch-etc/add-fstab.conf to setup additional mount points in the system as you see fit.

5. run immuarch-evolve -B. The script may complain about your system config and tell you to fix some stuff before accepting to run. Be very careful before running this command, because if something goes wrong, you will not be able to reboot and may suffer data loss.

6. restart the machine

7. The first start you will run immuarch-evolve, before the operations you requested, some post-bootstrap cleanup will occur. After this step, you're done, enjoy your immutable archlinux !

Useful things to do post install

1. • You may want to create a directory for you personal user account in /varhome, with same permission as your home directory. If you do want your homedir to stay mutable, you can either :
     • simply set the homedir of your user in /varhome/USER
     • immuarch-evolve -L /varhome/HOMEDIR /home/HOMEDIR
     • immuarch-evolve -M VARHOME_USER /home/HOMEDIR
       The first command will create a symlink, the second a btrfs subvolume and a mount point. The mount option may be a bit safer because sometimes things became a little bit complicated with symlinks (when using sandboxed app for instance).

Though I do recommand giving immutable home with a more discretionary approach to what you allow to mutate or not.

2. You will also have to consider how to manage parts of the filesystem hierarchy on disk that are usually changing during runtime and accross reboots. The most straightforward way is simple to allow /var to be mutable with immuarch-evolve -V VARVAR /var.
   You may also opt to not do it and tweak thinks with more granularity alike HOMEDIR management. Here are some thing thas usually need to mutate to function the intended way :

   • /var/log for system logs
   • /var/lib/systemd/timers for systemd timer last exec timestamps
   • /var/tmp for across-reboot temporary file (needed for instance if you use podman, to avoid filling your RAM)
   • /var/lib/docker, var/lib/containers for docker/podman, this one may be of particular interest if you want to contruct your system with OCI images
   • /var/lib/flatpak for flatpak...
   • wherever Nix store its stuff

3. Harden your system by setting up a complet boot chain-of-trust :

   • Install immuarch-verity-git. This will perform integrity check at boot time on immuarch files and root filesystem and stop the boot process if the system has been tampered.
     INFO : this will slowdown both boot and immuarch-evolve system updates.
     INFO : install this package using the immuarch-evolve -u option. Otherwise, this won't work.

   • Configure SecureBoot and GRUB kernel/initramfs/config signing

How to update Immuarch

1. run immuarch-evolve in mode -N and with option -u. The mode -N is interactive root fs update, option -u mount /immuarch in the transition chroot.

2. then, install/update immuarch-core-git as you would for any other arch package.

How to use

immuarch-evolve

immuarch-evolve.sh MODE [OPTIONS...] [MODE_ARGS...]

Evolve is Immuarch immutable system management utility. It features several 
critical operations related to such systems mostly atomic/transactional updates,
managing the GRUB bootloader config and setting the default system version 

Modes :
  -N                       : (next) edit the root fs in interactive shell
  -C [FILENAMES...]        : (commit) commit file(s) from root/home overlayfs
  -L TARGET LINK           : (link) create a symlink to a target in a read-write volume
  -U LINK                  : (unlink) remove a symlink to a target in a read-write volume
  -S [SV_NAME]               : (switch default root) switch default root subvolume (interactive if no arg given)
  -V SV_NAME DIR [OPTS...] : (makeVolume) make a btrfs subvolume mounted at directory in root filesystem (ex: /var), with OPTS options
  -I DIR                   : (makeImmutable) reverse makeVolume for specified directory (NOT IMPLEMENTED)
  -J [N]                   : (journal) display N last logs
  -R                       : (rebuild) create a new iteration without any particular operation
  -O                       : (importOCI) import and convert an OCI image (podman needed)
  -F PATH                  : (script File) update the system with a script to be executed in the transition env (NOT IMPLEMENTED)
  -Q                       : (query file) check if a file is mutable in the current setup or has been modified in the current overlay root (NOT IMPLEMENTED)
  -H                       : (help) print help
  -B                       : (bootstrap) install Immuarch on this system
  -T [SV_NAME]             : (softReboot) perform systemd soft-reboot, either in default root subvolume, or specified root subvolume (EXPERIMENTAL)

Options :
  -m MESSAGE   : (message) commit message to system update
  -f SV_NAME   : (fork) use another root as starting point
  -c           : (change) when using fork, make the resulting subvolume the default root sv
  -y           : (yes mode/non-interactive) 
  -d           : (dry-run) : Do a test run and do not perform changes
  -h           : (help) print help 
  -p           : (pretty) prettier journal output
  -u           : (update) mount /immuarch to in the transition environment to allow Immuarch update through pacman (or other mean).
  -i           : (inplace) Perform an operation inplace on existing subvolume instead of creating a new one (EXPERIMENTAL)
  -s           : (skipHooks) when performing update, do not execute hooks

Manual

INFO : SV_NAME design the name of the subvolume directory in /immuarch/immuarch-subvolumes

You will usually use -C mode after you modified configuration files in your system or home directory, for instance lets say you want to update /etc/hostname. The first step it to edit /etc/hostname normally, then, use immmuarch-evolve -C /etc/hostname to build a new root filesystem with the desired modification to the file. You can specify a commit message with -m option : immuarch-evolve -Cm "updated system hostname" /etc/hostname. Finally, once you understood how the command work, you can skip the interactive confirmation with the -y option : immuarch-evolve -Cym "updated system hostname" /etc/hostname.
TIP : If you're not sure if a file has been modified, you can check in /run/immuarch/cowspace{home_upper,root_upper}. If a file is present in this hierarchy, it has been modified.
TIP 2 : If you want to know exactly what has been modified, you can for instance diff /run/immuarch/root_ro/etc/passwd /run/immuarch/cowspace/root_upper/etc/passwd.

-L is usually used for linking files or directory in your home directory to /varhome or other volume to allow to be written. For instance, if you want to be able to edit your .vimrc and persist changes without needing to commit the file everytime, you can do immuarch-evolve -L /varhome/YOUR_USER/.vimrc ~/.vimrc. if /varhome/YOUR_USER/.vimrc does not exist, the file ~/.vimrc will be copied to the destination before the link is created.

-S allows you to interactively or non-interactively (by passing SV_NAME as argument) change the default subvolume (the one grub will boot to by default).

-f allows you to create a new root filesystem from another filesystem by passing SV_NAME as option argument. By default, -f does not change the default root. To change the default subvolume to the new root, you can pass -c option with -f.

-N allows you to interactively edit the root filesystem. You will be chrooted as root in the new root fs to do your things, then will be prompted for confirmation. I recommend avoiding using -N mode when alternative ways are available, because it's more risky and you could nuke the system with wrong manipulations. It's still useful for instance to update repo/aur packages.

-R is mostly useful after an immuarch update, or immuarch config change to rebuild the GRUB config, /etc/fstab in a new fs, and the initcpio.

-J displays the changelog of the system. You can use this before using -S to get the SV_NAME you want as the default subvolume. -Jp displays the changelog in a prettier, with generated name for each system iteration to be identified in GRUB menu when booting. It also make easier to find the parent of a filesystem.

-O allows to import an OCI image present on the system with podman. This feature is mostly intended to allow the user to build the root filesystem in a reproductible, declarative way using ContainerFile (Dockerfile), bash buildah script, or any tool that can build an OCI image, hence offering freedom and interfacing with well-known and defined standards.
If you want to use this feature, I suggest to set driver="btrfs" in your /etc/containers/storage.conf. This will make the better use of BTRFS copy-on-write and data deduplication feature but it's not a hard requirement. Please note that it's not possible by default (at least with podman/buildah) to write to the following files in an image as they are bind-mounted and cannot be unmounted :

• /etc/resolv.conf
• /etc/hostname
• /etc/hosts

As a workaround, you can put these files in your image with a .new suffix and Immuarch will take care of overriding the previous with these during import. Alternatively, you can use equivalents of options --dns=none --no-hostname --no-hosts (those are from buildah) in your build environment.
Please note : Importing an image will NOT set the newly created ROOT subvolumes to the default BTRFS subvolume. You'll have to switch manually with -S. You can find an example of toolset for building your system as an OCI image with buildah in the doc/examples/image-build directory of this repo.

-i option allow to perform an update on already existing subvolume inplace. While this sort of « break » Immuarch intended usage, it may be useful to fix grub/fstab/initcpios after a configuration error, or perform a batch of sequential changes. It is also useful for changing the commit message with option -m.
Please note that using -i option on the subvolume currently booted and mounted at /run/immuarch/root_ro/ is forbidden (overlayfs does not support writing to the lower layer and this result in undefined behaviour).

Immuarch allows you to setup hooks in the form of bash scripts, before and after any update. You can simply drop them in /immuarch/immuarch-etc/hooks pre-overlay.d pre.d or post.d post-overlay.d directory. This is useful for instance for updating services like AIDE or rkhunter that rely on a database of files hash to enforce verity/integrity checks. overlay are executed in the volatile FS currently mounted as /, while the other are executed in the evolution chroot. Hooks are executed in the lexicographic order of the files names. See doc/examples/hooks for examples.

More documentation will come.

immuarch-backup

Backup a choosen list of subvolumes to a remote host by SSH (need to have setup a passwordless key)

This is entirely configured by env.conf for now. I may add options later to allow users to customize the backup policy they see fit (for instance different frequency per subvolume type)

The script run without any argument.

More documentation will come.

Example worflow

My current workflow with Immuarch on my main machine :

• My homedir is immutable
• Most config files remain immutable as well, and are updated with the -C mode
• I have some dotfiles and config files in ~/.config, as well as stuff in ~/.local/share and ~/.cache I want to be mutable. In that case I just symlink those to /varhome/myuser/SAME_PATH
• In addition to /varhome/myuser, i have a /vault mounted subvolume with /vault/myuser directory that I use for personal data (photo,music,notes,otherstuff).
  Most applications I use are sandboxed using firejail and are strictly forbidden to access /vault at all.
• The following system directories are mounted to RW subvolumes :
  • /var/tmp (for podman)
  • /var/lib/containers (for podman as well)
  • /var/lib/systemd/timers
• For updating the immutable system :
  • I usually tweak and test stuff in the root system mounted on RAM
  • When I'm satistfied with the change, I commit them with evolve -C (and -L & -N mode if needed).
    I also copy the updated/new files to a git repository. This git repository purpose is to rebuild my system as a podman OCI image when needed.
  • Occasionally, I rebuild the system using buildah (see example in doc/example/image-build) and import it using evolve -O mode.
• I almost never update packages using pacman or AUR helper. I rebuild the whole system as previously stated. This removes the headache of dealing with AUR packages that need rebuild after a dependency is updated. In addition, I'm aware if there is a problem with a package I need before running my system into the problem as the image build will fail.

File System Hierarchy

Most Immuarch things are outside the FSH to avoid interactions with other system stuff.

Everything in the immuarch-core-git package is in /immuarch. Everything in the normal linux FHS is in immuarch-utils.

/immuarch is / on top-level subvolume (subvolid=5)

• /immuarch : Immuarch related stuff
• /immuarch/immuarch-bin : All scripts and other utilities
• /immuarch/immuarch-lib : All dependencies and non-executables components
• /immuarch/immuarch-etc : Configuration
• /immuarch/immuarch-subvolumes : All subvolumes
• /immuarch/immuarch-journal : System changes journal
• /immuarch/immuarch-verity : Immuarch AIDE database store
• /immuarch/.immuarch-backup-workspace : Working directory for backup system
• /varhome : mutable home (for dynlinking config and stuff, created by default, but not mandatory to have)
• /run/immuarch-ofs : directories for the overlays filesystem, created at the end of the initramfs stage

Tools

• /immuarch/immuarch-bin/evolve.sh : Mutate system
• /immuarch/immuarch-bin/backup.sh : Perform BTRFS snapshots system backup on remote host using SSH

Troubleshooting

Pacman failing to update in transition environment

If you encounter this error when trying to install/update packages with pacman in the transition chroot environment :

error: could not determine cachedir mount point /var/cache/pacman/pkg/download-xxxx
error: failed to commit transaction (not enough free disk space)
Errors occurred, no packages were upgraded.

You can comment/remove the following config line in /etc/pacman.conf :

CheckSpace

This seems to be occuring because we chroot directly in the BTRFS subvolume without mounting it before. I tried to fix this with a proper mount but encounter other issues so I recommand this for now.

The system boot but complain it is in a read-only root filesystem or couldn't mount /home

This is happening because your mkinitcpio.conf use the Busybox based hooks for generating the initial ramdisk (initramfs), currently only systemd-based initramfs are supported by this project. The transition is mostly painless, the only complication is for users with full disk encryption. In which case you have to follow the Arch documentation in the matter.

I plan to add support for the busybox-based hook as well. I choose systemd because I understood it was now the default but the last arch install I'v done from the official material still use the old hooks it seems.

FAQ

Why Immuarch ?

Rolling release distros are cool, but I don't fancy being some obscure project I have installed as a dependency of dependency beta-tester when they decide to do something extremely stupid that break userspace. In addition, I like tweaking and sometimes I also break stuff.

So two/three years ago while I was a student I started to hack something during boring marketing and management lessons to have immutable archlinux with atomic updates and rollback on my main machine in case something break.

I came up with some system conf, a custom file hierarchy and most importantly an absolutely garbage script « evolve.sh » that did the trick.

Since them, I wanted to iterate on this stuff to improve it and have a better version, easily installed over a basic arch install (with few requirements like having BTRFS, grub, overlaysfs and stuff...) so that I can easily use it on my others machines and distribute it on the AUR for people with similar needs.

Why should I use this instead of, let's say NixOS ?

It depends of which OS flavour you like.

I absolutely understand the love for infra as code and I am myself very fond of it and push for it where I can in professional context (mostly other techs for practical reasons tho) where what I and my employer need is reproductibility, accountability and fiability.

However, I am also a computer hobbyist and most importantly, an empiricist. I love to tinker, experiment and eventually learn from it. Arch is the perfect linux distro for me because it is uncomplicated and (almost) unopiniated. I can setup most thing the way I want and yet remain backed up by quality community work.

Immuarch to me is a great compromise between system fiability and Arch flexibility and community support.

Why bash ?

Mostly because when I started this project, before it became part of my daily driver setup, it started as a quick experiment with no such pretention at all. I grabbed bash for this because it's my default for system scripting and coreutils usally provides the most of what I need. I expanded it upon this crappy base.

I do plan thought to rewrite the tools eventually in a more robust, compiled, with a solid type system language. I don't know which one yet, but it will probably be either Rust or Haskell. Bash is fine thought, just a little bit messy.

What's the project stance towards systemd

The most unopiniated way for such a project : I'll try to stick to the default Archlinux level of integration with systemd and move with the wind.

Does this project support UKI ?

For the moment, no, but support is planned.

Does Immuarch protect the system from tampering between and after boot ?

Between boots

With the minimal setup, no. If you want protection against tampering, you can configure full-disk-encryption using cryptsetup/dm-crypt/LUKS2, which is supported by Immuarch. This will protect :

• the kernel
• the initramfs
• all data on the BTRFS partition.

Additionnaly, Immuarch support a full boot chain-of-trust with :

• Secure-boot signature of the GRUB payload
• GPG Signing of kernels and initramfs, GPG signature check with GRUB.
• Integrity/Verity checkup at boot with immuarch-verity package (beware, this will slow boot and updates)

After boots

The /immuarch subvolume is always set to read-only, using BTRFS ro property after updates. The newly created subvolume is also set to read-only after all operations are completed. While it is an interesting failsafe, it's not enough and is mostly cosmetic for security.

Therefore, Immuarch provide a package : immuarch-verity, that will perform verity/integrity check at boot (and during runtime if user setup a systemd unit for that).

In addition, Immuarch allow users to harden the system by mounting /usr, /etc, /boot read-only instead of volatile.

With the upper layer of the root/home overlays being tmpfs, do you eventually run out of RAM ?

On my main machine, with 16GB memory, I never ran out of memory in several years writing in the tmpfs, despite sometimes working with several-GB payload I « write » in the overlayfs upper-layer, and my system running for days without reboot.

As a failsafe, I have a btrfs swapfile, configured according to archwiki guide, with very low swapiness, but my system almost never swap.

Why is there two different package ?

Because everything under /immuarch is unique and persist across ROOT filesystem versions. Therefore if your usage of Immuarch is maintaining two or more lines of ROOT fs, you will have a divergence in the package manager database regarding what is supposed to be there and what is actually is. Therefore the immuarch-utils package is to be installed on all those ROOT fs, but you don't have to necessarily install imumuarch-core everywhere to avoid dealing with package manager headache.

Does this project use generative AI ?

Nope.