Migrating from Proxmox+Docker to CoreOS+Podman

Previous Setup

I started my home server journey with an Ubuntu server, then quickly moved to Proxmox. Both being Debian-based, it made it easy to switch. I then took advantage of the Proxmox features, the web-ui, Proxmox Backup Server, and LXC container management. Managing LXC containers was made easy in Proxmox, but each service was still maintained and updated individually. This drew me to docker.

docker made services easy to setup. Docker kept the abstraction from the operating system, but the setup for each container was already done. I found that all of my services could be done in docker, and were actually supported the best through it.

Why Migrate

I didn’t need some of the major features of Proxmox anymore, and didn’t like my relience those features for my setup. The web-ui is proxmox dependent, but one able to be setup on any operating system would be preferable. I had almost no LXC containers in use anymore, opting to use docker. Docker can be used on a wide variety of operating systems and environments, so my proxmox setup should be reproducible on the next OS of choice. However, Docker is not the only container management tool. Proxmox Backup was very useful for data replication of services but required a dedicated server for its setup, which was another depence.

Choosing Alternatives

Since I no longer needed Proxmox features, I no longer needed Proxmox. Proxmox provided a stable foundation for my containers, but packages were not as updated (like nvidia drivers). A similar rolling release distribution is Fedora, which has a specific server version called CoreOS. The major difference is that this distribution is immutable, having image based and automatic updates.

This approach to my base system will allow for a very stable base, one that will be kept up-to-date in a stable hands-free way. However, this does come with a price. The system image cannot be changed, so updates to these packages cannot typically happen without reboot, some configurations may be harder to set up there, and the system comes with some opionionated choices.

One such choice is that of podman over docker. Podman is a container management tool like Docker, but can run totally rootless by default. Podman utilizes a ‘fork/execute’ model instead of the Docker ‘client/server’ approach. Meaning, podman will run the processes of a container from the user that runs the command, where as docker requires every container command to communicate with the Docker Daemon — a child of the init system. This just means that podman has a better security model for containers. As Docker has Compose, Podman has Quadlets. Quadlets are a podman container configuration file that integrates directly with systemd to run the containers as services. This allows me to leverage the stengths of container defined in file with the added benefit of working directly with my OS service manager.

A web-ui for server management has been beneficial for system resource graphs, log management, and other administrative features without ssh. uCore comes with Cockpit web-ui by default, which does all that the proxmox-ui did, but integrates with the Fedora operating systems and podman containers.

Promox Backup Server needed a replacement as well, of which I found restic and autorestic. Autorestic allows you to define restic commands in YAML format, making automatic backups easy. Restic allows you backup files locally, remotely through sftp, and through many other cloud services. Restic includes backup configuration such as pruning, compression, and verification, plus easy restoration. Proxmox Backup server required both a Proxmox server and dedicated backup server, where as restic allows for much greater backup flexibility.

Migration

uCore Setup

The installation of uCore is more involved than that of a typical Linux distribution, lacking any gui or interactive install. The tool ‘Ignition’ is a provisioning utility that reads a JSON configuration file and produces a Fedora CoreOS system from it. Butane is another tool that converts a more readable YAML config into the Ignition JSON config.

I followed the Auto-Rebase installation method defined on the uCore Github. It gives an example config, where I just changed the password hash and version of uCore. The version ucore-hci:stable-nvidia-zfs comes with all available features for uCore, including Nvidia drivers for hardware acceleration and ZFS drivers for my ZRAID. The config was then converted with ‘Butane’ and then installed with coreos-installer.

Container Management Rollover

Rolling over my docker services to podman required tinkering with permissions and incremental steps. The first of which, was to recreate my current setup with docker. Although CoreOS supports mostly podman over docker, uCore also has moby-engine (a free Docker implementation) installed by default. I was previously using a set of docker-compose files that defined my services, which after migrating my service files, worked mostly on first run. The largest issue was that of selinux permissions with volumes, where the typical volume declaration did not work. The solution was to add volume labels ‘:z’ or ‘:Z’, these add the selinux permissions to the bind mount.

I could have stuck with my implementation in docker-compose, but the support for podman in CoreOS is better, the security of podman is better, and the Podman Quadlets integrates well with the OS. The next step to migrating to podman is through podman-compose. This is different than Quadlets, where it tries to allow the compose syntax but convert the commands to podman.