Docker-compose to podman

I migrated my services from docker-compose files to podman1.

Why? Essentially because it was easier getting rootless mode working with podman.

Actually, I initially tried to get docker to work in rootless mode2 and in user namespace isolation mode3, failing in both tries. More precisely:

For docker rootless mode, installation procedure is not really smooth with archlinux arm (I had to edit the installation script). Even after a successfull installation, I ended up with segfault, from this point I lost confidence in getting it to work correcly.

For user namespace isolation mode, the issue lied in mounting volumes which failed even with the rights correcly setup4.

I am not an expert in container technology nor in linux namespace. Maybe I overlooked something important, thus the issues I faced.

From this point I decided to let podman a chance.

Road to podman

Podman is an alternative to docker, it offers similar features to docker but it lacks docker-compose niceities… This is disappointing since alot of my self-hosted service uses docker-compose features extensively.

However, podman has a concept of pod5 as in kubernetes. The basic idea behind a pod is that it holds several container in a tightly coupled environment, the containers in a pod shares the same resources and are co-located.
For exemple, two containers inside the same pod can reference themselves through localhost since it shares the same network namespace6 7.

Although not the same feature set than docker-compose. The concept of pod is sufficient for encapsulating software stacks in the context of a single machine self hosted environment.

Additionnaly, Podman has been thought to integrate well with systemd and offers tools to generate systemd’s units. This can be used to replace the basic orchestrating abilities of docker-compose.

Table 1: Equivalent of some docker and podman's features
Docker feature Equivalent in podman’s world
docker build buildah bud
docker run podman run
docker-compose podman’s pods and systemd

Exemple: migrating docker-compose of miniflux8 service

We’ll see the migration more in details taking miniflux as an example.

Create miniflux user and setting it up

Since we are going full ret… rootless, we will be assigning dedicated user for the miniflux service.

# We will be assigning uid 100000 to miniflux user
useradd -m -u 100000 miniflux
groupmod -g 100000 miniflux
passwd miniflux
loginctl enable-linger miniflux # Mandatory for miniflux being able to launch systemd services without having to login manually
# see podman rootless mode documentation to understand subuids
echo "miniflux:100000:65536" \
  | sudo tee -a /etc/subuid \
  | sudo tee -a /etc/subgid
runuser -l miniflux -c "mkdir -p ~/.config/systemd/user" # Create folder holding definition of systemd unit
sudo machinectl login # login as miniflux
Code Snippet 1: Miniflux user creation

We are using sudo machinectl login to log as miniflux since a simple su - miniflux will not set environment correctly and you won’t be able to use user instance of systemd.

A quick look at existing docker-compose service definition

version: '2'

services:
  miniflux:
    image: miniflux/miniflux:2.0.25
    mem_limit: 128m
    restart: always
    env_file:
      - ./miniflux.secrets.env
    ports:
      - 8680:8080
    depends_on:
      - db
  db:
    image: postgres:13.1-alpine
    mem_limit: 96m
    restart: always
    volumes:
      - /mnt/disks/miniflux-db:/var/lib/postgresql/data
    env_file:
      - ./postgres.secrets.env
Code Snippet 2: docker-compose.yml file for miniflux stack

Note the exposed port: 8680 and the bind mount for db data.

Assign correct permission to the bind mount source

Since we will run rootless container. root inside the container doesn’t correspond to the root of the host. This has an impact on volumes and bind mounts.

Table 2: Correspondance of users in host and in db container
username:uid in container corresponding uid in host
root:0 100000
postgres:70 100070

Actually, in our case, we want to grant correct permissions to /mnt/disks/miniflux-db allowing db container to access it.
The db container runs under postgres user, which corresponds to uid 100070 in host. Fortunately, we can chown directly to an uid:

sudo chown -R 100070 /mnt/disks/miniflux-db
Code Snippet 3: chown to user 100070

Create equivalent services in podman

We will be creating a pod holding miniflux application and the database.
First, the pod:

pod create --name miniflux -p 8680:8080
podman run -d --pod=miniflux --memory 96m --name db -v /mnt/disks/miniflux-db:/var/lib/postgresql/data --env-file ./postgres.env postgres:13.1-alpine
podman run -d --pod=miniflux --memory 128m --name app --env-file ./miniflux.env miniflux/miniflux:2.0.25
Code Snippet 4: Creation of miniflux in podman

Note that the port exposition is defined at pod’s level, port 8680 on the host will correspond to 8080 within the pod.

Environment files being:

DATABASE_URL=postgres://miniflux:password@127.0.0.1/miniflux?sslmode=disable
BASE_URL=https://miniflux.nimamoh.net/
RUN_MIGRATIONS=1
CREATE_ADMIN=1
ADMIN_USERNAME=username
ADMIN_PASSWORD=password
Code Snippet 5: miniflux.env
POSTGRES_USER=miniflux
POSTGRES_PASSWORD=password
Code Snippet 6: postgres.env

Note that the app container address the db container with 127.0.0.1 since both container are on the same pod.

If everything went OK, you’ll have one pod with two running container and host port 8680 pointing to miniflux service.

Service lifecycle with systemd

We’ll now use systemd to control miniflux service. Podman is able to generate the systemd’s unit files via command:

$ podman generate systemd --new --name --files --restart-policy no miniflux
/home/miniflux/pod-miniflux.service
/home/miniflux/container-app.service
/home/miniflux/container-db.service
Code Snippet 7: Command to generate systemd's unit files

It will give you pod-miniflux.service, container-db.service and container-app.service. Respectively responsible of the pod, db and app lifecycles.

Our service definition lacks the notion of app being dependent of db container, in docker-compose we would have used depends_on9.
We can achieve similar result using After option of the unit file in container-app.service. Adding container-db.service to it will tell systemd that container-app should be started after container-db.

# container-app.service
# autogenerated by Podman 3.2.2
# Wed Jun 30 19:11:42 CEST 2021

[Unit]
Description=Podman container-app.service
...
BindsTo=pod-miniflux.service
After=pod-miniflux.service container-db.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
...
Code Snippet 8: container-app.service file

You can edit systemd service files further according to your need. Personnaly I would recommand to commentting the ExecStopPost step, in order to let the container live in case of failure (to be able to debug it).
Once finished, install, enable and start the services:

$ mv *.service ~/.config/systemd/user
$ systemctl --user daemon-reload && systemctl --user enable --now pod-miniflux.service
Code Snippet 9: systemd's services installation

Conclusion

Now, we have a regular systemd service running with user privileges corresponding to our miniflux service. It better fits the principle of least privilege principle. Additionnaly, podman have no daemon, which is a plus in term of simplicity.

On the other hand, systemd’s unit files are less expressive than docker-compose files and require some systemd knowledge.

Other techniques exists, like using kubernetes yaml10 (which podman is able to read) or using podman-compose project.

Troubleshooting, gotchas, tips and tricks

Address host from a rootless container

The container can target the host using its IP address, we can expose host ip through an env variable like this

$ podman run -ti --rm --env HOST_IP=(ip -br -4 addr | grep -v 'lo' | awk '{ print $3 }' | grep -P -o "[0-9\.]+(?=/)") --name container ubuntu bash
Code Snippet 10: Run podman container passing host ip as environment variable. (this is fish shell)

Local volume with options cannot be used in rootless mode

It can be interesting to customize volume creation, for example create a volume which points to a btrfs subvolume:

podman volume create --driver local --opt type=btrfs --opt device=/dev/sda --opt o=subvol=/data data
Code Snippet 11: Create a volume pointing to btrfs subvolume.

However, these kind of volume cannot be mounted in rootless mode, you will get this kind of error trying to do so:

$ podman run -ti --rm --name container -v data:/data ubuntu bash
Error: error mounting volume data for container xxx: cannot mount volumes without root privileges: operation requires root privileges
Code Snippet 12: Error when trying to mount custom local volume

OCI permission denied

On podman’s run command, I had this error.

Error: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied
Code Snippet 13: Error when running container.

Solution: replace runc by crun

As stated here and here, some version of runc appears to be too old. Replacing the runtime by crun fixed the issue.

pacman -S crun
echo 'runtime = "crun"' | sudo tee -a /etc/containers/containers.conf
Code Snippet 14: Commands to change runtime container to crun

  1. Podman ↩︎

  2. docker rootless documentation ↩︎

  3. docker user namespace isolation ↩︎

  4. Docker user namespace isolation: permission denied with bind mount ↩︎

  5. Kubernetes definition of a pod ↩︎

  6. manpage of linux namespace ↩︎

  7. namepage of network namespace ↩︎

  8. Miniflux website ↩︎

  9. Docker compose reference - depends_on ↩︎

  10. From Docker compose to Kubernetes with Podman ↩︎