Colin Walters, Red Hat, Inc. | walters@{,}

Types of software

  • Host: kernel, systemd, lvm, PAM, sshd
  • Extensions: VPNs, pcscd+ykclient, tmux, powerline; and fonts, codecs
  • Host or (system) container: docker, kubernetes
  • Tracing/debugging: perf, systemtap
  • Development: gcc, foo-devel
  • Misc cmdline: git, ansible, gpg2, fdisk
  • Unprivileged apps (desktop + server)

Delivery mechanisms

  • OS image
  • App framework
  • Packages (deb/RPM etc.)

Delivery mechanisms (blurred)

  • Packages in a privileged container (tracing/debugging, misc CLI)
  • Privileged applications
  • Packages: Possible but not good for all types
  • OS image + Explicit extension "slots"
  • Custom built OS image

For example: Wireguard VPN

For desktops, may want GUI config tool. And across server/desktop, want headless backend. Currently e.g. flatpak is "pure desktop app" only and doesn't have a "host dependency" mechanism.

Except systemd-networkd now supports it

OS image + apps

Android, iOS, CoreOS, Ubuntu Core, ChromeOS, Resin OS

And of course Atomic Host!

Most have different app/OS frameworks

Most have "blurred" OS/app/extension mechanisms
(e.g. iOS VPN)

ChromeOS - extensions

arstechnica: "Malicious Chrome extension is next to impossible to manually remove"

Ubuntu Core - snappy

debs are just one way to build snaps

"classic" confinement = full access to the system

Kubernetes DaemonSet

Run a container on each node

Often privileged, primary examples are
storage agents, monitoring

Compromise of registry = compromise of cluster
Maybe require GPG signatures for these?

Atomic system containers

OCI, but explictly designed for system bringup/management

Also, non-clustered scenarios:
"standalone" vs "orchestrated" containers

Goal: Lifecycle of container runtime distinct from host
(see also Fedora Modularity)

Hence, like "classic" snaps, RPMs, etc.: not confined

We're debating installing RPMs for host-side part
e.g. systemd units in /etc

OS image + app framework

Like peanut better and jelly

Focus on OS image

Atomic, online (but background) updates + rollback = fearless updates with low downtime

OS version number (and rpm -q pkg?)

Clear integration testing target

Accidental damage protection (e.g. sudo pip install)

Maybe dm-verity, fs-verity

File bugs against OS, not package

Can revert packages (version numbers not important)

OS image implementations

Dual partition: ChromeOS, (uses Docker/OCI for host)

Other: OLPC updater, Conary, Clear Linux, ...

My project: OSTree

More related projects

Also comparisons with e.g. Snapper


Gets content from A to B, GPG signatures

Works on filesystem level, not block level
i.e. anywhere rpm/dpkg work, ostree works

Deduplicating object store in /ostree/repo, hardlinks

Implements transactional updates, handles bootloader

C shared library

Flexible (e.g. EndlessOS, QtOTA, flatpak), also content-agnostic

Writing a new package manager? link to libostree!

Atomic Host base: rpm-ostree

True hybrid image + package system

BTW, ostree = libostree now

libdnf (C) provides RPM functionality (parts used by dnf)

Combination: Lots of radical changes, but leverages existing
(perhaps a bit boring) RPM ecosystem

"hybrid": like gas+electric cars

Images = electric, Packaging = gas (obviously!)

Hybrid types: Full hybrid, PHEV, mild hybrid, ...

Everyone knows gas cars, and classic packaging systems - fewer for e.g. PHEVs

Hybrid ideal: Best of both worlds

So rpm-ostree supports "OS Extensions" (RPMs)
e.g. PAM modules

Also most existing RPMs (apps, profiling/debugging, ...)

(And maybe enable system container + RPM)

rpm-ostree features

  • Atomic upgrades
  • Background (cancellable!) upgrades
  • Always read-only /usr
  • Reliable rollback (your data in /var)
  • Not opinionated about config mgmt: /etc works; use kickstart/ansible/puppet...
  • Client-side: package layering/override
  • ostree: static deltas (better than deltarpm)
  • Easily build your own server-side

The basics

Server: rpm-ostree compose = list of RPMs → OSTree commit

fedora-atomic & fedora-atomic-ws

Dependency resolution, %post scripts (ldconfig), SELinux labeling all server side; client side bit-for-bit reproducibility

Client side view

OSTree commit has GPG sig, version number, timestamp, checksum

Upgrade pulls from "remote" + branch (like git)

# rpm-ostree status
State: idle
● fedora-atomic:fedora/27/x86_64/atomic-host
                   Version: 27.44 (2018-01-01 21:54:31)
                    Commit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473
              GPGSignature: Valid signature by 860E19B0AFA800A1751881A6F55E7430F5282EE4
# rpm-ostree upgrade
... Version: 27.61 (2018-01-17 15:52:47)

Originally, "rpm-" part was just pkg diffs

Fearless rebases

# rpm-ostree rebase fedora-atomic-27:fedora/27/x86_64/atomic-host
# rpm-ostree status
                   Version: 27.44 (2018-01-01 21:54:31)
                    Commit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473

● fedora-atomic:fedora/26/x86_64/atomic-host
                   Version: 26.141 (2017-10-05 09:29:15)
                    Commit: 541abd650d1ffb3929e2ba8114436a0b04ee41da76a691af669dd037589a1421

Also if you want real excitement, rpm-ostree ex livefs --replace

Tap existing RPM ecosystem

# rpm-ostree install libvirt
...  (Maybe other system prep, followed by reboot)
# rpm-ostree status
State: idle
● fedora-atomic:fedora/27/x86_64/atomic-host
                   Version: 27.44 (2018-01-01 21:54:31)
                BaseCommit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473
           LayeredPackages: libvirt

Hybrid engaged, upgrade pulls from both (ostree remote + yum.repos.d)

Package installation engine: 🆕

Every operation checks out base tree, packages "reinstalled" (hardlinks) then scripts are run in a container with /var read-only, and the base /etc defaults.

Hence for example we survive rm -rf / in a %post!

We never run %postun, scripts don't see your SSH keys

Packages are imported into ostree branches, checkouts are hardlinked

More on live updates


# rpm-ostree override remove docker
# rpm-ostree status
     RemovedBasePackages: docker-2:1.13.1-44.git584d391.fc27.x86_64
# curl -L -O
# rpm-ostree override replace ./iptables-patched.x86_64.rpm

NB: Explicitly distinct from layering

Back to base image

# rpm-ostree override reset --all
# rpm-ostree uninstall libvirt # (...and other layered packages)

Hybrid disengaged, back to pure ostree

  • bit-for-bit identical in /usr
  • Unused config files in /etc gone
  • /var left unchanged

Other hybrid bits

# ostree show --raw fedora-atomic:fedora/27/x86_64/atomic-host
  ... <'Fedora-27-updates...'>, 'timestamp': <uint64 1516203727>

rpm-ostree injects OSTree metadata with rpm-md info

Down the line: Add versions to rpm-md?

Project status

  • Upstream releases ~monthly (single stream)
  • (Contrast update_engine,, snapper - boring git logs)
  • Stable
  • Supported on Fedora/CentOS/EL family
  • Headlining project: Atomic Host
  • Now used in side project: Atomic Workstation

Representative issues

Briefly: server-side composes

  • Stable, but requires learning investment
  • libostree is multi-vendor upstream project, incl. embedded/IoT community (rpm-ostree is separate)
  • Win: the same client + server tools works for you
  • Win: We've tested the package set in this mode
  • But not (yet) a Red Hat product

Big picture Project Atomic delivery methods

Host + Extensions: RPM(-OSTree)

System integration (RPM, OCI): atomic CLI, Kube daemonsets

Apps: via RPM, OCI

What's next for rpm-ostree

#247: Really polished 🆕⌚ automatic updates

Better auto rolling reboots of Kube cluster. Obvious default for desktops.

#1081: rpm-ostree jigdo ♲📦

RPM on wire, not ostree. Easier mirroring, change "base" presentation to be more "metapackage" like.

After that

yum vs atomic vs rpm-ostree (vs flatpak)

syscontainers + RPMs

rpm-ostree on github

ostree on github

Red Hat Enterprise Linux Atomic Host