Devconf.cz 2018
Colin Walters, Red Hat, Inc.
keybase.io/walters | walters@{redhat.com,verbum.org}
Types of software
- Host: kernel, systemd, lvm, PAM, sshd
- Extensions: VPNs, pcscd+ykclient, tmux, powerline; and fonts, codecs
- Host or (system) container: docker, kubernetes
- Tracing/debugging: perf, systemtap
- Development: gcc, foo-devel
- Misc cmdline: git, ansible, gpg2, fdisk
- Unprivileged apps (desktop + server)
Delivery mechanisms
- OS image
- App framework
- Packages (deb/RPM etc.)
Delivery mechanisms (blurred)
- Packages in a privileged container (tracing/debugging, misc CLI)
- Privileged applications
- Packages: Possible but not good for all types
- OS image + Explicit extension "slots"
- Custom built OS image
For example: Wireguard VPN
For desktops, may want GUI config tool. And across server/desktop, want headless backend. Currently e.g. flatpak is "pure desktop app" only and doesn't have a "host dependency" mechanism.
OS image + apps
Android, iOS, CoreOS, Ubuntu Core, ChromeOS, Resin OS
And of course Atomic Host!
Most have different app/OS frameworks
Most have "blurred" OS/app/extension mechanisms
(e.g. iOS VPN)
ChromeOS - extensions
arstechnica: "Malicious Chrome extension is next to impossible to manually remove"
Ubuntu Core - snappy
debs are just one way to build snaps
"classic" confinement = full access to the system
Kubernetes DaemonSet
Run a container on each node
Often privileged, primary examples are
storage agents, monitoring
Compromise of registry = compromise of cluster
Maybe require GPG signatures for these?
Atomic system containers
OCI, but explictly designed for system bringup/management
Also, non-clustered scenarios:
"standalone" vs "orchestrated" containers
Goal: Lifecycle of container runtime distinct from host
(see also Fedora Modularity)
Hence, like "classic" snaps, RPMs, etc.: not confined
We're debating installing RPMs for host-side part
e.g. systemd units in /etc
OS image + app framework
Like peanut better and jelly
Focus on OS image
Atomic, online (but background) updates + rollback = fearless updates with low downtime
OS version number (and rpm -q pkg?)
Clear integration testing target
Accidental damage protection (e.g. sudo pip install)
File bugs against OS, not package
Can revert packages (version numbers not important)
OS image implementations
Dual partition: ChromeOS, mender.io
Resin.io/Balena (uses Docker/OCI for host)
Other: OLPC updater, Conary, Clear Linux, ...
My project: OSTree
Also comparisons with e.g. Snapper
(lib)ostree
Gets content from A to B, GPG signatures
Works on filesystem level, not block level
i.e. anywhere rpm/dpkg work, ostree works
Deduplicating object store in /ostree/repo, hardlinks
Implements transactional updates, handles bootloader
C shared library
Flexible (e.g. EndlessOS, QtOTA, flatpak), also content-agnostic
Writing a new package manager? link to libostree!
Atomic Host base: rpm-ostree
True hybrid image + package system
BTW, ostree = libostree now
libdnf (C) provides RPM functionality (parts used by dnf)
Combination: Lots of radical changes, but leverages existing
(perhaps a bit boring) RPM ecosystem
"hybrid": like gas+electric cars
Images = electric, Packaging = gas (obviously!)
Hybrid types: Full hybrid, PHEV, mild hybrid, ...
Everyone knows gas cars, and classic packaging systems - fewer for e.g. PHEVs
Hybrid ideal: Best of both worlds
So rpm-ostree supports "OS Extensions" (RPMs)
e.g. PAM modules
Also most existing RPMs (apps, profiling/debugging, ...)
(And maybe enable system container + RPM)
rpm-ostree features
- Atomic upgrades
- Background (cancellable!) upgrades
- Always read-only /usr
- Reliable rollback (your data in /var)
- Not opinionated about config mgmt: /etc works; use kickstart/ansible/puppet...
- Client-side: package layering/override
- ostree: static deltas (better than deltarpm)
- Easily build your own server-side
The basics
Server: rpm-ostree compose = list of RPMs → OSTree commit
fedora-atomic & fedora-atomic-ws
Dependency resolution, %post scripts (ldconfig), SELinux labeling all server side; client side bit-for-bit reproducibility
Client side view
OSTree commit has GPG sig, version number, timestamp, checksum
Upgrade pulls from "remote" + branch (like git)
# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora/27/x86_64/atomic-host
Version: 27.44 (2018-01-01 21:54:31)
Commit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473
GPGSignature: Valid signature by 860E19B0AFA800A1751881A6F55E7430F5282EE4
# rpm-ostree upgrade
...
... Version: 27.61 (2018-01-17 15:52:47)Originally, "rpm-" part was just pkg diffs
Fearless rebases
# rpm-ostree rebase fedora-atomic-27:fedora/27/x86_64/atomic-host
...
# rpm-ostree status
fedora-atomic-27:fedora/27/x86_64/atomic-host
Version: 27.44 (2018-01-01 21:54:31)
Commit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473
● fedora-atomic:fedora/26/x86_64/atomic-host
Version: 26.141 (2017-10-05 09:29:15)
Commit: 541abd650d1ffb3929e2ba8114436a0b04ee41da76a691af669dd037589a1421Also if you want real excitement, rpm-ostree ex livefs --replace
Tap existing RPM ecosystem
# rpm-ostree install libvirt
... (Maybe other system prep, followed by reboot)
# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora/27/x86_64/atomic-host
Version: 27.44 (2018-01-01 21:54:31)
BaseCommit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473
LayeredPackages: libvirt
#Hybrid engaged, upgrade pulls from both (ostree remote + yum.repos.d)
Package installation engine: 🆕
Every operation checks out base tree, packages "reinstalled" (hardlinks) then scripts are run in a container with /var read-only, and the base /etc defaults.
Hence for example we survive rm -rf / in a %post!
We never run %postun, scripts don't see your SSH keys
Packages are imported into ostree branches, checkouts are hardlinked
Overrides
# rpm-ostree override remove docker
...
# rpm-ostree status
...
RemovedBasePackages: docker-2:1.13.1-44.git584d391.fc27.x86_64
# curl -L -O http://myrepo.example.com/iptables-patched.x86_64.rpm
# rpm-ostree override replace ./iptables-patched.x86_64.rpmNB: Explicitly distinct from layering
Back to base image
# rpm-ostree override reset --all
# rpm-ostree uninstall libvirt # (...and other layered packages)Hybrid disengaged, back to pure ostree
- bit-for-bit identical in /usr
- Unused config files in /etc gone
- /var left unchanged
Other hybrid bits
# ostree show --raw fedora-atomic:fedora/27/x86_64/atomic-host
... <'Fedora-27-updates...'>, 'timestamp': <uint64 1516203727>
#rpm-ostree injects OSTree metadata with rpm-md info
Down the line: Add versions to rpm-md?
Project status
- Upstream releases ~monthly (single stream)
- (Contrast update_engine, mender.io, snapper - boring git logs)
- Stable
- Supported on Fedora/CentOS/EL family
- Headlining project: Atomic Host
- Now used in side project: Atomic Workstation
Representative issues
- Packages installing into /opt
- Live updates are experimental
- DKMS needs total overhaul for this
- Docs + familiarity
Briefly: server-side composes
- Stable, but requires learning investment
- libostree is multi-vendor upstream project, incl. embedded/IoT community (rpm-ostree is separate)
- Win: the same client + server tools works for you
- Win: We've tested the package set in this mode
- But not (yet) a Red Hat product
Big picture Project Atomic delivery methods
Host + Extensions: RPM(-OSTree)
System integration (RPM, OCI): atomic CLI, Kube daemonsets
Apps: via RPM, OCI
What's next for rpm-ostree
#247: Really polished 🆕⌚ automatic updates
Better auto rolling reboots of Kube cluster. Obvious default for desktops.
#1081: rpm-ostree jigdo ♲📦
RPM on wire, not ostree. Easier mirroring, change "base" presentation to be more "metapackage" like.
After that
yum vs atomic vs rpm-ostree (vs flatpak)
syscontainers + RPMs
