From e628c4592b989147a91920537555c84ebd760185 Mon Sep 17 00:00:00 2001 From: Will Woods Date: Mon, 18 Jan 2016 14:03:30 -0500 Subject: add weld-overview.md This probably repeats a bunch of stuff from weld-design, but weld-design is probably going to get split into a bunch of separate documents as each of the various sections gets fleshed out further. --- weld-overview.md | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 weld-overview.md diff --git a/weld-overview.md b/weld-overview.md new file mode 100644 index 0000000..c4bf2a8 --- /dev/null +++ b/weld-overview.md @@ -0,0 +1,123 @@ +# WELD DESIGN OVERVIEW + +_Super-early first draft edition_ +_Will Woods , 18 January 2016_ + +This is a high-level overview of `weld`, an experimental Linux distribution. + +## Goals + +Make a linux distribution that's **easy to work on**, **easy to customize**, +**quick to build or rebuild**, and **designed for the 21st century**. + +### Easy to work on +* packager and developer operations should be `git`-like + +### Easy to customize +* respins should just be a `git clone` + `git branch` away +* fewer fights over competing visions - fork and do what you want + +### Quick to build or rebuild +* Rapid development and iteration allows better testing and faster fixes + +### Designed for the 21st century +* Free of legacy cruft whenever it interferes with the above goals + +## The super-simple overview: + +1. Clone upstream packages into our own `git` mirrors of their repos +2. Build instructions + package metadata for all packages lives in a single + `git` repo + * this is all _metadata_, not _code_ +3. Build tools make binaries and dump them into a big binary store +4. Image build tools take binaries out of the binary store and make images + * Build environments are also images that get built out of the binary store + +## What we actually need to _do_ + +### Metadata (`.spec` + `comps.xml`) + +We need to maintain metadata about packages and their relationships, +functionally equivalent to `.spec` files. + +1. Research and define our metadata format + * What file format? (maybe [TOML]?) + * What data is necessary? (BuildRequires, Requires, upstream URLs, etc.) + * What's the schema? (required / optional keys and sections?) +1. Gather/convert metadata from existing distro + +### Binary storage (`.rpm` + `repodata/`) + +We want an efficient way to store our build output and its associated metadata. + +1. Research and define what metadata we need attached to binaries + * package version, build time, build environment ID, build flags, ... + * basically all the `rpm` headers (that we actually care about) +1. Research efficient ways of storing versioned binary files + * possibly `ostree`? +1. Research efficient ways of maintaining metadata alongside those files + * Key-value files in the binary store? + * Key-value pairs stored by `ostree` itself? + * External key-value store? + +Functionally equivalent to binary `.rpm` packages. + +### Packaging tools (`.src.rpm`) + +We need tools and policies for pulling upstream sources into a local mirror. + +* Write tool that uses metadata to pull upstream sources into a local git repo + * Everything in git, even if it's not git upstream! + * This means we need tools to handle svn, hg, or even just tarball + patches + * (looking at you here, [`bash`]) +* Decide how exactly to use branches and tags in our git mirrors + * `upstream` branch that tracks upstream `master` + * `distro` branch where we put our patches + * what about projects that have multiple branches upstream? + * `upstream-[branch]` and/or `distro-[branch]`? + * Tag `upstream` branches with the appropriate upstream versions + * Distro patches go onto our `distro` branches + * what about `master` tho? do we care? + +### Package build tools (`rpmbuild`) + +We'll need tools that use our metadata to build binaries for individual +packages. + +1. Write tool to clone/check-out the source on our `distro` branch +1. Write tool to build binaries as per the metadata +1. Write tool to install/copy binaries into the Binary Store + * Tag binaries with appropriate metadata + +### Compose tools (`pungi`, `lorax`, `mock`, ...) + +We need tools that pull binaries out of the Content Store to create system +images, containers, build environments, and eventually RPMs for legacy users. + +## How do we get there? + +If we replace only one part of the current system at a time - keeping the inputs +and outputs compatible with existing infrastructure - we can work on each +part in parallel and replace things piece by piece. + +_TODO: staffing/hardware/time estimates_ + +#### Replacing source RPMs +* Write tool to pull code from upstream into a local `git` mirror +* Use this to generate tarball+patchset for existing `.spec` + +#### Replacing `spec` +* Define our own metadata +* Write tool that generates a valid `spec` from our metadata + +#### Replacing binary RPMs +* Research and design/implement/use a Binary Content Store + * Take `rpm`s in - store files, metadata, signatures, etc. + * Dump `rpm`s back out, with proper headers and signatures + +#### Replacing compose tools +* Requires the Binary Content Store +* Write tools that pull data/metadata direct from the Binary Content Store + +[TOML]: https://github.com/toml-lang/toml +[`bash`]: http://ftp.gnu.org/gnu/bash/ -- cgit