diff options
Diffstat (limited to 'docs/Guide.md')
| -rw-r--r-- | docs/Guide.md | 279 |
1 files changed, 279 insertions, 0 deletions
diff --git a/docs/Guide.md b/docs/Guide.md new file mode 100644 index 0000000..d8e3976 --- /dev/null +++ b/docs/Guide.md @@ -0,0 +1,279 @@ +# Building a swupd-based OS + +This guide will help introduce the Software Updater (swupd) and how meta-swupd +can be used to develop a custom Linux-based OS with swupd updates using the +Yocto Project tools. + +## What is swupd? + +The [SoftWare UPDater](https://clearlinux.org/features/software-update) — swupd +— from [Clear Linux](https://clearlinux.org/) provides a new way of adding +functionality to and updating a Linux-based OS. + +swupd uses binary-delta technology to efficiently update only the files that +have changed between OS updates. This means that updates are small, resulting in +fast downloads, and fast to apply. + +Functionality is added to the OS via bundles, rather than individual packages, +which compose a set of projects into a logical unit which can be added or +removed from the OS wholesale. + +## swupd concepts + +swupd encourages a certain way of constructing a Linux-based OS, via the +following concepts: + +### Bundles + +Bundles in swupd are used to define logical units of functionality. All +swupd-based operating systems will define at least os-core bundle, whilst most +will also define several other bundles containing complete sets of functionality +composed of one or more packages generated by the build system. + +### Immutable os-core + +Building on the bundle concept, swupd requires that the base operating system is +immutable, enabling software deployed on the system to rely on the base OS not +changing unexpectedly. + +Therefore files which are included in the os-core bundle must be the final +version of a file which supports all bundles that may be installed on top of it. +Adding a bundle to the OS *must not* modify a file provided by os-core. Most +notably this means that the passwd and groups should contain all potential +account information, regardless of whether or not these users and groups are +utilised by the base OS. + +### Single, atomic, OS-wide version number + +swupd based operating systems atomically upgrade the entire OS from one version +to the next, rather than upgrading at a more granular, package-based, level. +This means that the entire contents of the OS can be determined from a single +version number. + +### Stateless OS + +Clear Linux, the Linux-based OS which hosts swupd development, is a [stateless +OS](https://clearlinux.org/features/stateless) — that is it comes with an empty +`/etc` by default and all default OS configuration is enabled until overridden +by per-system configuration of files in `/etc`. + +Due to this swupd by default will not update files `/etc` under the assumption +that these are local configuration and the default configuration is provided by +read-only files in other parts of the directory hierarchy in os-core. + +### The swupd way + +The concepts described above lead to a swupd using OS being constructed a +certain way which provides several advantages to the OS developer, not least of +all the reduces test matrix through a much smaller set of combinations of +software that might be installed on the OS. + +### References + +For more details see the Clear Linux Project's documentation: +* [Bundles overview](https://clearlinux.org/documentation/bundles_overview.html). +* [About software update](https://clearlinux.org/documentation/swupdate_about_sw_update.html) +* [Stateless feature description](https://clearlinux.org/features/stateless) + +## OpenEmbedded/Yocto Project considerations + +Several of the concepts in the swupd way contradict the default configuration of +operating systems built with the Yocto Project tools and OpenEmbedded. + +When building a distribution with swupd the OS developer should be aware of the +following requirements: + +### systemd + +swupd assumes the use of systemd both by using systemd units to trigger update +checks and by explicitly calling the systemd binary to restart the +`update-triggers.target` after an OS update has completed (see the swupd helper +scripts section for more information about this target). + +Therefore at this time one must use systemd as init manager in order to make +use of swupd. + +### Version number + +As the OS version number is atomic and identifies an entire OS build we generate +a new set of swupd update artefacts only when the `OS_VERSION` variable is +changed (by default refusing to write over an existing set of generated +artefacts with the same version number). + +swupd-client checks the `VERSION_ID` in the the `os-release` file when checking +for updates and we need to be certain that this is changed when building an +OS update for deployment, to that end the swupd-image class automatically writes +the `OS_VERSION` to the `VERSION_ID` field in the `os-release` file on each +image creation. + +### update-alternatives + +As the `os-core` bundle must provide an unchanging (at least within the same OS version) which other bundles can't modify we must be wary of recipes which use `update-alternatives`. + +Due to the way meta-swupd constructs the swupd-based OS and the resulting images +and update stream artefacts we can find ourselves in a situation where the +symlink for an alternative is installed pointing to a file which doesn't exist +in the os-core (because it's provided by a bundle). + +It is recommended that the OS developer carefully review all uses of +`update-alternatives` in their metadata and ensure that: + +1. the `ALTERNATIVE_PRIORITY` is set appropriately so that the desired +provider of an `ALTERNATIVE_LINK_NAME` is correctly chosen and consistent +across bundles. +2. the highest `ALTERNATIVE_PRIORITY` provider for an `ALTERNATIVE_LINK_NAME` is +included in the os-core bundle. + +To help detect the dangling symlink scenario the `swupd-image` class includes a +mechanism to check for such dangling symlinks in a constructed image, enable it +by adding the `swupd_check_dangling_symlinks` sanity check to +`SWUPD_IMAGE_SANITY_CHECKS` i.e.: +``` +SWUPD_IMAGE_SANITY_CHECKS += " \ + swupd_check_dangling_symlinks \ +" +``` + +### Users and groups + +Another aspect of the immutable os-core bundle is that the account information +in the passwd and group files installed in the os-core bundle must include user +and group information required for all bundles provided by the OS. + +We would also recommend the use of `useradd-staticids` to ensure that the uid +and gid are consistent across rebuilds so that os-core has better +reproducibility. + +See the Yocto Project Reference Manual section on +[useradd*.bbclass](http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#ref-classes-useradd) for more details. + +**NOTE**: due to the stateless nature of swupd group and passwd will not be +updated, for more information see the *stateful* section below. + +### Stateful + +Linux-based OS built with OpenEmbedded and the Yocto Project Tools are stateful +OS, that is the default configuration is provided by files in `/etc` and without +extra work on top of the default packages the OS won't boot without a populated +`/etc`. This contradicts the swupd way and means that currently any files a +package installs into `/etc` won't be updated between OS version. + +swupd-client checks the `VERSION_ID` in the the `os-release` file when checking +for updates. As this file is usually installed in `/etc` we include a `bbappend` +which instead creates this file in `/usr/lib/` (a location swupd-client prefers) +and symlink it to `/etc`. + +In order to support deploying updates to files in `/etc` via swupd we have +developed patches for both swupd-client and swupd-server to include files in +`/etc` in updates. These changes are enabled in swupd-server (resulting in swupd +manifests which list files in `/etc`) but disabled by default in swupd-client. + +This is because enabling updating of files in `/etc` requires some forethought +by the OS developer to ensure that files in `/etc` which shouldn't be updated +are not included in the bundle directories. For example if you would like users +of your OS to be able to add users and groups without OS updates overwriting +those users and groups you would need to ensusre that the passwd and group files +are not included in the bundle directories during the `do_swupd_update` task. + +You might achieve this by having a `prefunc` for the `do_swupd_update` task +which removes said files from the *os-core* bundle directory. + +To enable stateful OS support in swupd-client disable the `stateless` +`PACKAGE_CONFIG` option for the swupd-client recipe, i.e. with a +`swupd-client_%.bbappend` file: + +``` +PACKAGE_CONFIG_remove = "stateless" +``` + +### swupd helper scripts + +swupd-client tries to call out to certain helper scripts during an update: +* before the update `clr_pre_update.sh` is called +* after the update `systemdboot_updater.sh` and `kernel_updater.sh` are called +* the systemd `update-triggers.target` is restarted once the update is complete + +The `oe-swupd-helpers` recipe provides skeletal implementations of the scripts +(`clr_pre_update.sh`, `systemdboot_updater.sh` and `kernel_updater.sh`) which +simply print out that they have been called, these are clearly insufficient for +a production environment and should be replaced with OS-specific +implementations. + +The systemd units in `oe-swupd-helpers` are a little more useful, being based +on the units used by the Clear Linux project, but likely need tweaking and +adding to in order to better accommodate the OS being deployed. + +## Quick steps +1. add the meta-swupd layer to bblayers.conf +2. designate an image recipe as the base OS image (os-core, in swupd parlance) +and `inherit swupd-image` in that recipe +3. ensure the `OS_VERSION` variable is assigned an integer value and that this +number is increased before each build which should generate swupd update +artefacts +4. Vet the os-core bundle to address the considerations documented above: +* review any included recipes that inherit update-alternatives and ensure that +the desired provider of a binary is correctly chosen and consistent across +bundles with the highest priority provider being included in the os-core bundle. +(we recommend the use of the `swupd_check_dangling_symlinks` to help catch this +issue at image construction time) +* ensure all users and groups required by all bundles in the OS are defined in +the os-core bundle (we recommend the use of `useradd-staticids`). +5. Implement appropriate versions of the scripts and units in oe-swupd-helpers +6. (optional) Define additional bundles for any features you want to add +assign a list of bundle names to `SWUPD_BUNDLES` and for each named bundle, +assign a list of packages for which their content should be included in the +bundle to a varflag of `BUNDLE_CONTENTS` which matches the bundle name i.e: +``` +SWUPD_BUNDLES = "feature_one feature_two" +BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six" +``` +**NOTE**: beware of reserved bundle names, both '*full*' and '*mega*' have +special meaning and cannot be used for bundle names. +7. (optional) Define extra images, consisting of the os-core with any number of +additional bundles installed, which can be built. Do this by setting the +`SWUPD_IMAGES` variable to a list of additional image name suffixes and +assigning the names of bundles to inclde to a varflag matching the defined name. +For example: +``` +SWUPD_IMAGES = "product1" +SWUPD_IMAGES[product1] = "product1" +``` +defined in an image named *myco-image-core* would enable bitbake to be invoked +with the target *myco-image-core-product1*. +The *myco-image-core-product1* image would consist of the base OS (*os-core* +bundle) and the contents of the *product1* bundle and might, for example, be +used to provide an image which can be directly flashed to the product hardware. + +## Control variables + +Several variables can be set to tune the way swupd-image works: + +* `SWUPD_GENERATE` — if set to *0* i.e. `SWUPD_GENERATE = "0"` swupd update +artefact processing will be skipped but all tasks of the `swupd-image` class +will be executed. This is useful both for debugging the `swupd-image` class and +in a scenario where it might be desirable to generate the chroot-like bundle +directories without performing an processing with swupd. +* `SWUPD_DELTAPACKS` — if set to *0* i.e. `SWUPD_DELTAPACKS="0"` swupd +delta-packs will not be generated. +* `SWUPD_N_DELTAPACKS` — the number of previous releases against which to +generate delta-packs, defaults to 2. +* `SWUPD_VERSION_STEP` — Amount the OS_VERSION should be increased by for each +release. Used by the delta pack looping to generate delta packs going back up to +SWUPD_N_DELTAPACK releases. + +## Using swupd client + +The swupd-client package installs a binary named *swupd* which can be used to +perform updates, etc on the target. + +Take extra care with swupd-client versions prior to v3.4.0 as they hard-code the +update URLs to clearlinux.org. When steps haven't been taken to change the +default update URLs be sure to operate swupd-client with the **-u** switch and +pass a URL to your update server. + +Examples of swupd-client use follow: +``` +swupd check-update -u example.com/updates +swupd update -u example.com/updates +swupd verify -u example.com/updates +``` |