diff options
| -rw-r--r-- | README | 10 | ||||
| -rw-r--r-- | docs/GUIDE.md | 76 | ||||
| -rw-r--r-- | docs/Guide.md | 279 | ||||
| -rw-r--r-- | docs/Implementation.md | 13 | ||||
| -rw-r--r-- | docs/Known Issues.md | 26 |
5 files changed, 325 insertions, 79 deletions
@@ -2,7 +2,7 @@ meta-swupd ========== OpenEmbedded meta layer for swupd software-update. For information on how to -best make use of this layer see HOWTO.md +best make use of this layer see docs/Guide.md This layer depends on: @@ -12,5 +12,9 @@ This layer depends on: Layer Maintainer: Joshua Lock <joshua.g.lock@intel.com> -Please send changes to the Yocto Project mailing list <yocto@yoctoproject.org> -with [meta-swupd] in the subject line. +Please file bugs on the Yocto Project Bugzilla in the 'Layers' Product, +selecting the 'layers' Component and prefixing the Summary with '[meta-swupd]' +https://bugzilla.yoctoproject.org/enter_bug.cgi?product=Layers + +Patches are welcome, please send them to the Yocto Project mailing list +<yocto@yoctoproject.org> with '[meta-swupd]' in the subject line. diff --git a/docs/GUIDE.md b/docs/GUIDE.md deleted file mode 100644 index f136cb3..0000000 --- a/docs/GUIDE.md +++ /dev/null @@ -1,76 +0,0 @@ -What is swupd? -============== - -The [SoftWare UPDater](https://clearlinux.org/features/software-update) — swupd — from [Clear Linux](https://clearlinux.org/) provides a new way of adding functionality to and updating a Linux-based OS. - -swupd uses binary-delta technology to efficiently update only the files that have changed between OS updates. This means that updates are small, resulting in fast downloads, and fast to apply. - -Bundles allow a system adminstrator to easily add or remove a complete set of functionality to the OS image, rather than worrying about individual packages. - -What is this meta-swupd? -=================== - -[meta-swupd](http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd) provides a metadata layer to aide Open Embedded and Yocto Project users in developing Linux-based operating systems built around swupd. - -meta-swupd can be used to implement both of the key features of swupd; efficient binary-delta based updates and bundle-based feature management. - -meta-swupd can easily be used to only deploy efficient update streams *or* to deploy a bundle-based OS with efficient updates. - -Swupd Concepts -============== - -Single, OS-wide, version number -------------------------------- - -swupd based operating systems atomically upgrade the entire OS from one version to the next, rather than upgrading at a more granular, package-based, level. -This means that the entire contents of the OS can be determined from a single version number. - -How do I use it? -================ - -The following steps are required to develop an OS that uses a swupd update stream: -* add the meta-swupd layer to bblayers.conf -* designate an image recipe as the base OS image (os-core, in swupd parlance) and `inherit swupd-image` in that recipe -* ensure the `OS_VERSION` variable is assigned an integer value which matches the `VERSION_ID` value in the *os-release* recipe and that this number is increased before each build which should generate swupd update artefacts - -In addition, to make use of swupd's bundle functionality, you will need to: -* assign a list of bundle names to `SWUPD_BUNDLES` i.e: - - ```SWUPD_BUNDLES = "feature_one feature_two"``` - -* for each named bundle, assign a list of packages for which their content should be included in the bundle to a varflag of `BUNDLE_CONTENTS` which matches the bundle name i.e: - - ```BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six"``` - - -What might go wrong? (a.k.a. known issues) -========================================== - -* refuses to overwrite generated data — because the act of generating swupd updates is time-consuming meta-swupd, by design, refuses to overwrite an existing update stream warning that the `OS_VERSION` number should be incremented. -* hardlinks and extended attributes — there's a bug in [pseudo](https://www.yoctoproject.org/tools-resources/projects/pseudo) which results in the extended attributes for a hardlink being incorrectly associated with the directory entry, rather than the inode. See [bug #9317](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9317) -* alternatives — when a bundle adds a package which pulls in an alternative with a higher priority than the one in the base image (i.e. coreutils takes precedence over busybox) we end up with a situation where the base image contains the symlink to the higher priority binary, but not the binaries themselves. See [bug #9320](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9320) -* potentially copies a lot of duplicate files — when copying bundle contents around it's very likely that the same file will exist in the swupd working directory multiple times. We may be able to make this less disk space intensive by using a utility like hardlink to deduplicate identical files. See [bug #9189](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9189) -* copies then deletes, rather than just copying — the current/initial implementation copies the entire mega-image rootfs for each bundle chroot-like directory and then prunes away unwanted files. It will be much more efficient to only copy the files which are required in the first instance. See [bug #9325](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9325) - -Control Variables -================= - -Several variables can be set to tune the way swupd-image works: -* `SWUPD_GENERATE` — if set to *0* i.e. `SWUPD_GENERATE = "0"` swupd update artefact processing will be skipped but all tasks of the `swupd-image` class will be executed. This is useful both for debugging the `swupd-image` class and in a scenario where it might be desirable to generate the chroot-like bundle directories without performing an processing with swupd. -* `SWUPD_DELTAPACKS` — if set to *0* i.e. `SWUPD_DELTAPACKS="0"` swupd delta-packs will not be generated. -* `SWUPD_N_DELTAPACKS` — the number of previous releases against which to generate delta-packs, defaults to 2. -* `SWUPD_VERSION_STEP` — Amount the OS_VERSION should be increased by for each release. Used by the delta pack looping to generate delta packs going back up to SWUPD_N_DELTAPACK releases. This could be replaced by a more elegant algorithm, see [bug #9322](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9322). - -Implementation Overview -======================= -An image that inherits this class will automatically have bundle 'chroots' created which contain the filesystem contents of the specified bundles. -The mechanism to achieve this is that several virtual image recipes are created, one for each defined bundle plus a 'mega' image recipe. -The 'mega' image contains the base image plus all of the bundles (and their contents), whilst bundle images contain only the base image plus the contents of a single bundle. - -We build the mega image first, then the base image (the one which inherits the `swupd-image` class) and finally all of the bundle images . Each non-mega image has a manifest generated that lists all of the file contents of the image. - -Each bundle 'chroot'-like directory and the rootfs of the base image are all populated from the contents of the mega image's rootfs. The reason for this is to ensure that all files which are modified during some kind of post-processing step, i.e. passwd and groups updated during postinsts, are fully populated. -This may not be an ideal compromise and requires further thought (i.e. see the known issue about alternatives above). - -Once the images and their manifests have been created each bundle image manifest is compared to the base image manifest in order to generate a delta list of files in the bundle image which don't exist in the base image. -Files in this list are then preserved in the bundle directory for processing by swupd-server in order to generate update artefacts. diff --git a/docs/Guide.md b/docs/Guide.md new file mode 100644 index 0000000..d8e3976 --- /dev/null +++ b/docs/Guide.md @@ -0,0 +1,279 @@ +# Building a swupd-based OS + +This guide will help introduce the Software Updater (swupd) and how meta-swupd +can be used to develop a custom Linux-based OS with swupd updates using the +Yocto Project tools. + +## What is swupd? + +The [SoftWare UPDater](https://clearlinux.org/features/software-update) — swupd +— from [Clear Linux](https://clearlinux.org/) provides a new way of adding +functionality to and updating a Linux-based OS. + +swupd uses binary-delta technology to efficiently update only the files that +have changed between OS updates. This means that updates are small, resulting in +fast downloads, and fast to apply. + +Functionality is added to the OS via bundles, rather than individual packages, +which compose a set of projects into a logical unit which can be added or +removed from the OS wholesale. + +## swupd concepts + +swupd encourages a certain way of constructing a Linux-based OS, via the +following concepts: + +### Bundles + +Bundles in swupd are used to define logical units of functionality. All +swupd-based operating systems will define at least os-core bundle, whilst most +will also define several other bundles containing complete sets of functionality +composed of one or more packages generated by the build system. + +### Immutable os-core + +Building on the bundle concept, swupd requires that the base operating system is +immutable, enabling software deployed on the system to rely on the base OS not +changing unexpectedly. + +Therefore files which are included in the os-core bundle must be the final +version of a file which supports all bundles that may be installed on top of it. +Adding a bundle to the OS *must not* modify a file provided by os-core. Most +notably this means that the passwd and groups should contain all potential +account information, regardless of whether or not these users and groups are +utilised by the base OS. + +### Single, atomic, OS-wide version number + +swupd based operating systems atomically upgrade the entire OS from one version +to the next, rather than upgrading at a more granular, package-based, level. +This means that the entire contents of the OS can be determined from a single +version number. + +### Stateless OS + +Clear Linux, the Linux-based OS which hosts swupd development, is a [stateless +OS](https://clearlinux.org/features/stateless) — that is it comes with an empty +`/etc` by default and all default OS configuration is enabled until overridden +by per-system configuration of files in `/etc`. + +Due to this swupd by default will not update files `/etc` under the assumption +that these are local configuration and the default configuration is provided by +read-only files in other parts of the directory hierarchy in os-core. + +### The swupd way + +The concepts described above lead to a swupd using OS being constructed a +certain way which provides several advantages to the OS developer, not least of +all the reduces test matrix through a much smaller set of combinations of +software that might be installed on the OS. + +### References + +For more details see the Clear Linux Project's documentation: +* [Bundles overview](https://clearlinux.org/documentation/bundles_overview.html). +* [About software update](https://clearlinux.org/documentation/swupdate_about_sw_update.html) +* [Stateless feature description](https://clearlinux.org/features/stateless) + +## OpenEmbedded/Yocto Project considerations + +Several of the concepts in the swupd way contradict the default configuration of +operating systems built with the Yocto Project tools and OpenEmbedded. + +When building a distribution with swupd the OS developer should be aware of the +following requirements: + +### systemd + +swupd assumes the use of systemd both by using systemd units to trigger update +checks and by explicitly calling the systemd binary to restart the +`update-triggers.target` after an OS update has completed (see the swupd helper +scripts section for more information about this target). + +Therefore at this time one must use systemd as init manager in order to make +use of swupd. + +### Version number + +As the OS version number is atomic and identifies an entire OS build we generate +a new set of swupd update artefacts only when the `OS_VERSION` variable is +changed (by default refusing to write over an existing set of generated +artefacts with the same version number). + +swupd-client checks the `VERSION_ID` in the the `os-release` file when checking +for updates and we need to be certain that this is changed when building an +OS update for deployment, to that end the swupd-image class automatically writes +the `OS_VERSION` to the `VERSION_ID` field in the `os-release` file on each +image creation. + +### update-alternatives + +As the `os-core` bundle must provide an unchanging (at least within the same OS version) which other bundles can't modify we must be wary of recipes which use `update-alternatives`. + +Due to the way meta-swupd constructs the swupd-based OS and the resulting images +and update stream artefacts we can find ourselves in a situation where the +symlink for an alternative is installed pointing to a file which doesn't exist +in the os-core (because it's provided by a bundle). + +It is recommended that the OS developer carefully review all uses of +`update-alternatives` in their metadata and ensure that: + +1. the `ALTERNATIVE_PRIORITY` is set appropriately so that the desired +provider of an `ALTERNATIVE_LINK_NAME` is correctly chosen and consistent +across bundles. +2. the highest `ALTERNATIVE_PRIORITY` provider for an `ALTERNATIVE_LINK_NAME` is +included in the os-core bundle. + +To help detect the dangling symlink scenario the `swupd-image` class includes a +mechanism to check for such dangling symlinks in a constructed image, enable it +by adding the `swupd_check_dangling_symlinks` sanity check to +`SWUPD_IMAGE_SANITY_CHECKS` i.e.: +``` +SWUPD_IMAGE_SANITY_CHECKS += " \ + swupd_check_dangling_symlinks \ +" +``` + +### Users and groups + +Another aspect of the immutable os-core bundle is that the account information +in the passwd and group files installed in the os-core bundle must include user +and group information required for all bundles provided by the OS. + +We would also recommend the use of `useradd-staticids` to ensure that the uid +and gid are consistent across rebuilds so that os-core has better +reproducibility. + +See the Yocto Project Reference Manual section on +[useradd*.bbclass](http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#ref-classes-useradd) for more details. + +**NOTE**: due to the stateless nature of swupd group and passwd will not be +updated, for more information see the *stateful* section below. + +### Stateful + +Linux-based OS built with OpenEmbedded and the Yocto Project Tools are stateful +OS, that is the default configuration is provided by files in `/etc` and without +extra work on top of the default packages the OS won't boot without a populated +`/etc`. This contradicts the swupd way and means that currently any files a +package installs into `/etc` won't be updated between OS version. + +swupd-client checks the `VERSION_ID` in the the `os-release` file when checking +for updates. As this file is usually installed in `/etc` we include a `bbappend` +which instead creates this file in `/usr/lib/` (a location swupd-client prefers) +and symlink it to `/etc`. + +In order to support deploying updates to files in `/etc` via swupd we have +developed patches for both swupd-client and swupd-server to include files in +`/etc` in updates. These changes are enabled in swupd-server (resulting in swupd +manifests which list files in `/etc`) but disabled by default in swupd-client. + +This is because enabling updating of files in `/etc` requires some forethought +by the OS developer to ensure that files in `/etc` which shouldn't be updated +are not included in the bundle directories. For example if you would like users +of your OS to be able to add users and groups without OS updates overwriting +those users and groups you would need to ensusre that the passwd and group files +are not included in the bundle directories during the `do_swupd_update` task. + +You might achieve this by having a `prefunc` for the `do_swupd_update` task +which removes said files from the *os-core* bundle directory. + +To enable stateful OS support in swupd-client disable the `stateless` +`PACKAGE_CONFIG` option for the swupd-client recipe, i.e. with a +`swupd-client_%.bbappend` file: + +``` +PACKAGE_CONFIG_remove = "stateless" +``` + +### swupd helper scripts + +swupd-client tries to call out to certain helper scripts during an update: +* before the update `clr_pre_update.sh` is called +* after the update `systemdboot_updater.sh` and `kernel_updater.sh` are called +* the systemd `update-triggers.target` is restarted once the update is complete + +The `oe-swupd-helpers` recipe provides skeletal implementations of the scripts +(`clr_pre_update.sh`, `systemdboot_updater.sh` and `kernel_updater.sh`) which +simply print out that they have been called, these are clearly insufficient for +a production environment and should be replaced with OS-specific +implementations. + +The systemd units in `oe-swupd-helpers` are a little more useful, being based +on the units used by the Clear Linux project, but likely need tweaking and +adding to in order to better accommodate the OS being deployed. + +## Quick steps +1. add the meta-swupd layer to bblayers.conf +2. designate an image recipe as the base OS image (os-core, in swupd parlance) +and `inherit swupd-image` in that recipe +3. ensure the `OS_VERSION` variable is assigned an integer value and that this +number is increased before each build which should generate swupd update +artefacts +4. Vet the os-core bundle to address the considerations documented above: +* review any included recipes that inherit update-alternatives and ensure that +the desired provider of a binary is correctly chosen and consistent across +bundles with the highest priority provider being included in the os-core bundle. +(we recommend the use of the `swupd_check_dangling_symlinks` to help catch this +issue at image construction time) +* ensure all users and groups required by all bundles in the OS are defined in +the os-core bundle (we recommend the use of `useradd-staticids`). +5. Implement appropriate versions of the scripts and units in oe-swupd-helpers +6. (optional) Define additional bundles for any features you want to add +assign a list of bundle names to `SWUPD_BUNDLES` and for each named bundle, +assign a list of packages for which their content should be included in the +bundle to a varflag of `BUNDLE_CONTENTS` which matches the bundle name i.e: +``` +SWUPD_BUNDLES = "feature_one feature_two" +BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six" +``` +**NOTE**: beware of reserved bundle names, both '*full*' and '*mega*' have +special meaning and cannot be used for bundle names. +7. (optional) Define extra images, consisting of the os-core with any number of +additional bundles installed, which can be built. Do this by setting the +`SWUPD_IMAGES` variable to a list of additional image name suffixes and +assigning the names of bundles to inclde to a varflag matching the defined name. +For example: +``` +SWUPD_IMAGES = "product1" +SWUPD_IMAGES[product1] = "product1" +``` +defined in an image named *myco-image-core* would enable bitbake to be invoked +with the target *myco-image-core-product1*. +The *myco-image-core-product1* image would consist of the base OS (*os-core* +bundle) and the contents of the *product1* bundle and might, for example, be +used to provide an image which can be directly flashed to the product hardware. + +## Control variables + +Several variables can be set to tune the way swupd-image works: + +* `SWUPD_GENERATE` — if set to *0* i.e. `SWUPD_GENERATE = "0"` swupd update +artefact processing will be skipped but all tasks of the `swupd-image` class +will be executed. This is useful both for debugging the `swupd-image` class and +in a scenario where it might be desirable to generate the chroot-like bundle +directories without performing an processing with swupd. +* `SWUPD_DELTAPACKS` — if set to *0* i.e. `SWUPD_DELTAPACKS="0"` swupd +delta-packs will not be generated. +* `SWUPD_N_DELTAPACKS` — the number of previous releases against which to +generate delta-packs, defaults to 2. +* `SWUPD_VERSION_STEP` — Amount the OS_VERSION should be increased by for each +release. Used by the delta pack looping to generate delta packs going back up to +SWUPD_N_DELTAPACK releases. + +## Using swupd client + +The swupd-client package installs a binary named *swupd* which can be used to +perform updates, etc on the target. + +Take extra care with swupd-client versions prior to v3.4.0 as they hard-code the +update URLs to clearlinux.org. When steps haven't been taken to change the +default update URLs be sure to operate swupd-client with the **-u** switch and +pass a URL to your update server. + +Examples of swupd-client use follow: +``` +swupd check-update -u example.com/updates +swupd update -u example.com/updates +swupd verify -u example.com/updates +``` diff --git a/docs/Implementation.md b/docs/Implementation.md new file mode 100644 index 0000000..b972542 --- /dev/null +++ b/docs/Implementation.md @@ -0,0 +1,13 @@ +# Implementation Overview + +An image that inherits this class will automatically have bundle 'chroots' created which contain the filesystem contents of the specified bundles. +The mechanism to achieve this is that several virtual image recipes are created, one for each defined bundle plus a 'mega' image recipe. +The 'mega' image contains the base image plus all of the bundles (and their contents), whilst bundle images contain only the base image plus the contents of a single bundle. + +We build the mega image first, then the base image (the one which inherits the `swupd-image` class) and finally all of the bundle images . Each non-mega image has a manifest generated that lists all of the file contents of the image. + +Each bundle 'chroot'-like directory and the rootfs of the base image are all populated from the contents of the mega image's rootfs. The reason for this is to ensure that all files which are modified during some kind of post-processing step, i.e. passwd and groups updated during postinsts, are fully populated. +This may not be an ideal compromise and requires further thought (i.e. see the known issue about alternatives above). + +Once the images and their manifests have been created each bundle image manifest is compared to the base image manifest in order to generate a delta list of files in the bundle image which don't exist in the base image. +Files in this list are then preserved in the bundle directory for processing by swupd-server in order to generate update artefacts. diff --git a/docs/Known Issues.md b/docs/Known Issues.md new file mode 100644 index 0000000..dacecdc --- /dev/null +++ b/docs/Known Issues.md @@ -0,0 +1,26 @@ +# Known issues and future work + +For the latest list of issues in the meta-swupd layer, please see the Yocto +Project Bugzilla: +https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=[meta-swupd]&list_id=577685 + +* refuses to overwrite generated data — because the act of generating swupd +updates is time-consuming meta-swupd, by design, refuses to overwrite an +existing update stream warning that the `OS_VERSION` number should be +incremented. +* hardlinks and extended attributes — there's a bug in +[pseudo](https://www.yoctoproject.org/tools-resources/projects/pseudo) which +results in the extended attributes for a hardlink being incorrectly associated +with the directory entry, rather than the inode. See [bug #9317](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9317) +* potentially copies a lot of duplicate files — when copying bundle contents +around it's very likely that the same file will exist in the swupd working +directory multiple times. We may be able to make this less disk space intensive +by using a utility like hardlink to deduplicate identical files. See [bug #9189](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9189) +* copies then deletes, rather than just copying — the current/initial +implementation copies the entire mega-image rootfs for each bundle chroot-like +directory and then prunes away unwanted files. It will be much more efficient to +only copy the files which are required in the first instance. See [bug #9325](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9325) +* Various other performance issues including `system()` calls in swupd-server +resulting in slow operation as pseudo is loaded multiple times [bug #9449](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9449), inefficient +algorithm for determining previous OS versions [bug #9322](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9322), sequential calls +to `swupd_make_pack` [bug #9224](https://bugzilla.yoctoproject.org/show_bug.cgi?id=9224). |