# Building a swupd-based OS

This guide will help introduce the Software Updater (swupd) and how meta-swupd
can be used to develop a custom Linux-based OS with swupd updates using the
Yocto Project tools.

## What is swupd?

The [SoftWare UPDater](https://clearlinux.org/features/software-update) — swupd
— from [Clear Linux](https://clearlinux.org/) provides a new way of adding
functionality to and updating a Linux-based OS.

swupd uses binary-delta technology to efficiently update only the files that
have changed between OS updates. This means that updates are small, resulting in
fast downloads, and fast to apply.

Functionality is added to the OS via bundles, rather than individual packages,
which compose a set of projects into a logical unit which can be added or
removed from the OS wholesale.

## swupd concepts

swupd encourages a certain way of constructing a Linux-based OS, via the
following concepts:

### Bundles

Bundles in swupd are used to define logical units of functionality. All
swupd-based operating systems will define at least os-core bundle, whilst most
will also define several other bundles containing complete sets of functionality
composed of one or more packages generated by the build system.

### Immutable os-core

Building on the bundle concept, swupd requires that the base operating system is
immutable, enabling software deployed on the system to rely on the base OS not
changing unexpectedly.

Therefore files which are included in the os-core bundle must be the final
version of a file which supports all bundles that may be installed on top of it.
Adding a bundle to the OS *must not* modify a file provided by os-core. Most
notably this means that the passwd and groups should contain all potential
account information, regardless of whether or not these users and groups are
utilised by the base OS.

### Single, atomic, OS-wide version number

swupd based operating systems atomically upgrade the entire OS from one version
to the next, rather than upgrading at a more granular, package-based, level.
This means that the entire contents of the OS can be determined from a single
version number.

### Stateless OS

Clear Linux, the Linux-based OS which hosts swupd development, is a [stateless
OS](https://clearlinux.org/features/stateless) — that is it comes with an empty
`/etc` by default and all default OS configuration is enabled until overridden
by per-system configuration of files in `/etc`.

Due to this swupd by default will not update files `/etc` under the assumption
that these are local configuration and the default configuration is provided by
read-only files in other parts of the directory hierarchy in os-core.

### The swupd way

The concepts described above lead to a swupd using OS being constructed a
certain way which provides several advantages to the OS developer, not least of
all the reduces test matrix through a much smaller set of combinations of
software that might be installed on the OS.

### References

For more details see the Clear Linux Project's documentation:
* [Bundles overview](https://clearlinux.org/documentation/bundles_overview.html).
* [About software update](https://clearlinux.org/documentation/swupdate_about_sw_update.html)
* [Stateless feature description](https://clearlinux.org/features/stateless)

## OpenEmbedded/Yocto Project considerations

Several of the concepts in the swupd way contradict the default configuration of
operating systems built with the Yocto Project tools and OpenEmbedded.

When building a distribution with swupd the OS developer should be aware of the
following requirements:

### systemd

swupd assumes the use of systemd both by using systemd units to trigger update
checks and by explicitly calling the systemd binary to restart the
`update-triggers.target` after an OS update has completed (see the swupd helper
scripts section for more information about this target).

Therefore at this time one must use systemd as init manager in order to make
use of swupd.

### Version number

As the OS version number is atomic and identifies an entire OS build we generate
a new set of swupd update artefacts only when the `OS_VERSION` variable is
changed (by default refusing to write over an existing set of generated
artefacts with the same version number).

swupd-client checks the `VERSION_ID` in the the `os-release` file when checking
for updates and we need to be certain that this is changed when building an
OS update for deployment, to that end the swupd-image class automatically writes
the `OS_VERSION` to the `VERSION_ID` field in the `os-release` file on each
image creation.

### update-alternatives

As the `os-core` bundle must provide an unchanging (at least within the same OS version) which other bundles can't modify we must be wary of recipes which use `update-alternatives`.

Due to the way meta-swupd constructs the swupd-based OS and the resulting images
and update stream artefacts we can find ourselves in a situation where the
symlink for an alternative is installed pointing to a file which doesn't exist
in the os-core (because it's provided by a bundle).

It is recommended that the OS developer carefully review all uses of
`update-alternatives` in their metadata and ensure that:

1. the `ALTERNATIVE_PRIORITY` is set appropriately so that the desired
provider of an `ALTERNATIVE_LINK_NAME` is correctly chosen and consistent
across bundles.
2. the highest `ALTERNATIVE_PRIORITY` provider for an `ALTERNATIVE_LINK_NAME` is
included in the os-core bundle.

To help detect the dangling symlink scenario the `swupd-image` class includes a
mechanism to check for such dangling symlinks in a constructed image, enable it
by adding the `swupd_check_dangling_symlinks` sanity check to
`IMAGE_QA_COMMANDS` i.e.:
```
IMAGE_QA_COMMANDS += " \
     swupd_check_dangling_symlinks \
"
```

### Users and groups

Another aspect of the immutable os-core bundle is that the account information
in the passwd and group files installed in the os-core bundle must include user
and group information required for all bundles provided by the OS.

We would also recommend the use of `useradd-staticids` to ensure that the uid
and gid are consistent across rebuilds so that os-core has better
reproducibility.

See the Yocto Project Reference Manual section on
[useradd*.bbclass](http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#ref-classes-useradd) for more details.

**NOTE**: due to the stateless nature of swupd group and passwd will not be
updated, for more information see the *stateful* section below.

### Stateful

Linux-based OS built with OpenEmbedded and the Yocto Project Tools are stateful
OS, that is the default configuration is provided by files in `/etc` and without
extra work on top of the default packages the OS won't boot without a populated
`/etc`. This contradicts the swupd way and means that currently any files a
package installs into `/etc` won't be updated between OS version.

swupd-client checks the `VERSION_ID` in the the `os-release` file when checking
for updates. As this file is usually installed in `/etc` we include a `bbappend`
which instead creates this file in `/usr/lib/` (a location swupd-client prefers)
and symlink it to `/etc`.

In order to support deploying updates to files in `/etc` via swupd we have
developed patches for both swupd-client and swupd-server to include files in
`/etc` in updates. These changes are enabled in swupd-server (resulting in swupd
manifests which list files in `/etc`) but disabled by default in swupd-client.

This is because enabling updating of files in `/etc` requires some forethought
by the OS developer to ensure that files in `/etc` which shouldn't be updated
are not included in the bundle directories. For example if you would like users
of your OS to be able to add users and groups without OS updates overwriting
those users and groups you would need to ensusre that the passwd and group files
are not included in the bundle directories during the `do_swupd_update` task.

You might achieve this by having a `prefunc` for the `do_swupd_update` task
which removes said files from the *os-core* bundle directory.

To enable stateful OS support in swupd-client disable the `stateless`
`PACKAGE_CONFIG` option for the swupd-client recipe, i.e. with a
`swupd-client_%.bbappend` file:

```
PACKAGE_CONFIG_remove = "stateless"
```

### swupd helper scripts

swupd-client tries to call out to certain helper scripts during an update:
* before the update `clr_pre_update.sh` is called
* after the update `systemdboot_updater.sh` and `kernel_updater.sh` are called
* the systemd `update-triggers.target` is restarted once the update is complete

The `oe-swupd-helpers` recipe provides skeletal implementations of the scripts
(`clr_pre_update.sh`, `systemdboot_updater.sh` and `kernel_updater.sh`) which
simply print out that they have been called, these are clearly insufficient for
a production environment and should be replaced with OS-specific
implementations.

The systemd units in `oe-swupd-helpers` are a little more useful, being based
on the units used by the Clear Linux project, but likely need tweaking and
adding to in order to better accommodate the OS being deployed.

## Quick steps
1. add the meta-swupd layer to bblayers.conf
2. designate an image recipe as the base OS image (os-core, in swupd parlance)
and `inherit swupd-image` in that recipe
3. ensure the `OS_VERSION` variable is assigned an integer value and that this
number is increased before each build which should generate swupd update
artefacts
4. Vet the os-core bundle to address the considerations documented above:
* review any included recipes that inherit update-alternatives and ensure that
the desired provider of a binary is correctly chosen and consistent across
bundles with the highest priority provider being included in the os-core bundle.
(we recommend the use of the `swupd_check_dangling_symlinks` to help catch this
issue at image construction time)
* ensure all users and groups required by all bundles in the OS are defined in
the os-core bundle (we recommend the use of `useradd-staticids`).
5. Implement appropriate versions of the scripts and units in oe-swupd-helpers
6. (optional) Define additional bundles for any features you want to add
assign a list of bundle names to `SWUPD_BUNDLES` and for each named bundle,
assign a list of packages for which their content should be included in the
bundle to a varflag of `BUNDLE_CONTENTS` which matches the bundle name i.e:
```
SWUPD_BUNDLES = "feature_one feature_two"
BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six"
```
**NOTE**: beware of reserved bundle names, both '*full*' and '*mega*' have
special meaning and cannot be used for bundle names.
**EXTRA**: it's also possible to define extra features for inclusion in a bundle
via the `BUNDLE_FEATURES` variable, however this **must** be used with caution
as the os-core is immutable any `BUNDLE_FEATURES` must only introduce additional
files into the system and not somehow alter the core image's contents.
7. (optional) Define extra images, consisting of the os-core with any number of
additional bundles installed, which can be built. Do this by setting the
`SWUPD_IMAGES` variable to a list of additional image name suffixes and
assigning the names of bundles to inclde to a varflag matching the defined name.
For example:
```
SWUPD_IMAGES = "product1"
SWUPD_IMAGES[product1] = "product1"
```
defined in an image named *myco-image-core* would enable bitbake to be invoked
with the target *myco-image-core-product1*.
The *myco-image-core-product1* image would consist of the base OS (*os-core*
bundle) and the contents of the *product1* bundle and might, for example, be
used to provide an image which can be directly flashed to the product hardware.

## Control variables

Several variables can be set to tune the way swupd-image works:

* `SWUPD_GENERATE` — if set to *0* i.e. `SWUPD_GENERATE = "0"` swupd update
artefact processing will be skipped but all tasks of the `swupd-image` class
will be executed. This is useful both for debugging the `swupd-image` class and
in a scenario where it might be desirable to generate the chroot-like bundle
directories without performing an processing with swupd.
* `SWUPD_DELTAPACKS` — if set to *0* i.e. `SWUPD_DELTAPACKS="0"` swupd
delta-packs will not be generated.
* `SWUPD_N_DELTAPACKS` — the number of previous releases against which to
generate delta-packs, defaults to 2.
* `SWUPD_VERSION_STEP` — Amount the OS_VERSION should be increased by for each
release. Used by the delta pack looping to generate delta packs going back up to
SWUPD_N_DELTAPACK releases.

## Using swupd client

The swupd-client package installs a binary named *swupd* which can be used to
perform updates, etc on the target.

Take extra care with swupd-client versions prior to v3.4.0 as they hard-code the
update URLs to clearlinux.org. When steps haven't been taken to change the
default update URLs be sure to operate swupd-client with the **-u** switch and
pass a URL to your update server.

Examples of swupd-client use follow:
```
swupd check-update -u example.com/updates
swupd update -u example.com/updates
swupd verify -u example.com/updates
```