| Age | Commit message (Collapse) | Author |
|---|
|
The format version isn't static, and indeed will be bumped in the
next swupd release.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
remove_unlisted_files_from_directory() failed to remove unwanted
symlinks to directories because it only looked at files, whereas
Python lists symlinks to directories as directories.
In Ostro OS, that affected for example /usr/lib/perl5 -> perl and
was found when adding a QA check for dangling symlinks.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
As a special optimization, updating the rootfs of the base image was
skipped if actual image creation was disabled for the base
image. However, there are reasons beyond image creation for updating
the rootfs, for example to run QA checks on it.
Therefore the change gets reverted and the code now only gets skipped
for the virtual bundle recipes.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Changing os-release in a .bbappend is undesirable for two reasons:
1. it changes the content also for images which do not use swupd
2. it causes full rebuilds even when the only thing that changed
is the OS_VERSION
The latter may or may not be desired, so we should better leave that
choice to the user, which is possible by setting VERSION_ID to
OS_VERSION in local.conf.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
We had a request to print more of the debugging around long-running
tasks to the console when using knotty using bbplain. This is undesirable
as a default but this patch enables a user to opt-in to this behaviour by
setting SWUPD_LOG_FN = "bbplain" producing output like:
NOTE: Executing RunQueue Tasks
jgl-image-swupd-1.0-r0 do_swupd_update: New OS_VERSION is 20
jgl-image-swupd-1.0-r0 do_swupd_update: Generating update from 0 to 20
jgl-image-swupd-1.0-r0 do_swupd_update: Generating fullfiles for 20
jgl-image-swupd-1.0-r0 do_swupd_update: Generating zero packs, this can take some time.
jgl-image-swupd-1.0-r0 do_swupd_update: Generating zero pack for os-core
jgl-image-swupd-1.0-r0 do_swupd_update: Generating zero pack for connman
jgl-image-swupd-1.0-r0 do_swupd_update: Generating zero pack for server
jgl-image-swupd-1.0-r0 do_swupd_update: Generating zero pack for developer
NOTE: Tasks Summary: Attempted 2857 tasks of which 2582 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
It is not guaranteed that OS versions always increment in steps of 10
and that deltas are desired for "current version - 10, -20, etc.". For
example, previous versions might be 10, 20, and the current one 21
because it is a minor update. In such a case no delta packs would be
generated because OS versions 11 and 1 do not exist.
Ostro OS outside of the CI derives the OS version from datetime, which
leads to high version numbers like 33014500. Counting down from that
in steps of ten in shell under pseudo was surprisingly slow and
unusable, running for minutes without even getting close to completion
when it does not hit existing version numbers.
So now the simplified approach is to generate delta packs for exactly
the last SWUPD_N_DELTAPACK older versions, regardless what their
numbering is.
If a more sophisticated approach is needed, it can still be added
later on, for example as part of YOCTO #9321.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
By convention, any bundle called "foobar-dev" is equal to the "foobar"
bundle plus the "dev-pkgs" image feature, i.e. it has the same content
as the base bundle plus the corresponding development files.
Without this feature it is impossible to generate separate bundles for
development files; the only option would be to put "dev-pkgs" into the
shared image features, which would add development files also to the
os-core and all bundles.
Using this fixed naming convention was chosen over additional
configuration options because a consistent naming automatically gets
enforced and it was easier to implement than additional configuration
options.
A future optimization will be to declare a dependency of the -dev
bundle on the base bundle and only provide the additional development
files in the -dev bundle. Right now, the -dev bundle is simply a
larger variant of the base bundle because all non-development files
are present in both bundles.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Because we skip the normal do_rootfs, the file is not created. But
several use cases depend on it, so now we create it based on the
manifest files of the individual bundles which get copied into the
image.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
There are scenarios in which these are useful for use by other
parts of this system, not least of all the next patch which uses
them to compose a manifest for bundle-based images.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The code in 5b48b103b9b940232cf65699db2adcab9c224124 fails to ever
delete the manifest symlinks, which causes errors on builds beyond
the first. The reason for this is that os.path.exists() on a broken
symlink returns False.
Fix this by trying to remove the manifest symlink first and then
removing the linked to file.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
We now have HOWTO.md to describe usage of the layer in detail.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
As we don't actually generate any images for the bundle images
(only the rootfs) writing an image manifest to the deploy directory
isn't very useful.
Add a task to remove the image manifests, ideally we'd just unset
IMAGE_MANIFEST in swupdbundle.bbclass but that isn't fully supported
in write_image_manifest() of rootfs-postcommands.bbclass in OE-Core.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
tar's option --preserve-permissions must be used on both sides
of the pipe: when creating an archive and when extracting it.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
|
|
Pseudo does not handle xattrs on hardlinks correctly. We work around
that for now by replacing hardlinks with symlinks. Only affects a few
entries, for example /sbin/e2fsck.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
To keep the "os-core" bundle small, the base image should be as
minimal as possible. It might not even allow logging in. In such a
setup, actual images have to have additional software installed, which
in a swupd-based image implies installing additional bundles.
The new SWUPD_IMAGES variable can be used to request the creation of
additional virtual image recipes which, if building them gets
explicitly requested, create a suitable rootfs by copying from the
mega image rootfs and then applying the normal image creation.
In contrast to the internal bundle-<baseimage>-<bundle> recipes, the
SWUPD_IMAGES are user-visible, so they are named
<baseimage>-<imageextension> where <imageextension> is the name given
in SWUPD_IMAGES.
To make this work, some changes are required in the code:
- the special swupd methods need to ignore the virtual image
recipes which produce images
- MEGA_IMAGE_ROOTFS needs to be set for all image recipes
- /usr/share/clear/bundles is populated consistently in all
rootfs directories (including os-core and also for the mega
rootfs)
- manipulating the rootfs must be split into code which sets
up the clear/bundles directory (can and should be in do_rootfs)
and copying files (should better be in do_image, because it needs
to replace work done there) and the latter must be available
as an utility function because the new virtual image recipes
replace the entire do_image with it
- the shared pseudo database can no longer be removed, because the
image creation also depends on it, and it is unknown when that is
done (might even be in a separate, future bitbake invocation)
The "swupdimage.bbclass" name is consistent with
"swupdbundle.bbclass", but can easily be confused with
"swupd-image.bbclass". The goal is to rename "swupd-image.bbclass" to
just "swupd.bbclass".
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
All that we need from virtual image recipes is the output of do_image.
That will run regardless whether there are image types defined.
Instead of producing image files and removing them later (which, besides
wasting time and IO bandwidth, was also not done completely when using
this class in Ostro OS) we can skip the entire image file creation
by unsetting IMAGE_FSTYPES.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Having virtual recipes named "${PN}-<bundle>" (example:
ostro-image-iotivity for ostro-image.bb) is potentially problematic,
because there might also be a real recipe with that name
(ostro-image-iotivity.bb).
Now the virtual recipes use "bundle-" as prefix in their name.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Convert do_copy_bundle_contents to python and use the same
functions for creating the swupd bundle manifest and copying
a tree with xattrs included.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
We should also call this when copying bundle directories to ensure
behaviour is consistent.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This prevents swupd-server failing with an error when it can't find
groups.ini if SWUPD_BUNDLES is unset.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Create a custom copy of oe.path.copytree() which also copies any
extended attributes (xattr) on files and directories in the tree.
Use this new copyxattrtree() function to copy the mega rootfs
contents in do_rootfs_append ().
To ensure that we have access to a version of GNU tar which supports
including xattr (1.27 or newer) we add a DEPENDS on
tar-replacement-native.
Note: this change removes the postfunc to delete the pseudo database
as the database is required in order to know what the xattr are and
life-cycles aren't as clear cut as we'd like (i.e. if the build is
run in stages by calling individual tasks).
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This parameter became mandatory in bitbake commit
fab717d303df0bcef737661f6917f275f35215a4
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
These changes don't affect the functionality but improve the code
and improve maintainability.
* remove a redundant comment
* keep the do_swupd_update task and its varflags together in the code
* remove a spurious double copy of the os-core bundle contents. This is now
handled by the do_copy_bundle_contents task.
* move os-core pack generation into the loop which handles other bundles,
this ensures all calls are the same regardless of bundle and aides future
refactoring.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If bundles haven't been defined for an image which inherits
swupd-image there's no point in creating a mega image, which should
be the same as the base image. Add some logic to avoid creating the
mega image when SWUPD_BUNDLES is unset to save time and space.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Fix reporting of the previous release which the update is being
generated against in a debug statement by correctly expanding the
PREVREL shell variable.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
It should be possible to use swupd-image to create images with an
update stream regardless of whether bundles have been defined for
the image or not.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The last push included some commented out code which should have
been removed before pushing, tidy that up now.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The bundle contents and the base image's root filesystem should be populated
from the mega image's rootfs in order to ensure that fully populated
versions of any files which undergo post-processing during rootfs/image
construction (i.e. /etc/passwd) are included.
[YOCTO #9193]
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If a user were to define a bundle named 'mega' the logic for
creating the mega-image (base image + all bundles) would become
confused. Therefore raise an error if a user-defined bundle is
named mega.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The virtual images are created as a side-effect of the swupd
update — we don't need the base image + bundle contents images
any more once the bundle contents have been extracted into the
swupd bundle directories, therefore delete them in order to
reduce disk footprint.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
We need to use a shared pseudo database whilst all of the virtual
images are created and the swupd post-processing is performed so
that the permissions are correct across recipes, however we also
need to be careful about the database's lifecycle to ensure that
operations run outside of pseudo control don't cause the database
to become invalidated.
As the do_swupd_image task only runs for the base image *and* runs
after all of the virtual images have been created we can safely
clean up the database once it's finished — add a task to do so.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This initial layer version provides an initial set of metadata to enable
integration of the swupd (https://clearlinux.org/features/software-update)
software updater into an image.
Approach:
An image that inherits the swupd-image bbclass will automatically have
bundle 'chroots' created which contain the filesystem contents of the
specified bundles, with the contents of the inheriting image forming the
default os-core bundle.
The mechanism to achieve this is that several virtual image recipes are
created using the swupdbundle class, one for each defined bundle plus a
'mega' image recipe. The 'mega' image contains the base image plus the
contents of all of the bundles, whilst bundle images contain only the
base image plus the contents of a single bundle.
We build the mega image first, then the base image (the one which
inherits this class) and finally all of the bundle images. Each
non-mega image has a manifest generated that lists the file contents
of the image.
We took the approach of building images, rather than populating the
chroot-like bundle directories with a package manager, because various
layers and recipes make changes to the rootfs contents outside of the
package manager, particularly with IMAGE_POSTPROCESS_COMMAND, etc.
Once the images and their manifests have been created each bundle image
manifest is compared to the base image manifest in order to generate a list
of files in the bundle image which don't exist in the base image.
Files in this list are then preserved in the bundle directory for
processing by swupd-server in order to generate update artefacts.
Finally the binaries from swupd-server are called on the bundle directories
to generate the artefacts for consumption by a swupd client.
How to:
* inherit the swupd-image class in your core OS image. swupd-based OS's use
bundles, the primary one of which, os-core, is defined as the contents of
this image.
* Assign a list of names for bundles you wish to generate to the
SWUPD_BUNDLES variable i.e.
SWUPD_BUNDLES = "feature_one feature_two"
* Assign a list of packages for which their content should be included in a
bundle to a varflag of BUNDLE_CONTENTS which matches the bundle name i.e.
BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six"
* Ensure the OS_VERSION variable is assigned an integer value and increased
before each image build which should generate swupd update artefacts.
This variable must echo the same version number as is used to set the
VERSION_ID field of os-release as swupd-client will use it to check for
updates.
* Publish the contents of
${DEPLOY_DIR}/swupd/${MACHINE}/${IMAGE_BASENAME}/www on a server for
consumption by swupd-client
* Use swupd client sub-commands with the -u argument pointing to the
contents published above
Known issues:
* shared pseudo database: the bundle chroot-like directories are generated
per-recipe and processed by a task of the inheriting recipe. In order for
the files generated outside of the base recipe to have correct
permissions when processed by swupd-server we need to share a pseudo
database across the recipes.
This database is currently never cleaned up, which is likely to cause
headaches due to the way pseudo operates on inodes that could be reused
outside of pseudo's influence. We have yet to determine an appropriate
time to perform housekeeping on this database (we essentially need the
database to be removed when DEPLOY_DIR_SWUPD is removed).
* oe-swupd-helpers: this recipe provides stub implementations only of some
swupd-client helpers. Anyone wishing to utilise swupd in a deployed image
will need to at least override kernel_updater.sh and
systemdboot_updater.sh.
* hard-coded paths: swupd assumes Clear Linux as a host OS and hard-codes
several paths to directories, programs and configuration files on Clear.
* builds a lot of images: due to the approach taken there are n+2 images
built, where n is the number of bundles defined in SWUPD_BUNDLES.
* creates a lot of duplicate files: due to the way swupd works by
processing chroot-like bundle directories on each os release we
potentially end up carrying a lot of duplicate files in DEPLOY_DIR_SWUPD.
We intend to look at using the hardlink program to replace duplicate
files in that directory with hard links in order to save disk space.
* requires far more testing: there are a lot of combinations of bundle
contents, rootfs modification commands (IMAGE_POSTPROCESS_COMMAND,
IMAGE_PREPROCESS_COMMAND, etc) that can affect the inputs to swupd and
our testing has likely missed areas of issue.
* OS_VERSION: introduces a new variable for the OS version number when we
already have a DISTRO_VERSION variable. This was done because swupd makes
various assumptions about the version number which aren't necessarily
true for traditional DISTRO_VERSION values in OE et al.
Co-authored-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
Joshua Lock
Patrick Ohly
Dmitry Rozhkov