| Age | Commit message (Collapse) | Author |
|---|
|
This removes the storing of previous build information in sstate. It
was conceptually questionable (sstate is a cache which does not need
to be backed up, while the information about previous builds is
crucial and must not get lost) and not working:
- the -map.inc file wasn't actually included anywhere and thus
the old build information wasn't getting restored
- restoring all previous builds would have made building slower
and slower as the number of previous builds grows
- the old build information lacked the www/Manifest files that
incremental updates need
The replacement puts previous build information into the image deploy
directory. That's tentative and also not fully working.
The automatic selection of old versions to build deltas against also
gets replaced with an explicit choice that has to be made by the user
of meta-swupd. That's because in practice, incremental updates are
more useful when prepared for the releases that actually run on the
target device, like major milestones.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
deploy/swupd
During development it is useful to wipe out deploy/swupd. This
simulates the "start from scratch" situation in the Ostro OS
CI. Previously it was necessary to force-run do_stage_swupd_inputs and
do_swupd_update after removing the directory, now this is fully
automatic.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
That PN is different in the base image and virtual images led to
various places which had to distinguish between the two. We can
simplify that by introducing a variable SWUPD_IMAGE_PN which
always has the PN value of the base image.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When the first tar in a pipe fails, its error code is getting lost.
Detect that by checking for output: normally, all operations should be
silent, so if there is output, something unusual happened.
This also catches warnings.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When reading the manifest .content.txt files in swupd-server, it
matters whether they list entries with or without leading
slash. Without it, matching files against the full content fails
because that is read from a directory and happens to use a leading
slash inside swupd_create_update, and then swupd created bundle
manifests without hashsums, leading to 404 errors during updates.
Fixing this in meta-swupd is easier than in swupd-server.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
do_swupd_update itself unpacks the tar archives that swupd-server
needs and therefore does no longer depend on sharing the pseudo
database with the other tasks and virtual images.
Using a separate pseudo DB speeds up
"ostro-image-swupd:do_stage_swupd_inputs
ostro-image-swupd:do_swupd_update ostro-image-swupd-dev" (two tasks
which run in parallel because both depend on the same full rootfs and
which used to share the same pseudo instance) from 25 to 16 minutes.
The pseudo data directory is intentionally inside the deploy/swupd
directory. There it can be deleted and re-created for testing swupd
update generation with:
rm -rf tmp*/deploy/swupd
bitbake -f <image>:do_stage_swupd_inputs <image>:do_swupd_update
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When creating bundle images, we need to know and copy also the entries
that we exclude from processing by swupd-server. This could be done
with a more complex syntax for the .content.txt files, but that would
also make the swupd-server patches more complicated.
Instead, an .extra-content.txt gets written alongside the
.content.text and meta-swupd uses that when copying files into
images. Due to the way how this is implemented, the .extra-content.txt
of bundles also lists the files that were excluded from the bundle
because they were already in the os-core. This may or may not be
desirable.
This change also includes some other improvements (consistent use of the
helper method, sorting the content of the file lists).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Storing the build rootfs in the sstate-cache has drawbacks:
- it's questionable whether storing data that cannot be re-created really
belongs into a cache which (by definition) should only contain data which
may get lost
- while it looks attractive to re-use an existing mechanism for sharing data
across builds, it's not a complete solution because the map still needs to
be carried across builds
- using the sstate-cache mechanism adds additional, large copy operations on
the critical path towards completing a build
- the code isn't quite mature yet, sometimes do_stage_swupd_inputs_setscene
fails: sstate_setscene(d) fails: No such file or directory:
'.../tmp-glibc/work/qemux86-ostro-linux/ostro-image-swupd/1.0-r0/sstate-install-stage_swupd_inputs/92909520' ->
'.../tmp-glibc/work/qemux86-ostro-linux/ostro-image-swupd/1.0-r0/swupd-image/92909520'
It might be better to define an explicit "shared build directory"
where the current image directory of a build can be stored for future
use.
In the meantime, disable the mechanism by default to speed up builds
inside a CI system (like the one from Ostro) which is not prepared to
use the mechanism anyway.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Compression with xz is slowing down do_stage_swupd_inputs (on the
critical path) by keeping one CPU 100% with xz. gzip compresses
faster and (at least for now) on-disk usage matters less than speed.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
do_stage_swupd_updates works with the entire full tree multiple times:
copying into the staging area, packing it as sstate archive, copying
to the swupd deploy directory.
Copying directory trees is slow, in particular when running under
pseudo, and do_stage_swupd_updates is on the critical path for
completing a build. Therefore it should be as fast as possible.
Storing the directory as compressed archive is faster: it cuts down
the time for do_stage_swupd_updates from 11min in the Ostro CI to
6min. This is with xz as compression method, which is suitable for
long-term archival (good compression) but a lot slower than gzip
(https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/). When
favoring speed, using gzip may be better.
The long-term goal (dream?) is to have swupd work directly with tar
archives, in which case expanding the archive and pseudo could be
avoided altogether.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Include the log output of the swupd tools in the normal stdout/stderr
logfile. That way errors are immediately visible when invoked from
bitbake and in the Ostro OS CI (which only shows the bitbake output).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When a swupd command fails, bitbake doesn't show what the error was
because the tools only write it into an internal log file. Logging it
to stderr will capture the error also in the logs shown by bitbake and
thus the Jenkins CI.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
OE-core recently introduced an intermediate IMGDEPLOYDIR into which
images and image manifests are meant to be written. IMAGE_MANIFEST already
uses it, but the manifest creation code was not using that variable
and also ignoring IMGDEPLOYDIR.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Sharing of pseudo databases was broken, leading to files with
wrong attributes: ${IMAGE_BASENAME} is different among all virtual
recipes and thus updating PSEUDO_LOCALSTATEDIR did not have the
desired effect.
Bundle recipes do not need to copy from anything (and thus they do not
depend on the mega image do_image) and also do not need to share the
pseudo database, because all that matters is the list of entries in
their rootfs. Being very specific about the task dependencies allows
more long-running image creation tasks to run in parallel.
Distinguishing between the various virtual image recipes and the base
image is a bit tricky. Therefore the "(virtual) swupd image recipes"
(called so because they get created by swupdimage.bbclass) now unsets
BUNDLE_NAME (thus removing the default "os-core" which is set in the
base recipe) and the usage of PN, PN_BASE, and BUNDLE_NAME is
explained in a comment.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Creating individual bundle directories as input for swupd is a waste
of resources and time, because swupd is just going to recreate the
"full" tree anyway.
With an improved swupd-server, we can just copy the full tree once and
then define the content of each bundle with a text file. This replaces
the "files-in-image" files. Those were used only by meta-swupd before.
They were renamed because they not only list files, but also
directories. "content" is a bit more neutral. Creating them is now
done in pure Python and integrated with the SWUPD_FILE_BLACKLIST
mechanism. That way, the content files are correct right away, which
allows removing the post-processing code (for example,
sanitise_file_list()).
The special mode of obtaining bundle content from the package manager
instead of a full rootfs gets dropped for now. If that mode can be
shown to be noticably faster then full rootfs creation, then it can be
re-added such that it also only produces a content file for the
bundle.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Splitting up the "mega" image just so that the original
swupd-create-update can be used unmodified creates lots of redundant
file operations, which are noticably slow under pseudo.
This path is meant to go upstream. For now it is included here as POC.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Using libarchive directly avoids one fork/exec per file in
swupd-make-fullfiles, which improves performance. Several
regressions in the new upstream version had to be fixed
as part of the version update.
The version got updated to make it easier to upstream the libarchive
patch. The latest upstream version actually is 3.2.7, but that version
introduces a format change. Updating to that will require further work
and preparations. Luckily, the source code patches apply cleanly to
3.2.5 and 3.2.7.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
The swupd client itself does not depend on bash anymore since
version 3.3.0. Any posix shell is fine. So let's move the
runtime dependency to the appropriate place.
If some layer's oe-swupd-helpers.bbappend does introduce
a bash dependency, it should just state that dependency
itself.
As the shell now be provided by bash or busybox, also add
an appropriate entry to SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS.
Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
These scripts don't do much and there's no reason for
them to require bash as interpreter.
Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
This allows us to completely remove the build time
depenency on libcheck when not needed, reducing
overall build time, and in addition tests can be
converted into a PACKAGECONFIG to enable them if
needed.
Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When buildstats detects TaskSucceeded for a do_rootfs task it
will try and determine the size of the rootfs using du, if the
rootfs directory isn't present the call to du fails which triggers
a bb.error.
Since e587c50c2639989d02d282c7a91134d5934eb042 do_rootfs for
swumpdimage based virtual images has been an empty function which
simply returns as soon as it's invoked, as we populate the rootfs
from the mega rootfs contents in do_image. Due to this the rootfs
directory wasn't yet present at the time buildstats detects the
TaskSucceeded and the subsequent call to du fails.
Work around this by creating an empty rootfs directory during
do_rootfs in swupdimage.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
OE-core commit 6d969bacc718e changed do_rootfs so that it creates
IMGDEPLOYDIR. That change broke the creation of additional swupd
images, because setting do_rootfs to empty caused the entire task to
be skipped, including the evaluation of the 'cleandirs' task
attribute.
It remains to be seen whether that's really the desired behavior (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10256), but as it is
what it is right now, we need to avoid the situation by overwriting
do_rootfs with non-empty code that doesn't do anything. That way, the
directory gets created.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
SWUPD server may move to a new location where a different pubkey
needs to be used and the hardcoded one won't work.
This makes pinned pubkey configurable.
Changes in v2: add explicit 'else' clause to the last statement
of do_install_append() to avoid returning exit code 1.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
|
|
The patch puts intended values to the config files with
swupd-client's default values.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
|
|
In case IMAGE_BASENAME is set on image recipe level the files ownership on
target rootfs is incorrect for recipes inheriting swupd-image.bbclass.
Depending on the context swupd-image.bbclass used either PN (PN_BASE) or
IMAGE_BASENAME when generating path to pseudo shared state directory. This
seems correct only when IMAGE_BASENAME is not set as it defaults to PN.
This patch resolves above problem.
Addresses [YOCTO #10108].
Signed-off-by: Piotr Figiel <p.figiel@camlintechnologies.com>
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Vopying->Copying
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If SWUPD_BUNDLES is empty no mega-image will be built and there's
no need to try and copy the os-core bundle contents from there,
instead the original image rootfs should be used as the source of
the copy.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If the user hasn't defined any bundles a mega-image won't be
created and there's no need to recopy the mega image rootfs.
The original rootfs can be used in this case.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Add a simple image which turns core-image-minimal into a swupd-image,
this should make it easier to start investigating and understanding
meta-swupd.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Ensure pseudo is available in the sysroot for all tasks which have
the fakeroot flag by adding virtual/fakeroot-native:do_populate_sysroot
to the depends of the tasks.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Drop and the swupd_sanity_check_image task and related
SWUPD_IMAGE_SANITY_CHECKS variable in favour of the recently added
OE-Core image QA mechanism.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The most recent version of swupd-client should now support
signed manifests.
For more details, refer to https://github.com/clearlinux/swupd-client
Signed-off-by: Igor Stoppa <igor.stoppa@intel.com>
|
|
This was a copy-and-paste of the check_output() method of the subprocess
module in order to support Python versions prior to 2.7 -- we should just
use the method from subprocess directly.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Rather than trying to determine the manifest link name by removing the value
of the DATETIME variable from the image name duplicate the logic from
rootfs-postcommands.bbclass in OE-Core to derive the manifest name in the
same way.
This prevents issues when the IMAGE_NAME (and thus manifest name) differ
significantly from the IMAGE_NAME's used in OE-Core.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This reverts commit ba2aeec203b09d96c6e81c85cdd57054bc670c93.
Manifest files are not currently writting with IMAGE_NAME_SUFFIX
|
|
Improve the usability of fetching swupd inputs from sstate objects by
writing all known OS_VERSION-->sstate object hash mappings to a variable
assignment in an inc file.
Utilising the ability to fetch swupd inputs for previous versions then
becomes a simple case of including this file, i.e. in an auto.conf or
local.conf.
As the inc file is parsed during a build all known mappings, i.e. those
loaded from the inc file and a new mapping generated by the current build,
will be written to any newly generated inc file during a build.
Preventing swupd inputs from OS_VERSIONS from being fetched and staged
becomes a simple matter of editing the inc file to remove the no longer
required maps.
An expected workflow for building new OS_VERSIONS with a CI would be:
* copy any existing inc file from previous builds to a conf dir such as
${BUILDDIR}/conf
* edit the file to ensure only versions we care about are fetched
* ensure that file is included/required by a conf file such as local.conf
or auto.conf
* build
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The addtask line for do_swupd_update contained a task which is no longer
defined by swupd-image.bbclass
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
We don't run do_stage_swupd_inputs for derived images, resulting in
no sstate object being genereted. Therefore we must also skip
do_stage_swupd_inputs_setscene for derived images.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Ensure the manifest files are written with a uniform filename
pattern for all swupd-image derived images.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Use manifest files written to DEPLOY_DIR_IMAGE, rather than
SWUPDMANIFESTDIR, during create_rootfs as the latter are (currently)
only written for package based bundles.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The image name suffix is modifiable via the IMAGE_NAME_SUFFIX
variable, therefore when constructing image filenames we should
use the same variable.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
We now ensure the do_swupd_update task is run for the base image as
a dependency of the do_swupd_update task for any derivative image.
Therefore this message is now informative, rather than indicative
of an issue.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
We already have a variable which lists all of the bundles for the
image including os-core, make use of ALL_BUNDLES instead of
assigning a new variable to use.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This change aims to help support more complex OS development
workflows where update artefacts:
* may not be sequential
* may have been generated on a separate system, or in a separate
build directory
With this change we now generate a file after the swupd inputs have
been staged and an sstate object for the inputs created which maps
the OS_VERSION of the build to the name of the sstate object.
A new task will read in these 'map' files and try to ensure that
the swupd inputs are available before do_swupd_update, first by
checking for an sstate objects in SSTATE_DIR and when not present
attempting to fetch the object from an sstate mirror, before
unpacking the object into the swupd directory for processing.
[YOCTO #9321]
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Use tempfile.mkstemp() from the standard library, rather than our
own logic, to create a unique filename in workdir to use as the
copyfile for the tar operation.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The copyxattrtree() method will copy the entire contents of the src
directory to dst preserving extended attributes on files.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Consistently use docstrings for documenting methods in lib/swupd/*
and ensure every method is documented.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Dictionaries don't have an iteritems() method in Python 3, use the
items() method instead.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The manifest files are used for various things so we must be sure
they are available, even if the swupd inputs were staged from a
shared state.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
Patrick Ohly
André Draszik
Joshua Lock
Dmitry Rozhkov
Piotr Figiel
Igor Stoppa