Age | Commit message (Collapse) | Author |
|
This removes the storing of previous build information in sstate. It
was conceptually questionable (sstate is a cache which does not need
to be backed up, while the information about previous builds is
crucial and must not get lost) and not working:
- the -map.inc file wasn't actually included anywhere and thus
the old build information wasn't getting restored
- restoring all previous builds would have made building slower
and slower as the number of previous builds grows
- the old build information lacked the www/Manifest files that
incremental updates need
The replacement puts previous build information into the image deploy
directory. That's tentative and also not fully working.
The automatic selection of old versions to build deltas against also
gets replaced with an explicit choice that has to be made by the user
of meta-swupd. That's because in practice, incremental updates are
more useful when prepared for the releases that actually run on the
target device, like major milestones.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When the first tar in a pipe fails, its error code is getting lost.
Detect that by checking for output: normally, all operations should be
silent, so if there is output, something unusual happened.
This also catches warnings.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When reading the manifest .content.txt files in swupd-server, it
matters whether they list entries with or without leading
slash. Without it, matching files against the full content fails
because that is read from a directory and happens to use a leading
slash inside swupd_create_update, and then swupd created bundle
manifests without hashsums, leading to 404 errors during updates.
Fixing this in meta-swupd is easier than in swupd-server.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When creating bundle images, we need to know and copy also the entries
that we exclude from processing by swupd-server. This could be done
with a more complex syntax for the .content.txt files, but that would
also make the swupd-server patches more complicated.
Instead, an .extra-content.txt gets written alongside the
.content.text and meta-swupd uses that when copying files into
images. Due to the way how this is implemented, the .extra-content.txt
of bundles also lists the files that were excluded from the bundle
because they were already in the os-core. This may or may not be
desirable.
This change also includes some other improvements (consistent use of the
helper method, sorting the content of the file lists).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Compression with xz is slowing down do_stage_swupd_inputs (on the
critical path) by keeping one CPU 100% with xz. gzip compresses
faster and (at least for now) on-disk usage matters less than speed.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
do_stage_swupd_updates works with the entire full tree multiple times:
copying into the staging area, packing it as sstate archive, copying
to the swupd deploy directory.
Copying directory trees is slow, in particular when running under
pseudo, and do_stage_swupd_updates is on the critical path for
completing a build. Therefore it should be as fast as possible.
Storing the directory as compressed archive is faster: it cuts down
the time for do_stage_swupd_updates from 11min in the Ostro CI to
6min. This is with xz as compression method, which is suitable for
long-term archival (good compression) but a lot slower than gzip
(https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/). When
favoring speed, using gzip may be better.
The long-term goal (dream?) is to have swupd work directly with tar
archives, in which case expanding the archive and pseudo could be
avoided altogether.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
OE-core recently introduced an intermediate IMGDEPLOYDIR into which
images and image manifests are meant to be written. IMAGE_MANIFEST already
uses it, but the manifest creation code was not using that variable
and also ignoring IMGDEPLOYDIR.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Creating individual bundle directories as input for swupd is a waste
of resources and time, because swupd is just going to recreate the
"full" tree anyway.
With an improved swupd-server, we can just copy the full tree once and
then define the content of each bundle with a text file. This replaces
the "files-in-image" files. Those were used only by meta-swupd before.
They were renamed because they not only list files, but also
directories. "content" is a bit more neutral. Creating them is now
done in pure Python and integrated with the SWUPD_FILE_BLACKLIST
mechanism. That way, the content files are correct right away, which
allows removing the post-processing code (for example,
sanitise_file_list()).
The special mode of obtaining bundle content from the package manager
instead of a full rootfs gets dropped for now. If that mode can be
shown to be noticably faster then full rootfs creation, then it can be
re-added such that it also only produces a content file for the
bundle.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Vopying->Copying
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If SWUPD_BUNDLES is empty no mega-image will be built and there's
no need to try and copy the os-core bundle contents from there,
instead the original image rootfs should be used as the source of
the copy.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If the user hasn't defined any bundles a mega-image won't be
created and there's no need to recopy the mega image rootfs.
The original rootfs can be used in this case.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This was a copy-and-paste of the check_output() method of the subprocess
module in order to support Python versions prior to 2.7 -- we should just
use the method from subprocess directly.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Rather than trying to determine the manifest link name by removing the value
of the DATETIME variable from the image name duplicate the logic from
rootfs-postcommands.bbclass in OE-Core to derive the manifest name in the
same way.
This prevents issues when the IMAGE_NAME (and thus manifest name) differ
significantly from the IMAGE_NAME's used in OE-Core.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This reverts commit ba2aeec203b09d96c6e81c85cdd57054bc670c93.
Manifest files are not currently writting with IMAGE_NAME_SUFFIX
|
|
Use manifest files written to DEPLOY_DIR_IMAGE, rather than
SWUPDMANIFESTDIR, during create_rootfs as the latter are (currently)
only written for package based bundles.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The image name suffix is modifiable via the IMAGE_NAME_SUFFIX
variable, therefore when constructing image filenames we should
use the same variable.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Use tempfile.mkstemp() from the standard library, rather than our
own logic, to create a unique filename in workdir to use as the
copyfile for the tar operation.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The copyxattrtree() method will copy the entire contents of the src
directory to dst preserving extended attributes on files.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Consistently use docstrings for documenting methods in lib/swupd/*
and ensure every method is documented.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The manifest files are used for various things so we must be sure
they are available, even if the swupd inputs were staged from a
shared state.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
When creating a composed rootfs for swupdimage images we also write
an updated manifest of the composed image's content. To better match
behaviour of other manifests in DEPLOY_DIR_IMAGE also create a
symlink to the latest manifest.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
When swupdimage rootfs are constructed they rely on reading the
contents of the rootfs.manifest files for each bundle they contain
in order to determine the image's contents. However it's possible,
especially with the addition of a sstate accelerated
do_stage_swupd_inputs, that the manifests will have been generated
in a different bitbake invocation and therefore that the DATETIME
in the IMAGE_NAME will be different.
This change accounts for that eventuality by falling back to the
manifest's symlink when the composed manifest filename (including
DATETIME) doesn't exist.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
When writing the rootfs.manifest for package manager staged bundles
also create a symlink to the manifest to better match the pattern
of manifests created for images.
We'll use this in a future commit to make the composed rootfs
construction in swupdimage based images more robust.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
SWUPD_FILE_BLACKLIST allows a user to list files, with a path in
the target rootfs, that they do not wish to be copied into the
swupd state directory for processing.
This mechanism can be used to prevent files being processed by
swupd which should not be tracked in a manifest file and thus not
processed by swupd-client.
The primary use for this is to exclude files in /etc which are
runtime modified (/etc/mtab) or generated at boot (/etc/machine-id)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
swupd-client doesn't support bundles being removed from the server,
if a bundle receipt still exists on the client but the manifest
disapears from the server the client doesn't know how to handle it.
Prevent a workaround for this by adding a SWUPD_EMPTY_BUNDLES
variable which can be used to continue to provide a manifest entry
for a bundle which is otherwise empty. With this workaround the
client can update to the empty bundle, removing files that it used
to provide (unless they are now provided by an alternative bundle).
Note: this was implemented as a separate variable, rather than
allowing SWUPD_BUNDLES to be defined without a corresponding
BUNDLE_CONTENTS varflag as it is expected the latter is more likely
to lead to unexpected results due to accidental misconfiguration.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|