Age | Commit message (Collapse) | Author |
|
When SWUPD_BUNDLES is not set, exceptions are thrown, stopping the
build - there are two issues in here:
- the debug statement references a non-existent variable, 'bundle' was
renamed to fulltar
- copyxattrfiles() needs a list of files as 2nd argument, not a filename
of a file which contains this list
This got broken in commit 6c097131ad39
("meta-swupd: support rm_work.bbclass") while support for rm_work
was being added.
Signed-off-by: André Draszik <git@andred.net>
|
|
Use try: finally: to make sure the file is removed even
on exceptions.
Signed-off-by: André Draszik <git@andred.net>
|
|
Sharing data between virtual images only worked when rm_work.bbclass
was not active.
To support rm_work.bbclass, the new do_swupd_list_bundle generates the
necessary information about the rootfs before do_rm_work removes the
rootfs. The output files and the mega image rootfs.tar get excluded
from the cleanup via the new RM_WORK_EXCLUDE_ITEMS.
While at it, some inaccurate comments get removed.
As a side effect of the more granular work split, it is now possible
to make swupd images depend on exactly those bundle images that they
contain. Now it is possible to build a swupd image without first
having to build all swupd images, which might speed up a build (less
work on the critical path).
Fixes: [YOCTO #10799]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
The recent enhancements broke the case where meta-swupd
is only used as update mechanism, without any additional
bundles installed.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
The previous approaches all relied on somehow carrying additional data
across from one build to the next (sstate or additional archives in
the deploy directory).
The new approach replaces that with downloading required content on a
file-by-file basis from the normal update repo when (and not sooner)
it is needed by swupd_create_pack.
That works for meta-swupd because the format of the files (compressed
archive created with bsdtar) is expected to be stable. If a change
ever becomes necessary, some backward compatibilty mode would have to
be added or deltas simply would be skipped again.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Instead of granting all virtual images access to the mega rootfs under
a shared pseudo instance, archive the mega rootfs in an archive and
extract from that the subset of entries that are needed.
Sharing pseudo instances is slow: using more than one avoids a
potential bottleneck (the pseudo daemon is often 100% busy on a CPU
during heavy use). Extracting files with full attributes also is
faster when merely sweeping through a tar archive, at least when most
of the content is needed.
This change therefore increases performance.
bsdtar with support for --no-recursive in combination with -x is
needed for that. Current bsdtar master does not support that yet, but
adding it was easy.
GNU tar already supports that, but had bugs in that mode ("Not found
in archive" errors for entries that were in the archive).
bsdtar is also nicer for other reasons and therefore was extended instead
of trying to fix GNU tar:
- no need to explicitly add xattrs
- guaranteed to auto-detect compression, even when reading from
stdin (GNU tar can only do that when working with files); not
that relevant here, though
- uses less system calls when creating files, which should
help a bit with performance under pseudo
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Creating updates based on the Manifest.full of the previous build
allows reusing unchanged files, i.e. work on compressing these file
and the storing the result again under "files" gets avoided.
This works by referencing the previous version in the new Manifest
files. The implication of that is that versions no longer can be
published separately. The content produced by all previous builds must
also be available to the client.
This is independent of computing deltas. Nothing besides the previous
"www" content needs to be available. It gets downloaded automatically
when starting a build without a previous swupd deploy directory, so no
extra work is needed to enable this mode besides publishing the
previous build results.
Fixes [YOCTO #9189]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
This removes the storing of previous build information in sstate. It
was conceptually questionable (sstate is a cache which does not need
to be backed up, while the information about previous builds is
crucial and must not get lost) and not working:
- the -map.inc file wasn't actually included anywhere and thus
the old build information wasn't getting restored
- restoring all previous builds would have made building slower
and slower as the number of previous builds grows
- the old build information lacked the www/Manifest files that
incremental updates need
The replacement puts previous build information into the image deploy
directory. That's tentative and also not fully working.
The automatic selection of old versions to build deltas against also
gets replaced with an explicit choice that has to be made by the user
of meta-swupd. That's because in practice, incremental updates are
more useful when prepared for the releases that actually run on the
target device, like major milestones.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When the first tar in a pipe fails, its error code is getting lost.
Detect that by checking for output: normally, all operations should be
silent, so if there is output, something unusual happened.
This also catches warnings.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When reading the manifest .content.txt files in swupd-server, it
matters whether they list entries with or without leading
slash. Without it, matching files against the full content fails
because that is read from a directory and happens to use a leading
slash inside swupd_create_update, and then swupd created bundle
manifests without hashsums, leading to 404 errors during updates.
Fixing this in meta-swupd is easier than in swupd-server.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
When creating bundle images, we need to know and copy also the entries
that we exclude from processing by swupd-server. This could be done
with a more complex syntax for the .content.txt files, but that would
also make the swupd-server patches more complicated.
Instead, an .extra-content.txt gets written alongside the
.content.text and meta-swupd uses that when copying files into
images. Due to the way how this is implemented, the .extra-content.txt
of bundles also lists the files that were excluded from the bundle
because they were already in the os-core. This may or may not be
desirable.
This change also includes some other improvements (consistent use of the
helper method, sorting the content of the file lists).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Compression with xz is slowing down do_stage_swupd_inputs (on the
critical path) by keeping one CPU 100% with xz. gzip compresses
faster and (at least for now) on-disk usage matters less than speed.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
do_stage_swupd_updates works with the entire full tree multiple times:
copying into the staging area, packing it as sstate archive, copying
to the swupd deploy directory.
Copying directory trees is slow, in particular when running under
pseudo, and do_stage_swupd_updates is on the critical path for
completing a build. Therefore it should be as fast as possible.
Storing the directory as compressed archive is faster: it cuts down
the time for do_stage_swupd_updates from 11min in the Ostro CI to
6min. This is with xz as compression method, which is suitable for
long-term archival (good compression) but a lot slower than gzip
(https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/). When
favoring speed, using gzip may be better.
The long-term goal (dream?) is to have swupd work directly with tar
archives, in which case expanding the archive and pseudo could be
avoided altogether.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
OE-core recently introduced an intermediate IMGDEPLOYDIR into which
images and image manifests are meant to be written. IMAGE_MANIFEST already
uses it, but the manifest creation code was not using that variable
and also ignoring IMGDEPLOYDIR.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Creating individual bundle directories as input for swupd is a waste
of resources and time, because swupd is just going to recreate the
"full" tree anyway.
With an improved swupd-server, we can just copy the full tree once and
then define the content of each bundle with a text file. This replaces
the "files-in-image" files. Those were used only by meta-swupd before.
They were renamed because they not only list files, but also
directories. "content" is a bit more neutral. Creating them is now
done in pure Python and integrated with the SWUPD_FILE_BLACKLIST
mechanism. That way, the content files are correct right away, which
allows removing the post-processing code (for example,
sanitise_file_list()).
The special mode of obtaining bundle content from the package manager
instead of a full rootfs gets dropped for now. If that mode can be
shown to be noticably faster then full rootfs creation, then it can be
re-added such that it also only produces a content file for the
bundle.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
|
|
Vopying->Copying
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If SWUPD_BUNDLES is empty no mega-image will be built and there's
no need to try and copy the os-core bundle contents from there,
instead the original image rootfs should be used as the source of
the copy.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
If the user hasn't defined any bundles a mega-image won't be
created and there's no need to recopy the mega image rootfs.
The original rootfs can be used in this case.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This was a copy-and-paste of the check_output() method of the subprocess
module in order to support Python versions prior to 2.7 -- we should just
use the method from subprocess directly.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Rather than trying to determine the manifest link name by removing the value
of the DATETIME variable from the image name duplicate the logic from
rootfs-postcommands.bbclass in OE-Core to derive the manifest name in the
same way.
This prevents issues when the IMAGE_NAME (and thus manifest name) differ
significantly from the IMAGE_NAME's used in OE-Core.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
This reverts commit ba2aeec203b09d96c6e81c85cdd57054bc670c93.
Manifest files are not currently writting with IMAGE_NAME_SUFFIX
|
|
Use manifest files written to DEPLOY_DIR_IMAGE, rather than
SWUPDMANIFESTDIR, during create_rootfs as the latter are (currently)
only written for package based bundles.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The image name suffix is modifiable via the IMAGE_NAME_SUFFIX
variable, therefore when constructing image filenames we should
use the same variable.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Use tempfile.mkstemp() from the standard library, rather than our
own logic, to create a unique filename in workdir to use as the
copyfile for the tar operation.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The copyxattrtree() method will copy the entire contents of the src
directory to dst preserving extended attributes on files.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Consistently use docstrings for documenting methods in lib/swupd/*
and ensure every method is documented.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
The manifest files are used for various things so we must be sure
they are available, even if the swupd inputs were staged from a
shared state.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
When creating a composed rootfs for swupdimage images we also write
an updated manifest of the composed image's content. To better match
behaviour of other manifests in DEPLOY_DIR_IMAGE also create a
symlink to the latest manifest.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
When swupdimage rootfs are constructed they rely on reading the
contents of the rootfs.manifest files for each bundle they contain
in order to determine the image's contents. However it's possible,
especially with the addition of a sstate accelerated
do_stage_swupd_inputs, that the manifests will have been generated
in a different bitbake invocation and therefore that the DATETIME
in the IMAGE_NAME will be different.
This change accounts for that eventuality by falling back to the
manifest's symlink when the composed manifest filename (including
DATETIME) doesn't exist.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
When writing the rootfs.manifest for package manager staged bundles
also create a symlink to the manifest to better match the pattern
of manifests created for images.
We'll use this in a future commit to make the composed rootfs
construction in swupdimage based images more robust.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
SWUPD_FILE_BLACKLIST allows a user to list files, with a path in
the target rootfs, that they do not wish to be copied into the
swupd state directory for processing.
This mechanism can be used to prevent files being processed by
swupd which should not be tracked in a manifest file and thus not
processed by swupd-client.
The primary use for this is to exclude files in /etc which are
runtime modified (/etc/mtab) or generated at boot (/etc/machine-id)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
swupd-client doesn't support bundles being removed from the server,
if a bundle receipt still exists on the client but the manifest
disapears from the server the client doesn't know how to handle it.
Prevent a workaround for this by adding a SWUPD_EMPTY_BUNDLES
variable which can be used to continue to provide a manifest entry
for a bundle which is otherwise empty. With this workaround the
client can update to the empty bundle, removing files that it used
to provide (unless they are now provided by an alternative bundle).
Note: this was implemented as a separate variable, rather than
allowing SWUPD_BUNDLES to be defined without a corresponding
BUNDLE_CONTENTS varflag as it is expected the latter is more likely
to lead to unexpected results due to accidental misconfiguration.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|
|
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
|