aboutsummaryrefslogtreecommitdiffstats
path: root/lib
AgeCommit message (Collapse)Author
2016-12-08meta-swupd: revise delta computationPatrick Ohly
This removes the storing of previous build information in sstate. It was conceptually questionable (sstate is a cache which does not need to be backed up, while the information about previous builds is crucial and must not get lost) and not working: - the -map.inc file wasn't actually included anywhere and thus the old build information wasn't getting restored - restoring all previous builds would have made building slower and slower as the number of previous builds grows - the old build information lacked the www/Manifest files that incremental updates need The replacement puts previous build information into the image deploy directory. That's tentative and also not fully working. The automatic selection of old versions to build deltas against also gets replaced with an explicit choice that has to be made by the user of meta-swupd. That's because in practice, incremental updates are more useful when prepared for the releases that actually run on the target device, like major milestones. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-12-08path.py: better error handling in copyxattrfiles()Patrick Ohly
When the first tar in a pipe fails, its error code is getting lost. Detect that by checking for output: normally, all operations should be silent, so if there is output, something unusual happened. This also catches warnings. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-12-08utils.py: list files with leading slash in manifestsPatrick Ohly
When reading the manifest .content.txt files in swupd-server, it matters whether they list entries with or without leading slash. Without it, matching files against the full content fails because that is read from a directory and happens to use a leading slash inside swupd_create_update, and then swupd created bundle manifests without hashsums, leading to 404 errors during updates. Fixing this in meta-swupd is easier than in swupd-server. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-12-08fix meta-swupd: avoid splitting up mega rootfsPatrick Ohly
When creating bundle images, we need to know and copy also the entries that we exclude from processing by swupd-server. This could be done with a more complex syntax for the .content.txt files, but that would also make the swupd-server patches more complicated. Instead, an .extra-content.txt gets written alongside the .content.text and meta-swupd uses that when copying files into images. Due to the way how this is implemented, the .extra-content.txt of bundles also lists the files that were excluded from the bundle because they were already in the os-core. This may or may not be desirable. This change also includes some other improvements (consistent use of the helper method, sorting the content of the file lists). Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-12-08swupd input: compress with gzPatrick Ohly
Compression with xz is slowing down do_stage_swupd_inputs (on the critical path) by keeping one CPU 100% with xz. gzip compresses faster and (at least for now) on-disk usage matters less than speed. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-12-08meta-swupd: put compressed rootfs into sstatePatrick Ohly
do_stage_swupd_updates works with the entire full tree multiple times: copying into the staging area, packing it as sstate archive, copying to the swupd deploy directory. Copying directory trees is slow, in particular when running under pseudo, and do_stage_swupd_updates is on the critical path for completing a build. Therefore it should be as fast as possible. Storing the directory as compressed archive is faster: it cuts down the time for do_stage_swupd_updates from 11min in the Ostro CI to 6min. This is with xz as compression method, which is suitable for long-term archival (good compression) but a lot slower than gzip (https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/). When favoring speed, using gzip may be better. The long-term goal (dream?) is to have swupd work directly with tar archives, in which case expanding the archive and pseudo could be avoided altogether. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-12-08rootfs.py: adapt to IMGDEPLOYDIR changePatrick Ohly
OE-core recently introduced an intermediate IMGDEPLOYDIR into which images and image manifests are meant to be written. IMAGE_MANIFEST already uses it, but the manifest creation code was not using that variable and also ignoring IMGDEPLOYDIR. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-12-08meta-swupd: avoid splitting up mega rootfsPatrick Ohly
Creating individual bundle directories as input for swupd is a waste of resources and time, because swupd is just going to recreate the "full" tree anyway. With an improved swupd-server, we can just copy the full tree once and then define the content of each bundle with a text file. This replaces the "files-in-image" files. Those were used only by meta-swupd before. They were renamed because they not only list files, but also directories. "content" is a bit more neutral. Creating them is now done in pure Python and integrated with the SWUPD_FILE_BLACKLIST mechanism. That way, the content files are correct right away, which allows removing the post-processing code (for example, sanitise_file_list()). The special mode of obtaining bundle content from the package manager instead of a full rootfs gets dropped for now. If that mode can be shown to be noticably faster then full rootfs creation, then it can be re-added such that it also only produces a content file for the bundle. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
2016-08-10swupd.bundles: fix typo in debug entryJoshua Lock
Vopying->Copying Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-08-10swupd.bundles: handle empty SWUPD_BUNDLES in copy_core_contentsJoshua Lock
If SWUPD_BUNDLES is empty no mega-image will be built and there's no need to try and copy the os-core bundle contents from there, instead the original image rootfs should be used as the source of the copy. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-08-10swupd.rootfs: handle empty SWUPD_BUNDLES in create_rootfsJoshua Lock
If the user hasn't defined any bundles a mega-image won't be created and there's no need to recopy the mega image rootfs. The original rootfs can be used in this case. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-07-04Drop use of oe.path.check_output()Joshua Lock
This was a copy-and-paste of the check_output() method of the subprocess module in order to support Python versions prior to 2.7 -- we should just use the method from subprocess directly. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-27lib: fix manifest link namesJoshua Lock
Rather than trying to determine the manifest link name by removing the value of the DATETIME variable from the image name duplicate the logic from rootfs-postcommands.bbclass in OE-Core to derive the manifest name in the same way. This prevents issues when the IMAGE_NAME (and thus manifest name) differ significantly from the IMAGE_NAME's used in OE-Core. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-27Revert "lib: don't hard-code .rootfs, use IMAGE_NAME_SUFFIX"Joshua Lock
This reverts commit ba2aeec203b09d96c6e81c85cdd57054bc670c93. Manifest files are not currently writting with IMAGE_NAME_SUFFIX
2016-06-22lib/rootfs: fix manifest paths in create_rootfsJoshua Lock
Use manifest files written to DEPLOY_DIR_IMAGE, rather than SWUPDMANIFESTDIR, during create_rootfs as the latter are (currently) only written for package based bundles. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-20lib: don't hard-code .rootfs, use IMAGE_NAME_SUFFIXJoshua Lock
The image name suffix is modifiable via the IMAGE_NAME_SUFFIX variable, therefore when constructing image filenames we should use the same variable. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-13lib/swupd/path: use tempfile for temporary file creationJoshua Lock
Use tempfile.mkstemp() from the standard library, rather than our own logic, to create a unique filename in workdir to use as the copyfile for the tar operation. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-13lib/swupd/path: add method to copy a directory preserving attributesJoshua Lock
The copyxattrtree() method will copy the entire contents of the src directory to dst preserving extended attributes on files. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-13lib/swupd: add docstrings for all methodsJoshua Lock
Consistently use docstrings for documenting methods in lib/swupd/* and ensure every method is documented. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-03swupd-image: include manifest files in the shared stateJoshua Lock
The manifest files are used for various things so we must be sure they are available, even if the swupd inputs were staged from a shared state. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-02swupd/rootfs: create composed rootfs manifest symlinkJoshua Lock
When creating a composed rootfs for swupdimage images we also write an updated manifest of the composed image's content. To better match behaviour of other manifests in DEPLOY_DIR_IMAGE also create a symlink to the latest manifest. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-02swupd/rootfs: more robust composed rootfs for swupdimage imagesJoshua Lock
When swupdimage rootfs are constructed they rely on reading the contents of the rootfs.manifest files for each bundle they contain in order to determine the image's contents. However it's possible, especially with the addition of a sstate accelerated do_stage_swupd_inputs, that the manifests will have been generated in a different bitbake invocation and therefore that the DATETIME in the IMAGE_NAME will be different. This change accounts for that eventuality by falling back to the manifest's symlink when the composed manifest filename (including DATETIME) doesn't exist. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-02swupd/bundles: write manifest symlink for bundle manifestsJoshua Lock
When writing the rootfs.manifest for package manager staged bundles also create a symlink to the manifest to better match the pattern of manifests created for images. We'll use this in a future commit to make the composed rootfs construction in swupdimage based images more robust. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-01swupd-image: mechanism to prevent files entering swupd manifestsJoshua Lock
SWUPD_FILE_BLACKLIST allows a user to list files, with a path in the target rootfs, that they do not wish to be copied into the swupd state directory for processing. This mechanism can be used to prevent files being processed by swupd which should not be tracked in a manifest file and thus not processed by swupd-client. The primary use for this is to exclude files in /etc which are runtime modified (/etc/mtab) or generated at boot (/etc/machine-id) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-01swupd-image: support dummy bundlesJoshua Lock
swupd-client doesn't support bundles being removed from the server, if a bundle receipt still exists on the client but the manifest disapears from the server the client doesn't know how to handle it. Prevent a workaround for this by adding a SWUPD_EMPTY_BUNDLES variable which can be used to continue to provide a manifest entry for a bundle which is otherwise empty. With this workaround the client can update to the empty bundle, removing files that it used to provide (unless they are now provided by an alternative bundle). Note: this was implemented as a separate variable, rather than allowing SWUPD_BUNDLES to be defined without a corresponding BUNDLE_CONTENTS varflag as it is expected the latter is more likely to lead to unexpected results due to accidental misconfiguration. Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2016-06-01swupd-image: move various helper methods to lib/swupd moduleJoshua Lock
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>