summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/tests
AgeCommit message (Collapse)Author
2024-01-05bitbake: data: Add missing dependency handling of remove operatorInsu Park
A recipe variable handles its dependencies even on the "contains" variables within the "inline Python expressions" like bb.utils.filter(). And it also handles those in the append operator correctly, but the problem is that it does not so in the remove operator. Fix it by adding the missing dependencies every time the remove operator has been handled. Also add a test case to check if the override operators handle dependencies correctly. (Bitbake rev: 48799c68b69b7921c809e0fc970303866643eb2a) Signed-off-by: Insu Park <insu0.park@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Cherry-picked from master: b90520eedb1dbc7f6a3928d089fe74fafb864eb5 - Conflicts in data.py are resolved as the master branch moved handle_contains() and handle_remove() out of the try block and added the 3rd argument, "exclusions", to handle_contains(). - The test code in codeparser.py are modified as the master branch added three more arguments to the build_dependencies(). Signed-off-by: Insu Park <insu0.park@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-05-02bitbake: tests/fetch.py: fix link to project documentationSteve Sakoman
(Bitbake rev: e16a9ca7e9286790ac37a067fdc8fde3a35a1c44) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-27bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie
CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 8ea8e443005ad92f4ad264d9abd9e90e33fb5c17) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-15bitbake: fetch/git: Fix usehead for non-default namesJoey Degges
The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: a247f56df680382d62910bb9a174e0fdd29e4ca8) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-23bitbake: tests/fetch: Handle upstream master -> main branch changeRichard Purdie
(Bitbake rev: d22cc1e587c650fd5f90cda32f5720f8a3105aac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-17bitbake: tests/fetch: Drop gnu urls from wget connectivity testRichard Purdie
These urls are no longer adding much to the test coverage but the intermittent network issues connecting to them are painful. Drop the urls. (Bitbake rev: e6d75e0beb95aa0cdf82bbc0a6b767c7f6cfcfc0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit bdf5739c5d831dc97a7d81568f94a0953c71017f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-09bitbake: tests/fetch: Update pcre.org address after github changesRichard Purdie
vcs.pcre.org was a redirect to github which we use for subversion testing. With the protocol changes at github and the removal of the redirect, use a direct address for github. (Bitbake rev: fa471399d41efdf61e95e0be541b45f0621756f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6230ca71eb7eb2a6db162e28a01727d00af5299b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-08bitbake: tests/fetch: Update github urlsRichard Purdie
(Bitbake rev: f1a3e9d22b5f4fb01c4a0e4ba03afb1afbba47f1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 07fca7e3ab696ba985b3ef86ab9031d688bf2df2) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-08bitbake: tests/fetch2: Fix quoting warningRichard Purdie
Fix: lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence for several lines of the fetch tests. (Bitbake rev: bd8883d756328ca4c8f6bf97f77e17133a6bfb45) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9d84fd557a3fcbae2cdd70b24e69325ad737a01e) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-23bitbake: compat.py: remove file since it no longer actually implements anythingChris Laplante
Now that compat.py just imports Python standard library stuff, get rid of the layer of indirection. (Bitbake rev: 2a40a776efd7c28fa93a8556accccb79e3a7d0f0) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e2be6defbb9fcf25f9df04c3b452d0dba48dfd03) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-19bitbake: test/fetch: Update urls to match upstream branch name changesRichard Purdie
(Bitbake rev: 93094705c1c66688e0730a1efb17805778c9fa9f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 473e2a5486bd972ad0f808db089abcb8945d3a48) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-22bitbake: tests/fetch2: Use our own git server for dtc test repoRichard Purdie
(Bitbake rev: 3c1799b2576f80b6dcb310e03f77105a58b9fa8e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30bitbake: data_smart: Allow colon in variable expansion regexRichard Purdie
Now that ":" is a valid character in variable key names, it needs to be allowed by the variable expansion code too, to match. (Bitbake rev: c5418eae56cc50dbae7951c055434a0c456c53a4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-13bitbake: bitbake: tests/fetch: remove write protected files tooMikko Rapeli
For some reason several git-annex files in Debian 10 buster are read-only and removing them with "rm -rf" fails. Fixes test failures like: $ bitbake-selftest ... rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied EE..................................ssss.sssssssssssssss.sssss....................................................................................................... ====================================================================== ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex fetcher, ud = self.fetch_shallow(uri) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow bb.utils.remove(ud.clonedir, recurse=True) File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path)) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1. Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod the whole temporary directory which should cover any directory name differences between different git-annex versions. Fixes tests failing after chmod call: Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/ (Bitbake rev: 1fae1c812138f35c35ea4c0586e21f022524c5f1) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-13bitbake: bitbake: tests/fetch: fix test execution without .gitconfigMikko Rapeli
A CI user validating changes does not have any git push rights or even a .gitconfig file so fix tests so that they run by setting the user.name and user.email for the repo before committing changes. Fixes errors like: ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_no r_shallow_tarball_exist self.add_empty_file('a') File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file self.git(['commit', '-m', msg, path], cwd) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git return bb.process.run(cmd, cwd=cwd)[0] File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run raise ExecutionError(cmd, pipe.returncode, stdout, stderr) bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128: *** Please tell me who you are. Run git config --global user.email "you@example.com" git config --global user.name "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. (Bitbake rev: e341afcce40edf8078661630af1e0a780e8c1910) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-05bitbake: fetch/git: download LFS content too during do_fetchMatt Hoosier
Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] (Bitbake rev: 2351b496bb63b96920d4ae67bec816f00d510df2) Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-26bitbake: tests/fetch: Update upstream master->main branchname transitionRichard Purdie
(Bitbake rev: 89fc9450abefd682266efcbfa031d1ef115ff1a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-13bitbake: bitbake: tests/fetch: add unit tests for SRC_URI with spaces in urlCharlie Davies
(Bitbake rev: d8daad57bdbceec041c4c2d288ed5d487ad8e6dc) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e7dab75c8d1923abcbbc7c9ac7de215d720ccf26) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-18bitbake: tests/fetch: Move away from problematic freedesktop.org urlsRichard Purdie
We're either hitting rate limiting with freedesktop.org or the servers have intermittent network connections. Use our own mirror of these repositories instead. (Bitbake rev: 18e1957337fd9f06bc673d28dd4f8277321d07bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a1b7ab5c9d5e64969f5ca0e41c0ac13c723e3761) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-17bitbake: fetch2: Drop cups.org from wget status checksRichard Purdie
Its becomming clear the upstream server doesn't like this, drop these two urls from the test, not sure we need them here anyway. (Bitbake rev: 16774d4d41f2011e8db7d2986a1a701df123dfc1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ab2ef942dc21f9639793c972f2e546edf9444783) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02bitbake: test/fetch: change to better svn sourceakuster
fixes: svn: warning: W175002: Unexpected HTTP status 504 'Gateway Timeout' on '/openembedded/bitbake/!svn/vcc/default' svn: E205011: Failure occurred processing one or more externals definitions picked pcre2 [Yocto #13948] (Bitbake rev: 767aa9316603a1c92c9c433a0c11ae98089bbc3a) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1483d17108da02f5d615e83403d5fd6288ca957c) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-22bitbake: tests/fetch: Switch from git.infradead.org to a YP mirrorRichard Purdie
Upstream is unavailable, breaking tests. Switch to a YP mirror since if we can't reach that there are bigger problems. This should remove a source of intermittent failures on the autobuilder. (Bitbake rev: 83296870bede70e31bdf6e73683bcc30681023fc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06bitbake: data/siggen: Don't expand ignored variablesRichard Purdie
If a variable is in the signature whitelist, we'd currently expand it, then later ignore the data. This is problemtic for code which has effects when expanded, recently source date epoch in OE-Core for example. We don't actually need to do this, if we pass the whitelist into the earlier function it can avoid the expansion. This also also give a small performance boost since we avoid running code in some cases. [YOCTO #13581] (Bitbake rev: f483ee4a869fb1dafbe4bdf2da228cdaa40b38bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24bitbake: tinfoil: Simplify remote datastore connectionsRichard Purdie
The current approach to remote datastores used in tinfoil is breaking. For example, adding a devupstream extension to a recipe with a git upstream, making it the preferred version and then running "devtool modify" on it causes get_srcrev() circular dependency issues. The problem is the override handling in the datastore is broken. This gets broken since remotedata:recieve_datastore() sets d.dict but doesn't update d.overridedata (or d.inchistory or d.varhistory). We could play whack-a-mole but the current implementation seems to be flawed to me. It also doesn't cover, or only partially covers some datastore operations and each needs new dedicated command API. Instead, step back and reimplement the way the datastore connector works. With this change, the datastore is either remote or local but the data is not spread on two sides of the connection. All the API is proxied over the connection by a single function for the datastore (and two to support variable history and include history). This code does not support using the datastore as a parameter to any data store functions. We did have one case of that but its just bad code and can be replaced. The result is something which is much simpler and less invasive to the datastore code itself, meaning its behaviour should be much more consistent. The existing tests for the remote data no longer make any sense and are removed. The one bug this code would have is if key/value pairs are returned over the IPC and those values contained a DataSmart object since we don't recurse into return values to find such things. Nothing appears to do that currently so lets worry about it if its ever an issue. This change should simplfy a ton of other issues and avoid a ton of other bugs so is a huge net gain. Tested with bitbake's and OE's selftests. (Bitbake rev: 85e03a64dd0a4ebe71009ec4bdf4192c04a9786e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-11bitbake: tests: Add test for gitsm fetcher with shallow mirror tarballsPaul Barker
(Bitbake rev: 2e26e97129d4c54bf86cdea8f9791696a06a36b4) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27bitbake: tests/fetch: add npmsw testsJean-Marie LEMETAYER
This commit adds some tests to validate the npmsw fetcher: - bb.tests.fetch.NPMTest.test_npmsw - bb.tests.fetch.NPMTest.test_npmsw_bad_checksum - bb.tests.fetch.NPMTest.test_npmsw_destsuffix - bb.tests.fetch.NPMTest.test_npmsw_dev - bb.tests.fetch.NPMTest.test_npmsw_mirrors - bb.tests.fetch.NPMTest.test_npmsw_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npmsw_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npmsw_npm_reusability - bb.tests.fetch.NPMTest.test_npmsw_premirrors (Bitbake rev: ba205df20b6a07a4b1125332601c6c54c7b019b5) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27bitbake: tests/fetch: add npm testsJean-Marie LEMETAYER
This commit adds some tests to validate the npm fetcher: - bb.tests.fetch.NPMTest.test_npm - bb.tests.fetch.NPMTest.test_npm_bad_checksum - bb.tests.fetch.NPMTest.test_npm_destsuffix_downloadfilename - bb.tests.fetch.NPMTest.test_npm_mirrors - bb.tests.fetch.NPMTest.test_npm_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npm_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npm_package_invalid - bb.tests.fetch.NPMTest.test_npm_package_none - bb.tests.fetch.NPMTest.test_npm_premirrors - bb.tests.fetch.NPMTest.test_npm_registry_alternate - bb.tests.fetch.NPMTest.test_npm_registry_invalid - bb.tests.fetch.NPMTest.test_npm_registry_none - bb.tests.fetch.NPMTest.test_npm_version_invalid - bb.tests.fetch.NPMTest.test_npm_version_latest - bb.tests.fetch.NPMTest.test_npm_version_none (Bitbake rev: b166bd3cc6cc1ca63e885319091f17daaaaa2537) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-22bitbake: tests/fetch: Allow wget upgrade tests to run against a local server3.1_M2Richard Purdie
Currently these tests rely upon multiple uptream webservers which may change or be unavailable. Add local copies of the test data, copy the httpserver from OE-Core (used for testing there) and run these tests against a local server instead. (Bitbake rev: d5a4a352723258b4d499d3a51f340109c4f36f60) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19bitbake: tests/utils: add tests for bb.utils.get_referenced_varsChris Laplante via bitbake-devel
(Bitbake rev: 346e74e5d751aadf7881de70b5ab6670dfc463ce) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19bitbake: lib: remove unused importsFrazer Clews
removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-27bitbake: tests/runqueue: Fix to match recent task migration fixesRichard Purdie
(Bitbake rev: 35784582fdbb2092eddec094deb6ab9c87666b5e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-21bitbake: tests: add test for the hashing functionsRoss Burton
Add a basic test for bb.utils.md5_file() etc. (Bitbake rev: e944d02fe678f7c6b05c62419f8bceb0709f3037) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-28bitbake: tests/fetch: add test for fetching shallow revsChristopher Larson
[YOCTO #13586] (Bitbake rev: f7a973604fbfd1059875ff1fabb3f885df9c5b95) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02bitbake: tests/runqueue: Fix hashserve shutdown raceRichard Purdie
The hashserve can delete its socket whilst the cleanup us happening leading to backtraces and test failures. Add code to avoid this race condition. [YOCTO #13542] (Bitbake rev: efd7b025cee25d0ee668c09476395d08fcf5ae1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30bitbake: runqueue: Change task migration behaviour for rerunning setscene tasksRichard Purdie
Currently runqueue will rerun setscene tasks multiple times as hashes change. This has caused numerous problems since a setscene task may become "unavailable" for some future signature combination and the code then can't easily "unskip" tasks its already passed into the execution queue. At least for now, only run setscene once and assume they're equivalent at that point. In practise that has been much more stable in testing. Tweak the test to match the change in behaviour. (Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30bitbake: runqueue: Fix task migration problemsRichard Purdie
Tasks were not migrating consistently, particularly: * if a task was rehashed which had already run * if a task which was valid became invalid due to a rehash We need to always run the migration code for rehashed tasks and then reprocess them for hash validity. This means rearranging the code. It also means several tests are no longer correct and can't be written correctly to work on all possible workflows so those are removed. (Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27bitbake: tests/data: Test combinations of _append together with overrideJacob Kroon
(Bitbake rev: f31f35e8527c60a95931a4a8311a4cd237770b42) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27bitbake: tests/fetch: add test case for git-lfs handlingRoss Burton
Add a test case to exercise the detection of git-lfs repositories and the behaviour of the lfs parameter. (Bitbake rev: a7cf4fc72cce357c425084dc2c5f35b5ed1a4b7b) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18bitbake: bitbake: Rework hash equivalenceJoshua Watt
Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-01bitbake: tests/fetch: Resolve fetch error in bitbake-selftestArmin Kuster
FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) AssertionError: '' is not true : Could not find upstream version for db [YOCTO #13496] The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget. Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the recipe. Also change the version being checked. (Bitbake rev: 4cf5bb761c561ddea86f2875be35d05abc8486e1) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-15bitbake: tests/runqueue: Fix testsRichard Purdie
There were paths being accidentally included in some of the hashserv tests. Remove that and update the hashes so the tests work independently of paths. (Bitbake rev: 6ddb9f09cb60c2354fa6a67cce412c4dc1e7dc2d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14bitbake: runqueue: Fix event timing raceRichard Purdie
The event from the task notifiing of hash equivalency should only be processed when the task completes. This can otherwise result in a race where a dependent task may run before the original task completes causing various failures. To make this work reliably, the code had to be restructured quite a bit. (Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14bitbake: tests/runqueue: Add further hash equivalence testsRichard Purdie
Add some extra hash equivalence runqueue tests based on recent scenarios that caused problems during testing. (Bitbake rev: 373b085ead992a725b2230ededd992b4c61a1a05) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06bitbake: tests/runqueue: Add hashserv+runqueue testRichard Purdie
Add a test which tests the runqueue adaptations for hash equivalency. (Bitbake rev: 477321d0780df177c1582db119c2bb6795912fc6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06bitbake: siggen: Clean up task reference formatsRichard Purdie
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the separator everywhere. This is an API breaking change since the cache is affected, as are siginfo files and any custom signature handlers such as those in OE-Core. Ultimately this will let us clean up and the accessor functions from runqueue, removing all the ".rsplit(".", 1)[0]" type code currently all over the place. Once a standard is used everwhere we can update the code over time to be more optimal. (Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06bitbake: runqueue: Clean up BB_HASHCHECK_FUNCTION APIRichard Purdie
This function uses an old API which uses offsets into lists as a communication mechanism. Update the API to use "tid" which is used universally in runqueue now. We can also add kwargs support to the funciton definition to drop some of the backwards compaiblility hoops we had to jump though with different function argument combinations. (Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-18bitbake: runqueue: Fix non setscene tasks targets being lostRichard Purdie
If you specify both setscene and non-setscene tasks on the commandline, the non-setscene tasks could be missed, e.g. "bitbake X:do_patch X:do_populate_sysroot" and do_patch would fail to run. Fix the problem in runqueue and add a testcase. (Bitbake rev: 75292fdec5d9c0b5b3c554c4b7474a63656f7e12) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16bitbake: tests/runqueue: Allow common sstate tasks to become validRichard Purdie
As the logic in bitbake improves, the logic in the tests needs to as well. Afer we built a task for the first time, allow its setscene hash verification status to change, mirroring what would happen in a multiconfig build. (Bitbake rev: 27ec2e69ab3e32972caf8b072b2945736696d83d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16bitbake: runqueue: Optimise multiconfig with overlapping setsceneRichard Purdie
Currently if a multiconfig build contains different configurations which have overlapping sstate artefacts, it will build them multiple times. This is clearly suboptimal and not what users want/expect. This adds code to detect this and stall all but one of the setscne tasks so that once its built, it can be found by the other tasks. We take care to iterate the multiconfigs in order so try and avoid dependency loops. We also match on PN+taskname+taskhash since this is what we know sstate in OE-Core would use. There are some tasks even within a multiconfig which match hashes (mostly do_populate_lic tasks) but those have a much higher chance of circular dependency so aren't work attempting to optimise. If a deadlock does occur the build will be slower but there is code to unbreak such a deadlock so it hopefully doens't break anything. Comments are injected into the test tasks so they have different task hashes and a new test for this optimisation is added. (Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15bitbake: tests: Add initial scenario based test for runqueueRichard Purdie
We need some tests for runqueue, its been something which has been hard to test for a long time. Add some dummy metadata to allow this, mirroring the OE structure in spirit. (Bitbake rev: 37564d7440c5d7aa05ec537f3b79026b1c83bb68) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-25 20:36:55 +0000