summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/tests/fetch.py
AgeCommit message (Collapse)Author
2022-02-23bitbake: tests/fetch: Handle upstream master -> main branch changeRichard Purdie
(Bitbake rev: 3603da4b5c4b7cbb29f93a1b8374198ee82e18eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-07bitbake: tests/fetch: Drop gnu urls from wget connectivity testRichard Purdie
These urls are no longer adding much to the test coverage but the intermittent network issues connecting to them are painful. Drop the urls. (Bitbake rev: 598ea8584f792b30799231c4d3151d40852fc43c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit bdf5739c5d831dc97a7d81568f94a0953c71017f) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-21bitbake: tests/fetch: Update pcre.org address after github changesRichard Purdie
vcs.pcre.org was a redirect to github which we use for subversion testing. With the protocol changes at github and the removal of the redirect, use a direct address for github. (Bitbake rev: 3ce3efde167e8cd109659b81275a6935875877eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6230ca71eb7eb2a6db162e28a01727d00af5299b) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-21bitbake: fetch: Handle mirror user/password replacements correctlyRichard Purdie
Username or password replacements in URIs were being appended rather than replaced in mirror url remapping. Fix this and add a test case. [YOCTO #13823] (Bitbake rev: 5a13bca0e9088be4191f6120336c79dadfb86f76) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 66ad58bb87e5158aced572be4f1d5726bc97fcce) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-21bitbake: tests/fetch: Update github urlsRichard Purdie
(Bitbake rev: a95f05359e65c3336eeb8366c0c35a23d3821dda) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 07fca7e3ab696ba985b3ef86ab9031d688bf2df2) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-19bitbake: test/fetch: Update urls to match upstream branch name changesRichard Purdie
(Bitbake rev: bcd4285116ea4990f10d53698e0a81ae1e7ce24c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-22bitbake: tests/fetch2: Use our own git server for dtc test repoRichard Purdie
(Bitbake rev: 19c5605b72e25e06f5ea57bcc1672650e7978a7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27bitbake: bitbake: tests/fetch: remove write protected files tooMikko Rapeli
For some reason several git-annex files in Debian 10 buster are read-only and removing them with "rm -rf" fails. Fixes test failures like: $ bitbake-selftest ... rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied EE..................................ssss.sssssssssssssss.sssss....................................................................................................... ====================================================================== ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex fetcher, ud = self.fetch_shallow(uri) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow bb.utils.remove(ud.clonedir, recurse=True) File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path)) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1. Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod the whole temporary directory which should cover any directory name differences between different git-annex versions. Fixes tests failing after chmod call: Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/ (Bitbake rev: 14c5f0735947307b9d69c57f7334fefaea7311b3) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7729ef2983c72867e99fad82d671069ba5cb32b2) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27bitbake: bitbake: tests/fetch: fix test execution without .gitconfigMikko Rapeli
A CI user validating changes does not have any git push rights or even a .gitconfig file so fix tests so that they run by setting the user.name and user.email for the repo before committing changes. Fixes errors like: ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist self.add_empty_file('a') File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file self.git(['commit', '-m', msg, path], cwd) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git return bb.process.run(cmd, cwd=cwd)[0] File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run raise ExecutionError(cmd, pipe.returncode, stdout, stderr) bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128: *** Please tell me who you are. Run git config --global user.email "you@example.com" git config --global user.name "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. (Bitbake rev: 1e1d1187e602aa1ef50c23551eec07f1a0cd81ef) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 57c0811f1ee19b6619f4840a39e01e3cb98c34c4) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23bitbake: doc: Update links to documentationMichael Opdenacker
(Bitbake rev: ed8e1fd4cf9d5ac8a8203638add99d686b4b3521) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23bitbake: fetch/git: download LFS content too during do_fetchMatt Hoosier
Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] (Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23bitbake: tests/fetch: add test for empty query parametersRoss Burton
To exercise the previous patch, add a test case. (Bitbake rev: efdf6d5c4cd8155bc54dc667346f58dccb138e07) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08bitbake: fetch/git: Fix usehead for non-default namesJoey Degges
The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08bitbake: tests/fetch: Test usehead with a non-default nameJoey Degges
Add tests for fetching a URL with the usehead parameter set and a non-default name set. We currently expect the local version of this test to fail since there is a bug in the usehead implementation that breaks for non-default names. (Bitbake rev: a2345110f217fac429f6ec15f699c87c39531e7c) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08bitbake: tests/fetch: Document behavior of test_gitfetch_useheadJoey Degges
The test `test_gitfetch_usehead' exercises a way to override the usehead feature by setting SRCREV. It may not be obvious that this is what is being exercised here so let's add some comments to document the expected behavior. (Bitbake rev: 1cd998c19101e3b093e81c126b3048c5d56058b0) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08bitbake: tests/fetch: Organize usehead tests by net requirementsJoey Degges
Move the local only usehead test to the FetcherLocalTest class so it will be run when BB_SKIP_NETTESTS=yes since it does not require network access. Rename the usehead tests to better match the new organization. (Bitbake rev: 137cfa13d5319bc91c3e5fe6c7062cb8c8484d64) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-26bitbake: tests/fetch: Update upstream master->main branchname transitionRichard Purdie
(Bitbake rev: 8b9471e02528320f6ef8d35840b5618883e85447) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-25bitbake: tests/fetch: backslash support in file:// URIsLeif Middelschulte
Implements backslashes in local filenames. A typical usecase for such a filename is a systemd unit. Example: `dev-disk-by\x2dlabel-FOO.device` (Bitbake rev: 14a35f273b579d5cd5fd92765b89c28f870dd577) Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-23bitbake: bitbake: tests/fetch: add unit tests for SRC_URI with spaces in urlCharlie Davies
(Bitbake rev: e7dab75c8d1923abcbbc7c9ac7de215d720ccf26) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-15bitbake: tests/fetch: Move away from problematic freedesktop.org urlsRichard Purdie
We're either hitting rate limiting with freedesktop.org or the servers have intermittent network connections. Use our own mirror of these repositories instead. (Bitbake rev: a1b7ab5c9d5e64969f5ca0e41c0ac13c723e3761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26bitbake: fetch2: Drop globbing supprt in file:// SRC_URIsRichard Purdie
Globbing in file:// urls is terminally broken. Currently when its used, the file checksum code is basically bypassed. This means changes to the source files don't change the task checksum, the task doesn't rebuild when the inputs change and things generally break. To make globbing work generically, we'd have to scan the file system for all possible matches to the glob and log whether they exist or not. We can't simply log the files which exist, we have to also know which files could later exist and influence the choice of file so we know when to reparse. For a simple file://xxx/*, this could be done but for bigger patterns, it becomes much more problemtic. We already support file://xxx/ in urls. So, lets decide we'll not support globs in file://urls. Worse case users can put files in a directory and reference that, moving files into place if needed. Remove all the glob special cases (see the comments if anyone doesn't believe this is terminally broken) and error to the user if they have such urls. (Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-24bitbake: fetch2: Drop cups.org from wget status checksRichard Purdie
Its becomming clear the upstream server doesn't like this, drop these two urls from the test, not sure we need them here anyway. (Bitbake rev: ab2ef942dc21f9639793c972f2e546edf9444783) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-28bitbake: test/fetch: change to better svn sourceakuster
fixes: svn: warning: W175002: Unexpected HTTP status 504 'Gateway Timeout' on '/openembedded/bitbake/!svn/vcc/default' svn: E205011: Failure occurred processing one or more externals definitions picked pcre2 [Yocto #13948] (Bitbake rev: 1483d17108da02f5d615e83403d5fd6288ca957c) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-22bitbake: tests/fetch: Switch from git.infradead.org to a YP mirrorRichard Purdie
Upstream is unavailable, breaking tests. Switch to a YP mirror since if we can't reach that there are bigger problems. This should remove a source of intermittent failures on the autobuilder. (Bitbake rev: 232471083d6c574c7ada6320f9379ad7d7862a9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-11bitbake: tests: Add test for gitsm fetcher with shallow mirror tarballsPaul Barker
(Bitbake rev: 2e26e97129d4c54bf86cdea8f9791696a06a36b4) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27bitbake: tests/fetch: add npmsw testsJean-Marie LEMETAYER
This commit adds some tests to validate the npmsw fetcher: - bb.tests.fetch.NPMTest.test_npmsw - bb.tests.fetch.NPMTest.test_npmsw_bad_checksum - bb.tests.fetch.NPMTest.test_npmsw_destsuffix - bb.tests.fetch.NPMTest.test_npmsw_dev - bb.tests.fetch.NPMTest.test_npmsw_mirrors - bb.tests.fetch.NPMTest.test_npmsw_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npmsw_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npmsw_npm_reusability - bb.tests.fetch.NPMTest.test_npmsw_premirrors (Bitbake rev: ba205df20b6a07a4b1125332601c6c54c7b019b5) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27bitbake: tests/fetch: add npm testsJean-Marie LEMETAYER
This commit adds some tests to validate the npm fetcher: - bb.tests.fetch.NPMTest.test_npm - bb.tests.fetch.NPMTest.test_npm_bad_checksum - bb.tests.fetch.NPMTest.test_npm_destsuffix_downloadfilename - bb.tests.fetch.NPMTest.test_npm_mirrors - bb.tests.fetch.NPMTest.test_npm_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npm_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npm_package_invalid - bb.tests.fetch.NPMTest.test_npm_package_none - bb.tests.fetch.NPMTest.test_npm_premirrors - bb.tests.fetch.NPMTest.test_npm_registry_alternate - bb.tests.fetch.NPMTest.test_npm_registry_invalid - bb.tests.fetch.NPMTest.test_npm_registry_none - bb.tests.fetch.NPMTest.test_npm_version_invalid - bb.tests.fetch.NPMTest.test_npm_version_latest - bb.tests.fetch.NPMTest.test_npm_version_none (Bitbake rev: b166bd3cc6cc1ca63e885319091f17daaaaa2537) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-22bitbake: tests/fetch: Allow wget upgrade tests to run against a local server3.1_M2Richard Purdie
Currently these tests rely upon multiple uptream webservers which may change or be unavailable. Add local copies of the test data, copy the httpserver from OE-Core (used for testing there) and run these tests against a local server instead. (Bitbake rev: d5a4a352723258b4d499d3a51f340109c4f36f60) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19bitbake: lib: remove unused importsFrazer Clews
removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-28bitbake: tests/fetch: add test for fetching shallow revsChristopher Larson
[YOCTO #13586] (Bitbake rev: f7a973604fbfd1059875ff1fabb3f885df9c5b95) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27bitbake: tests/fetch: add test case for git-lfs handlingRoss Burton
Add a test case to exercise the detection of git-lfs repositories and the behaviour of the lfs parameter. (Bitbake rev: a7cf4fc72cce357c425084dc2c5f35b5ed1a4b7b) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-01bitbake: tests/fetch: Resolve fetch error in bitbake-selftestArmin Kuster
FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) AssertionError: '' is not true : Could not find upstream version for db [YOCTO #13496] The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget. Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the recipe. Also change the version being checked. (Bitbake rev: 4cf5bb761c561ddea86f2875be35d05abc8486e1) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28bitbake: tests/fetch.py: add missing skipIfNoNetwork tags to tests that try ↵Luca Boccassi
to git clone test_git_submodule_dbus_broker, test_git_submodule_CLI11, test_git_submodule_update_CLI11, test_git_submodule_aktualizr and test_git_submodule_iotedge try to access the network via git clone, which fails when there is no network available. Add the relevant skip tag. (Bitbake rev: 9b0538753da0514e6518723dac537007abf7a649) Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21bitbake: svn.py: Stop SVN from directly pulling from an external layer w/o ↵Mark Hatle
fetcher Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems. By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user. It is up to the user to determine if these are necessary for the recipe. They may disable the warning by adding "externals=nowarn" to the url. In the future we would like to parse this list and see if the items are already in the SRC_URI for that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble. Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded. (Bitbake rev: bf53f07c3647e57d8452a7743a2b04bcb72c80d6) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-16bitbake: gitsm: Fix a bug where the wrong path was used for the submodule initMark Hatle
Because we are trying to avoid network activity and use our own fetcher, the system emulates the behavior of 'git submodule init'. git submodule init uses the .gitmodules file, where typically the module name and path are the same. However, in this case the module name and path (in the tree) were different. i.e.: [submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"] path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared url = https://github.com/Azure/azure-c-shared-utility.git Previously the code assumed the 'path' was both the checkout location under .git/modules, as well as the path to extract the components. This proved to be incorrect as the .git/modules path needs to match the submodule 'name'. This causes the components that were fetched to be initialized in the wrong location, which later caused the 'git submodule update' process to skip not properly initialized modules. A test case was added for this specific case to ensure a regression does not appear in the future. (Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Strip old editor directives from file headersRichard Purdie
There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie
With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie
This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-01bitbake: gitsm: Add need_update method to determine when we are going to a ↵Mark Hatle
new SRCREV If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly. The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch. To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present. This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work. This issue was discovered by William A. Kennington III <wak@google.com>. The included test case was also written by him, and included unmodified. (Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12bitbake: bitbake: remove True option to getVar callsAndré Draszik
getVar() has been defaulting to expanding by default for a long time (2016), thus remove the True option from getVar() calls with a regex search and replace. Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28bitbake: gitsm: The fetcher did not process some recursive submodules properly.Mark Hatle
Move the submodule processing outside of the if statement to avoid any optimizations that may happen. Update the test cases to include the additional case, and split the other test cases into individual tests to make it easier to figure out what the failure may be. (Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24bitbake: gitsmy.py: Fix unpack of submodules of submodulesMark Hatle
If the submodule is in a subdirectory, it needs to have that structure preserved. This means the unpack path needs to be in the 'dirname' of the final path -- since the unpack directory name is specified in the URI. Additional specific test cases were added to ensure this is working properly based on two recent error reports. (Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24bitbake: gitsm.py: Fix relative URLsMark Hatle
Prior code happened to work with relative URLs, when the code was recently restructured it caused all relative urls to no longer work. Restore the prior code flow for relative support and better comment why that code is there. (Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: Refactor the functions and simplify the classMark Hatle
The update_submodules and unpack_submodules functions were nearly indentical, so we made a common function where the different behavior could be passed in by the download and unpack users. The new function is process_submodules. Moved the parse_gitmodules function under the new process_submodules, since there are no external callers. Refactor the file relative path processing to the URL translation code. We also add a warning to the translation if a relative ssh URL has been detected. Since this can cause a problem. In the case of a relative URL that does not work after being translated, it should be possible to use the MIRROR functions to manual translate the generated relative URL into one that works properly. Remove 'git config' processing on download contents. It turns out this is not necessary since all of the later components work using the git fetcher. Limit the 'git submodule update' call to only when unpacking a non-bare repository. Submodules are always loaded as bare, so this prevents intermediate unpacks from being attempted. Finally, the test cases were updated and the new commit ids in the test repository were updates as well. (Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: Rework the shallow fetcher and test caseMark Hatle
A custom shallow submodule is no longer necessary, as the regular git fetcher is used and shallow handling works with the same code. The only general difference between the regular change is simply declaring a clone as shallow, when appropriate. This also removes a potential race condition in copying repositories vs cloning them. The gitsm shallow fetcher test was revised to verify that the submodule is shallow cloned along with the primary repository. The first step of this change was to be sure to clean the gitsubmodule download directory, as was previously done with the may gitsource directory. Additional test components were added to verify commit counts, and an obsolete (and likely incorrect) test for the .git/modules directory to be empty was also removed. (Bitbake rev: f9cc4684dcf4281acc557cda8cb35602354ac3d6) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: tests/fetch.py: Add alternative gitsm test caseMark Hatle
In order to test the ssh processing in gitsm, we add an alternative testcase that can be downloaded from git.yoctoproject.org. However, this test case requries (read) access, via ssh, to git.yoctoproject.org. (Bitbake rev: c8554cdc1287620fe8e8960561e614567879a010) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22bitbake: tests/fetch: Update after recent gitsm message changesRichard Purdie
Also use assetIn instead of assertTrue which aids debugging failures. (Bitbake rev: 19dee675bb9ad012d28e1e57a888931355a831cb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18bitbake: fetch2/git: provide information about missing sourcesUrs Fässler
Provide more information in the case the sources are not found in the unpack step. (Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18bitbake: fetch2/git: prevent access to non-existing clonedirUrs Fässler
A user friendly error is throw when neither the clonedir nor fullshallow exist. Without the check, a difficult to interpret error is throw from within the fetch command. (Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18bitbake: fetch2/git: use intention revealing names for premirror testsUrs Fässler
(Bitbake rev: 3434e64e7c077c1ecff7b36f02b6c6b59a7d1fe9) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>