summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
AgeCommit message (Collapse)Author
2019-06-19bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie
This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ac556588fac55e91b7ce4839a975eb9ebb5aa192) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18bitbake: fetch2: runfetchcmd(): unset _PYTHON_SYSCONFIGDATA_NAMEMatthias Schiffer
Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME; unfortunately, this also affects Python scripts run as fetch commands like git-make-shallow, breaking it with a message like Failed to import the site module Traceback (most recent call last): File "/usr/lib/python3.6/site.py", line 570, in <module> main() File "/usr/lib/python3.6/site.py", line 556, in main known_paths = addusersitepackages(known_paths) File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages user_site = getusersitepackages() File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages user_base = getuserbase() # this will also set USER_BASE File "/usr/lib/python3.6/site.py", line 254, in getuserbase USER_BASE = get_config_var('userbase') File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var return get_config_vars().get(name) File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars _init_posix(_CONFIG_VARS) File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0) ModuleNotFoundError: No module named '_sysconfigdata' on an Ubuntu 18.04 system (and likely others) when building with BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS. Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this. (Bitbake rev: 6783104f18a8a14ddd88932720cd80dffeada63d) Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18bitbake: gitsm: Fix a bug where the wrong path was used for the submodule initMark Hatle
Because we are trying to avoid network activity and use our own fetcher, the system emulates the behavior of 'git submodule init'. git submodule init uses the .gitmodules file, where typically the module name and path are the same. However, in this case the module name and path (in the tree) were different. i.e.: [submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"] path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared url = https://github.com/Azure/azure-c-shared-utility.git Previously the code assumed the 'path' was both the checkout location under .git/modules, as well as the path to extract the components. This proved to be incorrect as the .git/modules path needs to match the submodule 'name'. This causes the components that were fetched to be initialized in the wrong location, which later caused the 'git submodule update' process to skip not properly initialized modules. A test case was added for this specific case to ensure a regression does not appear in the future. (Bitbake rev: 2a823584a5dbc2427060352f3b4798dbb2640cfd) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11bitbake: bitbake: cooker: Rename __depends in all multiconfigsJoshua Watt
The renaming of the __depends variable to __base_depends and file watches needs to occurs for all multiconfigs, not just the base config. Failing to do this for all multiconfigs will result in a huge increase in the size of the parsing cache (about 5x for a single mulitconfig) because all multiconfig caches will still depend on the base config files. This will also seen a similar jump in the amount of time required to load the parsing cache from memory, both because the cache is larger and because of explosion of additional existence checks that must be done for the base files. [YOCTO #13359] (Bitbake rev: 2ea1e6720c66841baa5e01c523c86f8ecb03eb7d) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11bitbake: cooker: Ensure mcdeps are processed even if only one multiconfigRichard Purdie
If you have no BBMULTICONFIG set but set mcdepends, they're currently ignored. We can handle them correctly with this small tweak. (Bitbake rev: 2aaea2f72e0c37451e7c7851b3a32abd733b38e1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10bitbake: toaster: Fix Thud Bitbake release metadataDavid Reyna
Fix the Bitbake version ID for Thud (1.40). [YOCTO #13356] (Bitbake rev: 7cda98d64693d978ad1387fed05a83aaa681e097) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-15bitbake: bitbake: fetch2/git: git-lfs checkNaveen Saini
Build will fail if repository has lfs contents in absense of git-lfs tool on host. Build will pass if repository may or may not contains lfs content if host has git-lfs installed. [YOCTO #13198] (Bitbake rev: 7107c1953b4c949b81491295e03736beb085b055) Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-15bitbake: toaster: update to WarriorDavid Reyna
Update the fixture files to add Warrior and remove Sumo. [YOCTO #13287] (Bitbake rev: 014287facfcf4598b6fd24487f44109bfb4cff57) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12bitbake: bitbake: Update version to 1.42.0Richard Purdie
(Bitbake rev: 00b133af009f9e7c1a4c751b6ef4902f47a81fe2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12bitbake: utils: Let mkdirhier fail if existing path is not a folderAndre Rosa
Let mkdirhier fail if existing path is not a folder instead of assuming a directory hierarchy already exists. (Bitbake rev: a8d9b82ccf93dcb74258693f62d88be380b1c0b7) Signed-off-by: Andre Rosa <andre.rosa@lge.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09bitbake: utils: Make prune_suffix prune a suffixAndre Rosa
Make prune_suffix prune a suffix instead of replacing a substring that could happen more than once and not only when it ends with it. (Bitbake rev: 57e765e38c6382a9b36d5ee2a6f3fa96ac905b82) Signed-off-by: Andre Rosa <andre.rosa@lge.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-01bitbake: gitsm: Add need_update method to determine when we are going to a ↵Mark Hatle
new SRCREV If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly. The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch. To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present. This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work. This issue was discovered by William A. Kennington III <wak@google.com>. The included test case was also written by him, and included unmodified. (Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26bitbake: fetch2: Use bb.utils.to_boolean() for BB_NO_NETWORKRobert Yang
Make it consistent with BB_FETCH_PREMIRRORONLY. (Bitbake rev: 4c6013f47ef36b03f590c909d7c9a2f50b698620) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLYRobert Yang
The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous, and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case, so unify it to the previous one. (As BB_NO_NETWORK does). And also use bb.utils.to_boolean() to make them consistent. (Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24bitbake: fetch2: Print SCMs list when SRCREV_FORMAT is not setRobert Yang
This makes it easier to debug, especially when multipe SCMs like gitsm, otherwise we don't know why there are multiple SCMs. (Bitbake rev: 313fe5e86b254eadfdead706be4bd7b274d5e3c0) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24bitbake: fetch2/git: Fix clean to remove clonedirRobert Yang
The localpath is a symlink to clonedir when it is cloned from a mirror, for example: $ bitbake systemtap-native -cfetch $ ls downloads/git2 sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git mirror.path.git.sourceware.org.git.systemtap.git There are both sourceware.org.git.systemtap.git and mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink sourceware.org.git.systemtap.git is created by try_mirror_url(), but do_cleanall" only removed the symlink, didn't remove the real dir mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions, for example, I assumed that do_cleanall removed everything, but it didn't, and it would the re-used next time when do_fetch. This patch fixes the problem. (Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24bitbake: cooker: Fix bbfile_config_priorities when BBFILE_PATTERN is emptyRobert Yang
The layer was not in bbfile_config_priorities when BBFILE_PATTERN is empty, this caused "bitbake-layers show-layers" can't show these layers, this was incorrect since these layer did exist. Add these layer to bbfile_config_priorities can fix the problem. Fixed: Add BBFILE_PATTERN_core = "" in oe-core/meta/conf/layer.conf $ bitbake show-layers | grep oe-core There was nothing, now the layer is shown (Bitbake rev: 0ff5cdb0cca9266ca29127639494bcfd95e36831) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24bitbake: fetch2: runfetchcmd(): Print workdir in debug messageRobert Yang
The workdir is an important message when run git command, for exmample, before the patch: Running git -c core.fsyncobjectfiles=0 branch --contains [snip] We don't know where it is running, now it is: Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git Which is easier to debug. (Bitbake rev: f25c05b73218d0e8edf2dd63b36bc09e547c5455) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19bitbake: fetch2: Fix undefined variable issuesRichard Purdie
Various refactors have left hanging variables, often in debug messages, hence why they haven't been spotted. Fix them (pylint spotted the issues). (Bitbake rev: e1f252408982a62485bbf874a01fd30a02aeed32) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19bitbake: fetch2/wget: Remove pointless lambda functionRichard Purdie
(Bitbake rev: 329986c3f2d98c4cd1a43b725194003575e718d5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19bitbake: fetch2/wget: Clean up whitespace/commentsRichard Purdie
Clean up the whitespace and comments style, keep pylint happy. (Bitbake rev: ee59fdaae68543ade03cacfdbbf14fdc7e469412) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19bitbake: fetch2/wget: Fix typo in proxy handlingRichard Purdie
Fix a typo in a variable which would affect proxy handling spotted by pylint. (Bitbake rev: 19f4ca0119e33df64da629253d5002fcdb385f4f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19bitbake: fetch2/wget: Cleanup module importsRichard Purdie
(Bitbake rev: ef062a3251af474fdeb480149856be3887b8c1c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19bitbake: fetch2/wget: Fix authentication in checkstatus() of the wget fetcherStefan Klug
I wonder how this used to work for anybody. (Bitbake rev: 9481e9cd688f635a7fe2ac60d5826ac26f0cb9ba) Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25bitbake: bitbake: fix version comparison when one of the versions ends in .Alexander Kanavin
Previously, this would happen: ====================================================================== ERROR: test_vercmpstring (bb.tests.utils.VerCmpString) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alexander/development/poky/bitbake/lib/bb/tests/utils.py", line 45, in test_vercmpstring result = bb.utils.vercmp_string('1.', '1.1') File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 143, in vercmp_string return vercmp(ta, tb) File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 135, in vercmp r = vercmp_part(va, vb) File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 124, in vercmp_part elif ca < cb: TypeError: '<' not supported between instances of 'NoneType' and 'int' ---------------------------------------------------------------------- (Bitbake rev: fef56d28c3efec4876c379898cbc4d4c65303aee) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25bitbake: cooker: Tweak multiconfig dependency resolutionRichard Purdie
There were a couple of problems with the multiconfig dependency resolution: - the "if mc" condition triggering this code wasn't correct, it needs to be "if more than one multiconfig" configured - after adding providers we need to call add_unresolved again and rebuild mcdeps within the "while new" loop By fixing these issues we allow various other combinations of multiconfig builds to work which previously didn't. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: 4359b037de578095db2595f119dfb8e3340e1414) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25bitbake: siggen: Fix multiconfig corner caseRichard Purdie
There was already a fix to ignore some multiconfig dependencies but its 'opposite' case wasn't covered. Cover that combination to so as to avoid tracebacks in multiconfig builds. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: c883dfe378af9dfc192a8e392e84325d68648806) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25bitbake: runqueue: Filter out multiconfig dependencies from BB_TASKDEPDATARichard Purdie
The consumers of BB_TASKDEPDATA in OE metadata can't cope with multiconfig dependencies. The choice is either to start adding code to each of them to filter out multiconfig dependencies, or do this at source. After consideration we've decided to do this at source as doing otherwise is code duplication and error prone and in any case we've looked at, they don't make sense. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: 531dcd221a10853f45cc057b52bb2d5083e0ee42) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17bitbake: runqueue: Fix dependency loop analysis 'hangs'Richard Purdie
Currently the mechanism for breaking out of the dependnecy loop analysis code is broken and doesn't work leading to bitbake appearing to hang. Add in a custom exception for this purpose and fix the code to exit as intended, fixing the hang and making the dependency loop code usable again. (Bitbake rev: 8756e4ade67c16e35269ea0659e10b9ebaa6117f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15bitbake: bitbake: Allow arguments in FAKEROOTCMDJoshua Watt
Changes FAKEROOTCMD so that it can accept additional arguments to pass to the fakeroot implementation instead of being treated as a simple command (Bitbake rev: 4fa51afb56b090cf1f746842acd602c9536715d5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-14bitbake: bitbake: cookerdata: Check duplicated BBFILE_COLLECTIONSRobert Yang
It shouldn't work when there are duplicated BBFILE_COLLECTIONS. (Bitbake rev: a1f251e5ab859d6d4a2cb908408d4ddcab5a5de1) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-14bitbake: build.py: add unhandled exception information to the logger when ↵Alexander Kanavin
executing python tasks Previously this information was simply discarded, which in some cases resulted in generic, unhelpful failures. With this change the user would see what the exception is, but without ugly tracebacks or bulky, irrelevant information from the log file. (Bitbake rev: 8da9242702fbef8cf3156b95a1076802e0f653c7) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12bitbake: bitbake: remove True option to getVar callsAndré Draszik
getVar() has been defaulting to expanding by default for a long time (2016), thus remove the True option from getVar() calls with a regex search and replace. Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06bitbake: server/process: Add missing exception raiseRichard Purdie
The intent of the code was to catch one kind of error, it was actually swallowing all exceptions and looping indefinitely. Fix it to work as intended. This explains some mystery hangs we've been seeing. (Bitbake rev: d73dbc3580faa1225d95ae4cefac4879ca3c1b2f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06bitbake: bitbake: ConfHandler: Don't strip leading spacesRobert Yang
Fixed: - Add the following lines to conf/local.conf: FOO = "BAR1" FOO_append = "\ BAR2" $ bitbake -e | grep '^FOO' FOO="BAR1BAR2" The leading spaces in the second line have been removed. - But if add the previous two lines to base.bbclass: $ bitbake -e | grep '^FOO' FOO="BAR1 BAR2" The leading spaces in the second line are preserved, this is inconsistent, now fix ConfHandler to preserve leading spaces. [YOCTO #12380] (Bitbake rev: 8c3bc15a7b5e0a81d7b6c9d3fe43fbff63207156) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06bitbake: toaster: correctly register the thud branchDavid Reyna
There are typos in the fixture files that need to be fixed to correctly check out the 'thud' branch. [YOCTO #13064] (Bitbake rev: 649a304b4d0a35553f665f0a748a7f21fcab51ba) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28bitbake: gitsm: The fetcher did not process some recursive submodules properly.Mark Hatle
Move the submodule processing outside of the if statement to avoid any optimizations that may happen. Update the test cases to include the additional case, and split the other test cases into individual tests to make it easier to figure out what the failure may be. (Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24bitbake: gitsmy.py: Fix unpack of submodules of submodulesMark Hatle
If the submodule is in a subdirectory, it needs to have that structure preserved. This means the unpack path needs to be in the 'dirname' of the final path -- since the unpack directory name is specified in the URI. Additional specific test cases were added to ensure this is working properly based on two recent error reports. (Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24bitbake: gitsm.py: Fix relative URLsMark Hatle
Prior code happened to work with relative URLs, when the code was recently restructured it caused all relative urls to no longer work. Restore the prior code flow for relative support and better comment why that code is there. (Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-21bitbake: bb.tests.codeparser: add parameter expansion modifiers testChristopher Larson
We don't want references including shell parameter expansion modifiers (i.e. `:-`, `#`, `%%`, etc) to be added to our vardeps, so add a test to ensure this. YOCTO #12987 (Bitbake rev: be022085fe1ea1b9a9d519f0455883e2da363d2c) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-21bitbake: bb.data_smart: only try to expand refs to valid variable namesChristopher Larson
This aligns the behavior of expansion with the recipe parser, only attempting to expand references to valid variable names. This avoids adding references for things like `${foo#${TOPDIR}}` to our vardeps without imposing much additional processing overhead beyond the change to the expansion regexp. YOCTO #12987 (Bitbake rev: df2ac65370aa86cdbc1574fdede25e3519410e45) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: Refactor the functions and simplify the classMark Hatle
The update_submodules and unpack_submodules functions were nearly indentical, so we made a common function where the different behavior could be passed in by the download and unpack users. The new function is process_submodules. Moved the parse_gitmodules function under the new process_submodules, since there are no external callers. Refactor the file relative path processing to the URL translation code. We also add a warning to the translation if a relative ssh URL has been detected. Since this can cause a problem. In the case of a relative URL that does not work after being translated, it should be possible to use the MIRROR functions to manual translate the generated relative URL into one that works properly. Remove 'git config' processing on download contents. It turns out this is not necessary since all of the later components work using the git fetcher. Limit the 'git submodule update' call to only when unpacking a non-bare repository. Submodules are always loaded as bare, so this prevents intermediate unpacks from being attempted. Finally, the test cases were updated and the new commit ids in the test repository were updates as well. (Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: Rework the shallow fetcher and test caseMark Hatle
A custom shallow submodule is no longer necessary, as the regular git fetcher is used and shallow handling works with the same code. The only general difference between the regular change is simply declaring a clone as shallow, when appropriate. This also removes a potential race condition in copying repositories vs cloning them. The gitsm shallow fetcher test was revised to verify that the submodule is shallow cloned along with the primary repository. The first step of this change was to be sure to clean the gitsubmodule download directory, as was previously done with the may gitsource directory. Additional test components were added to verify commit counts, and an obsolete (and likely incorrect) test for the .git/modules directory to be empty was also removed. (Bitbake rev: f9cc4684dcf4281acc557cda8cb35602354ac3d6) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: revise unpackMark Hatle
Greatly simply the unpack rule by copying the general functionality of update_submodules as unpack_submodules. This will recursively construct a set of urls and unpack them using the standard system behaviors. The overall code may be slightly bigger, but this ensures that all of the standard locks are inplace, ensuring the code doesn't change out from under the unpack function. (This could have happened before due to using 'cp' instead of further unpacks on submodules. This may still happen in shallow clones.) (Bitbake rev: 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: Optimize code and attempt to resolve locking issueMark Hatle
It was reported that a race condition on a shared download directory could occur with the gitsm fetcher, the result happened with a call to git config that occured within the update_submodules. Since the fetch is locked by the upper level, it was probably the prior need_update(...) function causing this because of some old code. The gitsm class inherits the git class. The need_update was overridding the version in gitsm, so that it forceably checked the submodules. It's clear we can optimize the code by only updating if the primary repository needs updating. Since we don't care if the submodule repository has changed because if the primary hasn't, references to the submodule won't change. (Bitbake rev: 346338667edca1f58ace769ad417548da2b8d981) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: tests/fetch.py: Add alternative gitsm test caseMark Hatle
In order to test the ssh processing in gitsm, we add an alternative testcase that can be downloaded from git.yoctoproject.org. However, this test case requries (read) access, via ssh, to git.yoctoproject.org. (Bitbake rev: c8554cdc1287620fe8e8960561e614567879a010) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: Add support for alternative URL formats from submodule filesMark Hatle
The following appear to be the git supported formats: proto://user:pass@host/path (URI format) user@host:path (SSH format) /path or ./path or ../path (local file format) We adjust the parsing to find out if we have a URI format or not. When we are NOT in URI format, we do our best to determine SSH or file format by looking for a ':' in the overall string. If we find a ':' we assume SSH format and adjust accordingly. Note, in SSH format we simply replace the ':' with a '/' when constructing the URL. However, if the original path was ":/...", we don't want '//' so we deal with this corner case as well. (Bitbake rev: dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: gitsm.py: Fix when a submodule is defined, but not initializedMark Hatle
It is possible for a submodule to be defined in the .gitmodules file, but never initialized in the repository itself. This shows itself when searching for the defined module hash you will get back a empty value. Similarly we need to identify and skip defined but not initialized submodules during the unpack stages as well. Thanks to raphael.lisicki@siemens.com for their help is figuring out how to resolve this issue. Additionally a problem was found where, while unlikely, it may be possible for the wrong revision to have been searched using ls-tree. This has been resolved in the update_submodules function by keeping the correct revision along with the submodule path. (Bitbake rev: 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16bitbake: bitbake: Fix Deprecated warnings from regexsRichard Purdie
Fix handling of escape characters in regexs and hence fix python Deprecation warnings which will be problematic in python 3.8. (Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14bitbake: bs4/element: Fix DeprecationWarningRichard Purdie
./lib/bs4/element.py:1565: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working (Bitbake rev: 52a144a7daa94b2bd239d582cb71d1f03119918f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-27 16:51:04 +0000