aboutsummaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
AgeCommit message (Collapse)Author
2012-07-11bitbake: fetch2/__init__.py: Warn user if SRC_URI is using "proto" and not ↵Andrei Gherzan
"protocol" As well, if "proto" is used, get the associated value as "protocol" (Bitbake rev: 53e6b630f0463d2d07cdaa9c9eb36794dc9b6b69) Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-11bitbake: fetch2/bzr.py: Use "protocol" parameter to get the protocolAndrei Gherzan
(Bitbake rev: 630876b40ed181312e84f902c4cfb97361afbe81) Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-11bitbake: fetch2/hg.py: Use "protocol" parameter to get the protocolAndrei Gherzan
(Bitbake rev: f4a780a7f50fdd8f2bd75888dad790bcfe95b873) Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-11bitbake: fetch2/osc.py: Use "protocol" parameter to get the protocolAndrei Gherzan
(Bitbake rev: 01040efce82cfbaa76f46d9f95c984be8f9ea810) Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-10bitbake: fetch2/wget: Use FETCHCMD_wget to provide the commandline and optionsRichard Purdie
This also changes to use -t 2 -T 30 as the defaults which are more sane for a modern fetcher and already specified in OpenEmbedded metadata. (Bitbake rev: bf0e5dddf0f63cdb0648fb6d872af5ceef6fbfb0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-10bitbake: fetch2/cvs.py: Switch to use FETCHCMD_cvs and handle parameters in ↵Richard Purdie
the fetcher itself This brings the cvs fetcher more into line with the others and allows consistent usage of the FETCHCMD variable and option handling. (Bitbake rev: 390ad59739356422852e976fa246375abc6aba08) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-04bitbake: fetch2/svn: Enhance to cope with subversion 1.7 upgradeRichard Purdie
svn changed working checkout formats between 1.6 and 1.7. Its convoluted to detect what format a given working copy is in so the simplest solution is simply to run "svn upgrade" within the working copy. The base svn command variable is relocated slightly to enable this new code to work effectively. (Bitbake rev: ebd3ecdb5f3c6d3fe1fad27cbed4d80f4277e92e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-04bitbake: git: do not execute 'git remote prune'Enrico Scholz
'git remote prune' at this location does not make much sense because the following 'git remote rm' will prune stale and non-stale branches. The 'prune' can cause trouble because it will access the network bypassing the no-network code in bitbake. When this operation fails and throws an exception, the next command (--> 'git remote rm') will be skipped. This in turn, will make all the following operations fail, because they assume that the remote does not exist yet. (Bitbake rev: 2ba23df5fad4b94d38a6aed97f7822226d72eb89) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-28bitbake: fetch2: Revert the regexp removal for the type field and instead ↵Richard Purdie
anchor regexp People are using regexps in the url type field so we need to preserve this bitbake behaviour. To address the issues with https:// urls mapping badly to file:// urls we anchor the regexp if its not already anchored. There should be no expressions in the wild which would break with this change. (Bitbake rev: ce0579dc256251e523c6330641f98b9f5a0e5761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-25bitbake: fetch2: Ensure star character doesn't end up in mirror tarball namesRichard Purdie
If '*' does end up in mirror urls accidently, some strange things can break since supports_checksum() looks for this, ud.localpath can then get ignored and this can lead to empty directories being downloaded "successfully". '*' is a special case for file urls only at this point so remove any entries that accidentlly make it in through url mapping. (Bitbake rev: 1369bec2404d942acc3618a8d005ec6868dcfd41) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-25bitbake: fetch2: Add new mirror syntax to simplify mirror specificationsRichard Purdie
When writing mirror specifications, the current regexp syntax can be awkward and hard to get it to do what you want. For example, extracting the 'basename' of a repository: PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file" can now become: PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file" which is much clearer. A MIRRORNAME substitution is also added which contains an encoded form of both host and path. One of the problems with the existing regexp syntax is you couldn't access HOST information from PATH and vice-versa which is an issue this patch also addresses. Tests for the new syntax are also added. (Bitbake rev: c6b1acbad7b3d2698530eb8b5249adb4ab95da21) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-25bitbake: fetch2: Handle errors orruring when building mirror urlsRichard Purdie
When we build the mirror urls, its possible an error will occur. If it does, it should just mean we don't attempt this mirror url. The current code actually aborts *all* the mirrors, not just the failed url. This patch catches and logs the exception allowing things to continue. (Bitbake rev: c35cbd1a1403865cf4f59ec88e1881669868103c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-25bitbake: git.py: Remove -l option, its not necessaryRichard Purdie
If -l is specified and the source and destination are not on a common filesystem an error occurs. The -l option is however the default for git for local paths which the fetcher already now ensures in the file:// case. We can therefore safely drop the -l option. (Bitbake rev: 3aeb268b2aaab4bb8b1cfff1450e0b76aa8ce855) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2/git.py: Optimize clone fall back when it is localJason Wessel
A file:// url should use "clone -l" to greatly speed up the clone in the case of a kernel when it is local. (Bitbake rev: 2bab2cc3ffe67ee2a308074a6e4c2c7be5636d2f) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Update replace_uri to handle uri types explicitlyRichard Purdie
For mirror mapping, we never use regexps for the type component of the url. Doing so causes various slightly bizarre behaviour such as https:// urls being translated to files:// urls which we have no handler for. This patch forces the type matches to be direct, not regexp based and gives the expected bahvriour. (From Poky rev: 695d8aca0999d2d61970b990e83c3132ba6f12cb) (Bitbake rev: 604df1b25cf114e083f52917df2df64e01279c25) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: uri_replace() only consider ud.localpath where its a fileRichard Purdie
Using ud.localpath as a basename when it points at a directory causes problems. The supports_checksum() method gives a good indication of whether ud.localpath can be used in the way we need. (From Poky rev: 933ec8a44634e33f92f6f76de3a34094c3d63aa6) (Bitbake rev: dcd79ae20ab2c72c3312b2251c2b6dc4cabe988e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: uri_replace() improve mirrortarball handlingRichard Purdie
We only consider mirror tarballs when the source and target urls are of differing types. We also should clear all url paramters when handling mirror tarballs. (From Poky rev: da140b8b0b3dda5429f9eee68829ef5247cdfe12) (Bitbake rev: 7619dcad29a6c2405b15a8fbadfa11e81b399ae1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: uri_replace() remove what amounts to a null operation and ↵Richard Purdie
add some comments (From Poky rev: 6d67200d052ba72258f5a0a178542ef99500a9cc) (Bitbake rev: e67af4d12f390ce6083965509ca9ea85a76dc351) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Add parameter handling to uri_replace()Richard Purdie
This means that parameters in the source expression are used as part of the match. Parameters in the destination are used explicitly in the final url. (From Poky rev: c465cb0c5c927dd41d96ad6d6fa1566349574bb7) (Bitbake rev: 3c468ac3ed6d045561afce19b85ae9dd18d87cea) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Remove basestring test and simplify uri_replaceRichard Purdie
(From Poky rev: d5657883d34bfef6beec594ac8d799f617b6b3ad) (Bitbake rev: 84ffc261f376429b3a6b5d7bf2f6217cd10ca12a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Simplify some looping constructs in uri_replace()Richard Purdie
(From Poky rev: c6bd25150a842a530f958d7233b15ae50d42c6c8) (Bitbake rev: 075296f3e5e5e3662290a888a6ba11229a36b95a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Improve mirror looping to consider more casesRichard Purdie
Currently we only consider one pass through the mirror list. This doesn't catch cases where for example you might want to setup a mirror of a mirror and allow multiple redirection. There is no reason we can't support this and the patch loops through the list recursively now. As a safeguard, it will stop if any duplicate urls are found, hence avoiding circular dependency looping. (From Poky rev: 0ec0a4412865e54495c07beea1ced8355da58073) (Bitbake rev: e585730e931e6abdb15ba8a3849c5fd22845b891) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Explicitly check for mirror tarballs in mirror handling codeRichard Purdie
With support for things like git:// -> git:// urls, we need to be more explicity about the mirrortarball check since we need to fall through to the following code in other cases. (From Poky rev: 28e858cd6f7509468ef3e527a86820b9e06044db) (Bitbake rev: a2459f5ca2f517964287f9a7c666a6856434e631) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Split try_mirrors into two partsRichard Purdie
There are no functionality changes in this change (From Poky rev: d222ebb7c75d74fde4fd04ea6feb27e10a862bae) (Bitbake rev: db62e109cc36380ff8b8918628c9dea14ac9afbc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Ensure when downloading we are consistently in the same ↵Richard Purdie
directory This assists with build reproducuility. It also avoids errors if cwd happens not to exist when we call into the fetcher. That situation would be unusual but I hit it with the unit tests. (From Poky rev: 86517af9e066c2da1d580fa66b7c7f0340f3403e) (Bitbake rev: b886c6c15a58643e06ca5ad7a3ff1f7766e4f48c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Only cache data if fn is set, its pointless caching it ↵Richard Purdie
against a None value (From Poky rev: c2df30bf6d1f8c263a38c45866936c1bf496ece5) (Bitbake rev: f4b59cc6e1c3ddc168a1678ce39ff402ea1ff4cc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2: Fix error handling in uri_replace()Richard Purdie
(From Poky rev: 1bfba28a583cb167f60e05ecdf34d0786dc1eec5) (Bitbake rev: aa7467a764ddcbc7d65af99e88cf093b6ec6d24e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21bitbake: fetch2/__init__: Make it clearer when uri_replace doesn't return a ↵Richard Purdie
match (From Poky rev: dc9976331c5cbb0983adb54f6deb97b9203bacbc) (Bitbake rev: eb96609864dec95a516e6e687dd6a2f31d523acf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-14fetch2/local: Ensure quoting is handled correctly from the urlRichard Purdie
The commit 420eb112a4f (fetch2: quote/unquote url paths) caused a regression since local file urls containing quoted characters are no longer being handled correctly. This fixes things so the url is dequoted correctly, fixing various fetcher failures with recipes like "gtk+". [YOCTO #2558] (Bitbake rev: 95c0595d66b3f8a5f0716662ba2a878600f312ea) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-14fetch2: Fix missing output from stderr in fetcher logsJason Wessel
There are actually two problems to fix 1) The exception for bb.process.ExecutionError must be processed first because it is a derived from the bb.process.CmdError class and we never reach the ExecutionError otherwise. 2) The stderr needs to be printed as well as stdout to determine the root cause of a fetch failure. The example I have is that I got a log that looked like: -- ERROR: Function failed: Network access disabled through BB_NO_NETWORK but access requested with command /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /localds 'http://downloads.yoctoproject.org/[...CLIPPED...] url None) -- That really didn't tell me much, but with this patch I get error above plus the following: -- STDERR: /net/[...CLIPPED...]kernel-tools.git: Read-only file system -- (Bitbake rev: af2133a04e1f4b22b181adf9c71f89c439bb88cf) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-30lib/bb/fetch2: ignore remote URIs when doing file checksumsPaul Eggleton
Skip evaluating remote URIs when doing local file checksums, because we don't need to process them and doing so will trigger a parse failure if SRCREV is not fully specified. Whilst this is just delaying a check until runtime (when do_fetch runs for the recipe) we're only validating this here accidentally and if we did wish to check it during parsing it ought to be done explicitly. Fixes [YOCTO #2512] (Bitbake rev: 99feb77c2de707f2d825566cf48371c48f450e3e) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-25lib/bb/fetch2: fix comments for verify_checksumPaul Eggleton
This function no longer returns a value, it raises errors instead, so update the comments to reflect this. (Bitbake rev: 8b61b9584a8b00378f9fd8f165dbd65dcfd3c14e) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-23replace os.popen with subprocess.PopenRobert Yang
Replace os.popen with subprocess.Popen since the older function would fail (more or less) silently if the executed program cannot be found There is a bb.process.run() which will invoke the Popen to run command, use it for simplify the code. For the: p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot)) ... for file in p4file: list = file.split() in bitbake/lib/bb/fetch2/perforce.py, it should be an error in the past, since it didn't use readline() to read the pipe, but directly used the split() for the pipe. Use the bb.process.run would fix the problem since bb.process.run will return strings. More info: http://docs.python.org/library/subprocess.html#subprocess-replacements [YOCTO #2075] (Bitbake rev: 8d6700255a6d4dda403c89b171a6d4a1883e5aae) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-23replace os.system with subprocess.callRobert Yang
Replace os.system with subprocess.call since the older function would fail (more or less) silently if the executed program cannot be found More info: http://docs.python.org/library/subprocess.html#subprocess-replacements [YOCTO #2075] (Bitbake rev: f5b3bf115dc1ffbfb241a49cec0fc3654cb71021) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-23bitbake: implement checksums for local files in SRC_URIPaul Eggleton
Gathers a list of paths to have checksums calculated at parse time, and processes these when calculating task hashes. Checksums are cached with the file's current mtime. Thus, changing any local file in SRC_URI will now cause the do_fetch taskhash to change, thus forcing a rebuild. This change adds very roughly about an 8% increase in parse time (a few seconds) and maybe a few seconds during runqueue generation, so a fairly moderate performance hit. Note that since paths are resolved at parse time, this will not force a rebuild when files are introduced which would cause that resolved path to be different - for example, where a machine-specific version of a file was added without otherwise changing the recipe. This will need to be handled in a future update. Code to hook this into the signature generator was courtesy of Richard Purdie <richard.purdie@linuxfoundation.org>. Implements [YOCTO #2044]. (Bitbake rev: c993b7c457f8b7776e8a5dff253bfa0724bc2cae) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-22fetch2: quote/unquote url pathsChristopher Larson
This ensures we can handle things like %-encoded characters in the path portion of urls. (Bitbake rev: b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-22fetch2: Also exclude urls containing wildcards from checksumming for nowRichard Purdie
Without this change, bitbake will try and checksum a wildcard which will lead to fetch failures. (Bitbake rev: ac53b88be58b0bed21730c0b61a8fc8e801a2f1b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-21bb/fetch2/__init__.py: Don't try to compute checksums for directoriesAndrei Gherzan
In this way we avoid failing the build while trying to fetch local directories. [YOCTO #2475] (Bitbake rev: 39adb5741d9eee0879d3181be505400dffc60804) Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-20bitbake/fetch: Spell out which fetcher backends support and recommend checksumsRichard Purdie
There were some hardcoded behaviours in the system for which backends support checksums verses which backends recommend them verses which don't recommend them. This moves the functionality into specific fetchers and then makes the general code generic. This cleans up the codebase and fixes some corner cases such as trying to checksum directories returned by the git fetcher. (Bitbake rev: ef6d268f7b8527541a7fb044cf95a973be4097f4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-20bitbake/fetch: If checksums are available, check them for any file, local or ↵Richard Purdie
remote Currently, checksums are only checked for remote files. This changes the check to apply to any file we have checksum data for. A mismatch against a file is fatal but in the local case, no warnings are shown about missing checksums. (Bitbake rev: 80411b3acf2173a4a7d415102d16676eb98363e3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-20bitbake/fetch2: Improve visibility of checksum warnings when fetching from ↵Richard Purdie
mirrors When fetching from mirrors, checksum errors would get buried in the logs. This raises their profile so a warning is logged on the console when fetcher checksum issues are encountered, even if other attempts are made to get the file (which may or may not have the same issue). (Bitbake rev: d43fafd7f01b5534499b45213197d8ccececdbc4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-20bitbake/fetch2: Improve stamp creation in mirror codeRichard Purdie
Under certain contitions such as local file:// mirrors of all files, bitbake might not check checksums. This is not desirable and we should always check the checksum where possible. This only exception to this is the git mirror tarball case where the checksum is unknown but we need to create the .done stamp for other reasons. This patch preserves thta functionality but in a more specific section of code where this doesn't interfere in normal checksums. (Bitbake rev: 2592a43e422c06a6f6174ada562e9c8ba25e1ec1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-20bitbake/fetch2: Remove hardcoded file removal and use clean methodRichard Purdie
Under certain circumstances, bitbake could remove files which were outside its control since it unconditionally removes ud.localpath. In the file:// case this can point at external paths. This patch converts it to use the clean() method which will remove files when its safe to do so but not in the file:// case since the file:// handler has an empty clean method. This means bitbake no longer removes files outside its control and is generally much safer. (Bitbake rev: 9ad1cebe2220b41da2141cdb3f0a403abb46d77c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-13bitbake/fetch2/git: Set a default for the GITDIR variableRichard Purdie
(Bitbake rev: 1a0cdc65812f1f12bf4bbea6540a3aaf0f81b4f7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-13bitbake/fetch2/git: Don't set the branch name to None, that makes no senseRichard Purdie
With floating revisions and no specified branch, the fetcher could fail with some obtuse errors. This was due to the branch name being set to None which makes no sense. This patch reworks some conditions to avoid this. (Bitbake rev: 740c58d43cfb1445dd126e4827bb70ce988ca107) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-07bitbake/fetch2/wget: Add fallback/default wget commandsRichard Purdie
(Bitbake rev: 9586808572d06de4127f6a524e87e9ad75232423) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-07bitbake/fetch2/wget: Remove unneeded function indirection and reindentRichard Purdie
(Bitbake rev: 769b92b14a1dfbbf697b8f4bf9a5a828807fd885) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-07bitbake/fetch2/wget: Remove unnecessary use of OVERRIDESRichard Purdie
This allows some simplifications of the function and is slightly faster. (Bitbake rev: 5999dc9985ad087c036611bfaa59b090a08781a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-07bitbake/fetch2: When unpacking, only use PATH variable if its setRichard Purdie
(Bitbake rev: ffec38675c0d78ee9fcd1d8f5a746d162145554e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-07bitbake/fetch2: Remove WORKDIR reference, should use passed parameterRichard Purdie
(Bitbake rev: 9384dfaf644365c731be26572443287ce8f190b2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-03 12:14:12 +0000