| Age | Commit message (Collapse) | Author |
|---|
|
Using backslashes in file:// URIs was broken.
Either the resolver would fail or the subsequent `cp` command.
Try to avoid this by putting the filenames into quotes.
Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=8161
(Bitbake rev: aa857fa2e9cf3b0e43a9049b04ec4b0b3c779b11)
Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Microsoft's TFS VCS system allows for spaces in a git repository url.
An example of a valid url is:
ssh://tfs-my-company.org:22/tfs/My Projects/FooBar
This commit adds support for such urls by implementing two changes.
Firstly, when bitbake makes a git command line call the url is
surrounded by quotes so that the url, regardless of spaces, is
treated as one argument. Secondly, additional parsing of various
filepath variables, which are based off of the url, are now
completed with any spaces in the url replaced with underscores.
(Bitbake rev: eb38b6f0935763f7ba19e5618f376fcae1dac41a)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Globbing in file:// urls is terminally broken. Currently when its used, the
file checksum code is basically bypassed. This means changes to the source
files don't change the task checksum, the task doesn't rebuild when the
inputs change and things generally break.
To make globbing work generically, we'd have to scan the file system for
all possible matches to the glob and log whether they exist or not. We can't
simply log the files which exist, we have to also know which files could
later exist and influence the choice of file so we know when to reparse.
For a simple file://xxx/*, this could be done but for bigger patterns,
it becomes much more problemtic. We already support file://xxx/ in urls.
So, lets decide we'll not support globs in file://urls. Worse case users
can put files in a directory and reference that, moving files into place
if needed.
Remove all the glob special cases (see the comments if anyone doesn't
believe this is terminally broken) and error to the user if they have
such urls.
(Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.
(Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5)
Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The buffering parameter was removed in python 3.1 and made default
so we can clean up the code. This removes weird looking double
exceptions when connections fail.
(Bitbake rev: 06b7bafbd18a47c8db2f7b943dc535c65df176bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.
We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.
I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.
(Bitbake rev: 820ab886e79eea516560c0c008e4cf059c6e11a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch implements three new SRC_URI options for the perforce
fetcher, namely:
* module
* remotepath
The options are intended to provide the user more control over the
downloaded file paths by allowing the user to specify how much of the remote
path should be preserved locally.
The changes in this patch are backwards compatible, i.e. if none of the
introduced options is specified, the default (old) behavior is enforced.
(Bitbake rev: aab228822d2f221c01337dd57d7582c51ce9a505)
Signed-off-by: Alexandru N. Onea <onea.alex@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch adds a basic implementation of a progress handler for the
perforce fetcher, based on the number of files to be downloaded and the
output behavior of the p4 print command used in the fetcher
implementation.
(Bitbake rev: f0582292bf79b0988048683dfd086aa3b9787344)
Signed-off-by: Alexandru N. Onea <onea.alex@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the bitbake.srcrev nugget is not present for the commit we're
interested in we should not just bail out and say that an update is
needed. Instead we can recursively walk through the submodules and check
for the presence of the required commits.
(Bitbake rev: cfc78316309556bec487ef0a5a9205e41f1be86f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This prevents multiple bitbake.srcrev entries being created when a
repository has more than one submodule. It also ensures that the
bitbake.srcrev entry is not added unless all submodules are correctly
fetched.
(Bitbake rev: 860ff1193fe53f04696d41635a720c2d1f29fa7f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some fetchers may download additional sources along with those
explicitly listed in SRC_URI. These "implicit URLs" will be needed by
the archiver to ensure that all sources can be archived.
We can't just return a list of URL strings since each URL may need its
own SRCREV data so we return a list of FetchData objects.
Each fetcher can override the implicit_urldata() function to provide the
additional FetchData objects. For now this is just needed in the gitsm
fetcher to walk git submodules recursively.
(Bitbake rev: 1350f241b7d991bd191ce9e44f6662e4376c6e24)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
objects.
Function "contains_lfs" was only looking at the master branch when searching for LFS
content. LFS may be configured in specific branches only, so we need to use the
correct branch.
(Bitbake rev: 4fa67c2af830035a1ddedc14592ee25a15ebff22)
Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The message "Repository %s has LFS content but it is not being fetched" was
being printed, even when Git-LFS was available and "lfs=1" was set. In those
situations, we want to fetch LFS content, so that message would not make sense.
(Bitbake rev: 45028dfda5a29a34ab408cb3f11d72ae17963340)
Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Git-LFS objects were being fetched even when lfs=0 was not set.
This patch disables LFS smudging when lfs=0. That way, only the LFS pointers
are downloaded during checkout.
(Bitbake rev: 646d86df7de774255246a3d7051c308e43eb257d)
Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Clean method assumes that download directory should be determined from
CVSDIR variable, but this is handled differently in download method.
Now we set download directory for the whole class in urldata_init
(Bitbake rev: 6d7ad9b06b82612f0dd17ffccea7ab98f9077198)
Signed-off-by: Vyacheslav Yurkov <Vyacheslav.Yurkov@bruker.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When a website is behind a CDN like Cloudflare there may be a "Browser
Integrity Check" or other test applied to requests before they are
allowed through to the server. Downloading via wget passes these tests
as headers are set appropriately, however the Python urllib module may
fail these tests unless additional headers are set. This causes
Wget.checkstatus() to fail where Wget.download() would actually succeed.
For Cloudflare in particular a valid User-Agent is needed, it's easy to
add this to the headers in Wget.checkstatus(). The user agent string is
copied from Wget._fetch_index().
(Bitbake rev: 4679d3cdb9cdf23f3962aa61c599ad7474591f9f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Upon inspection its clear the way the keys for this cache were being handled
would break it and cause the cache to never be used. Fix this.
(Bitbake rev: 9a5dd1be63395c76d3fac2c3c7ba6557fe47b442)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
is "cache""
As per mailing list discussion, the cache policy was behaving correctly before
and wouldn't expect to update after the initial fetch even for AUTOREV.
This reverts commit ba093a38539960e645e994a66ed7872a604c00a9.
(Bitbake rev: 51f827911b7202de3e855e683fdbd732d7a84e09)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked temporarily so that the .gitmodules file can be examined.
(Bitbake rev: 3987db953e414255ce278bc25a5f6cec0f2a30c7)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently bitbake will list many checksums for a recipe when none are
present, encouraging users to add them all to a recipe. We don't
need/want them all.
We used to show md5 and sha256 but given the concerns about md5,
switch to showing just sha256 going forward which seems like the
sensible one to standardise upon.
There will be no change to existing recipe functionality.
(Bitbake rev: 47f0c849ed13ba554d9523b926d92405e8251702)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The update codepath would trigger:
Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment
Fix this so the code functions as intended in both fetch and update cases.
[YOCTO #13798]
(Bitbake rev: 16c4e930ff37ea6eac2ac0cb2197908ce3a1cc53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Documentation says:
"path_spec": A specific directory in which to checkout the specified
svn module.
but existing svn fetcher uses "module" always as path of checked out
svn-module, regardless whether path_spec was given or not.
(Bitbake rev: 75223644ab9bc94fc268f1bab775e66c4188f279)
Signed-off-by: Jens Rehsack <sno@NetBSD.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Mark any keys used to cache the srcrevs for a recipe as "dontcache" if
BB_DONT_CACHE is set for the recipe. Remove any such keys upon the
next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will
make sure the srcrev is updated as expected if ${AUTOREV} is used.
(Bitbake rev: ba093a38539960e645e994a66ed7872a604c00a9)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This seems to have been broken for a very long time. Now it also works
regardless of BB_SRCREV_POLICY.
(Bitbake rev: ffd663a8e07e2e39e8ca2d2493f4f98037c5f9e4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit adds a new npmsw fetcher that fetches every npm dependencies
described in a npm shrinkwrap file:
https://docs.npmjs.com/files/shrinkwrap.json.html
The main package must be fetched separately:
SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \
npmsw://${THISDIR}/npm-shrinkwrap.json"
Since a separation has been created between the package and its
dependencies, the package can also be fetched with a non npm fetcher
without impacting the general behavior:
SRC_URI = "git://github.com/foo/bar.git;protocol=https \
npmsw://${THISDIR}/npm-shrinkwrap.json"
(Bitbake rev: f5223be54450bf20e0bfbd53b372a7748a44b475)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit refactors the npm fetcher to improve some points and fix
others:
- The big change is that the fetcher is only fetching the package
source and no more the dependencies. Thus the npm fetcher act as the
other fetchers e.g git, wget. The dependencies will be handled later.
- The fetcher only resolves the url of the package using 'npm view' and
then forwards it to a proxy fetcher.
- This commit also fixes a lot of issues with the package names (exotic
characters, scoped packages) which were badly handled.
- The validation files - lockdown.json and npm-shrinkwrap.json - are no
longer used by the fetcher. Instead, the downloaded tarball is
verified with the 'integrity' and 'shasum' provided in the 'npm view'
of the package [1][2].
1: https://docs.npmjs.com/files/package-lock.json#integrity
2: https://www.w3.org/TR/SRI
(Bitbake rev: 0f451cdc43130d503ada53ed1b4fc5a24943f6ef)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When using a download filename with characters which can be interpreted
by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the
filename fixes the issue.
(Bitbake rev: ed652dce5200161068eccdbfaaaefde33136eb09)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the done condition to a
proxy fetcher.
(Bitbake rev: ee3a2545e99e6e99559a72bcda64797ae674ec71)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "try_mirrors" functions to
a proxy fetcher.
(Bitbake rev: 462c9a2b368a1720da276310b1d5d0423b7cefea)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "verify_donestamp" and
"update_stamp" functions to a proxy fetcher.
(Bitbake rev: f7612c0704b4252bba5157ce9a94d8888c6d0760)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit enables the "sha1", "sha384" and "sha512" hash functions in
the supported checksum list. This allows to use more SRC_URI checksums
functions for a url:
SRC_URI[sha1sum] = "..."
SRC_URI[sha384sum] = "..."
SRC_URI[sha512sum] = "..."
The npm fetcher needs this to support subresource integrity:
https://www.w3.org/TR/SRI/
(Bitbake rev: cd80a646aa841b71e68282bb8d11194abb5df0e4)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit refactors the way checksums are verified to be more generic.
The support of new hash functions is now limited to the update of the
CHECKSUM_LIST variable.
(Bitbake rev: debd9eeaf5638755d8956b2d65b904fe02826966)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
>From a SRCREV caching point of view, there is no reason to treat the
following upstreams as different:
SRC_URI = "git://github.com/file/file.git"
SRC_URI = "git://github.com//file/file.git"
(Bitbake rev: 425e21c14955dd38868c6e97637df3bbe0f89fac)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Prior to this change, two different recipes pulling from the same
exact repo could get a different SRCREV during a single parse session.
This was originally observed using git. For git at least, it still
allows recipes to pull from the same repo, but with different branches
or tags, since the form of the srcrev cache key for git is:
"git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name]
Where the 'unresolvedrev' part is the branch or tag name.
(Bitbake rev: 6c938e6fd29beebe09b32be839dae008fe6491d2)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
directories
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.
Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"
(Bitbake rev: 923aff060d8aba8456979c35b16d300ba7c13ff9)
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to ensure the correct working directory is used for this
tar command to succeed.
(Bitbake rev: e1a2a0e29a75dead3426ae083387181ad52da91f)
Signed-off-by: Christian Herzig <cherzig@gauselmann.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake/lib/bb/fetch2/clearcase.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="?
if command is 'mkview':
bitbake/lib/bb/fetch2/clearcase.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif command is 'rmview':
bitbake/lib/bb/fetch2/clearcase.py:159: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif command is 'setcs':
Python 3.8 is quite correct and we so mean "==" here, fix it to
avoid the warnings.
(Bitbake rev: 1fab03f6e10eaa13b8a89ce0b2f9fe8ce5157189)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix mercurial fetching after breakage from changes to the core fetcher.
Fix username and password usage and setting moddir needed by setup_revisions.
(Bitbake rev: 8962b27abc68427eae085624f26300108ae88aa1)
Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There was a weird error in OE-Core where "devtool modify virtual/kernel"
was showing basehash mismatch errors. This was due to SRCPV sometimes being:
AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28.
The latter hash comes from KBRANCH and meant sometimes the correct branch
was seen, sometimes it was not. The issue was complicated by the execution
using a remote datastore over tinfoil.
The problem turns out to be a fetcher caching error. If the datastore
changes, the cached url data may not be valid.
We therefore ensure we match cached url data against the datastore that
generated it, which appears to fix this issue.
(Bitbake rev: 0540abd338ed2ffd822edbd2947cab2f18873422)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When bitbake determines if a git clone needs updating, it only checks for the
needed srcrevs, not the revs listed in BB_GIT_SHALLOW_REVS, which will fail if
using shallow and the needed rev was added to the upstream git repo after a
previous fetch. Ensure that we also check for shallow revs.
[YOCTO #13586]
(Bitbake rev: 93b65e3c6ca64f644946953980595c44fbbcc748)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Refactor the git-lfs checking: this means both clearer code in the download()
function and allows unit testing to monkeypatch the functionality.
(Bitbake rev: 33cf9172ded50a869f7201ba463ab9ecc69b8252)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a new 'lfs' option to the git fetcher so that we can optionally not fetch
git-lfs content, for repositories that contain LFS data that we don't
actually need for building.
By default lfs is set to 1, so if the repository has LFS content then git-lfs is
required. Setting lfs to 0 will mean that git-lfs won't be required to fetch,
and some files will be missing.
(Bitbake rev: be0b78ccfc5ede98041bc0545a15092494b12b26)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Not all servers and repositories have this property set
which results in failures like this when actual svn checkout
command succeeded:
svn: warning: W200017: Property 'svn:externals' not found on ''
svn: E200000: A problem occurred; see other errors for details
(Bitbake rev: 238636f033cbf18e5741f0ea0e64db40e84f5838)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* noticed on read-only sshfs premirror
* it was showing the warning about renaming the file:
WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441
and then failed because of movefile() issue with python3 (fixed in previous commit):
ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:
with movefile() fixed, it let do_fetch continue and re-fetch locally with the right
checksum, but still the renamed file didn't exist, because of movefile failure - add
another warning when the movefile fails - for whatever reason - unfortunately movefile
prints error messages with just print() so the real error is hidden only in log.do_fetch
in this case:
movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441'
(Bitbake rev: 9a1bf4ba9ec00c2a222d820f8f83d1f056b021d6)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
handling 403 codes
The code says that some servers respond with 403 codes when they really
mean 405 codes. But we still need to account for legitimate 403 codes.
Before this change, I noticed that sstate mirror checking was taking a
very long time when I purposely entered incorrect credentials into my
.netrc file for our sstate mirror. Instrumenting the code, I discovered
tracebacks like the following for every mirror access attempt:
File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus
fetcher.checkstatus()
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus
ret = try_mirrors(self, self.d, ud, mirrors, True)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors
ret = try_mirror_url(fetch, origud, uds[index], ld, check)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url
found = ud.method.checkstatus(fetch, ud, ld)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus
opener.open(r)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 504, in error
result = self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
unverifiable=True))
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 504, in error
result = self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
unverifiable=True))
... (repeats until recursion depth is reached)
Solution is to make sure we only attempt the GET request once when handling 403/405 error codes.
(Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using npm pack to download the main node module and its dependencies
allow for the use of private npm modules and access to them via .npmrc
(Bitbake rev: e5eda3871893e4eadeb311aeb997e183675598f4)
Signed-off-by: Mads Andreasen <mads@andreasen.cc>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 9a5152fa4613a1164cbf2a0248460e75207b2624)
Signed-off-by: Christian Herzig <cherzig@gauselmann.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME;
unfortunately, this also affects Python scripts run as fetch commands like
git-make-shallow, breaking it with a message like
Failed to import the site module
Traceback (most recent call last):
File "/usr/lib/python3.6/site.py", line 570, in <module>
main()
File "/usr/lib/python3.6/site.py", line 556, in main
known_paths = addusersitepackages(known_paths)
File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages
user_site = getusersitepackages()
File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/usr/lib/python3.6/site.py", line 254, in getuserbase
USER_BASE = get_config_var('userbase')
File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var
return get_config_vars().get(name)
File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars
_init_posix(_CONFIG_VARS)
File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix
_temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'
on an Ubuntu 18.04 system (and likely others) when building with
BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS.
Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this.
(Bitbake rev: d94ccd506d04aff182ab48f501f6f366d5dd14f5)
Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Leif Middelschulte
Charlie Davies
Richard Purdie
Frazer Clews
Alexandru N. Onea
Paul Barker
Mauro Queirós
Vyacheslav Yurkov
Khem Raj
Jens Rehsack
Peter Kjellerstedt
Jean-Marie LEMETAYER
Chris Laplante
Frazer Clews
Aníbal Limón
CHerzig@Gauselmann.de
Volker Vogelhuber
Christopher Larson
Ross Burton
Mikko Rapeli
Martin Jansa
Chris Laplante via bitbake-devel
Mads Andreasen
Matthias Schiffer