aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2019-11-21recipes: add help button to explain search termspaule/fixes14Paul Eggleton
There's a bit of advanced functionality by now in recipe searching, so add a link that shows a popup with information on how it works. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21recipes: allow searching for layer:oe-corePaul Eggleton
When using the layer: advanced query term, if you want to match on OE-Core, its actual layer name is "openembedded-core", but people will naturally assume that "oe-core" should work, so make it so (case insensitive). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21recipes: support pn: query prefixPaul Eggleton
Allow filtering only on recipe name - i.e., searching for "git" finds any recipe with "git" in the name or description. Now, you can search for "pn:git" which will return only recipes with the name "git". Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21recipes: improved support for queries containing quotesPaul Eggleton
Use shlex.split() to split the query string so that quotes can be used to delimit strings containing spaces to be matched as a whole. This worked with the previous code, but it did not support single quotes - these caused an error in Django's filter code and thus an internal server error (as did querying for ""). Add some additional checks for single quotes as it is still possible to get them past shlex.split() e.g. with something like "'hello'" (with quotes). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21update: fix exception with -x/--nofetch optionPaul Eggleton
Fixes the bitbakepath variable not being defined with -x/--nofetch specified. (Regression introduced in c91372587bbddd4c595d7202e51a8740b787a06e.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21Drop LICENSE.diff2htmlPaul Eggleton
This was added when we brought over a patch from the Clear Linux Dissector, but here we're not using diff2html here so we shouldn't have this either. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21TODO: add some more taskspaule/recipesymbolPaul Eggleton
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: do not ignore non-numeric characters in versionsPaul Eggleton
The regex we were using here explicitly only matched numeric characters in version numbers - presumably the assumption was that any non-numeric characters were not significant. However, for upstream projects such as OpenSSL and BIND for example, alphabetic characters are an explicit part of the version number, so if we ignore them then we miss detecting most of the upgrades. Fix the regex so that that doesn't happen. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21Add recipe dependencies toolPaul Eggleton
Add an extra tool that lets you view all of the recipe dependencies in a layer. There is also a mode that shows only cross-layer dependencies, which can be useful to find dependencies on recipes in other layers that aren't declared in the layer's dependencies (or conversely where a layer dependency is no longer necessary). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: Handle two versions added on same day then later one deletedPaul Eggleton
We have at least one instance where two versions of a recipe were added at the same time and then later one was deleted - sed. We didn't detect more than one recipe being added and thus the delete was seen as removing the recipe entirely, causing the recipe to vanish. Fix the filter so that we see the other addition and adjust the debug printing so that we can see what type of deletions are occurring. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: enable grouping recipe upgrades by licensePaul Eggleton
Going back in OE-Core recipe upgrade history, we kept GPLv2 and GPLv3 versions of a number of recipes around, so this is the source of quite a few situations where we had multiple versions of recipes with the same recipe name around. Add means of grouping upgrades by license so that we can keep these versions separate in the upgrade history instead of detecting lots of apparent upgrades and downgrades if they are intermingled. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: Add tool to dump upgradesPaul Eggleton
By dumping the recipe upgrade data using the mostly same code as the web application but in plain text format, this tool gives us an easy way to compare recipe upgrade history data from multiple runs. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: detect changes in SRCREV as upgradesPaul Eggleton
Aligning with recent changes in the layer index proper, handle where PV is not changing but SRCREV is - typically this happens when PV does not contain ${SRCPV} - ncurses in OE-Core is one example. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: ensure default URLs for release/milestone are the latestPaul Eggleton
In the "Maintenance Plan" drop-down the maintenance plans point to the "default" release and milestone, but it was picking the most recently added record in the database rather than the latest one by date. Use an order_by() to ensure we get the most recent release/milestone by date rather than just the most recently added in case they have been added out-of-order. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: exclude lib/ subdirectory of layers to avoid picking up templatesPaul Eggleton
We were picking up lib/bblayers/example.bb in OE-Core, and it's possible we might add similar templates in future. There shouldn't ever be files we're interested in under lib/, and in the absence of the ability to follow BBFILES, just exclude the directory explicitly. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: handle when recipes get deleted and later re-addedPaul Eggleton
We need to capture the re-addition properly or the recipe simply won't show up in the recipe list. Examples from meta-oe: * psqlodbc removed in ec9e5ed06256ad92c818474cdb490dc0d3a0d0a3 and added back in 16a6fee6c0455863ed5df15afc49efe8cc617d9c * libgxim removed in 5dd01c5175f518658d8ee5627ede4f593111b872 and added back in af602920594a9cc2e9b397fe311fda7f531be7f3 Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21recipeparse: handle recipes at root of repositoryPaul Eggleton
You'd think this is very unlikely to happen, but back in meta-openembedded commit 415e213ad75ec9a93171c963395a1c4b92c6233b and the commits preceding it, a recipe was added to the root of the repository and then moved into place, and os.path.relpath() does not like to be called with a blank path and thus raises an exception. To avoid the exception, get the relative path to the filename and then chop that off instead of the other way around. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: avoid historical parsing bug in bitbakePaul Eggleton
In bitbake commit 5796ed550d127853808f38257f8dcc8c1cf59342, line numbering functionality was improved with the starting line number for python functions being stored in a "lineno" varflag; however, mapped functions (using EXPORT_FUNCTIONS) did not have a line number set, which caused parse failures. This bug was not fixed until 547128731e62b36d2271c4390b3fee2b16c535dc so we should be avoiding any bitbake commit inside that range. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: ensure upgrades recorded at exact same time are correctly orderedPaul Eggleton
In meta-oe there are two commits (d91f92cf04 and 57492d40b5) which have the same commit date and thus don't deterministically order; the result was that the mercurial-native recipe might or might not show up. Add id to the order_by to make it deterministic. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: fix some more bad OE-Core commitsPaul Eggleton
In OE-Core commit 309a02931779f32d1139cc1169a039cbe4638706, a reference to BBINCLUDED was added to HOSTTOOLS in conf/bitbake.conf, however when we use tinfoil to parse this BBINCLUDED is not set (probably too early) and the result is an immediate parsing failure. The issue was eventually fixed in 40a904bf8bc1279c3da0893c003f740f1d2066c2 however there are some commits in this range that we care about, so within this range we hack bitbake.conf to have a default for BBINCLUDED since it's an easy workaround. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: fixup handling of upgrades where recipe moved to incPaul Eggleton
Sometimes in the past it has been desirable to create a shared .inc file from a recipe, e.g. d5a95dc8985a42bb7e50bc4e7dc6b012d711ff08 in OE-Core for tzdata. Git detects this type of change as a rename of the .bb to a .inc with some changes, and an addition of a new .bb with new content; however we want to treat it as a change to the .bb file and ignore the .inc, otherwise it can look like the recipe was renamed and the history becomes broken (it wasn't, the recipe name stayed the same). Detect this situation and handle it properly. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21rrs_upgrade_history: add stop commit optionPaul Eggleton
Add an option to stop at a particular commit (so we can then repeat a specific commit afterwards easily for debugging purposes). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: record previous versionPaul Eggleton
Record the previous version in RecipeUpgrades, and use it to more accurately record upgrades where there are multiple versions present at a given time (common with e.g. kernel recipes). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21rrs_upgrade_history: implement file path filteringPaul Eggleton
Make it possible to re-collect all the history for a given path. (Typically this would only be used for debugging, as it saves time if you are trying to correct an issue with upgrade data collection for a single recipe.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: detect PN changing without movePaul Eggleton
If PN changes we need to mark the old recipe as deleted (since it is effectively gone.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: use RecipeUpgradeGroup to determine downgradesPaul Eggleton
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: support grouping upgrades by version for multi-version recipesPaul Eggleton
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: handle recipe moves without overwriting dataPaul Eggleton
If we ever want to analyse the upgrade chain later on then we need to avoid overwriting the paths when we identify a moved recipe - instead, store a "move" upgrade record (not shown in the UI) that we can later pick up when we are going through and deleting. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21rrs_upgrade_history: record start marker in log filePaul Eggleton
Make it easier to see where each session starts when looking back through the log. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: use more robust method of getting last upgrade recordPaul Eggleton
Silently catching all exceptions like this is evil. Use .first() and check the return instead. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: handle downgradesPaul Eggleton
Version downgrades (or what appear to be downgrades) do occasionally happen, and if they did then the RRS was previously simply ignoring them, resulting in the latest version being reported incorrectly. Allow downgrades to be recorded as an upgrade with a new 'Downgrade' type option set, and display a label on such records in the UI. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: skip problematic OE-Core commits (when a dependency)Paul Eggleton
There are a range of commits in OE-Core which cause parsing problems; map them to the one that fixes it in order to avoid the problem. (This will only be done if we're dealing with OE-Core as a dependency, rather than the actual layer we're parsing). (The second set are some commits during the python 3 conversion time.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: use more robust RFC2822 date conversionPaul Eggleton
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: Add deleted recipe handlingPaul Eggleton
Now that we're using RecipeSymbols we have the complete list of recipes that ever existed in a layer. We only want to see the ones that are valid for the selected milestone, so when a recipe gets deleted (or renamed or moved outside of the layer subdirectory, if any) we need to record that - do so using a RecipeUpgrade record with a new field upgrade_type set to 'R'. Additionally we need to store the file path so that deletion events (where we don't parse the contents of the recipe, thus we don't have PN) are easy to match up with RecipeUpgrade records; naturally we need to keep the paths "up-to-date" when we notice recipe files being moved around. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-11-21RRS: collect history independent of current recipesPaul Eggleton
Recipes come and go over time, so when a recipe gets deleted the history for it goes away, which means that if you look back in time you do not see an accurate picture - you only see the subset of recipes that are currently present. Introduce an indirection between recipes and history that allows for old recipes to persist (mostly in name only). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-29requirements.txt: bump a couple more versionspaule/requirements1Paul Eggleton
Update pytz and beautifulsoup4. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-29requirements.txt: update to fix CVE-2019-16865Paul Eggleton
Update Pillow version to incorporate a fix for a denial-of-service vulnerability (which should not affect this application however, as it does not use Pillow to process external images): https://nvd.nist.gov/vuln/detail/CVE-2019-16865 Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-21Fix cgit commit URL settingPaul Eggleton
Typo, there has to be a ? in front of the id or otherwise we don't get linked to the right commit. This would have affected recently added layers with a cgit web frontend. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-21Add branch comparison functionPaul Eggleton
Add the ability to compare available recipes and their versions between two branches for a selection of layers (default is just OE-Core). This was mainly intended to help us with the Yocto Project release notes preparation (hence the "Plain text" button at the bottom of the page) but is also useful in its own right. Note: for readability, SRCREVs are only shown when PV has not changed. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-21Track SRCREV for each recipePaul Eggleton
For the purposes of the branch comparison function I'm about to add it would be useful to track the value of SRCREV, so we can see if it has changed even if PV hasn't. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-14editlayer: Be more specific on the searchesMark Hatle
Just because git.yoctoproject.org is in the URL, doesn't mean we can or should force the vcs_web_url to be a specific value. If it starts with git://git.yoctoproject.org then we can do this. git.openembedded.org already did this. This also changes github, gitlab and bitbucket references. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-14update.py: Allow bitbake to live in a subdirectory of a repositoryMark Hatle
Add a new BITBAKE_PATH to the settings file to specify the path within the BITBAKE_REPO_URL where bitbake lives. This is useful when using a combined repository, such as poky, that contains bitbake, openembedded-core and other layers. This change also changes the default path, in the fetch directory, for the bitbake checkout. It no longer uses the path 'bitbake', but instead uses the same URL processing as the layer fetching. There is a side effect that, when using a shared fetch, the branch of the layer will be used instead of the specified bitbake branch. Generally this is a reasonable compromise, since in a combined repository bitbake and openembedded-core component should already match. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-14layerindex/urls.py: Allow branches with a '.' in the nameMark Hatle
Without this change the system will fail parsing various URL components Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02Add a space between text and glyphiconsPaul Eggleton
Where we use glyphicons to mark items in a list, ensure there is a space between the item and the icon to make things look a bit neater. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02requirements.txt: updatePaul Eggleton
* Bump a few versions where we can * Drop anyjson - this used to be a dependency of kombu but not anymore, and nothing else needs it. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02README: add basic documentation on db backup and restorePaul Eggleton
Make it a bit easier for the user to back up and restore the database. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02Track PE and PR for recipesPranay Mankad
This change is to record PE (epoch) and PR (release) values for recipes. These values will be reflected in the REST API but will not be exposed in the UI. The aim of this change is to enrich metadata availability to consumers of the feed. Signed-off-by: Pranay Mankad <pranaymankad@gmail.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02update_classic_status: ensure fields are correctly orderedPaul Eggleton
On some systems it seems the order of fields in a values() query aren't guaranteed, so in order to avoid the contents dancing around too much, explicitly write out the fields in the order we expect them to be in. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02README.devel: touch up a few thingsPaul Eggleton
* Drop some explicit outdated version numbers * You do actually have to create the database with MariaDB etc. so add instructions as to how for convenience * Add pointer to virtualenv in python requirements item * Add libssl-dev package to install command * Use python3-venv instead of virtualenv package in install command Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02import_layers: tweak debug logging to be a bit more usefulPaul Eggleton
* Drop most "Skipping..." messages * Use layerbranch name instead of ID Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>