aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2019-10-21Fix cgit commit URL settingpaule/branch_comparisonPaul Eggleton
Typo, there has to be a ? in front of the id or otherwise we don't get linked to the right commit. This would have affected recently added layers with a cgit web frontend. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-21Add branch comparison functionPaul Eggleton
Add the ability to compare available recipes and their versions between two branches for a selection of layers (default is just OE-Core). This was mainly intended to help us with the Yocto Project release notes preparation (hence the "Plain text" button at the bottom of the page) but is also useful in its own right. Note: for readability, SRCREVs are only shown when PV has not changed. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-21Track SRCREV for each recipePaul Eggleton
For the purposes of the branch comparison function I'm about to add it would be useful to track the value of SRCREV, so we can see if it has changed even if PV hasn't. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-14editlayer: Be more specific on the searchesMark Hatle
Just because git.yoctoproject.org is in the URL, doesn't mean we can or should force the vcs_web_url to be a specific value. If it starts with git://git.yoctoproject.org then we can do this. git.openembedded.org already did this. This also changes github, gitlab and bitbucket references. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-14update.py: Allow bitbake to live in a subdirectory of a repositoryMark Hatle
Add a new BITBAKE_PATH to the settings file to specify the path within the BITBAKE_REPO_URL where bitbake lives. This is useful when using a combined repository, such as poky, that contains bitbake, openembedded-core and other layers. This change also changes the default path, in the fetch directory, for the bitbake checkout. It no longer uses the path 'bitbake', but instead uses the same URL processing as the layer fetching. There is a side effect that, when using a shared fetch, the branch of the layer will be used instead of the specified bitbake branch. Generally this is a reasonable compromise, since in a combined repository bitbake and openembedded-core component should already match. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-14layerindex/urls.py: Allow branches with a '.' in the nameMark Hatle
Without this change the system will fail parsing various URL components Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02Add a space between text and glyphiconsPaul Eggleton
Where we use glyphicons to mark items in a list, ensure there is a space between the item and the icon to make things look a bit neater. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02requirements.txt: updatePaul Eggleton
* Bump a few versions where we can * Drop anyjson - this used to be a dependency of kombu but not anymore, and nothing else needs it. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02README: add basic documentation on db backup and restorePaul Eggleton
Make it a bit easier for the user to back up and restore the database. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02Track PE and PR for recipesPranay Mankad
This change is to record PE (epoch) and PR (release) values for recipes. These values will be reflected in the REST API but will not be exposed in the UI. The aim of this change is to enrich metadata availability to consumers of the feed. Signed-off-by: Pranay Mankad <pranaymankad@gmail.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02update_classic_status: ensure fields are correctly orderedPaul Eggleton
On some systems it seems the order of fields in a values() query aren't guaranteed, so in order to avoid the contents dancing around too much, explicitly write out the fields in the order we expect them to be in. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02README.devel: touch up a few thingsPaul Eggleton
* Drop some explicit outdated version numbers * You do actually have to create the database with MariaDB etc. so add instructions as to how for convenience * Add pointer to virtualenv in python requirements item * Add libssl-dev package to install command * Use python3-venv instead of virtualenv package in install command Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02import_layers: tweak debug logging to be a bit more usefulPaul Eggleton
* Drop most "Skipping..." messages * Use layerbranch name instead of ID Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02import_layers: use vcs_last_rev to check if record is updatedPaul Eggleton
Unfortunately the "updated" date gets changed every time the repo is fetched (i.e. vcs_last_fetch changes), but that doesn't mean the layerbranch or anything under it has actually changed. Use vcs_last_rev instead although unfortunately that won't catch manual edits to the layerbranch alone. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02dockersetup: ensure letsencrypt volume change gets undonePaul Eggleton
If you enable the --letsencrypt option when you run dockersetup.py, the script will modify the volume mount for the certificates to point to /etc/letsencrypt instead of /opt/cert. If you then run dockersetup.py again (with -r/--reinstall) without --letsencrypt, we want the path to be set back to /opt/cert, so ensure that it does. Additionally, the code wasn't actually setting the path for the layerscertbot service since editing that section was done separately. (Admittedly, the letsencrypt functionality has not been well-tested.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02update_classic_status: add cover status import/exportPaul Eggleton
Provide a way to export and import other distro recipe coverage in update_classic_status.py. (Based on the Clear Linux Dissector commit ff38e9582093453e13b90e06af125374ca4b0a16). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02update_classic_status: switch to argparsePaul Eggleton
I'm about to add a few command line options to this script, so switch over to argparse before that which is more modern and a bit more flexible. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02update_classic_status: tweak single-object queriesPaul Eggleton
.first() returns None if the query is empty, so we can simplify this. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02dockersetup: minor command line comment fixPaul Eggleton
We do allow compressed SQL files. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02dockersetup: allow updating if proxy config set in environmentPaul Eggleton
Don't confuse proxy options being defaulted through from the environment and the user explicitly specifying them. Also look at no_proxy option. Fixes https://github.com/intel/clear-linux-dissector-web/issues/13 Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02dockersetup: handle ArgumentTypeError properlyPaul Eggleton
Print ArgumentTypeError (which we are raising) instead of showing a traceback. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02dockersetup: add uninstall optionPaul Eggleton
Add the ability to uninstall the application for the user's convenience. (Note that this does not undo the changes to the configuration, it only removes the Docker containers and volumes.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02dockersetup: proxy support fixesPaul Eggleton
* We need a SOCKS proxy to support fetching from git:// or ssh:// URLs within the container, so add an option to specify it * It's possible for the http and https proxy settings to be the same, so set one from the other if only one of them is set. * If we want to be able to fetch from internal servers inside the proxy then we also need a "no-proxy" list, so add support for that. * It's not unlikely that machines within networks requiring use of a proxy for external network access will have all of the proxy settings set in the environment, so we can try to pick up the defaults from there. * Ensure that we can switch from proxy to no proxy (when reinstalling) which means we always need to edit the config files and ensure the proxy options get commented out if we don't want them set. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02docker: fix comment at top of DockerfilePaul Eggleton
docker/README is no more - the info is all in the main README now. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-10-02dockersetup.py: add connectivity checkPaul Eggleton
It's easy to get the proxy settings wrong and not realise until you've got quite a long way into the process of setting things up. Thus, add a check where we actually try to fetch various things within the container environment and fail reasonably early if things aren't working. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-09-18API: fix recipes view to really point to recipes ViewSetPaul Eggleton
I forgot the lesson of f5922091b4c8b888849df158548453b73f92a377 which was that djangorestframework has a very silly default of registering the model name as the key for finding the ViewSet, with the result that the recently created RecipesExtendedViewSet was also being used for the recipes URL which we expect to point to RecipeViewSet; this broke the bitbake-layers layerindex-* commands. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-09-17import_layers: fix updating build dependenciesPaul Eggleton
Apparently I forgot that filter() returns a QuerySet and not an actual object. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-09-16import_layers: fix output when importing layers from scratchpaule/restapi-fixesPaul Eggleton
Fixes "Importing None" message since we had not yet initialised the layerbranch variable. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-09-16import_layers: use recipesExtended viewsetPaul Eggleton
The new viewset has a different URL and uses pagination, so we need to take that into account. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-09-16API: fix recipes API performance regressionPaul Eggleton
Adding these extra child items to the standard "recipes" viewset (which we did recently in 684a06a383fd2a8da80490dff5f0dbc47750934e) means that some current usage is impractical due to the size of the returned list of items. Instead, create a recipesExtended viewset, move the new child items to that and add pagination to avoid result size issues. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06README: add reference to using import_layers to updatepaule/importlayersPaul Eggleton
This is now be practical to use for updating as well as the initial import. Additionally the user doesn't need to be using the debug option by default so drop it from the example command. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: show basic progressPaul Eggleton
Show an info message with a counter when importing layers. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: add reload optionPaul Eggleton
Make it possible to reload the data even if the updated field says it's up-to-date. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: delete non-matching LayerBranch objectsPaul Eggleton
If a LayerBranch is on a branch that is in the remote layer index (and that branch is in any branch list specified with -b/--branch) and the layer for the LayerBranch is not found in the remote layerindex then it should be deleted, otherwise we could end up with stale data. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: allow filtering layersPaul Eggleton
Add an option to import only certain layers - mostly for debugging (can make testing a lot quicker). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: allow filtering branchesPaul Eggleton
We only work with branches that exist in both the local database and the remote index, but allow the user to restrict the list further if desired. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: switch to argparsePaul Eggleton
No functionality changes, just a code upgrade. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06Add simple script for testing web APIPaul Eggleton
This is not a comprehensive test, but does provide a way to easily compare that what was originally fetched for a recipe from the API is the same once it's been imported into a new instance using import_layers. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06admin: add layerbranch to BBAppend string representationPaul Eggleton
This makes it a little bit easier to see which BBAppend objects belong to which layer. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: import other missing itemsPaul Eggleton
Import Machine, Distro, BBClass, BBAppend, IncFile objects. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-06import_layers: import recipesPaul Eggleton
Import recipe data from the REST API, including sub-recipe objects (sources, patches, PACKAGE_CONFIGs, dependencies etc.) now that they are available from the API. Also, with the data being more or less complete, it's also reasonable to also set the vcs_* fields so that we can tell how up-to-date it is. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-05import_layers: enable incremental LayerItem/LayerBranch importPaul Eggleton
Don't assume we need to create these objects - check the updated field value against what the API reports and update the existing object if the latter is newer. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-05import_layers: avoid importing duplicate objectsPaul Eggleton
For LayerDependency, LayerMaintainer and LayerNote objects, properly sync the objects in the database with the objects fetched from the API. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-05API: add BBAppend and IncFile objects to REST APIPaul Eggleton
These are a little less useful than the other items but if we want to be able to create a clone of a layer index then we need them. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-08-05API: add sources, patches, dependencies to recipesPaul Eggleton
We're tracking these items for each recipe, so they should be accessible via the API as well. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-07-31dockersetup: fix regression with https enabledpaule/dissector-backportsPaul Eggleton
Fix a regression introduced in bfecbb091bde86a6df9980095379b31480dbb759. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-07-25dockersetup: fix error when printing URL with https enabledPaul Eggleton
Copy-paste error. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-07-25import_otherdistro: try-specfile: handle files in current dirPaul Eggleton
The try-specfile subcommand is for testing, so it's entirely possible the spec file we are testing with will be in the current directory; os.path.dirname() returns '' in this case, which os.path.relpath() then chokes on. Avoid this in the easiest way possible by just prepending ./ to the path in this case. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-07-25import_otherdistro: handle non-UTF8 encoded spec filesPaul Eggleton
It's not common, but if someone saves a spec file with a non-UTF8 encoding then the surrogate characters were preventing the data from being saved to the database (at least with MariaDB). Instead of assuming UTF8, try a few common encodings on the first 100 lines. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2019-07-25Optionally allow accounts without security questions to reset passwordPaul Eggleton
Add a SECURITY_QUESTIONS_REQUIRED setting that defaults to True, but if set to False then a user who has not set security questions will still be allowed to reset their password. This is convenient for the OE Layer index because there are a number of existing accounts, none of which will have security questions set. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>