| Age | Commit message (Collapse) | Author |
|---|
|
This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10,
this is the fetch time only, I hacked it to stop when the fetch is done to get
the data (124 layers):
$ update.py -b <branch>
Before: 2m30
Now: 16s
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
* Allow the model's __str__() function to control what is shown for each
dependency item as per other model admins
* Enable searching for PACKAGECONFIGs by recipe name
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Use the more typical handling of the return of get_or_create() and check
if the item was created before saving.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Fix duplicate PackageConfig records being created each time a recipe
is being updated - we need to delete the old ones first.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Since we added the "packageconfig" to Recipe we need to exclude it from
the admin or we see an error when we try to open ClassicRecipe in the
admin site.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Added a model for the PACKAGECONFIG variable, which has a one to
many relationship with the Recipe model.
Added models for static build dependencies and dynamic build
dependencies, both of which have a many to many relationship with
the Recipe model.
These objects are created in update_layer.py and are displayed on the
Recipe detail page.
Added a depends search option for recipes, allowing users to search for
recipes that depend on any particular recipe. Use "depends:recipename"
in the recipe search to activate this.
Fixes [YOCTO #12129]
Fixes [YOCTO #11415]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Use JavaScript to check if the search box for recipe search is
empty before querying the database. This will prevent the "502
Bad Gateway" error that occurs when the query takes too long due
to the large list of recipes. Since there are so many recipes
spread across the layers in the OE index, there's no point in
allowing a user to search without a keyword in order to browse
the list; it simply isn't digestible as a whole.
For the Machines, Classes, and Distros pages, the search behaviour is
unaffected, however to make it more obvious that you can browse the list
add an explicit "browse" button.
Fixes [YOCTO #11930]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Add another tab to search for classes.
Fixes [YOCTO #11207]
Signed-off by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Don't assume that every model will have a "search_allowed_fields"
attribute - if it doesn't, default to all CharFields in the model. This
fixes searching via the REST API.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
When publishing a layer, send an email notification to all of that
layer's maintainers. Include information on how to edit the layer, plus
contact details for the first active staff user if there are any
problems (we could make this configurable in future, but for now this is
sufficient).
Fixes [YOCTO #11208]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
For newly added layers it may not be immediately obvious that you have
to wait for it to be indexed (since the update script is usually run on
a scheduled basis). If the layer branch hasn't been indexed, add a note
mentioning this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
It's really irritating to be forced to specify a value for this field
especially as it'll get auto-populated by the update script. Set
blank=True to allow that. While we're at it, touch up the description a
bit to make more sense.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If you're diagnosing problems with the bitbake server when running the
update script, then you need to be able to look at
bitbake-cookerdaemon.log, but you couldn't do that after the fact
because the temporary directory it gets written out to was being
unconditionally deleted. Add a --keep-temp option which preserves it and
some debug messages to tell you where it is.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Allow admin to create Yocto Project versions with the version name,
description, an icon link that represents that version, and a link that
contains more information about the version.
Admins who have the "set_yp_compatibility" permission can then
decide if a layer has Yocto Project compatible certification,
and if it does, admin can choose which version of Yocto Project
the layer is compatible with. If a layer is deemed compatible,
the version's icon will appear next to the layer's name, and
the icon be a clickable link to a page with more information.
Fixes [YOCTO #11452]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If a log level is set on the command line with -q/-d,
set tinfoil's log level to the appropriate log level.
Fixes [YOCTO #11931]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
This patch adds asynchronous task execution using a Celery backend
and RabbitMQ task queue, so that the layer submission process to
proceed even in the event that sending the notification email fails,
and establishing an asynchronous execution mechanism that we can use
in the future e.g. for triggering parse operations from the web UI.
This pertains to bug 11197:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11197
It updates the README to reflect the installation and configuration
of a basic RabbitMQ setup, adds a 'tasks.py' file to contain task
definitions, updates the 'edit_layer_view' function to send
emails to administrators about new and updated layers asynchronously,
modifies the 'settings.py' to include a default configuration
for a RabbitMQ connection, and updates the Dockerfile to start a Celery
worker alongside the Gunicorn daemon.
Fixes [YOCTO #11197].
Signed-off-by: Diana Thayer <garbados@gmail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Patches have been recently merged from both.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Before, if a user edited a layer's name, they would be redirected
to a url utilizing the old name and then receive a 404 Page not
found error. Now, the url utilizes the new name.
Fixes [YOCTO #11932]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Robert's changes just got merged recently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
At the moment the URL that is presented in the review email will have
http:// or https:// prefix depending on what the user who submitted the
layer was using, but that's irrelevant - we actually want https:// if
the server is capable of it since the reviewer may be redirected to log
in (and Django's login_required decorator will always redirect to the
login page if http is being used as far as I observed, which is a bit
annoying if you are already logged in.) Add a setting which if enabled
will substitute https:// as the prefix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The branch is not needed any more when it has been removed from the repo, so we
also need remove its layerbranch, otherwise it still can be got from the web,
which causes confusions.
Note, we have to move the location of tinfoil's code to avoid creating tinfoil
when not needed, otherewise, if tinfoil is created (but not needed), and the
program exists earlier before "try ... finally" block, then tinfoil.shutdown()
doesn't run so that it is not shutdown. Move the code back, right before where
it is needed can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
-p, --prune
Before fetching, remove any remote-tracking references that no longer exist on the remote.
Fixed:
$ git push origin :test_branch
$ ./update.py
The test_branch was still in fetched local repo which was incorrect, it should
be gone since it has been removed by upstream.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
It doesn't need to be so ahead since we only need it when writing database, and
a following patch will remove layerbranch from database when the branch had
been removed from the repo, it's not easy to do the work in
transaction.atomic() block.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
It never works since it is in the middle of transaction.atomic() block, and
update.py doesn't need it any more, so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers orderly to update_layer.py according to dependencies
can fix the problem, we can get intial dependencies from tinfoil, add an option
"-i, --initial" to update_layer.py to get them.
Not only new branch, but also existing branches may have the problem, because
collections and dependencies maybe changed in the coming update, so we can't
trust database when the layer is going to be updated, for example, if there are
10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collections data from database, we need get
them from tinfoil.
* Performance improvement:
It should be the same as before in theory, I have tested it with 97 layers:
- Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(11 * 60 + 6)/75*97/60 = 14m35s
- Now 12m10.350s, all the layers are added in the first run.
So about 2 minutes are saved.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If we don't restore the current working directory, it may cause trouble
for the calling function.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The _add_dependency() uses:
if layerbranch.collection:
var_name = layerbranch.collection
The layerbranch.collection is none if it is newly created, thus it can't get
LAYERDEPENDS, because what defined in layer.conf is LAYERDEPENDS_<collection>,
but what it would get is LAYERDEPENDS_<layer_name>, this patch can fix the
problem.
Reproducer:
$ python3 update_layer.py -l mete-xfce -b <newbranch> --fullreload -d
It would get None LAYERDEPENDS.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Fixed:
1) set LAYERDEPENDS_openembedded-layer = "core"
2) $ "update.py -l meta-oe -b master"
Check from web, its dependency is "openembedded-core"
3) Change LAYERDEPENDS_openembedded-layer = "foo"
4) Run "update.py -l meta-oe -b master"
5) Check from web, its dependency is "openembedded-core and foo", this might
be incorrect, now if set REMOVE_LAYER_DEPENDENCIES to true, the old
dependency openembedded-core will be removed, the default is False which
prints warnings to notify users.
And also the existing checking should filter(required=required), otherwise it
can't work well when a layer is in both depends and recommends, this can't
happen in a normal case, but it would surprise the user when this happens.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The layerbranch is not used in parse_layer(), so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Add an option "-a" to update actual branch for layer and bitbake, it is
useful when there are many layers and need update actual branches
frequently. We only can update them via website without this patch,
which is not fun and easy to make mistakes.
* It works with "-l", and "-l bitbake" means update bitbake branch.
* It requires "-b" to work, and only one branch is supported in a run.
For example:
$ update.py -b master -a branch_20170526
All the layers which have branch master and actual_branch branch_20170526
will be updated to branch_20170526.
$ update.py -b master -l meta-oe -a branch_20170526
Only meta-oe layer will be updated.
$ update.py -b master -l bitbake -a branch_20170526
The bitbake's bitbake_branch will be updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
This change supports querying recipes that belong to a specific layer
by using the prefix "layer:" + the desired layer name, for example:
"layer: openembedded-core" and this string can be used by itself or
combined with other supported options.
A descriptive error message is displayed when the query string has an
unexpected formatting or a non-valid layer name is searched.
[YOCTO #6618]
Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Drop a couple of items that got implemented recently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
In OE-Core master, HOSTTOOLS specifies a list of host-side tools to be
symlinked into a directory that is placed in a filtered path, so that
binaries used from the host are a controlled subset. This variable is
acted upon at configuration parse time and if any tools are missing,
parsing is terminated.
We're not actually running any tasks, so we won't need to call most of
the tools specified by HOSTTOOLS. There is one we definitely do need to
run (gcc) so keep that one, but the rest can be moved to
HOSTTOOLS_NONFATAL.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Add a status for a layer indicating it should not be updated. I don't
expect this to be widely used (and is only settable from the admin
interface) but would be useful if you have a legacy sub-layer that you
want to prevent from being visible on certain branches - it will prevent
the update script from doing anything with the layer and thus avoid
branch records from being auto-created on branches where you've deleted
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
When reading conf/distro/*.conf to create distro records, attempt to
parse the config file and use DISTRO_NAME (if set) to populate the
description field for the record. If that's not set then fall back to
the less commonly used meta-comment that we supported previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If we want to parse a configuration file (e.g. a distro conf file) then
we need convenient access to bitbake's conf parsing code, so create a
parse_conf() function to provide that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
When the recipe search returns a single result, redirect to it
instead of showing the full result list view.
Part of the fix for [YOCTO #6618].
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The branch meta change made in b91b09a0aec11c0f8ee0cb99aa4ec81be9d23ddc
apparently requires this migration - not entirely sure why, but Django's
manage.py makemigrations wants to create it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Make minor (non-structural) improvements to the model, mostly for the
benefit of the admin interface:
* Set default ordering so that branches appear in the same order they do
in the user-facing drop-down
* Include the description in the default string representation (so you
know which branch name matches with which Yocto Project version, in
the OE index).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
We should always be using four spaces.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
We can't decode UTF-8 characters byte-by-byte, as soon as we hit a
character that's more than one byte long then we'll fail. Use a reader
object to do it properly. This fixes parsing current meta-angstrom on
master. At the same time, specify errors="surrogateescape" to avoid the
update process dying at this point in case of characters that aren't
valid UTF-8.
Thanks to Jiajie Hu <jiajie.hu@intel.com> who fixed this in devtool's
very similar code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The fido branch of OE-Core expects BitBake 1.26.x. Unfortunately that
version had a commit backported which introduced a non-working
tinfoil.shutdown() method, so we can't simply rely on its presence to
determine whether or not it should be called. Use a check on the BitBake
version number instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Layers that aren't published shouldn't be visible via the API. (We don't
need to apply that filter to recipes, machines or distros though since a
layer's content won't automatically be indexed unless it has been
published).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
On the main layer index page we want openembedded-core to appear before
meta-oe - naturally openembedded-core is the layer everyone starts from.
We already have an "index_preference" column that we use for search
results, so just sort by that here before sorting by name.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
In the layer detail page, show counts on all but the updates tab (i.e.
for recipes, machines, appends, classes, and distros).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Since some recent infrastructure changes, "/cgit.cgi" should no longer
be part of cgit.openembedded.org URLs, apparently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
This wasn't documented anywhere so I've added it here. I'm not sure
it'll get used outside of the live OE index but it's at least useful for
anyone else who needs to debug it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Fix the transaction handling code to work with Django 1.8 in most of the
tools scripts. (Some of these are no longer used, but still serve as
examples of how to import data and update the database.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
I neglected to update this at the time I updated the README recently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
Robert Yang
Paul Eggleton
Amanda Brindle
Diana Thayer
Jose Lamego
Alex Franco