| Age | Commit message (Collapse) | Author |
|---|
|
Worryingly, if you SIGKILL the bitbake cooker, an autostarted PR server
will remain behind. It turns out there are a few things we should do:
* The PR service doesn't need to daemonize when started from cooker,
it just complicated the process lifecycle. Add a fork() method
to handle this and use the non-daemon mode for the singleton.
* Reset the sigterm and sigint handlers. Bitbake cooker installs its
own which we inherit meaning PR server was ignoring SIGTERM. Installing
our own handlers which include a sync makes most sense here. Since
we're in the code, make it sync the database on SIGINT.
* Use the new bb.utils.signal_on_parent_exit() call so that we get a
SIGTERM when the parent (usually cooker) exits and we can shutdown
too. Alternatives would be having an open pipe or polling
os.getppid() for changes but this seems more effective.
(Bitbake rev: 05d31fa1f56bd3d3d363a16a421d9ba7541d4293)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I keep having to dig into the archives to remember this information.
Add it as a comment to the file instead.
(Bitbake rev: 21dce82056887d8d28edde61b1b82f78bdf7613c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Remove unused xmlrpclib, atexit and duplicated threading module imports
(Bitbake rev: 3e12f4e8e3ec66d1df772a64be04b90ec72462ae)
Signed-off-by: Maxin B. John <maxin.john@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, I'm not sure how the prserver managed to shut down cleanly. These
issues may explain some of the hangs people have reported.
This change:
* Ensures the connection acceptance thread monitors self.quit
* We wait for the thread to exit before exitting
* We sync the database when the thread exits
* We do what the comment mentions, timeout after 30s and sync the database
if needed. Previously, there was no timeout (the 0.5 applies to sockets,
not the Queue object)
(Bitbake rev: 0926492295d485813d8a4f6b77c7b152e4c5b4c4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Ideally, we want the PR service to have minimal influence from
queued disk IO. sqlite tends to be paranoid about data loss and
locks/fsync calls. There is a "WAL mode" which changes the journalling
mechanism and would appear much better suited to our use case.
This patch therefore switches the database to use WAL mode. With this
change, write overhead appears significantly reduced.
(Bitbake rev: 0cdd48261daeb17efc528b5de0ac81c8836e8565)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The sync/commit calls are happening in the submission thread which can
race against the handler. The handler may start new transactions which
then causes the submission thread to error with "cannot start a
transaction within a transaction".
The fix is to move the calls to the correct thread.
(Bitbake rev: 92e128a0e331e563cfe48827e95939041c16c88e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In the commit 'prserv: Ensure data is committed', the PR server moved to
only committing transactions to the database when the PR server is
stopped. This improves performance, but it means that if the machine
running the PR server loses power unexpectedly or if the PR server
process gets SIGKILL, the uncommitted package revision data is lost.
To fix this issue, sync the database periodically, once per 30 seconds
by default, if it has been marked as dirty. To be safe, continue to
sync the database at exit regardless of its status.
(Bitbake rev: 424df64f2e9679043f0ce2b4f7dfc59c3d404304)
Signed-off-by: Ben Shelton <ben.shelton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
supplied host address
If localhost resolves to a remote address (due to a misconfigured network),
starting the pr server will fail without useful information.
To reproduce, add '<bogus ip> localhost' to /etc/hosts and run
'bitbake -p'. The error message will be:
ERROR: Timeout while attempting to communicate with bitbake server
ERROR: Could not connect to server False:
Running 'bitbake-prserv --host=localhost --port=0 --start' will fail with:
error: [Errno 99] Cannot assign requested address
Since these errors does not show the IP address of the attempted socket
binding, this results in a lot of wasted time looking at firewall rules, etc.
This patch results in the following error message if the socket binding fails:
PR Server unable to bind to <bogus ip>:0
(Bitbake rev: fae5914030bcf4c061c22fc61034c40c87b7121a)
Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a '\n' to the last line of the file to fix:
No newline at end of file
(Bitbake rev: 54f1359ed2e9d47980cd221b7b43ef56543fe06d)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the power were to fail, it doesn't matter to us much if the data
makes it to disk or not, we'd have other problems. However an fsync()
call on a multi build autobuilder is painful so lets avoid them.
(Bitbake rev: 4eb2dc8048e2722d64d589f453df1ce6262c71b8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We shouldn't immediately remove the pid file when stopping the server, if we do, this
causes a traceback within the server itself which can then hang. Fix this by removing
the stale pid file as the last thing we do.
Also:
* don't printing a new "waiting" line every 0.5 seconds.
* make the loop more granular since the user can 'feel' the 0.5 seconds
[YOCTO #5984]
(Bitbake rev: 81f41a806aeddcc38992163557672e296bcbc967)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The default value of 0.5 seconds before sending the pr-server a
SIGTERM is not enough to guarantee that sqlite has committed all
the pr data to the database. By polling the pid to see if it is
still running, this allows the pr-server process to shutdown
cleanly and finish the final pr data commit.
(Bitbake rev: 22eec978e70794923c85689928c6be0cfe71cdcd)
Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In exclusive mode, we need to complete the transaction for writes to make
it to the database. Therefore add sync calls to ensure this happens.
Autocommit mode is significantly (100 times) slower so caching the
data is of significant benefit.
(Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using the threading mixin class resulted in large amounts of memory
being used by the PR server for no good reason. Using a receiver thread
and a thread to do the actual database operations on a single connection
gives the same performance with a much saner memory overhead so
switch to this.
(Bitbake rev: e08455d5f3b8e96765942b9c3b9767c30650557d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since we don't support using multiple servers on the same database file,
don't use the BEGIN/COMMIT syntax and allow writes to the database
to work ~100 times faster with no transaction locking.
(Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We only support one server using the database at a time so take an exclusive
lock and avoid later lock overhead.
(Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the database is locked we will get an immediate error indicating so,
there is no retry timeout. The looping code is therefore useless, the loop
count is near instantly exceeded.
Using a time based retry means we can wait a sensible time, then gracefully
exit.
(Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist"
by retrying if we see the string "is locked".
(Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This makes the PR server multithreaded and able to handle multiple connections
at once which means its no longer a build bottle neck when serving one connection
at a time. I've experimented and database connection for each thread seems to
cause the least issues, pushing the contention for sqllite to handle itself.
This means moving the db/table connection code into the actual function methods.
It doesn't abstract well as a function since we need the db object around for
the lifetime of the function as well as the table else we lose the connection.
(Bitbake rev: bf9be2029b2bded5f532bdda4c38ae3dff5d1cf6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Enabling threading for the PRServer causes a number of issues. Firstly is
the obtuse error:
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type
which is due to the class not being derived from object. See:
http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable
Secondly, we want to enable multithreadded access to the database so we do this
when we open it. This opens the way up to multithreading the PR server.
(Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
commit 972bc43e6d5b1207b944b3baa8f9805adb35dda7 (serv.py: Fix hang
when spawned dynamically with bitbake) introduced a regression,
because the wrong patch was submitted. The syntax was incorrect in
the original patch. The logger iterator must be used with a call to
getLogger().
[YOCTO #5059]
(Bitbake rev: 85fed8acc3af3e15bf119db2f51c486a9de3646b)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The PRServer has the possibility to hang indefinitely blocking on a
semaphore processing a xmlrpc request to send an event back to the
main bitbake instance. This was observed during a "bitbake -e" on a
heavily loaded machine and the main bitbake instance and cooker exited
before the PRServer emitted its first log.
The stack trace is provided below as to show what happens every time a
logger.info() is executed in the PRServer. Not only does it write to
the stream handler but it also tries to send the event to the main
event processor.
self._notempty.acquire()
self.queue.put(event)
_ui_handlers[h].event.send(event)
fire_ui_handlers(event, d)
fire(record, None)
self.emit(record)
hdlr.handle(record)
self.callHandlers(record)
self.handle(record)
self._log(INFO, msg, args, **kwargs)
(self.dbfile, self.host, self.port, str(os.getpid())))
self.work_forever()
pid = self.daemonize()
self.prserv.start()
singleton.start()
self.prhost = prserv.serv.auto_start(self.data)
cooker.pre_serve()
bb.cooker.server_main(self.cooker, self.main)
self.run()
code = process_obj._bootstrap()
self._popen = Popen(self)
self.serverImpl.start()
server.detach()
server = start_server(servermodule, configParams, configuration)
ret = main()
It was never intended for the PRServer to send its logs anywhere but
its own log file. The event processing is an artifact of how the
PRServer was forked and it inherits the event log handlers. The
simple fix is to clean up and purge all the log handlers after the
fork() but before doing any of the typical PRServer work or logging.
(Bitbake rev: 972bc43e6d5b1207b944b3baa8f9805adb35dda7)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The standard python socket connect has long timouts which make sense for remote
connections but not local things like the PR Service. This adds a timeout
parameter to the common xmlrpc server creation function and sets it to a more
reasonable 5 seconds.
Making the PR server instantly exit is a good way to test the effect of this
on bitbake.
We can remove the bodged timeout in the PRServer terminate function which
has the side effect of affecting global scope.
(Bitbake rev: 8c01cff94787abbb64fbdf0c16cd63f8f97a7e03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Mark Hatle spotted there were pid files being left around. This patch
fixes things so the removal function is called correctly, the code
contained a typo.
(Bitbake rev: c696a16c8200c31c52750037eeafe07e065b6517)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the change to bitbake-worker we need to ensure the workers know
how to contact the PR service, the magic 0 port and singleton is
no longer enough.
(Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: e2cc22fb8b2e97b068b6037540c746ecb1856de6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
StandardError doesn't exist in python 3, use Exception instead.
(Bitbake rev: 4a40046036493f0cdf0f66487ad5ce083461a5c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Current if the PR Service fails to start, bitbake carries on regardless or
hangs with no error message. This adds an exception and then handles it correctly
so the UIs correctly handle the error and exit cleanly.
[YOCTO #4010]
(Bitbake rev: 949c01228a977c3b92bfc0802f6c71b40d8e05b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In deamon mode we need to ensure the logging module is sending log data to the
log file. These changes ensure this happens correctly.
(Bitbake rev: bb53b47710ca4579e20284668cb354f734c3d502)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using the threading module interacts badly with multiprocessing used elsewhere
in bitbake under certain machine loads. This was leading to bitbake hanging on
Ctrl+C when the PR Server was being used.
This patch converts it to always use the daemonize code which
then means the threading code isn't required.
[YOCTO #3742]
(Bitbake rev: 2d0bbd9398ab839bd2d1e29e50b25d52efb1ce2a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
- remove PRSERV_PORT variable
- use 'hostname:port' as PRSERV_HOST format
- use 'localhost:0' for enabling the local PRServer
[YOCTO #3744]
(Bitbake rev: ad62eed9e1f0867a406e9fbfa44916a0f1ad1282)
Signed-off-by: Constantin Musca <constantinx.musca@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #1942]
Ping the PRService only if PRSERV_HOST and PRSERV_PORT are set.
(Bitbake rev: 20f24de0bdafac21f5d8a58701f977efa7041288)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* this is used in all other bitbake parts where sqlite3 is used, don't
know why it wasn't used here, but it fails e.g. on Gentoo
Traceback (most recent call last):
File "bin/bitbake", line 39, in <module>
from bb import cooker
File "lib/bb/cooker.py", line 39, in <module>
import prserv.serv
File "lib/prserv/serv.py", line 4, in <module>
import xmlrpclib,sqlite3
ImportError: No module named sqlite3
(Bitbake rev: 9a57ec705cf5c932d8c2a35852db7a4627c57937)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #1126]
A local PR service will be started and stopped automatically along
with the bitbake invocation/ternimation.
This local PR service will be started only and if only when the
PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'.
When started, the sqlite3 database is stored at
"${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3".
(Bitbake rev: 9d8f45407c67ed0d3c4f820cf646de3c385067c7)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #1556]
1. Added the package_arch into the index to the DB table. Because the
change in PACKAGE_ARCH will results in different checksum, and it is
better to have seperate PR value domains for differnt PACKAGE_ARCH of
the same pakcage.
2. Changed the PR service to operate in no history mode. In this mode,
the for a given query tuple (version, pkgarch, checksum), the returned
value will be the largest among all the values of the same (version,
pkgarch). This means the PR value returned can NOT be decremented.
3. Added export function. For each (version, pkgarch) tuple, only the
record with the maximum value will be exported.
4. Added import function. The record will only be imported if the
imported value is larger than the value stored in the DB with the same
(version, pkgarch, checksum) tuple.
(Bitbake rev: 379567ee879dcdc09a51f7f1212bde1076147a6f)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update exception handling syntax to use the modern style:
except ExcType as localvar
(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Added the initial implementation of the server side PR service.
(Bitbake rev: 4d0e79e5591ff58ce35c7fb96f6e9217ddc27466)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Maxin B. John
Ben Shelton
Konrad Scherer
Robert Yang
Jason Wessel
Constantin Musca
Lianhao Lu
Martin Jansa
Scott Garman