summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/server
AgeCommit message (Collapse)Author
2019-08-16bitbake: bitbake: server/process: Handle BBHandledException to avoid ↵Robert Yang
unexpected exceptions The parseBaseConfiguration() raises bb.BBHandledException(), but BitBakeServer() didn't handle it, so we always got unexpected exceptions when there were errors. For example: === Case 1: * Add "print "hello"' in base.bbclass' def oe_import() function def oe_import(d): print "hello" [snip] $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Last 60 lines of server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 21 print "hello" ^ SyntaxError: Missing parentheses in call to 'print' <The first exception> During handling of the above exception, another exception occurred: <Tracebacks> <The second exception> During handling of the above exception, another exception occurred: <Tracebacks> <The third exception> During handling of the above exception, another exception occurred: <Tracebacks> [snip] Now it looks like: $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 21: The code lines resulting in this error were: 0001:def oe_import(d): *** 0002: print "hello" 0003: import sys 0004: 0005: bbpath = d.getVar("BBPATH").split(":") 0006: sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath] SyntaxError: Missing parentheses in call to 'print' (base.bbclass, line 21) === Case 2: * Add 'HOSTTOOLS += "hello"' to conf/local.conf: $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): <Tracebacks> [snip] During handling of the above exception, another exception occurred: [snip] <Tracebacks> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello The error message is printed by bb.fatal() which raises bb.BBHandledException(), but BitBakeServer() doesn't handle it, so we got it. Now it looks like: ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello No unexpected exceptions anymore. [YOCTO #13267] (Bitbake rev: 6e6865e6371dbd31a136eae64cc5b1fa5f5bee33) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie
With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie
This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06bitbake: server/process: Add missing exception raiseRichard Purdie
The intent of the code was to catch one kind of error, it was actually swallowing all exceptions and looping indefinitely. Fix it to work as intended. This explains some mystery hangs we've been seeing. (Bitbake rev: d73dbc3580faa1225d95ae4cefac4879ca3c1b2f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-26bitbake: process: Rewrite multiple connection handlingRichard Purdie
If the bitbake server recieved multiple connections, it currently closes ones it can't handle (while its dealing with another). This is rather antisocial behaviour which causes clients to quickly run through their retries and abort. Instead, queue any other connections until the current one is closed. This way the client can decide when it wants to stop waiting for the server. If the client is gone by the time we handle it, we handle that gracefully. This also fixes a number of bugs in the connection handling where connections which did drop early were badly handled causing tracebacks in the logs. Also, handle queue incomming connections in a loop to ensure that the main client handling doesn't starve that piece of the system. This code was stress tested by running 50 connection attempts in parallel at once, ensuring the code correctly handled them. (Bitbake rev: 02845a561b38658ac3edf5cc9c34625ed860d34f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-26bitbake: process: Handle EWOULDBLOCK in socket connectRichard Purdie
Now that we set a timeout for the socket, it can return EWOULDBLOCK if a signal or other event happens to wake up even if we don't timeout. If this happens, retry the connection, else we simply see it quickly loop through the retries and abort the connection in a very short interval. (Bitbake rev: d5b0a9a302ac0eafa4f797ac15ea77db87e82b3c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-26bitbake: process.py: Set socket timeout to 10 secondsRichard Purdie
The current value of 2 seconds has shown to be short in wider testing. (Bitbake rev: 8a1f2fcf35d61d83bbafa8fa3ae215fd5f51728b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-15bitbake: server/process: Ensure socket has a timeout setRichard Purdie
We're seeing hangs in oe-selftest where server startup and shutdown are racing. The assumption was a connect would timeout however no timeout is set which can leave processes hanging. Set a short timeout for the connection to avoid this. (Bitbake rev: f02114cb70e8f6f1d32e19c02b758fe0aadecd19) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08bitbake: server/process: Increase server startup time delayRichard Purdie
On loaded production systems we've seen bitbake server take over 40s to start up. Increase the timeout to 90s which tries to avoid failures in selftests. The delays come from setting up the inotify watches (31s) so can't really be avoided. After 5s delay we now warn the user we're waiting for 90s so the interactive exeperience shouldn't be much changed and its very unlikely the user would see that anyway. (Bitbake rev: 7a4e105093c3080c1087cd06abc2883852a23e28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05bitbake: process: Flush key output to logsRichard Purdie
Small tweak to ensure these items are printed into the log since there is other logging code which looks for the header and this makes it clearer the server did start but is slow somewhere in startup. (Bitbake rev: 7c0b84cac892744225fa0462f918ea9a79b356cb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05bitbake: server/process: Handle short readsRichard Purdie
Its possible the read may return a smaller number of characters. Remove the possibility by using a single character to signal the server is ready. (Bitbake rev: 767c9596d129d21ddf5d2e00c55f0a0525c641dc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05bitbake: server/process: Make lockfile handling clearerRichard Purdie
This simplifies the code and makes it easier to read but has the same functionality. (Bitbake rev: a0b9cfaf2d03fd047a79d32e668001718d02c4bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01bitbake: server/process: Fix ConnectionRefusedError tracebacksRichard Purdie
Improve connetion refused error handling: NOTE: Retrying server connection... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection... (Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 471, in connectProcessServer sock.connect(os.path.basename(sockname)) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 502, in connectProcessServer os.close(i) TypeError: an integer is required (got type NoneType) ) WARNING: /home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0> logger.info("Retrying server connection... (%s)" % traceback.format_exc()) (Bitbake rev: 6998fd816ceb0034c852a8fb994901fdf1975cfd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-27bitbake: server/process: Show the last 60 log lines, not the last 10Richard Purdie
10 log lines may not capture any full traceback, increase the number of lines to 60 which covers most tracebacks. (Bitbake rev: b8538c2088c199157404712a60a6740c926ce312) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-27bitbake: server/process: Show last 60 lines of the log if the server didn't ↵Richard Purdie
start We're seeing issues where the server doesn't start with no logs as to why. Allow the server to print the last 60 log lines just in case this shows us something useful about what is failing. (Bitbake rev: c8c80b404e38fe96f65d6314cd95f4069319f3d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-19bitbake: server/process: print a message when no logfileRobert Yang
[YOCTO #12898] There might be no bitbake-cookerdaemon.log, print a message for debugging. (Bitbake rev: 4adc582d2df7fdb9e51c4ebb5e66bbd21165b4dc) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-14bitbake: lib/bb/server: Avoid UnboundLocalError tracebackRichard Purdie
Traceback (most recent call last): File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 490, in connectProcessServer if command_chan_recv: UnboundLocalError: local variable 'command_chan_recv' referenced before assignment (Bitbake rev: ad79fadd855f5c10242ed17e9e0f3eb0274f26d2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-07bitbake: server/process: Fix unclosed socket warnings upon server connection ↵Richard Purdie
refused Extend the server error handling to avoid: Reconnecting to bitbake server... NOTE: Retrying server connection... (Traceback (most recent call last): File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 457, in connectProcessServer sock.connect(os.path.basename(sockname)) ConnectionRefusedError: [Errno 111] Connection refused ) WARNING: /home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0> logger.info("Retrying server connection... (%s)" % traceback.format_exc()) NOTE: Starting bitbake server... (Bitbake rev: afee3f594e1510051a0b18e430e92549caf72fa2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05bitbake: server/process: Various server startup logging fixesRichard Purdie
There were various problems in the server startup loggin: a) stdout/stderr were not being flushed before forking which could potentially duplicate output b) there were separate buffers for stdout/stderr leading to confusing logs where the entries could be reordered. This was particularly confusing due to the separator the logs use to idendify new messages c) an fd wasn't being closed during server startup meaning if the server failed to start, the closed fd wasn't detected as it was held open by the other reference d) If the pipe was detected as being closed, the code incorrectly retried server startup e) The event code would remap stdout/stderr without flushing them, leading to lose log messages (Bitbake rev: 0594faa0b52ce5dbd948d836d88617d38d9862d1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-25bitbake: server/process: Always place the server logfile in the build directoryuninative-2.2Richard Purdie
Currently the bitbake-cookerdaemon.log is placed into cwd. This seems like a bad idea, we can place it in the build directory alongside the lockfile that represents the server instead. (Bitbake rev: 1620dbc48ffb2a882371cf9174a7b12648befc8a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14bitbake: bitbake: cooker: don't stop file notifier when cooker is shutdownRobert Yang
It should be live/exited with server rather than cooker, fixed: $ bitbake --server-only -T -1 Set MACHINE = "qemux86" in conf/local.conf $ bitbake quilt Set MACHINE = "qemuppc" in conf/local.conf $ bitbake quilt [snip] ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed. [snip] This is because the server doesn't know local.conf is changed since the notifiers are stopped, so it doesn't reparse, and then we would get the errors, let the notifiers live/exited with server can fix the problem. (Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-25bitbake: server/process: Increase runCommand timeoutRichard Purdie
We were seeing cases where we could hit the 5s timeout on large/fast machines running many different tasks at once. Increase this to 30s since the main connection timeout path should no longer hit this slow path. [YOCTO #12116] (Bitbake rev: 131d4b8a5834781a93ed41e2967d8dcd4d80f29a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-12bitbake: server/process: ensure server failure log is limited to current sessionPaul Eggleton
Printing the last 10 lines of bitbake-cookerdaemon.log when the server fails to start can sometimes result in printing the output from a previous run, which could lead the user completely down the wrong path in terms of the cause of the failure. Use a known start text containing the time which we can then look for when scanning through the log, and then grab the last 10 lines of that part instead. Fixes [YOCTO #11903]. (Bitbake rev: 567f2cf1bc455b4f3cfb1cbd7f25145360b05a62) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-12bitbake: process/cooker: Improve readypipe handlingRichard Purdie
Issues in start are not being correctly detected by the current readypipe code. Change it to use specific "ready" or "fail" messages to correctly determine the correct failure mode and avoid bitbake seeming to hang (it does currently timeout eventually). [YOCTO #12062] (Bitbake rev: 60d4791e3dd05729d2a2adf6f3b203c80d466a73) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31bitbake: cooker: Change to consistent prefile/postfile handlingRichard Purdie
Currently the original prefile and postfile passed when starting bitbake server are 'sticky'. With the new memory resident model this doesn't make sense as the server the system is started with isn't special. This patch changes the code so the prefile/postfile are used if specified on the commandline and not used otherwise. This makes the behaviour much more predictable and expected and as an added bonus simplifies the code. (Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24bitbake: process: Clean up connection retry logicRichard Purdie
Its possible for a connection to connect to the server as its shutting down but before its removed the socket file. This patch: a) Removes the socket file earlier to avoid connections. b) Handles EOFError in initial connections gracefully. These occur if the socket is closed during the server shutdown. c) Ensure duplicate events aren't shown on the console. This makes debugging these issues very very confusing. With these changes the backtrace that was concerning users is hidden and the server works as expected with a reconnect when it catches it in a bad state. (Bitbake rev: f45196cf84669723382730944dddc7eaf50826f2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-22bitbake: process: Ensure we call select() to know which fds to readRichard Purdie
There is an interesting bug in the current code where a sync command is not seen until the current async command completes, by which time the UI may have shut down. The reason is that if there are idle commands, we may not end up sleeping in the select call at all, partiularly under heavy load like parsing. Fix this by calling select with a zero timeout so that we see active fds and know to read from them. This fixes various problems toaster was having with the recent server changes. [YOCTO #11898] (Bitbake rev: bbcce58e824b2793abf50efa52db158ae16e23e4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16bitbake: process: Increase server startup timeoutRichard Purdie
We're seeing the server fail to start within 8s on heavily loaded autobuilders so increase this timeout to 30s which should be more than enough time. (Bitbake rev: 8d4c120ec46d6d7a54947c64d33e18cb60b60505) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16bitbake: process: Improve client disconnectsRichard Purdie
There have been cases where the server could loop indefinitely and incorrectly handle client disconnects. In the EOFError case, ensure a full disconnect happens in the alternative disconnect path to avoid this. (Bitbake rev: 5e267f14bb0155889615f567a920af4a37eb3c6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13bitbake: server/process: Ensure we don't loop on client EOFErrorRichard Purdie
The server currently crashes if we hit an EOFError due to controllersock still being in ready and the continue meaning ready isn't re-evaluated. Setting the value to False can mean the shutdown code doesn't handle the situation cleanly. Clear ready to avoid the crash/loop instead and handle any OSError whilst we're in here. (Bitbake rev: 2bc47e887c3b41417edaa89a8708c223fd2085de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09bitbake: main: Handle BB_SERVER_TIMEOUT = -1 for no server timeoutRobert Yang
Make BB_SERVER_TIMEOUT = -1 mean no unload forever. (Bitbake rev: 923d5567be6fd9969ca74c166f36817ec09305e3) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09bitbake: process: Fix disconnect when BB_SERVER_TIMEOUTRobert Yang
Fixed: $ export BB_SERVER_TIMEOUT=10000 $ bitbake --server-only $ bitbake --status-only [snip] File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size)) OSError: [Errno 9] Bad file descriptor And: $ export BB_SERVER_TIMEOUT=10000 $ bitbake --server-only -B localhost:-1 $ bitbake --status-only # Everything is fine in first run $ bitbake --status-only [snip] File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size)) OSError: [Errno 9] Bad file descriptor This was because self.controllersock was not set to False, so it still ran sock.recvmsg() when sock was closed. And also need set command_channel to Flase, otherwise the self.command_channel.get() will always run when EOF, and cause infinite loop. (Bitbake rev: 7b739a38601b053d9bea4df2c0b44a952ab670c4) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31bitbake: process: Add some extra server startup logsRichard Purdie
We have cases where the server is being started but we're not seeing any messages from it. Add some earlier logging so we can try and better understand where issues may be occurring. (Bitbake rev: 10a6a48c0c5a2fe2051e90143e66075356853971) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31bitbake: process: Reorder server command processing and handle EOFErrorRichard Purdie
If the connection control socket and the command channel close together, we can race and hit EOFError exceptions before we close the channel. Reorder the code to handle this in the correct order and ignore the EOFError exceptions as they mean the client is disconnecting and shouldn't terminate the server. (Bitbake rev: 974281ed72d8366baa16ee85f7e93970689b5956) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30bitbake: process: Clean up server communication timeout errorsRichard Purdie
This timeout path was commonly hit due to errors starting the server. Now we have a better way to handle that, the retry logic can be improved and cleaned up. This patch: * Makes the timeout 5s rather than intervals of 1s with a message. Paul noted some commands can take around 1s to run on a server which has just been started on a loaded system. * Allows a broke connection to exit immediately rather than retrying something which will never work. * Drops the Ctrl+C masking, we shouldn't need that anymore and any issues would be better handled in other ways. This should make things clearer and less confusing for users and is much cleaner code too. (Bitbake rev: 8633b7cd03cfaba3e0359aa5da22fc76b66768c7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30bitbake: process: Don't leak open pipes upon reconnectionRichard Purdie
If we reconnect to the server, stop leaking pipes and clean up after ourselves. (Bitbake rev: f41e4e971e807157be68cf4496580494b8b60643) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30bitbake: process/cooker: Allow UI process to know if the cooker was started ↵Richard Purdie
successfully Currently if the server fails to start, the user sees no error message and the server will be repeatedly attempted to be started until some longer timeouts expire. There are error messages in the cookerdeamon log but nobody thinks to look there. Add in a pipe which can be used to tell the starting process whether the cooker did actually start or not. If it fails to start, no further attempts can be made and if present, the log file can be shown to the user. [YOCTO #11834] (Bitbake rev: 57000d44beb1aeba37dfc70782b0d6418943acc5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30bitbake: process: Move socket keep alive into BitBakeProcessServerConnectionRichard Purdie
This cleans up the socket keep alive into better class structured code and adds cleanup of the open file descriptors upon shutdown. (Bitbake rev: 77fd3a3a29a569e212374b27aea742ddbaafcdd5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30bitbake: process: Allow BBUIEventQueue to exit cleanlyRichard Purdie
Currently the monitoring thread exits with some error code or runs indefinitely. Allow closure of the pipe its monitoring to have the thread exit cleanly/silently. (Bitbake rev: 930d077637928213e13a07c78fee3bf7a8c37ebf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30bitbake: process: Ensure ConnectionReader/Writer have fileno() and close() ↵Richard Purdie
methods Expose the underlying close() and fileno() methods which allow connection monitoring and cleanup. (Bitbake rev: f79187f4ebfad7969be47b429995e7f7a3e33c1e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-24bitbake: process: Change timeout warning to a noteRichard Purdie
The warning message currently shown can occur more frequently than previously if a previous bitbake server is shutting down and we're reconnecting to a new server. Change it to a note message to match the higher level connection logging retry messages and so as not to interfer with selftests. (Bitbake rev: b7514340cd6a2753eb217b059229bb279c3849ec) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-24bitbake: cooker/process: Drop server_main functionRichard Purdie
Now that there is only one server, this abstraction is no longer needed and causes indrection/confusion. The server shutdown is also broken with the cooker post_server calls happening too late, leading to "lock held" warnings in the logs if PRServ is enabled. Remove the abstraction and put the shutdown calls in the right order with respect to the locking. (Bitbake rev: c0ddde7cf680225127d6285685652b905ed176c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21bitbake: server/process.py: fix self.bitbake_lock.write()Robert Yang
There is no global var "configuration", so the old code hang at self.bitbake_lock.write(), and nothing wrote to bitbake.lock. I didn't figure out why it hang (but not print errors). Reproducer: $ bitbake -B localhost:-1 world -k Check bitbake.log, there was nothing, now fixed. (Bitbake rev: dbdd9010663cd8dcb328e1b6fb40e3a777293bc5) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21bitbake: server: Rework the server API so process and xmlrpc servers coexistRichard Purdie
This changes the way bitbake server works quite radically. Now, the server is always a process based server with the option of starting an XMLRPC listener on a specific inferface/port. Behind the scenes this is done with a "bitbake.sock" file alongside the bitbake.lock file. If we can obtain the lock, we know we need to start a server. The server always listens on the socket and UIs can then connect to this. UIs connect by sending a set of three file descriptors over the domain socket, one for sending commands, one for receiving command results and the other for receiving events. These changes meant we can throw away all the horrid server abstraction code, the plugable transport option to bitbake and the code becomes much more readable and debuggable. It also likely removes a ton of ways you could hang the UI/cooker in weird ways due to all the race conditions that existed with previous processes. Changes: * The foreground option for bitbake-server was dropped. Just tail the log if you really want this, the codepaths were complicated enough without adding one for this. * BBSERVER="autodetect" was dropped. The server will autostart and autoconnect in process mode. You have to specify an xmlrpc server address since that can't be autodetected. I can't see a use case for autodetect now. * The transport/servetype option to bitbake was dropped. * A BB_SERVER_TIMEOUT variable is added which allows the server to stay resident for a period of time after the last client disconnects before unloading. This is used if the -T/--idle-timeout option is not passed to bitbake. This change is invasive and may well introduce new issues however I believe the codebase is in a much better position for further development and debugging. (Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21bitbake: server: Remove base classes and inline codeRichard Purdie
In preparation for rewriting this code, expand the relatively useless base classes into the code itself. (Bitbake rev: a1c6151420d86bac658c08ae714647062edd6ef2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21bitbake: event/command: Allow UI to request the UI eventhander IDRichard Purdie
The UI may want to change its event mask however to do this, it needs the event handler's ID. Tweak the code to allow this to be stored and add a command to query it. Use the new command in the process server backend. (Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-18bitbake: bb/server/process: Handle EINTR on idle_commands selectAníbal Limón
If a signal is sent like SIGWINCH the select could be interrupted so ignore the InterruptError like in XMLRPC server [1]. [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/server/xmlrpc.py#n307 (Bitbake rev: 96bb174325493764718c61c12c943c37b882cd61) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08bitbake: server/xmlrpc: Add Heartbeat event supportRichard Purdie
When heartbeat event support was added it was only added to process.py. Add it to server/xmlrpc too. There is duplicated code however since we're likely to combine the server abstractions soon its not worth worrying about now. This ensures the backends have the same event support. [YOCTO #10741] (Bitbake rev: 00bf2e60222767b4dee84fb3f958732a83544e80) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08bitbake: event: Queue offline events for the UIRichard Purdie
Messages printed when no UI is connected (e.g. memres) are currently lost. Use the existing queue mechanism to queue these until a UI attaches, then replay them. This isn't ideal but better than the current situation of losing them entirely. (Bitbake rev: cb241fb8544dfb05646dbae1a1b04e17878a466c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08bitbake: server/process: Fix waitEvent() calls with 0 timeoutRichard Purdie
You might think Queue.Queue.get(True, 0) would return an event immediately if present and otherwise return. It doesn't, it immediately "times out" and returns with nothing from the queue. The behaviour we want is not to wait but return anything present which is what .get(False) does so map to this. This fixes some odd behaviour observed in some of the tinfoil selftests. (Bitbake rev: 412bfab8721ea317898a1974f6a7a0d0bea763df) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>