summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/prserv
AgeCommit message (Collapse)Author
10 daysbitbake: prserv: remove unnecessary codeMichael Opdenacker
In db.py, the ifnull() statement guarantees that the SQL request will return a value. It's therefore unnecessary to test the case when no value is found. (Bitbake rev: e4ae5177861c9a27e93e5a2d3a6c393baecd6416) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 daysbitbake: prserv: correct error messageMichael Opdenacker
according to db.py, prserv.NotFoundError is returned here when adding a new value to the database failed (Bitbake rev: 4cc4069987edd14f51715dfaf0c6e1a3aa307106) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 daysbitbake: prserv: remove redundant exception handlerMichael Opdenacker
This exception handler is already present in db.py's get_value() code. (Bitbake rev: 2fd38b1bb685ec441f0eb0f28f3d84ba252ba90b) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 daysbitbake: prserv: add extra requestsMichael Opdenacker
Useful for connecting a PR server to an upstream one - "test-package" checks whether the specified package version and arch is known in the database. - "test-pr" checks a specified output hash is found in the database. Otherwise it returns 'None' instead of a new value. - "max-package-pr" returns the highest PR number for (version, arch) entries in the database, and None if not found Add new DB functions supporting the above, plus test_value() which tells whether a given value is available for the specified package and architecture. (Bitbake rev: 0f1474a30f741b760ca81c19dd1d8f3bd5647251) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 daysbitbake: prserv: capitalization and spacing improvementsMichael Opdenacker
Choosing only one style of capitalization Add extra space after some commas too Remove idle spaces (Bitbake rev: daad17bccec8cb98ef2fca4262641167500bd46e) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 daysbitbake: prserv: use self.logger instead of logger directlyMichael Opdenacker
In both the PRServerClient and PRClient objects. This aligns with what is done in hashserv/server.py and makes it possible to benefit from possible specializations of the logger in the corresponding super classes, instead of using always the global logger. (Bitbake rev: 5fc6d2b1a5db617e16c1eb9fbd25e821237611d8) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 daysbitbake: prserv: use double quotes by defaultMichael Opdenacker
To aligh with the hashserv code (Bitbake rev: 7a6999750791659eaffe49aabfbfba9f37f51913) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 daysbitbake: prserv: simplify the PRServerClient() interfaceMichael Opdenacker
serv.py: simplify the PRServerClient() interface by passing the server object instead of multiple arguments, and then retrieving the data through this object. This replicates what is done for ServerClient() in hashserv/server.py (Bitbake rev: d3be073218feb4d6e68a751832da4936da485dbc) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-03bitbake: prserv/serv: Fix a PID file removal race on prserv stopYoann Congal
A race condition has happened where the exiting server removed the PID file between the existence check and the removal, resulting in a FileNotFoundError exception. The fix is to ignore the FileNotFoundError exception, the existence check is now redundant so remove it to simplify. Fixes [YOCTO #14341] (Bitbake rev: 40d00bf9308e0bf73a00134a99a012a292daa1c5) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-14bitbake: asyncrpc: Add option to set log level when running as a processJoshua Watt
When running an asyncrpc server as a subprocess, it is often desired to run it with a lower logging level since the normal logging of clients connecting and disconnecting is not desired. As such, add an option to set the logging level of the server when running as a subprocess and set the level to WARNING when starting a local hashserver or PRserver (Bitbake rev: 61dac7b99ad6d2a858f85d8ed1b5524d558be6c8) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-09bitbake: asyncrpc: Add context manager APIJoshua Watt
Adds context manager API for the asyncrcp client class which allow writing code that will automatically close the connection like so: with hashserv.create_client(address) as client: ... Rework the bitbake-hashclient tool and PR server to use this new API to fix warnings about unclosed event loops when exiting (Bitbake rev: d01d684a0f6398270fe35ed59b7d28f3fd9b7e41) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-09bitbake: asyncrpc: Abstract socketsJoshua Watt
Rewrites the asyncrpc client and server code to make it possible to have other transport backends that are not stream based (e.g. websockets which are message based). The connection handling classes are now shared between both the client and server to make it easier to implement new transport mechanisms (Bitbake rev: 2aaeae53696e4c2f13a169830c3b7089cbad6eca) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12bitbake: bitbake: Add copyright headers where missingRichard Purdie
Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-02bitbake: prserv: make localhost workMingli Yu
After [1] introduced, the PR server doesn't work in docker when use below setting like before. PRSERV_HOST = "localhost:0" And it's because the localhost is resolved to an ipv6 address ::1 as the below bitbake-prserv shows. bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address So add the extra logic to make the localhost resolved as expected to make the PR service work especially in docker. [1] 6a2b23e2 prserv: Replace XML RPC with modern asyncrpc implementation (Bitbake rev: 0a11696e0898c3c5108e6d7c5ad28da50e00ea66) Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-01bitbake: prserv: handle PRSERV_HOST = "127.0.0.1:0" the same as "localhost:0"Martin Jansa
* When using PRSERV_HOST = "localhost:0" inside Docker container (tested with ubuntu 20.04 and 21.04) the self.loop.run_until_complete never completed, so self.address wasn't ever assigned few lines bellow and then self.port = int(self.prserv.address.rsplit(':', 1)[1]) in lib/prserv/serv.py caused a bit ugly exception: bitbake@599696cd20aa:~/nodistro/honister$ bitbake -k pkgconfig-native Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake", line 35, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "/OE/nodistro/honister/bitbake/lib/bb/main.py", line 385, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "/OE/nodistro/honister/bitbake/lib/bb/ui/knotty.py", line 397, in main params.updateToServer(server, os.environ.copy()) File "/OE/nodistro/honister/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer raise Exception("Unable to update the server configuration with local parameters: %s" % error) Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 286, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 491, in updateConfigOpts self.reset() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 1717, in reset self.handlePRServ() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 383, in handlePRServ self.prhost = prserv.serv.auto_start(self.data) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 318, in auto_start singleton.start() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 133, in start self.port = int(self.prserv.address.rsplit(':', 1)[1]) AttributeError: 'NoneType' object has no attribute 'rsplit' * the issue was caused by "localhost" being resolved as IPv6 address ::1 and then asyncio failing to bind it, the same is reproducible with hashserv, but hashserve at least shows nice error message: bitbake$ bitbake-hashserv -l DEBUG -b localhost:0 Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 59, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 53, in main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address * or by bitbake-prserv in prserv.log: bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address * while 127.0.0.1 works fine: bitbake$ bitbake-prserv --start --host=127.0.0.1 --port=42005 bitbake$ cat prserv.log DEBUG: Listening on ('127.0.0.1', 42005) 2021-08-26 22:28:05,828 Listening on ('127.0.0.1', 42005) DEBUG: Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' 2021-08-26 22:28:05,829 Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' NOTE: Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 2021-08-26 22:28:05,831 Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 but 127.0.0.1:0 wasn't handled as "autostart" like localhost:0 is update is_local_special to allow that * /etc/hosts file generated by docker contails localhost for both IPv4 and IPv6: $ grep localhost /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback even when ipv6 is disabled in dockerd as reported in: https://github.com/docker/for-linux/issues/250 * add a check for self.prserv.address to provide better error message: ERROR: Unable to start PR Server, exitting when something bad happens, but in this case you still need to read bitbake-cookerdaemon.log to see the actuall error, in this case: 90 22:30:39.008441 --- Starting bitbake server pid 90 at 2021-08-26 22:30:39.008419 --- 90 22:30:39.023734 Started bitbake server pid 90 90 22:30:39.024286 Entering server connection loop 90 22:30:39.024753 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([]) 90 22:30:39.026314 Processing Client 90 22:30:39.026456 Connecting Client 90 22:30:39.027509 Running command ['setFeatures', [2]] 90 22:30:39.027757 Command Completed 90 22:30:39.028711 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'DISTRO': '', 'PWD': '/OE/nodistro/honister', 'HOME': '/OE', 'MACHINE': 'qemux86', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'PATH': '/OE/nodistro/honister/oe-core/scripts:/OE/nodistro/honister/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'en_US.UTF-8', 'MACHINES': 'qemux86', 'HOSTNAME': '6a439759e3c6', 'TOPDIR': '/OE/nodistro/honister', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHLVL': '1', 'BITBAKE_HOME': '/OE', 'BUILDDIR': '/OE/nodistro/honister/BUILD', 'OLDPWD': '/OE/nodistro/honister/bitbake', '_': '/OE/nodistro/honister/bitbake/bin/bitbake'}, ['/OE/nodistro/honister/bitbake/bin/bitbake', '-k', 'zlib-native']] Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 255, in run self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address 90 22:30:39.530037 Command Completed 90 22:30:39.530913 Processing Client 90 22:30:39.531023 Disconnecting Client 90 22:30:39.531638 No timeout, exiting. 90 22:30:39.632137 Exiting 90 22:30:39.637562 Original lockfile contents: ['90\n'] 90 22:30:39.638107 Exiting as we could obtain the lock (Bitbake rev: c2cdda0e5fc720c60d3b8537fc702cb118981bd2) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-23bitbake: prserv: Add read-only modePaul Barker
[YOCTO #13659] (Bitbake rev: 44287430b9804fcbf2440f85a2424792140e4dc9) Signed-off-by: Paul Barker <pbarker@konsulko.com> [updated for asyncrpc changes] Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-23bitbake: prserv: Replace XML RPC with modern asyncrpc implementationPaul Barker
Update the prserv client and server classes to use the modern json and asyncio based RPC system implemented by the asyncrpc module. (Bitbake rev: 6a2b23e27bb61185b8afb382e20ce79f996d9183) Signed-off-by: Paul Barker <pbarker@konsulko.com> [updated for asyncrpc changes, client split to separate file] Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06bitbake: prserv: Handle requests in main threadPaul Barker
The prserver process is cleanly separated from the main bitbake process so requests can be handled in the main thread. This removes the need for a request queue and a separate request handling thread. (Bitbake rev: 6b09415bed6b5e7c12aaf39b677d9ef72844e233) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06bitbake: prserv: Extract daemonization from PRServer classPaul Barker
The code to start the prservice process as a daemon is extracted out of the PRServer class and simplified. This makes the PRServer class easier to modernise as it no longer needs to worry about process management. (Bitbake rev: 39c7c158c52157b18f5ccbbd673e3298e6402f52) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06bitbake: prserv: Use multiprocessing to auto start prserverPaul Barker
We can use the modern multiprocessing support in Python instead of manually using fork to start the prserver process. To do this we need to set up the signal handlers for the prserver process in the work_forever function (which is now used as the main function for this process). The old code to start the prserver process using fork is not removed in this commit as it is tightly intertwined with the daemonization code which will be refactored in a following commit. (Bitbake rev: b3da56240c0f92efab1c0b293738c35c0f1ee6ab) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27bitbake: prserv: Add connect functionPaul Barker
This function abstracts the setup of a PR service client connection so that openembedded-core doesn't need to be updated any time the details are changed. (Bitbake rev: d892287b31f81b075983ba500be265f75b53df64) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27bitbake: prserv: Drop unused dump_db methodPaul Barker
(Bitbake rev: ecb7bf34eac02ff58dbc27b3768ceaf4adb1c9cd) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27bitbake: prserv: Drop obsolete python version checkPaul Barker
Bitbake no longer supports Python 2 so this version check is obsolete. (Bitbake rev: 45eb6c6e124e507012df9c288f1fbde0e7899e5d) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19bitbake: lib: amend code to use proper singleton comparisons where possibleFrazer Clews
amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. (Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-30bitbake: prserv/serv: Use with while reading pidfileOla x Nilsson
(Bitbake rev: 6fa8a18ea4994031fdd1253fe363c5d8eeeba456) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-25bitbake: prserv/serv: Only restart the server if settings changeRichard Purdie
The server is now restarting when running commands which doesn't make sense. Only restart if its configuration has changed. This should potentially fix various memory resident bitbake usages too. (Bitbake rev: 0d2c67abf8c92386802eccfbb6b124dd65597941) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14bitbake: prserv: fix ResourceWarning due to unclosed socketGavin Li
With PRSERV_HOST = "localhost:0", this message would occasionally pop up during the initial cache read: WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)> value = pickled.load() The file location stated is irrelevant; it just happens to be wherever CPython decides to run the garbage collector. The issue is that after we fork off a PRServer, self.socket is also duplicated. The parent side of it also needs to be closed. (Bitbake rev: c45b0140c37f2d68c4fa08cd17f0321b8d6524a8) Signed-off-by: Gavin Li <gavin@matician.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-28bitbake: bitbake: prserv/serv: Use with while reading pidfileOla x Nilsson
(Bitbake rev: 405a9068ce5072050b9878f7cec15b2c2592bc4e) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06bitbake: prserv: Use a memory journalRichard Purdie
We've seen PR Server timeouts on the autobuilder, this is likely from the journal being blocked on disk IO generated by the build. Since we're running with synchronous off, we may as well put the journal into memory and avoid any IO related stalls. (Bitbake rev: ee3fc6030e653f3244b065fc89aafd2a7c36ae04) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie
This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31bitbake: prserv/serv: Gracefully handle the PR server exiting quicklyRichard Purdie
If the server exits quickly its PID may no longer exist. Handle this gracefully. (Bitbake rev: c1b00a9265fa4146b8db8b7d03a51bf2bfcf9f51) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31bitbake: prserv/serv: Rename self.quit -> self.quitflagRichard Purdie
self has a quit function and a variable. Separate this to two different things as the current setup is prone to breakage. (Bitbake rev: ba7e3c73d8f4d2bd1d7434b97c326e7ab935231a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31bitbake: prserv/serv: Send sentinel to stop handler threadRichard Purdie
Shutdown from SIGTERM currently has to wait for the handler thread to timeout. Add a sentinel value which triggers it to loop and allows for a quick exit. (Bitbake rev: a7591ef34ce70ff1d7aa9362d7473e6f16fbd10f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31bitbake: prserv/serv: Shut down any existing server before restartingRichard Purdie
This allows for cleaner code in cooker as any existing server is dealt with before a new one is started. (Bitbake rev: b8616931bc0e523a3a3bb23b4f623f8b6e71d690) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31bitbake: prserv/cooker: Drop unused paramRichard Purdie
Drop pointless unused function parameter. (Bitbake rev: 8104b33656de0b619943bd7a9884eb650ccafbf4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-24bitbake: prserv/serv: Improve process exit handlingRichard Purdie
The server shutdown is currenlty laggy and race prone. This patch: * adds a waitpid so that no zombie server is left around if its not running in daemon mode. * adds a quit "sentinal" using a pipe so that we're not sitting in a socket poll() until timeout in order just to quit. * use a select() call to poll the socket and the pipe for a quit signal. The net result of this change is that the prserv exits with the cooker server and it does so immediately and doesn't wait for the select/poll calls to timeout. This makes bitbake a lot more responsive for startup/shutdown and doesn't cause UI timeout errors as often when prserv is used. (Bitbake rev: 0b5a837477d18442caf97dd0fa14a806c54f5842) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21bitbake: server: Rework the server API so process and xmlrpc servers coexistRichard Purdie
This changes the way bitbake server works quite radically. Now, the server is always a process based server with the option of starting an XMLRPC listener on a specific inferface/port. Behind the scenes this is done with a "bitbake.sock" file alongside the bitbake.lock file. If we can obtain the lock, we know we need to start a server. The server always listens on the socket and UIs can then connect to this. UIs connect by sending a set of three file descriptors over the domain socket, one for sending commands, one for receiving command results and the other for receiving events. These changes meant we can throw away all the horrid server abstraction code, the plugable transport option to bitbake and the code becomes much more readable and debuggable. It also likely removes a ton of ways you could hang the UI/cooker in weird ways due to all the race conditions that existed with previous processes. Changes: * The foreground option for bitbake-server was dropped. Just tail the log if you really want this, the codepaths were complicated enough without adding one for this. * BBSERVER="autodetect" was dropped. The server will autostart and autoconnect in process mode. You have to specify an xmlrpc server address since that can't be autodetected. I can't see a use case for autodetect now. * The transport/servetype option to bitbake was dropped. * A BB_SERVER_TIMEOUT variable is added which allows the server to stay resident for a period of time after the last client disconnects before unloading. This is used if the -T/--idle-timeout option is not passed to bitbake. This change is invasive and may well introduce new issues however I believe the codebase is in a much better position for further development and debugging. (Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09bitbake: prserv/persist_data/utils: Drop obsolete python2 importsRichard Purdie
These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. (Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-05bitbake: prserv/serv: Tweak stdout manipulation to be stream safeRichard Purdie
We've been seeing oe-selftest failures under puzzling circumstances. It turns out if you run oe-selftest on a machine with xmlrunner installed and have the recent tinfoil2 changes, the launching of PR server crashes leading to selftest hanging if using an autoloaded PR server. The reason is that xmlrunner uses an io.StringIO object as stdout/stderr instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method. We have to deal with both cases and in the python way, we try and then seek forgivness if we see an AttributeError or UnSupportedOperation exception. Unfortunately we have to deal with both cases as we may be performing a traditiional double fork() from the commandline, or a larger python program. [YOCTO #10866] (Bitbake rev: 26243f04e3af652291d13e85c084057104fe155b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: bitbake: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02bitbake: bitbake: Convert to python 3Richard Purdie
Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13bitbake: daemonize/prserv/tests/fetch: Convert file() -> open()Richard Purdie
Use python3 compatible functions. (Bitbake rev: e6a0296ba29c3fbc8417d1df7a01d50562668a41) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26bitbake: prserv: Add dump_db()Haris Okanovic
Returns a script (string) that reconstructs the state of the entire database at the time this function is called. The script language is defined by the backing database engine, which is a function of server configuration. Returns None if the database engine does not support dumping to script or if some other error is encountered in processing. The SQLite3 implementation in db.py calls iterdump() [1] to generate a script. iterdump() is the library equivalent of the `sqlite3 .dump` shell command, and the scripts are compatible. Execute the script in an empty SQLite3 database using the sqlite3 utility to restore a backup of prserv. Use case: Backup a live PR server database in a non-racy way, such that one could snapshot the entire database after a set of bitbake builds all using a shared server. I.e. All changes made prior to the start of a dump_db() operation should be committed and captured in the script. Subsequent changes made during the backup process are not guaranteed to be captured. Testing: ~7MB database backs up in ~1s while PR server is under load from 32 thread bitbake builds on two separate machines. [1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump (Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3) Signed-off-by: Haris Okanovic <haris.okanovic@ni.com> Reviewed-by: Ken Sharp <ken.sharp@ni.com> Reviewed-by: Bill Pittman <bill.pittman@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04bitbake: bitbake: prserv: do not clear umask when daemonizingDiego Santa Cruz
Clearing the umask when daemonizing is not the correct thing to do, as it will create files writable by anyone by default. For instance the pid file was being created with mode 777. This could also potentially affect the sqlite database. Better let the calling process decide on the umask. [YOCTO #9036] (Bitbake rev: ff6d3f53a4504eae7ec4c190b9f7595b09aed017) Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04bitbake: bitbake: prserv: SIGTERM handling hung processDiego Santa Cruz
The current SIGTERM handler hungs the process instead of making it exit. The problem seems to be that the handler thread is not signaled to quit, so it stays there doing its work, as it is not a daemon thread. Setting the quit variable fixes this. While at it, to not use the SystemExit exception to terminate upon SIGTERM but instead left the quit flag do its job. This way the PID file is properly removed. [YOCTO #9035] (Bitbake rev: 655ec800d54da581229f12efb6f0baf54975fed4) Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04bitbake: bitbake: prserv: -wal and -shm sqlite lost when daemonizingDiego Santa Cruz
When daemonizing the PR service the -wal and -shm sqlite files were being deleted, although they should never be. While the daemonized process keeps the file descriptors open and thus a clean exit does not loose any data, a power outage would loose all data in the WAL. Removing these files also breaks sqlite collaboration between processes and furthermore prevents taking proper backups without stopping the PR service. The reason this happens is that the DB connection is opened in the initial process, before forking for daemonization. When the DB connection is closed by the exiting parent processes it can delete the -wal and -shm files if it considers itself to be the last connection to the DB. This is easily fixed by opening the DB connection after all forking. [YOCTO #9034] (Bitbake rev: bc867c81e3894da5bdd2e45fa695bb5f5f1bb58b) Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30bitbake: bitbake: Set process names to be meaninfulRichard Purdie
This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28bitbake: prserv/serv.py: Better messaging when starting/stopping the server ↵Leonardo Sandoval
with port=0 When starting the server using port=0, the server actually starts with a different port, so print a message with this new value. When stopping the server with port=0, advise the user which ports the server is listening to, so next time it tries to close it, user can pick up the correct one. [YOCTO #8560] (Bitbake rev: 851e53a216682fc9133f51c05a24527cfc677741) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28bitbake: prserv/serv: Close the DB connection out of class destructorLeonardo Sandoval
When launching the PR server daemon, the PRData __del__ function was being called (no reason found yet) where the DB connection closed, thus following PR updates were not getting into the DB. This patch closes the connection explicitly, not relaying on the __del__ function execution. Closing the connection in turn causes all WAL file transactions to be moved into the database (checkpoint), thus effectively updating the database. [YOCTO #8215] (Bitbake rev: c1b4754f69003df1a83fafc1c80a9ef74400b6dd) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-24bitbake: prserv/serv: Start/Stop daemon using ip instead of hostLeonardo Sandoval
In cases where hostname is given instead of an IP (i.e. localhost instead of 127.0.0.1) when stopping the server with bitbake-prserv --stop, the server shows a misleading message indicating that the daemon was not found, where it is actually stopped. This patch converts host to IP values before starting/stopping the daemon, so it will always work on IP, not on hostnames, avoiding problems like the latter. [YOCTO #8258] (Bitbake rev: bd6398e967c234e89d773f509512ebf460fa76ff) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-24 13:51:26 +0000