summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/persist_data.py
AgeCommit message (Collapse)Author
2020-08-26bitbake: compat.py: remove file since it no longer actually implements anythingChris Laplante
Now that compat.py just imports Python standard library stuff, get rid of the layer of indirection. (Bitbake rev: e2be6defbb9fcf25f9df04c3b452d0dba48dfd03) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19bitbake: persist_data.py: Immediately get exclusive lock in __setitem__Chris Laplante
To avoid races, SQLTable::__setitem__ needs an exclusive lock for the entire transaction, not just the INSERT/UPDATE part. (Bitbake rev: feb43e7c30f5bfab75d718896c45df621810d06f) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie
With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie
This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08bitbake: bitbake: persist_data: Fix Locking Protocol ErrorJoshua Watt
Under heavy load with process delays, sqlite can issues a "locking protocol" error (SQLITE_PROTOCOL). Unfortunately, it is impossible to distinguish between actual locking protocol errors and this race condition, so they best that can be done is to retry the operation when the error is detected. [YOCTO #13108] (Bitbake rev: 93cd15644f9d12b38abea276fee7b5bade0276df) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08bitbake: bitbake: persist_data: Retry database setupJoshua Watt
The configuration of the sqlite database can timeout due to locking under heavy load and should be subject to the same retry logic as the other statements. [YOCTO #13069] (Bitbake rev: 5a2a95b0396e39662968690b3065d2f88167a71c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-07bitbake: persist_data: Disable enable_shared_cacheJoshua Watt
Turns off the shared cache. It isn't a significant factor in performance (now that WAL is enabled), and is a really bad idea to have enabled in processes that fork() (as bitbake it prone to do). [YOCTO #13030] (Bitbake rev: 71b427bf01374973a971c10cb64024c8ef2a11eb) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-07bitbake: persist_data: Enable Write Ahead LogJoshua Watt
Enabling the write ahead log improves database reliability, speeds up writes (since they mostly happen sequentially), and speeds up readers (since they are no longer blocked by most write operations). The persistent database is very read heavy, so the auto-checkpoint size is reduced from the default (usually 1000) to 100 so that reads remain fast. [YOCTO #13030] (Bitbake rev: 79100fa67539f9654af9bf6d3e6842eb5c12e989) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-07bitbake: persist_data: Add key constraintsJoshua Watt
Constructs the "key" column in the persistent database as a non-NULL primary key. This significantly speeds up lookup operations in large databases. [YOCTO #13030] (Bitbake rev: f5ba7775cfcb90401522d977cc66fe0f5aeb7a66) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-07bitbake: persist_data: Fix leaking cursors causing deadlockJoshua Watt
The original implementation of persistent data executed all SQL statements via sqlite3.Connection.execute(). Behind the scenes, this function created a sqlite3 Cursor object, executed the statement, then returned the cursor. However, the implementation did not account for this and failed to close the cursor object when it was done. The cursor would eventually be closed when the garbage collector got around to destroying it. However, sqlite has a limit on the number of cursors that can exist at any given time, and once this limit is reached it will block a query to wait for a cursor to be destroyed. Under heavy database queries, this can result in Python deadlocking with itself, since the SQL query will block waiting for a free cursor, but Python can no longer run garbage collection (as it is blocked) to free one. This restructures the SQLTable class to use two decorators to aid in performing actions correctly. The first decorator (@retry) wraps a member function in the retry logic that automatically restarts the function in the event that the database is locked. The second decorator (@transaction) wraps the function so that it occurs in a database transaction, which will automatically COMMIT the changes on success and ROLLBACK on failure. This function additionally creates an explicit cursor, passes it to the wrapped function, and cleans it up when the function is finished. Note that it is still possible to leak cursors when iterating. This is much less frequent, but can still be mitigated by wrapping the iteration in a `with` statement: with db.iteritems() as it: for (k, v) in it: ... As a side effect, since most statements are wrapped in a transaction, setting the isolation_level when the connection is created is no longer necessary. [YOCTO #13030] (Bitbake rev: e8b9d3f534ef404780be23b601d5a4bb9cec928a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09bitbake: prserv/persist_data/utils: Drop obsolete python2 importsRichard Purdie
These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. (Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: bitbake: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11bitbake: persist_data: Fix py3 update stack overflowRichard Purdie
Revision d0f904d407f57998419bd9c305ce53e5eaa36b24 accidentally broke items() and values() and made them cause stack overflows. Undo that breakage. (Bitbake rev: 88c5beca705efa7df4a96fb2aaf3f13c336ac328) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02bitbake: bitbake: Convert to python 3Richard Purdie
Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06bitbake: persist_data: Return str instead of unicode for sqlite3 text queriesKlauer, Daniel
Python 2's sqlite3 module defaults to returning Unicode strings for SQL text queries, which could trickle down to other parts of bitbake code and cause unexpected Unicode conversions. Using byte strings avoids this issue. For example, the git fetcher's AUTOREV support caches HEAD SHA1's using bb.persist_data, so sometimes the git command strings passed to fetch2's runfetchcmd() were unicode, potentially causing UnicodeDecodeErrors when it appended the values of environment variables containing non-ASCII chars. [YOCTO #9382] (Bitbake rev: 09623a0811c613a47a01ae465b822d8156faca30) Signed-off-by: Daniel Klauer <daniel.klauer@gin.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-05-03bitbake: persist_data: Avoid fsync() callsRichard Purdie
If the power were to fail, it doesn't matter to us much if the data makes it to disk or not, we'd have other problems. However an fsync() call on a multi build autobuilder is painful so lets avoid them. This is particularly true in this case if a timeout causes a reconnect during a build. (Bitbake rev: ec28256ac2a30f047585e8f61200d764bc295ded) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-22bitbake: persist_data: add get_by_pattern method to APIConstantin Musca
- one can use get_by_pattern to get a list of values associated with keys that match the specified pattern (Bitbake rev: 6ee1f58698e2d782c54ce5aec271bcec26107eac) Signed-off-by: Constantin Musca <constantinx.musca@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-22persist_data: survive if sqlite3 couldn't enable the shared cacheChristopher Larson
(Bitbake rev: e870fd57bb84a17a871a5b5de3bf5202fe9b135f) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-19bitbake/persist_data: Reconnect when DB is lockedLianhao Lu
[YOCTO #1761] Reconnect to the backend Sqlite DB in 'database is locked' exception so the timeout can be leveraged in each time retry. (Bitbake rev: b310382764367b573c84f33d847c6eb821266f9e) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-27bitbake: Update users of getVar/setVar to use the data store functions directlyRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06bitbake/persist_data: Sync file with upstream bitbakeRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06persist_data: implement comparison, same as dictChris Larson
(Bitbake rev: 1190406c526c7bb7cf415867be83e0403812a7dd) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06persist_data: don't allow non-string keys/valuesChris Larson
(Bitbake rev: 28958cd55e592853c68f5f2ba79381d1b8dcfb8f) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06persist_data: add has_keyChris Larson
(Bitbake rev: 7942833ca0685cf4f3b243dde6203499ef97420c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06persist_data: raise KeyError on missing elementsChris Larson
(Bitbake rev: a4f62433845c29f98c6a9746d5d2847bf9506ea5) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06Initial work on getting bitbake working under pypyChris Larson
- use os.chmod, not os.fchmod, as the latter is missing under pypy - rearrange our imports a bit - don't die if sqlite3 is missing shared cache support (Bitbake rev: f229824dc9c453adf6067500e2bf6761536e4f2f) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-10persist_data: kill unreachable break lineChris Larson
(Bitbake rev: 7486b38603f2766adaf976a9f95e9276c83abe31) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-10persist_data: loop on database lock for table creationChris Larson
(Bitbake rev: d93fcbd64ab5d806288424170f55323b4297e7d6) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-05persist_data: resurrect the lock wait for selectsChris Larson
Think this got inadvertantly dropped when switching to the new API. (Bitbake rev: 628c5159d1151b89f2b7210c8819489e8dc9a84d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-05Rework the persist_data APIChris Larson
Rather than having to run .addDomain() and then .getValue(domain, key), .setValue(domain, key), etc, now it just works as mappings. As an example: setValue(domain, key) -> persist[domain][key] = value It also arranges things so we can construct objects of this type using any arbitrary filename/path for the sqlite3 database, rather than being so tightly bound to the metadata. (Bitbake rev: d9e8b8af308ae871efdc8ef0782be30af8c1f894) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-05Revert "persist_data: cache connection and use cursor"Chris Larson
Caching the database connection can cause serious issues if it results in multiple processes (e.g. multiple tasks) simultaneously using the same connection. This reverts commit 8a6876752b90efd81d92f0947bfc9527d8260969. (Bitbake rev: 60b9b18eafad5ac46c7cf1048d749d673c2ee0ad) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04persist_data: handle locked db for SELECTChris Larson
Parallel processes interacting with the persist_data db can quite easily explode without this. (Bitbake rev: b3d5432cff0ff28f4c8a5bcf10efa3e383b4fd4d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04Switch bitbake internals to use logging directly rather than bb.msgChris Larson
We use a custom Logger subclass for our loggers This logger provides: - 'debug' method which accepts a debug level - 'plain' method which bypasses log formatting - 'verbose' method which is more detail than info, but less than debug (Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-20bitbake/persist_data: Attempt to fix locking issuesRichard Purdie
It appears the timeout sometimes has no effect and we see database access failures. Combat this by wrapping the execute function in all cases and retrying manually ourselves. Thanks to Kevin Tian for help debugging this. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02persist_data: cache connection and use cursorBernhard Reutner-Fischer
Store database connection to persistent database in fetcher. (Bitbake rev: 8a6876752b90efd81d92f0947bfc9527d8260969) Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Apply some 2to3 transforms that don't cause issues in 2.6Chris Larson
(Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Switch some references to moved functionsChris Larson
(Bitbake rev: ddea2978cb969cf1381cfee5c055de0c9e56a4e3) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Formatting cleanupsChris Larson
(Bitbake rev: 2caf134b43a44dad30af4fbe33033b3c58deee57) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2009-07-30bitbake: Remove persist_data domain renaming code as it appears to expose ↵Richard Purdie
sqlite bugs and data consistency issues Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2009-07-30bitbake: Make sure tables exist when renamingRichard Purdie
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2009-07-29bitbake: Add renameDomain and getKeyValues functions to the persistent data codeRichard Purdie
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2008-05-13bitbake: Sync with 1.8 branchRichard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@4463 311d38ba-8fff-0310-9ca6-ca027cbcb966
2008-01-20bitbake: Update to 1.8.10 releaseRichard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@3555 311d38ba-8fff-0310-9ca6-ca027cbcb966
2007-08-04bitbake persist_data: Fix for python 2.4Richard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@2366 311d38ba-8fff-0310-9ca6-ca027cbcb966
2007-08-04bitbake persist_data: Fix for python 2.4Richard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@2365 311d38ba-8fff-0310-9ca6-ca027cbcb966
2007-08-04bitbake persist_data: Fix module referenceRichard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@2364 311d38ba-8fff-0310-9ca6-ca027cbcb966
2007-08-04bitbake persist_data: Add own retry logicRichard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@2363 311d38ba-8fff-0310-9ca6-ca027cbcb966
2007-08-04bitbake persist_data: Use immediate connection modeRichard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@2362 311d38ba-8fff-0310-9ca6-ca027cbcb966
2007-08-03bitbake: Update along 1.8 branchRichard Purdie
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@2345 311d38ba-8fff-0310-9ca6-ca027cbcb966
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-23 06:43:17 +0000