aboutsummaryrefslogtreecommitdiffstats
path: root/scripts/gen_compile_commands.py
AgeCommit message (Collapse)Author
2020-08-27Makefile: Add clang-tidy and static analyzer support to makefileNathan Huckleberry
This patch adds clang-tidy and the clang static-analyzer as make targets. The goal of this patch is to make static analysis tools usable and extendable by any developer or researcher who is familiar with basic c++. The current static analysis tools require intimate knowledge of the internal workings of the static analysis. Clang-tidy and the clang static analyzers expose an easy to use api and allow users unfamiliar with clang to write new checks with relative ease. ===Clang-tidy=== Clang-tidy is an easily extendable 'linter' that runs on the AST. Clang-tidy checks are easy to write and understand. A check consists of two parts, a matcher and a checker. The matcher is created using a domain specific language that acts on the AST (https://clang.llvm.org/docs/LibASTMatchersReference.html). When AST nodes are found by the matcher a callback is made to the checker. The checker can then execute additional checks and issue warnings. Here is an example clang-tidy check to report functions that have calls to local_irq_disable without calls to local_irq_enable and vice-versa. Functions flagged with __attribute((annotation("ignore_irq_balancing"))) are ignored for analysis. (https://reviews.llvm.org/D65828) ===Clang static analyzer=== The clang static analyzer is a more powerful static analysis tool that uses symbolic execution to find bugs. Currently there is a check that looks for potential security bugs from invalid uses of kmalloc and kfree. There are several more general purpose checks that are useful for the kernel. The clang static analyzer is well documented and designed to be extensible. (https://clang-analyzer.llvm.org/checker_dev_manual.html) (https://github.com/haoNoQ/clang-analyzer-guide/releases/download/v0.1/clang-analyzer-guide-v0.1.pdf) The main draw of the clang tools is how accessible they are. The clang documentation is very nice and these tools are built specifically to be easily extendable by any developer. They provide an accessible method of bug-finding and research to people who are not overly familiar with the kernel codebase. Signed-off-by: Nathan Huckleberry <nhuck@google.com> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Lukas Bulwahn <lukas.bulwahn@gmail.com> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
2020-08-27gen_compile_commands: remove the warning about too few .cmd filesMasahiro Yamada
This warning was useful when users previously needed to manually build the kernel and run this script. Now you can simply do 'make compile_commands.json', which updates all the necessary build artifacts and automatically creates the compilation database. There is no more worry for a mistake like "Oh, I forgot to build the kernel". Now, this warning is rather annoying. You can create compile_commands.json for an external module: $ make M=/path/to/your/external/module compile_commands.json Then, this warning is displayed since there are usually less than 300 files in a single module. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-27gen_compile_commands: support *.o, *.a, modules.order in positional argumentMasahiro Yamada
This script currently searches the specified directory for .cmd files. One drawback is it may contain stale .cmd files after you rebuild the kernel several times without 'make clean'. This commit supports *.o, *.a, and modules.order as positional parameters. If such files are given, they are parsed to collect associated .cmd files. I added a generator helper for each of them. This feature is useful to get the list of active .cmd files from the last build, and will be used by the next commit to wire up the compile_commands.json rule to the Makefile. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-27gen_compile_commands: move directory walk to a generator functionMasahiro Yamada
Currently, this script walks under the specified directory (default to the current directory), then parses all .cmd files found. Split it into a separate helper function because the next commit will add more helpers to pick up .cmd files associated with given file(s). There is no point to build and return a huge list at once. I used a generator so it works in the for-loop with less memory. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-27gen_compile_commands: make -o option independent of -d optionMasahiro Yamada
Change the -o option independent of the -d option, which is I think clearer behavior. Some people may like to use -d to specify a separate output directory, but still output the compile_commands.py in the source directory (unless the source tree is read-only) because it is the default location Clang Tools search for the compilation database. Also, move the default parameter to the default= argument of the .add_argument(). Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-27gen_compile_commands: reword the help message of -d optionMasahiro Yamada
I think the help message of the -d option is somewhat misleading. Path to the kernel source directory to search (defaults to the working directory) The part "kernel source directory" is the source of the confusion. Some people misunderstand as if this script did not support separate output directories. Actually, this script also works for out-of-tree builds. You can use the -d option to point to the object output directory, not to the source directory. It should match to the O= option used in the previous kernel build, and then appears in the "directory" field of compile_commands.json. Reword the help message. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-27gen_compile_commands: do not support .cmd files under tools/ directoryMasahiro Yamada
The tools/ directory uses a different build system, and the format of .cmd files is different because the tools builds run in a different work directory. Supporting two formats compilicates the script. The only loss by this change is objtool. Also, rename the confusing variable 'relative_path' because it is not necessarily a relative path. When the output directory is not the direct child of the source tree (e.g. O=foo/bar), it is an absolute path. Rename it to 'file_path'. os.path.join(root_directory, file_path) works whether the file_path is relative or not. If file_path is already absolute, it returns it as-is. I used os.path.abspath() to normalize file paths. If you run this script against the kernel built with O=foo option, the file_path contains '../' patterns. os.path.abspath() fixes up 'foo/bar/../baz' into 'foo/baz', and produces a cleaner commands_database.json. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-27gen_compile_commands: use choices for --log_levels optionMasahiro Yamada
Use 'choices' to check if the given parameter is valid. I also simplified the help message because, with 'choices', --help shows the list of valid parameters: --log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL} I started the help message with a lower case, "the level of log ..." in order to be consistent with the -h option: -h, --help show this help message and exit The message "show this help ..." comes from the ArgumentParser library code, and I do not know how to change it. So, I changed our code. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-27gen_compile_commands: parse only the first line of .*.cmd filesMasahiro Yamada
After the allmodconfig build, this script takes about 5 sec on my machine. Most of the run-time is consumed for needless regex matching. We know the format of .*.cmd file; the first line is the build command. There is no need to parse the rest. With this optimization, now it runs 4 times faster. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Nick Desaulniers <ndesaulniers@google.com>
2019-07-27gen_compile_commands: lower the entry count thresholdMasahiro Yamada
Running gen_compile_commands.py after building the kernel with allnoconfig gave this: $ ./scripts/gen_compile_commands.py WARNING: Found 449 entries. Have you compiled the kernel? Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-19scripts: add a tool to produce a compile_commands.json fileTom Roeder
The LLVM/Clang project provides many tools for analyzing C source code. Many of these tools are based on LibTooling (https://clang.llvm.org/docs/LibTooling.html), which depends on a database of compiler flags. The standard container for this database is compile_commands.json, which consists of a list of JSON objects, each with "directory", "file", and "command" fields. Some build systems, like cmake or bazel, produce this compilation information directly. Naturally, Makefiles don't. However, the kernel makefiles already create .<target>.o.cmd files that contain all the information needed to build a compile_commands.json file. So, this commit adds scripts/gen_compile_commands.py, which recursively searches through a directory for .<target>.o.cmd files and extracts appropriate compile commands from them. It writes a compile_commands.json file that LibTooling-based tools can use. By default, gen_compile_commands.py starts its search in its working directory and (over)writes compile_commands.json in the working directory. However, it also supports --output and --directory flags for out-of-tree use. Note that while gen_compile_commands.py enables the use of clang-based tools, it does not require the kernel to be compiled with clang. E.g., the following sequence of commands produces a compile_commands.json file that works correctly with LibTooling. make defconfig make scripts/gen_compile_commands.py Also note that this script is written to work correctly in both Python 2 and Python 3, so it does not specify the Python version in its first line. For an example of the utility of this script: after running gen_compile_commands.json on the latest kernel version, I was able to use Vim + the YouCompleteMe pluging + clangd to automatically jump to definitions and declarations. Obviously, cscope and ctags provide some of this functionality; the advantage of supporting LibTooling is that it opens the door to many other clang-based tools that understand the code directly and do not rely on regular expressions and heuristics. Tested: Built several recent kernel versions and ran the script against them, testing tools like clangd (for editor/LSP support) and clang-check (for static analysis). Also extracted some test .cmd files from a kernel build and wrote a test script to check that the script behaved correctly with all permutations of the --output and --directory flags. Signed-off-by: Tom Roeder <tmroeder@google.com> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>