1. 19 Dec, 2018 1 commit
    • Tom Roeder's avatar
      scripts: add a tool to produce a compile_commands.json file · b3020464
      Tom Roeder authored
      The LLVM/Clang project provides many tools for analyzing C source code.
      Many of these tools are based on LibTooling
      (https://clang.llvm.org/docs/LibTooling.html), which depends on a
      database of compiler flags. The standard container for this database is
      compile_commands.json, which consists of a list of JSON objects, each
      with "directory", "file", and "command" fields.
      
      Some build systems, like cmake or bazel, produce this compilation
      information directly. Naturally, Makefiles don't. However, the kernel
      makefiles already create .<target>.o.cmd files that contain all the
      information needed to build a compile_commands.json file.
      
      So, this commit adds scripts/gen_compile_commands.py, which recursively
      searches through a directory for .<target>.o.cmd files and extracts
      appropriate compile commands from them. It writes a
      compile_commands.json file that LibTooling-based tools can use.
      
      By default, gen_compile_commands.py starts its search in its working
      directory and (over)writes compile_commands.json in the working
      directory. However, it also supports --output and --directory flags for
      out-of-tree use.
      
      Note that while gen_compile_commands.py enables the use of clang-based
      tools, it does not require the kernel to be compiled with clang. E.g.,
      the following sequence of commands produces a compile_commands.json file
      that works correctly with LibTooling.
      
      make defconfig
      make
      scripts/gen_compile_commands.py
      
      Also note that this script is written to work correctly in both Python 2
      and Python 3, so it does not specify the Python version in its first
      line.
      
      For an example of the utility of this script: after running
      gen_compile_commands.json on the latest kernel version, I was able to
      use Vim + the YouCompleteMe pluging + clangd to automatically jump to
      definitions and declarations. Obviously, cscope and ctags provide some
      of this functionality; the advantage of supporting LibTooling is that it
      opens the door to many other clang-based tools that understand the code
      directly and do not rely on regular expressions and heuristics.
      
      Tested: Built several recent kernel versions and ran the script against
      them, testing tools like clangd (for editor/LSP support) and clang-check
      (for static analysis). Also extracted some test .cmd files from a kernel
      build and wrote a test script to check that the script behaved correctly
      with all permutations of the --output and --directory flags.
      Signed-off-by: default avatarTom Roeder <tmroeder@google.com>
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      b3020464
  2. 16 Dec, 2018 10 commits
  3. 10 Dec, 2018 1 commit
  4. 08 Dec, 2018 2 commits
  5. 02 Dec, 2018 1 commit
    • Masahiro Yamada's avatar
      kbuild: move .SECONDARY special target to Kbuild.include · 8e9b61b2
      Masahiro Yamada authored
      In commit 54a702f7 ("kbuild: mark $(targets) as .SECONDARY and
      remove .PRECIOUS markers"), I missed one important feature of the
      .SECONDARY target:
      
          .SECONDARY with no prerequisites causes all targets to be
          treated as secondary.
      
      ... which agrees with the policy of Kbuild.
      
      Let's move it to scripts/Kbuild.include, with no prerequisites.
      
      Note:
      If an intermediate file is generated by $(call if_changed,...), you
      still need to add it to "targets" so its .*.cmd file is included.
      
      The arm/arm64 crypto files are generated by $(call cmd,shipped),
      so they do not need to be added to "targets", but need to be added
      to "clean-files" so "make clean" can properly clean them away.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      8e9b61b2
  6. 01 Dec, 2018 21 commits
    • Masahiro Yamada's avatar
      kbuild: remove redundant 'set -e' from cmd_* defines · 5439f09f
      Masahiro Yamada authored
      These three cmd_* are invoked in the $(call cmd,*) form.
      
      Now that 'set -e' moved to the 'cmd' macro, they do not need to
      explicitly give 'set -e'.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      5439f09f
    • Masahiro Yamada's avatar
      kbuild: refactor if_changed · 67126965
      Masahiro Yamada authored
      '@set -e; $(echo-cmd) $(cmd_$(1)' can be replaced with '$(cmd)'.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      67126965
    • Masahiro Yamada's avatar
      kbuild: remove trailing semicolon from cmd_* passed to if_changed_rule · e5d28910
      Masahiro Yamada authored
      With the change of rule_cc_o_c / rule_as_o_S in the last commit, each
      command is executed in a separate subshell. Rip off unneeded semicolons.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      e5d28910
    • Masahiro Yamada's avatar
      kbuild: change if_changed_rule for multi-line recipe · 3a2429e1
      Masahiro Yamada authored
      The 'define' ... 'endef' directive is useful to confine a series of
      shell commands into a single macro:
      
        define foo
                [action1]
                [action2]
                [action3]
        endif
      
      Each action is executed in a separate subshell.
      
      However, rule_cc_o_c and rule_as_o_S in scripts/Makefile.build are
      written as follows (with a trailing semicolon in each cmd_*):
      
        define rule_cc_o_c
                [action1] ; \
                [action2] ; \
                [action3] ;
        endef
      
      All shell commands are concatenated with '; \' so that it looks like
      a single command from the Makefile point of view. This does not
      exploit the benefits of 'define' ... 'endef' form because a single
      shell command can be more simply written, like this:
      
        rule_cc_o_c = \
                [action1] ; \
                [action2] ; \
                [action3] ;
      
      I guess the intention for the command concatenation was to let the
      '@set -e' in if_changed_rule cover all the commands.
      
      We can improve the readability by moving '@set -e' to the 'cmd' macro.
      The combo of $(call echo-cmd,*) $(cmd_*) in rule_cc_o_c and rule_as_o_S
      have been replaced with $(call cmd,*). The trailing back-slashes have
      been removed.
      
      Here is a note about the performance: the commands in rule_cc_o_c and
      rule_as_o_S were previously executed all together in a single subshell,
      but now each line in a separate subshell. This means Make will spawn
      extra subshells [1]. I measured the build performance for
        x86_64_defconfig + CONFIG_MODVERSIONS + CONFIG_TRIM_UNUSED_KSYMS
      and I saw slight performance regression, but I believe code readability
      and maintainability wins.
      
      [1] Precisely, GNU Make may optimize this by executing the command
          directly instead of forking a subshell, if no shell special
          characters are found in the command line and omitting the subshell
          will not change the behavior.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      3a2429e1
    • Masahiro Yamada's avatar
      kbuild: simplify dependency generation for CONFIG_TRIM_UNUSED_KSYMS · bbda5ec6
      Masahiro Yamada authored
      My main motivation of this commit is to clean up scripts/Kbuild.include
      and scripts/Makefile.build.
      
      Currently, CONFIG_TRIM_UNUSED_KSYMS works with a tricky gimmick;
      possibly exported symbols are detected by letting $(CPP) replace
      EXPORT_SYMBOL* with a special string '=== __KSYM_*===', which is
      post-processed by sed, and passed to fixdep. The extra preprocessing
      is costly, and hacking cmd_and_fixdep is ugly.
      
      I came up with a new way to find exported symbols; insert a dummy
      symbol __ksym_marker_* to each potentially exported symbol. Those
      dummy symbols are picked up by $(NM), post-processed by sed, then
      appended to .*.cmd files. I collected the post-process part to a
      new shell script scripts/gen_ksymdeps.sh for readability. The dummy
      symbols are put into the .discard.* section so that the linker
      script rips them off the final vmlinux or modules.
      
      A nice side-effect is building with CONFIG_TRIM_UNUSED_KSYMS will
      be much faster.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Reviewed-by: default avatarNicolas Pitre <nico@linaro.org>
      bbda5ec6
    • Masahiro Yamada's avatar
      kbuild: refactor modversions build rules · ee3e46b7
      Masahiro Yamada authored
      Let $(CC) compile objects into normal files *.o instead of .tmp_*.o
      whether CONFIG_MODVERSIONS is enabled or not. With this, the input
      file for objtool is always *.o so objtool_o can go away.
      
      I guess the reason of using .tmp_*.o for intermediate objects was
      to avoid leaving incomplete *.o file (, whose timestamp says it is
      up-to-date) when the genksyms tool failed for some reasons.
      
      It no longer matters because any targets are deleted on errors since
      commit 9c2af1c7 ("kbuild: add .DELETE_ON_ERROR special target").
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      ee3e46b7
    • Masahiro Yamada's avatar
      kbuild: remove redundant 'set -e' from sub_cmd_record_mcount · 4317ee3b
      Masahiro Yamada authored
      This is executed inside the if_changed_rule, which already sets
      'set -e'.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      4317ee3b
    • Masahiro Yamada's avatar
      kbuild: remove redundant 'set -e' from filechk_offsets · f3fd4a3f
      Masahiro Yamada authored
      The filechk macro in scripts/Kbuild.include already sets 'set -e'.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      f3fd4a3f
    • Masahiro Yamada's avatar
      kbuild: let fixdep directly write to .*.cmd files · 392885ee
      Masahiro Yamada authored
      Currently, fixdep writes dependencies to .*.tmp, which is renamed to
      .*.cmd after everything succeeds. This is a very safe way to avoid
      corrupted .*.cmd files. The if_changed_dep has carried this safety
      mechanism since it was added in 2002.
      
      If fixdep fails for some reasons or a user terminates the build while
      fixdep is running, the incomplete output from the fixdep could be
      troublesome.
      
      This is my insight about some bad scenarios:
      
      [1] If the compiler succeeds to generate *.o file, but fixdep fails
          to write necessary dependencies to .*.cmd file, Make will miss
          to rebuild the object when headers or CONFIG options are changed.
          In this case, fixdep should not generate .*.cmd file at all so
          that 'arg-check' will surely trigger the rebuild of the object.
      
      [2] A partially constructed .*.cmd file may not be a syntactically
          correct makefile. The next time Make runs, it would include it,
          then fail to parse it. Once this happens, 'make clean' is be the
          only way to fix it.
      
      In fact, [1] is no longer a problem since commit 9c2af1c7 ("kbuild:
      add .DELETE_ON_ERROR special target"). Make deletes a target file on
      any failure in its recipe. Because fixdep is a part of the recipe of
      *.o target, if it fails, the *.o is deleted anyway. However, I am a
      bit worried about the slight possibility of [2].
      
      So, here is a solution. Let fixdep directly write to a .*.cmd file,
      but allow makefiles to include it only when its corresponding target
      exists.
      
      This effectively reverts commit 2982c953 ("kbuild: remove redundant
      $(wildcard ...) for cmd_files calculation"), and commit 00d78ab2
      ("kbuild: remove dead code in cmd_files calculation in top Makefile")
      because now we must check the presence of targets.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      392885ee
    • Masahiro Yamada's avatar
      kbuild: descend into scripts/gcc-plugins/ via scripts/Makefile · ce2fd53a
      Masahiro Yamada authored
      Now that 'archprepare' depends on 'scripts', Kbuild can descend into
      scripts/gcc-plugins in a more standard way.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Reviewed-by: default avatarKees Cook <keescook@chromium.org>
      ce2fd53a
    • Masahiro Yamada's avatar
      kbuild: make 'archprepare' depend on 'scripts' · 059bc9fc
      Masahiro Yamada authored
      Before start descending, Kbuild needs to run 'prepare' and 'scripts',
      which has been orthogonal to each other.
      
      Going forward, let's consider 'scripts' is a part of the preparation.
      This will allow more cleanups.
      
      Move 'scripts' to the prerequisite of 'archprepare', where UML starts
      compiling target *.c files.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      059bc9fc
    • Masahiro Yamada's avatar
      kbuild: move modpost out of 'scripts' target · 60df1aee
      Masahiro Yamada authored
      I am eagar to build under the scripts/ directory only with $(HOSTCC),
      but scripts/mod/ highly depends on the $(CC) and target arch headers.
      That it why the 'scripts' target must depend on 'asm-generic',
      'gcc-plugins', and $(autoksyms_h).
      
      Move it to the 'prepare0' stage. I know this is a cheesy workaround,
      but better than the current situation.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      60df1aee
    • Masahiro Yamada's avatar
      kbuild: fix UML build error with CONFIG_GCC_PLUGINS · 65bba042
      Masahiro Yamada authored
      UML fails to build with CONFIG_GCC_PLUGINS=y.
      
      $ make -s ARCH=um mrproper
      $ make -s ARCH=um allmodconfig
      $ make ARCH=um
        UPD     include/generated/uapi/linux/version.h
        WRAP    arch/x86/include/generated/uapi/asm/bpf_perf_event.h
        WRAP    arch/x86/include/generated/uapi/asm/poll.h
        WRAP    arch/x86/include/generated/asm/dma-contiguous.h
        WRAP    arch/x86/include/generated/asm/early_ioremap.h
        WRAP    arch/x86/include/generated/asm/export.h
        WRAP    arch/x86/include/generated/asm/mcs_spinlock.h
        WRAP    arch/x86/include/generated/asm/mm-arch-hooks.h
        SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
        SYSHDR  arch/x86/include/generated/asm/unistd_32_ia32.h
        SYSHDR  arch/x86/include/generated/asm/unistd_64_x32.h
        SYSTBL  arch/x86/include/generated/asm/syscalls_64.h
        SYSHDR  arch/x86/include/generated/uapi/asm/unistd_32.h
        SYSHDR  arch/x86/include/generated/uapi/asm/unistd_64.h
        SYSHDR  arch/x86/include/generated/uapi/asm/unistd_x32.h
        HOSTCC  scripts/unifdef
        CC      arch/x86/um/user-offsets.s
      cc1: error: cannot load plugin ./scripts/gcc-plugins/cyc_complexity_plugin.so
      ./scripts/gcc-plugins/cyc_complexity_plugin.so: cannot open shared object file: No such file or directory
      cc1: error: cannot load plugin ./scripts/gcc-plugins/structleak_plugin.so
      ./scripts/gcc-plugins/structleak_plugin.so: cannot open shared object file: No such file or directory
      cc1: error: cannot load plugin ./scripts/gcc-plugins/latent_entropy_plugin.so
      ./scripts/gcc-plugins/latent_entropy_plugin.so: cannot open shared object file: No such file or directory
      cc1: error: cannot load plugin ./scripts/gcc-plugins/randomize_layout_plugin.so
      ./scripts/gcc-plugins/randomize_layout_plugin.so: cannot open shared object file: No such file or directory
      make[1]: *** [scripts/Makefile.build;119: arch/x86/um/user-offsets.s] Error 1
      make: *** [arch/um/Makefile;152: arch/x86/um/user-offsets.s] Error 2
      
      Reorder the preparation stage (with cleanups) to make sure gcc-plugins
      is built before descending to arch/x86/um/.
      
      Fixes: 6b90bd4b ("GCC plugin infrastructure")
      Reported-by: default avatarkbuild test robot <lkp@intel.com>
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      65bba042
    • Masahiro Yamada's avatar
      modpost: move unresolved symbol checks to check_exports() · 3b415288
      Masahiro Yamada authored
      This will fit better in check_exports() than add_versions().
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      3b415288
    • Masahiro Yamada's avatar
      modpost: merge module iterations · c6826ad8
      Masahiro Yamada authored
      Probably, this is just a matter of the order of error/warning
      messages. Merge the two for-loops.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      c6826ad8
    • Masahiro Yamada's avatar
      modpost: refactor seen flag clearing in add_depends() · d2665ca8
      Masahiro Yamada authored
      You do not need to iterate over all modules for resetting ->seen flag
      because add_depends() is only interested in modules that export symbols
      referenced from the given 'mod'.
      
      This also avoids shadowing the 'modules' parameter of add_depends().
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      d2665ca8
    • Masahiro Yamada's avatar
      modpost: file2alias: check prototype of handler · f880eea6
      Masahiro Yamada authored
      Use specific prototype instead of an opaque pointer so that the
      compiler can catch function prototype mismatch.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Reviewed-by: default avatarMathieu Malaterre <malat@debian.org>
      f880eea6
    • Masahiro Yamada's avatar
      modpost: file2alias: go back to simple devtable lookup · ec91e78d
      Masahiro Yamada authored
      Commit e49ce141 ("modpost: use linker section to generate table.")
      was not so cool as we had expected first; it ended up with ugly section
      hacks when commit dd2a3aca ("mod/file2alias: make modpost compile
      on darwin again") came in.
      
      Given a certain degree of unknowledge about the link stage of host
      programs, I really want to see simple, stupid table lookup so that
      this works in the same way regardless of the underlying executable
      format.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Acked-by: default avatarMathieu Malaterre <malat@debian.org>
      ec91e78d
    • Masahiro Yamada's avatar
      kbuild: fix single target build for external module · e07db28e
      Masahiro Yamada authored
      Building a single target in an external module fails due to missing
      .tmp_versions directory.
      
      For example,
      
        $ make -C /lib/modules/$(uname -r)/build M=$PWD foo.o
      
      will fail in the following way:
      
        CC [M]  /home/masahiro/foo/foo.o
      /bin/sh: 1: cannot create /home/masahiro/foo/.tmp_versions/foo.mod: Directory nonexistent
      
      This is because $(cmd_crmodverdir) is executed only before building
      /, %/, %.ko single targets of external modules. Create .tmp_versions
      in the 'prepare' target.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      e07db28e
    • Paul Walmsley's avatar
      modpost: skip ELF local symbols during section mismatch check · a4d26f1a
      Paul Walmsley authored
      During development of a serial console driver with a gcc 8.2.0
      toolchain for RISC-V, the following modpost warning appeared:
      
      ----
      WARNING: vmlinux.o(.data+0x19b10): Section mismatch in reference from the variable .LANCHOR1 to the function .init.text:sifive_serial_console_setup()
      The variable .LANCHOR1 references
      the function __init sifive_serial_console_setup()
      If the reference is valid then annotate the
      variable with __init* or __refdata (see linux/init.h) or name the variable:
      *_template, *_timer, *_sht, *_ops, *_probe, *_probe_one, *_console
      ----
      
      ".LANCHOR1" is an ELF local symbol, automatically created by gcc's section
      anchor generation code:
      
      https://gcc.gnu.org/onlinedocs/gccint/Anchored-Addresses.html
      
      https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/varasm.c;h=cd9591a45617464946dcf9a126dde277d9de9804;hb=9fb89fa845c1b2e0a18d85ada0b077c84508ab78#l7473
      
      This was verified by compiling the kernel with -fno-section-anchors
      and observing that the ".LANCHOR1" ELF local symbol disappeared, and
      modpost no longer warned about the section mismatch.  The serial
      driver code idiom triggering the warning is standard Linux serial
      driver practice that has a specific whitelist inclusion in modpost.c.
      
      I'm neither a modpost nor an ELF expert, but naively, it doesn't seem
      useful for modpost to report section mismatch warnings caused by ELF
      local symbols by default.  Local symbols have compiler-generated
      names, and thus bypass modpost's whitelisting algorithm, which relies
      on the presence of a non-autogenerated symbol name.  This increases
      the likelihood that false positive warnings will be generated (as in
      the above case).
      
      Thus, disable section mismatch reporting on ELF local symbols.  The
      rationale here is similar to that of commit 2e3a10a1 ("ARM: avoid
      ARM binutils leaking ELF local symbols") and of similar code already
      present in modpost.c:
      
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/mod/modpost.c?h=v4.19-rc4&id=7876320f88802b22d4e2daf7eb027dd14175a0f8#n1256
      
      This third version of the patch implements a suggestion from Masahiro
      Yamada <yamada.masahiro@socionext.com> to restructure the code as an
      additional pattern matching step inside secref_whitelist(), and
      further improves the patch description.
      Signed-off-by: default avatarPaul Walmsley <paul.walmsley@sifive.com>
      Signed-off-by: default avatarPaul Walmsley <paul@pwsan.com>
      Acked-by: default avatarSam Ravnborg <sam@ravnborg.org>
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      a4d26f1a
    • Masahiro Yamada's avatar
      kbuild: announce removal of SUBDIRS if used · 0126be38
      Masahiro Yamada authored
      SUBDIRS has been kept as a backward compatibility since
      commit ("[PATCH] kbuild: external module support") in 2002.
      
      We do not need multiple ways to do the same thing, so I will remove
      SUBDIRS after the Linux 5.3 release. I cleaned up in-tree code, and
      updated the document so that nobody would try to use it.
      
      Meanwhile, display the following warning if SUBDIRS is used.
      
      Makefile:189: ================= WARNING ================
      Makefile:190: 'SUBDIRS' will be removed after Linux 5.3
      Makefile:191: Please use 'M=' or 'KBUILD_EXTMOD' instead
      Makefile:192: ==========================================
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Acked-by: Boris Brezillon <boris.brezillon@bootlin.com> # for scx200_docflash.c
      Acked-by: Guenter Roeck <linux@roeck-us.net> # for scx200_wdt.c
      0126be38
  7. 21 Nov, 2018 3 commits
  8. 13 Nov, 2018 1 commit