Skip to content

L
linux
Switch branch/tag
  1. 20 Oct, 2020 4 commits
  3. 14 Oct, 2020 2 commits
  5. 12 Oct, 2020 1 commit
  7. 11 Oct, 2020 1 commit
    • Jacob Keller's avatar
      scripts: remove namespace.pl · 7dfbea4c
      Jacob Keller authored 
      namespace.pl is intended to help locate symbols which are defined but
are not used externally. The goal is to avoid bloat of the namespace in
the resulting kernel image.

The script relies on object data, and only finds unused symbols for the
configuration used to generate that object data. This results in a lot
of false positive warnings such as symbols only used by a single
architecture, or symbols which are used externally only under certain
configurations.

Running namespace.pl using allyesconfig, allmodconfig, and
x86_64_defconfig yields the following results:

* allmodconfig
  * 11122 unique symbol names with no external reference
  * 1194 symbols listed as multiply defined
  * 214 symbols it can't resolve
* allyesconfig
  * 10997 unique symbol names with no external reference
  * 1194 symbols listed as multiply defined
  * 214 symbols it can't resolve
* x86_64_defconfig
  * 5757 unique symbol names with no external reference
  * 528 symbols listed as multiply defined
  * 154 symbols it can't resolve

The script also has no way to easily limit the scope of the checks to
a given subset of the kernel, such as only checking for symbols defined
within a module or subsystem.

Discussion on public mailing lists seems to indicate that many view the
tool output as suspect or not very useful (see discussions at [1] and
[2] for further context).

As described by Masahiro Yamada at [2], namespace.pl provides 3 types of
checks: listing multiply defined symbols, resolving external symbols,
and warnings about symbols with no reference.

The first category of issues is easily caught by the linker as any set
of multiply defined symbols should fail to link. The second category of
issues is also caught by linking, as undefined symbols would cause
issues. Even with modules, these types of issues where a module relies
on an external symbol are caught by modpost.

The remaining category of issues reported is the list of symbols with no
external reference, and is the primary motivation of this script.
However, it ought to be clear from the above examples that the output is
difficult to sort through. Even allyesconfig has ~10000 entries.

The current submit-checklist indicates that patches ought to go through
namespacecheck and fix any new issues arising. But that itself presents
problems. As described at [1], many cases of reports are due to
configuration where a function is used externally by some configuration
settings. Prominent maintainers appear to dislike changes modify code
such that symbols become static based on CONFIG_* flags ([3], and [4])

One possible solution is to adjust the advice and indicate that we only
care about the output of namespacecheck on allyesconfig or allmodconfig
builds...

However, given the discussion at [2], I suspect that few people are
actively using this tool. It doesn't have a maintainer in the
MAINTAINERS flie, and it produces so many warnings for unused symbols
that it is difficult to use effectively. Thus, I propose we simply
remove it.

[1] https://lore.kernel.org/netdev/20200708164812.384ae8ea@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com/
[2] https://lore.kernel.org/lkml/20190129204319.15238-1-jacob.e.keller@intel.com/
[3] https://lore.kernel.org/netdev/20190828.154744.2058157956381129672.davem@davemloft.net/
[4] https://lore.kernel.org/netdev/20190827210928.576c5fef@cakuba.netronome.com/Signed-off-by: default avatarJacob Keller <jacob.e.keller@intel.com>
Acked-by: default avatarRandy Dunlap <rdunlap@infradead.org>
Acked-by: default avatarJakub Kicinski <kuba@kernel.org>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      7dfbea4c
  9. 09 Oct, 2020 5 commits
  11. 24 Sep, 2020 8 commits
  13. 07 Sep, 2020 1 commit
  15. 26 Aug, 2020 12 commits
    • Nathan Huckleberry's avatar
      Makefile: Add clang-tidy and static analyzer support to makefile · 6ad7cbc0
      Nathan Huckleberry authored 
      This patch adds clang-tidy and the clang static-analyzer as make
targets. The goal of this patch is to make static analysis tools
usable and extendable by any developer or researcher who is familiar
with basic c++.

The current static analysis tools require intimate knowledge of the
internal workings of the static analysis. Clang-tidy and the clang
static analyzers expose an easy to use api and allow users unfamiliar
with clang to write new checks with relative ease.

===Clang-tidy===

Clang-tidy is an easily extendable 'linter' that runs on the AST.
Clang-tidy checks are easy to write and understand. A check consists of
two parts, a matcher and a checker. The matcher is created using a
domain specific language that acts on the AST
(https://clang.llvm.org/docs/LibASTMatchersReference.html).  When AST
nodes are found by the matcher a callback is made to the checker. The
checker can then execute additional checks and issue warnings.

Here is an example clang-tidy check to report functions that have calls
to local_irq_disable without calls to local_irq_enable and vice-versa.
Functions flagged with __attribute((annotation("ignore_irq_balancing")))
are ignored for analysis. (https://reviews.llvm.org/D65828)

===Clang static analyzer===

The clang static analyzer is a more powerful static analysis tool that
uses symbolic execution to find bugs. Currently there is a check that
looks for potential security bugs from invalid uses of kmalloc and
kfree. There are several more general purpose checks that are useful for
the kernel.

The clang static analyzer is well documented and designed to be
extensible.
(https://clang-analyzer.llvm.org/checker_dev_manual.html)
(https://github.com/haoNoQ/clang-analyzer-guide/releases/download/v0.1/clang-analyzer-guide-v0.1.pdf)

The main draw of the clang tools is how accessible they are. The clang
documentation is very nice and these tools are built specifically to be
easily extendable by any developer. They provide an accessible method of
bug-finding and research to people who are not overly familiar with the
kernel codebase.
Signed-off-by: default avatarNathan Huckleberry <nhuck@google.com>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
Tested-by: default avatarNick Desaulniers <ndesaulniers@google.com>
Tested-by: default avatarLukas Bulwahn <lukas.bulwahn@gmail.com>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      6ad7cbc0
    • Masahiro Yamada's avatar
      gen_compile_commands: remove the warning about too few .cmd files · 8b61f748
      Masahiro Yamada authored 
      This warning was useful when users previously needed to manually
build the kernel and run this script.

Now you can simply do 'make compile_commands.json', which updates
all the necessary build artifacts and automatically creates the
compilation database. There is no more worry for a mistake like
"Oh, I forgot to build the kernel".

Now, this warning is rather annoying.

You can create compile_commands.json for an external module:

  $ make M=/path/to/your/external/module compile_commands.json

Then, this warning is displayed since there are usually less than
300 files in a single module.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      8b61f748
    • Masahiro Yamada's avatar
      kbuild: wire up the build rule of compile_commands.json to Makefile · 3d32285f
      Masahiro Yamada authored 
      Currently, you need to manually run scripts/gen_compile_commands.py
to create compile_commands.json. It parses all the .*.cmd files found
under the specified directory.

If you rebuild the kernel over again without 'make clean',
.*.cmd files from older builds will create stale entries in
compile_commands.json.

This commit wires up the compile_commands.json rule to Makefile, and
makes it parse only the .*.cmd files involved in the current build.

Pass $(KBUILD_VMLINUX_OBJS), $(KBUILD_VMLINUX_LIBS), and modules.order
to the script. The objects or archives linked to vmlinux are listed in
$(KBUILD_VMLINUX_OBJS) or $(KBUILD_VMLINUX_LIBS). All the modules are
listed in modules.order.

You can create compile_commands.json from Make:

  $ make -j$(nproc) CC=clang compile_commands.json

You can also build vmlinux, modules, and compile_commands.json all
together in a single command:

  $ make -j$(nproc) CC=clang all compile_commands.json

It works for M= builds as well. In this case, compile_commands.json
is created in the top directory of the external module.

This is convenient, but it has a drawback; the coverage of the
compile_commands.json is reduced because only the objects linked to
vmlinux or modules are handled. For example, the following C files are
not included in the compile_commands.json:

 - Decompressor source files (arch/*/boot/)
 - VDSO source files
 - C files used to generate intermediates (e.g. kernel/bounds.c)
 - Standalone host programs

I think it is fine for most developers because our main interest is
the kernel-space code.

If you want to cover all the compiled C files, please build the kernel,
then run the script manually as you did before:

  $ make clean    # if you want to remove stale .cmd files [optional]
  $ make -j$(nproc) CC=clang
  $ scripts/gen_compile_commands.py

Here is a note for out-of-tree builds. 'make compile_commands.json'
works with O= option, but please notice compile_commands.json is
created in the object tree instead of the source tree.

Some people may want to have compile_commands.json in the source tree
because Clang Tools searches for it through all parent paths of the
first input source file.

However, you cannot do this for O= builds. Kbuild should never generate
any build artifact in the source tree when O= is given because the
source tree might be read-only. Any write attempt to the source tree
is monitored and the violation may be reported. See the commit log of
8ef14c2c.

So, the only possible way is to create compile_commands.json in the
object tree, then specify '-p <build-path>' when you use clang-check,
clang-tidy, etc.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Acked-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      3d32285f
    • Masahiro Yamada's avatar
      gen_compile_commands: support *.o, *.a, modules.order in positional argument · ecca4fea
      Masahiro Yamada authored 
      This script currently searches the specified directory for .cmd files.
One drawback is it may contain stale .cmd files after you rebuild the
kernel several times without 'make clean'.

This commit supports *.o, *.a, and modules.order as positional
parameters. If such files are given, they are parsed to collect
associated .cmd files. I added a generator helper for each of them.

This feature is useful to get the list of active .cmd files from the
last build, and will be used by the next commit to wire up the
compile_commands.json rule to the Makefile.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      ecca4fea
    • Masahiro Yamada's avatar
      gen_compile_commands: move directory walk to a generator function · fc2cb22e
      Masahiro Yamada authored 
      Currently, this script walks under the specified directory (default to
the current directory), then parses all .cmd files found.

Split it into a separate helper function because the next commit will
add more helpers to pick up .cmd files associated with given file(s).

There is no point to build and return a huge list at once. I used a
generator so it works in the for-loop with less memory.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      fc2cb22e
    • Masahiro Yamada's avatar
      gen_compile_commands: make -o option independent of -d option · 6fca36f1
      Masahiro Yamada authored 
      Change the -o option independent of the -d option, which is I think
clearer behavior. Some people may like to use -d to specify a separate
output directory, but still output the compile_commands.py in the
source directory (unless the source tree is read-only) because it is
the default location Clang Tools search for the compilation database.

Also, move the default parameter to the default= argument of the
.add_argument().
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      6fca36f1
    • Masahiro Yamada's avatar
      gen_compile_commands: reword the help message of -d option · 0a7d376d
      Masahiro Yamada authored 
      I think the help message of the -d option is somewhat misleading.

  Path to the kernel source directory to search (defaults to the working directory)

The part "kernel source directory" is the source of the confusion.
Some people misunderstand as if this script did not support separate
output directories.

Actually, this script also works for out-of-tree builds. You can
use the -d option to point to the object output directory, not to
the source directory. It should match to the O= option used in the
previous kernel build, and then appears in the "directory" field of
compile_commands.json.

Reword the help message.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      0a7d376d
    • Masahiro Yamada's avatar
      gen_compile_commands: do not support .cmd files under tools/ directory · 6ca4c6d2
      Masahiro Yamada authored 
      The tools/ directory uses a different build system, and the format of
.cmd files is different because the tools builds run in a different
work directory.

Supporting two formats compilicates the script.

The only loss by this change is objtool.

Also, rename the confusing variable 'relative_path' because it is
not necessarily a relative path. When the output directory is not
the direct child of the source tree (e.g. O=foo/bar), it is an
absolute path. Rename it to 'file_path'.

os.path.join(root_directory, file_path) works whether the file_path
is relative or not. If file_path is already absolute, it returns it
as-is.

I used os.path.abspath() to normalize file paths. If you run this
script against the kernel built with O=foo option, the file_path
contains '../' patterns. os.path.abspath() fixes up 'foo/bar/../baz'
into 'foo/baz', and produces a cleaner commands_database.json.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      6ca4c6d2
    • Masahiro Yamada's avatar
      gen_compile_commands: use choices for --log_levels option · ea6cedc5
      Masahiro Yamada authored 
      Use 'choices' to check if the given parameter is valid.

I also simplified the help message because, with 'choices', --help
shows the list of valid parameters:

  --log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}

I started the help message with a lower case, "the level of log ..."
in order to be consistent with the -h option:

  -h, --help            show this help message and exit

The message "show this help ..." comes from the ArgumentParser library
code, and I do not know how to change it. So, I changed our code.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      ea6cedc5
    • Masahiro Yamada's avatar
      gen_compile_commands: parse only the first line of .*.cmd files · 8a685db3
      Masahiro Yamada authored 
      After the allmodconfig build, this script takes about 5 sec on my
machine. Most of the run-time is consumed for needless regex matching.

We know the format of .*.cmd file; the first line is the build command.
There is no need to parse the rest.

With this optimization, now it runs 4 times faster.
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
Tested-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      8a685db3
    • Masahiro Yamada's avatar
      kbuild: hide commands to run Kconfig, and show short log for syncconfig · 23cd88c9
      Masahiro Yamada authored 
      Some targets (localyesconfig, localmodconfig, defconfig) hide the
command running, but the others do not.

Users know which Kconfig flavor they are running, so it is OK to hide
the command. Add $(Q) to all commands consistently. If you want to see
the full command running, pass V=1 from the command line.

syncconfig is the exceptional case, which occurs without explicit
command invocation by the user. Display the Kbuild-style log for it.
The ugly bare log will go away.

[Before]

scripts/kconfig/conf  --syncconfig Kconfig

[After]

  SYNC    include/config/auto.conf
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      23cd88c9
    • Sedat Dilek's avatar
      kbuild: Simplify DEBUG_INFO Kconfig handling · 695afd3d
      Sedat Dilek authored 
      While playing with [1] I saw that the handling
of CONFIG_DEBUG_INFO can be simplified.

[1] https://patchwork.kernel.org/patch/11716107/Signed-off-by: default avatarSedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      695afd3d
  17. 23 Aug, 2020 6 commits
    • Linus Torvalds's avatar
      Linux 5.9-rc2 · d012a719
      Linus Torvalds authored
      d012a719
    • Linus Torvalds's avatar
      Merge tag 'powerpc-5.9-3' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux · cb957121
      Linus Torvalds authored 
      Pull powerpc fixes from Michael Ellerman:

 - Add perf support for emitting extended registers for power10.

 - A fix for CPU hotplug on pseries, where on large/loaded systems we
   may not wait long enough for the CPU to be offlined, leading to
   crashes.

 - Addition of a raw cputable entry for Power10, which is not required
   to boot, but is required to make our PMU setup work correctly in
   guests.

 - Three fixes for the recent changes on 32-bit Book3S to move modules
   into their own segment for strict RWX.

 - A fix for a recent change in our powernv PCI code that could lead to
   crashes.

 - A change to our perf interrupt accounting to avoid soft lockups when
   using some events, found by syzkaller.

 - A change in the way we handle power loss events from the hypervisor
   on pseries. We no longer immediately shut down if we're told we're
   running on a UPS.

 - A few other minor fixes.

Thanks to Alexey Kardashevskiy, Andreas Schwab, Aneesh Kumar K.V, Anju T
Sudhakar, Athira Rajeev, Christophe Leroy, Frederic Barrat, Greg Kurz,
Kajol Jain, Madhavan Srinivasan, Michael Neuling, Michael Roth,
Nageswara R Sastry, Oliver O'Halloran, Thiago Jung Bauermann,
Vaidyanathan Srinivasan, Vasant Hegde.

* tag 'powerpc-5.9-3' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux:
  powerpc/perf/hv-24x7: Move cpumask file to top folder of hv-24x7 driver
  powerpc/32s: Fix module loading failure when VMALLOC_END is over 0xf0000000
  powerpc/pseries: Do not initiate shutdown when system is running on UPS
  powerpc/perf: Fix soft lockups due to missed interrupt accounting
  powerpc/powernv/pci: Fix possible crash when releasing DMA resources
  powerpc/pseries/hotplug-cpu: wait indefinitely for vCPU death
  powerpc/32s: Fix is_module_segment() when MODULES_VADDR is defined
  powerpc/kasan: Fix KASAN_SHADOW_START on BOOK3S_32
  powerpc/fixmap: Fix the size of the early debug area
  powerpc/pkeys: Fix build error with PPC_MEM_KEYS disabled
  powerpc/kernel: Cleanup machine check function declarations
  powerpc: Add POWER10 raw mode cputable entry
  powerpc/perf: Add extended regs support for power10 platform
  powerpc/perf: Add support for outputting extended regs in perf intr_regs
  powerpc: Fix P10 PVR revision in /proc/cpuinfo for SMT4 cores
      cb957121
    • Linus Torvalds's avatar
      Merge tag 'x86-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip · 550c2129
      Linus Torvalds authored 
      Pull x86 fix from Thomas Gleixner:
 "A single fix for x86 which removes the RDPID usage from the paranoid
  entry path and unconditionally uses LSL to retrieve the CPU number.

  RDPID depends on MSR_TSX_AUX. KVM has an optmization to avoid
  expensive MRS read/writes on VMENTER/EXIT. It caches the MSR values
  and restores them either when leaving the run loop, on preemption or
  when going out to user space. MSR_TSX_AUX is part of that lazy MSR
  set, so after writing the guest value and before the lazy restore any
  exception using the paranoid entry will read the guest value and use
  it as CPU number to retrieve the GSBASE value for the current CPU when
  FSGSBASE is enabled. As RDPID is only used in that particular entry
  path, there is no reason to burden VMENTER/EXIT with two extra MSR
  writes. Remove the RDPID optimization, which is not even backed by
  numbers from the paranoid entry path instead"

* tag 'x86-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/entry/64: Do not use RDPID in paranoid entry to accomodate KVM
      550c2129
    • Linus Torvalds's avatar
      Merge tag 'perf-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip · cea05c19
      Linus Torvalds authored 
      Pull x86 perf fix from Thomas Gleixner:
 "A single update for perf on x86 which has support for the broken down
  bandwith counters"

* tag 'perf-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  perf/x86/intel/uncore: Add BW counters for GT, IA and IO breakdown
      cea05c19
    • Linus Torvalds's avatar
      Merge tag 'efi-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip · 10c091b6
      Linus Torvalds authored 
      Pull EFI fixes from Thomas Gleixner:

 - Enforce NX on RO data in mixed EFI mode

 - Destroy workqueue in an error handling path to prevent UAF

 - Stop argument parser at '--' which is the delimiter for init

 - Treat a NULL command line pointer as empty instead of dereferncing it
   unconditionally.

 - Handle an unterminated command line correctly

 - Cleanup the 32bit code leftovers and remove obsolete documentation

* tag 'efi-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  Documentation: efi: remove description of efi=old_map
  efi/x86: Move 32-bit code into efi_32.c
  efi/libstub: Handle unterminated cmdline
  efi/libstub: Handle NULL cmdline
  efi/libstub: Stop parsing arguments at "--"
  efi: add missed destroy_workqueue when efisubsys_init fails
  efi/x86: Mark kernel rodata non-executable for mixed mode
      10c091b6
    • Linus Torvalds's avatar
      Merge tag 'core-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip · e99b2507
      Linus Torvalds authored 
      Pull entry fix from Thomas Gleixner:
 "A single bug fix for the common entry code.

  The transcription of the x86 version messed up the reload of the
  syscall number from pt_regs after ptrace and seccomp which breaks
  syscall number rewriting"

* tag 'core-urgent-2020-08-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  core/entry: Respect syscall number rewrites
      e99b2507
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7