- 29 Apr, 2015 10 commits
-
-
Jiri Olsa authored
For big data files the size of data allocated for stream instance could get really high. It's needed to flush the data out of the stream once in a while. Unfortunately there's no size indication in the stream object, so we govern the flush based on the number of stored events. Current flush limit is set ot 100000 events. Reviewed-by:
David Ahern <dsahern@gmail.com> Signed-off-by:
Jiri Olsa <jolsa@kernel.org> Acked-by:
Namhyung Kim <namhyung@kernel.org> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: He Kuang <hekuang@huawei.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Tom Zanussi <tzanussi@gmail.com> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/r/1429372220-6406-4-git-send-email-jolsa@kernel.orgSigned-off-by:
Arnaldo Carvalho de Melo <acme@redhat.com>
-
Sebastian Andrzej Siewior authored
Currently we store the data into single data strea/file. The cpu if data is stored within the event sample. The lttng puts the CPU number that belongs to the event into the packet context instead into the event. This patch makes sure that the trace produce by perf does look the same way. We now use one stream per-CPU. Having it all in one stream increased the total size of the resulting file. The test went from 416KiB (with perf_cpu event member) to 24MiB due to the required (and pointless) flush. With the per-cpu streams the total size went up to 588KiB. Signed-off-by:
Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: He Kuang <hekuang@huawei.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Tom Zanussi <tzanussi@gmail.com> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/r/1429372220-6406-3-git-send-email-jolsa@kernel.orgSigned-off-by:
-
Jiri Olsa authored
Adding build tests for following make commands: $ make -C <kernelsrc> tools/perf $ make -C <kernelsrc>/tools perf Signed-off-by:
-
Jiri Olsa authored
Several fixes were needed to allow following builds: $ make tools/tmon $ make -C <kernelsrc> tools/perf $ make -C <kernelsrc>/tools perf - some of the tools (perf) use same make variables as in kernel build, unsetting srctree and objtree - using original $(O) for O variable - perf build does not follow the descend function setup invoking it via it's own make rule I tried the rest of the tools/Makefile targets and they seem to work now. Reported-by:
Brian Norris <computersforpeace@gmail.com> Signed-off-by:
-
Jiri Olsa authored
The perf build handles its dependencies by itself. Also renaming libapi libapikfs to libapi as it got changed just recently. Signed-off-by:
-
Jiri Olsa authored
Introduce an 'alias' field to 'struct format_field' to be able to use alternative name for the field. It is initialized with same string pointer as 'name' field. The free logic checks the 'alias' pointer being reset by user and frees it. This will be handy when converting data into CTF, where each field within event needs to have a unique name (while this is not required for tracepoint). Converter can easily assign unique name into the format_field struct. Suggested-by:
Steven Rostedt <rostedt@goodmis.org> Signed-off-by:
-
He Kuang authored
Show message when errors occurred during conversion setup and conversion process. Before this patch: $ ./perf data convert --to-ctf=ctf $ echo $? 255 After this patch: $ ./perf data convert --to-ctf=ctf Error during conversion setup. Signed-off-by:
He Kuang <hekuang@huawei.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Tom Zanussi <tzanussi@gmail.com> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/n/tip-xvhr1vf7zav9kkeo9w1hv4uk@git.kernel.org Link: http://lkml.kernel.org/r/1429372220-6406-2-git-send-email-jolsa@kernel.orgSigned-off-by:
-
Namhyung Kim authored
The period_ratio_delta, period_ratio and wdiff are never by used at the same time. Instead, Just one of them is accessed according to a comparison method. So make it union to reduce memory footprint. Signed-off-by:
-
Namhyung Kim authored
It's not used anywhere, let's get rid of it. Signed-off-by:
-
David Ahern authored
0d68bc92 breaks compiles on RHEL6/OL6: cc1: warnings being treated as errors builtin-kmem.c: In function ‘search_page_alloc_stat’: builtin-kmem.c:322: error: declaration of ‘stat’ shadows a global declaration node = &parent->rb_left; /usr/include/sys/stat.h:455: error: shadowed declaration is here builtin-kmem.c: In function ‘perf_evsel__process_page_alloc_event’: builtin-kmem.c:378: error: declaration of ‘stat’ shadows a global declaration /usr/include/sys/stat.h:455: error: shadowed declaration is here builtin-kmem.c: In function ‘perf_evsel__process_page_free_event’: builtin-kmem.c:431: error: declaration of ‘stat’ shadows a global declaration /usr/include/sys/stat.h:455: error: shadowed declaration is here Rename local variable to pstat to avoid the name conflict. Signed-off-by:
David Ahern <david.ahern@oracle.com> Link: http://lkml.kernel.org/r/1429033773-31383-1-git-send-email-david.ahern@oracle.comSigned-off-by:
-
- 14 Apr, 2015 1 commit
-
-
Ingo Molnar authored
Merge tag 'perf-core-for-mingo-2' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/urgent Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo: New features: - Analyze page allocator events in 'perf kmem' (Namhyung Kim) User visible changes: - Fix retprobe 'perf probe' handling when failing to find needed debuginfo (He Kuang) - lazy_line probe fixes in 'perf probe' (Naohiro Aota, He Kuang) Infrastructure changes: - Record pfn instead of pointer to struct page in tracepoints (Namhyung Kim) Signed-off-by:
Ingo Molnar <mingo@kernel.org>
-
- 13 Apr, 2015 5 commits
-
-
He Kuang authored
The first argument passed to find_probe_point_lazy() should be CU die, which will be passed to die_walk_lines() when lazy_line matches. Currently, when we probe with lazy_line pattern to file without function name, NULL pointer is passed and causes a segment fault. Can be reproduced as following: $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;' [ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp 00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000] Segmentation fault After this patch: $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;' Added new event: probe:_stext (on @fs/super.c) You can now use it in all perf tools, such as: perf record -e probe:_stext -aR sleep 1 Signed-off-by:Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/r/1428925290-5623-3-git-send-email-hekuang@huawei.comSigned-off-by:
-
Naohiro Aota authored
If we use lazy matching, it failed to open a souce file if perf command is invoked outside of compilation directory: $ perf probe -a '__schedule;clear_*' Failed to open kernel/sched/core.c: No such file or directory Error: Failed to add events. (-2) OTOH, other commands like "probe -L" can solve the souce directory by themselves. Let's make it possible for lazy matching too! Signed-off-by:
Naohiro Aota <naota@elisp.net> Acked-by:
-
He Kuang authored
When perf probe searched in a debuginfo file and failed, it tried with an alternative, in function get_alternative_probe_event(): memcpy(tmp, &pev->point, sizeof(*tmp)); memset(&pev->point, 0, sizeof(pev->point)); In this case, it drops the retprobe flag and forgets to set it back in find_alternative_probe_point(), so the problem occurs. Can be reproduced as following: $ perf probe -v -k vmlinux --add='sys_write%return' ... Added new event: Writing event: p:probe/sys_write _stext+1584952 probe:sys_write (on sys_write%return) $ cat /sys/kernel/debug/tracing/kprobe_events p:probe/sys_write _stext+1584952 After this patch: $ perf probe -v -k vmlinux --add='sys_write%return' Added new event: Writing event: r:probe/sys_write SyS_write+0 probe:sys_write (on sys_write%return) $ cat /sys/kernel/debug/tracing/kprobe_events r:probe/sys_write SyS_write Signed-off-by: -
Namhyung Kim authored
The perf kmem command records and analyze kernel memory allocation only for SLAB objects. This patch implement a simple page allocator analyzer using kmem:mm_page_alloc and kmem:mm_page_free events. It adds two new options of --slab and --page. The --slab option is for analyzing SLAB allocator and that's what perf kmem currently does. The new --page option enables page allocator events and analyze kernel memory usage in page unit. Currently, 'stat --alloc' subcommand is implemented only. If none of these --slab nor --page is specified, --slab is implied. First run 'perf kmem record' to generate a suitable perf.data file: # perf kmem record --page sleep 5 Then run 'perf kmem stat' to postprocess the perf.data file: # perf kmem stat --page --alloc --line 10 ------------------------------------------------------------------------------- PFN | Total alloc (KB) | Hits | Order | Mig.type | GFP flags ------------------------------------------------------------------------------- 4045014 | 16 | 1 | 2 | RECLAIM | 00285250 4143980 | 16 | 1 | 2 | RECLAIM | 00285250 3938658 | 16 | 1 | 2 | RECLAIM | 00285250 4045400 | 16 | 1 | 2 | RECLAIM | 00285250 3568708 | 16 | 1 | 2 | RECLAIM | 00285250 3729824 | 16 | 1 | 2 | RECLAIM | 00285250 3657210 | 16 | 1 | 2 | RECLAIM | 00285250 4120750 | 16 | 1 | 2 | RECLAIM | 00285250 3678850 | 16 | 1 | 2 | RECLAIM | 00285250 3693874 | 16 | 1 | 2 | RECLAIM | 00285250 ... | ... | ... | ... | ... | ... ------------------------------------------------------------------------------- SUMMARY (page allocator) ======================== Total allocation requests : 44,260 [ 177,256 KB ] Total free requests : 117 [ 468 KB ] Total alloc+freed requests : 49 [ 196 KB ] Total alloc-only requests : 44,211 [ 177,060 KB ] Total free-only requests : 68 [ 272 KB ] Total allocation failures : 0 [ 0 KB ] Order Unmovable Reclaimable Movable Reserved CMA/Isolated ----- ------------ ------------ ------------ ------------ ------------ 0 32 . 44,210 . . 1 . . . . . 2 . 18 . . . 3 . . . . . 4 . . . . . 5 . . . . . 6 . . . . . 7 . . . . . 8 . . . . . 9 . . . . . 10 . . . . . Signed-off-by: -
Namhyung Kim authored
The struct page is opaque for userspace tools, so it'd be better to save pfn in order to identify page frames. The textual output of $debugfs/tracing/trace file remains unchanged and only raw (binary) data format is changed - but thanks to libtraceevent, userspace tools which deal with the raw data (like perf and trace-cmd) can parse the format easily. So impact on the userspace will also be minimal. Signed-off-by:
Joonsoo Kim <js1304@gmail.com> Acked-by:
-
- 12 Apr, 2015 1 commit
-
-
Ingo Molnar authored
Dan Carpenter pointed out that the control flow in pt_pmu_hw_init() is a bit messy: for example the kfree(de_attrs) is entirely superfluous. Another problem is the inconsistent mixing of label based and direct return error handling. Add modern, label based error handling instead and clean up the code a bit as well. Note that we'll still do a kfree(NULL) in the normal case - this does not matter as this is an init path and kfree() returns early if it sees a NULL. Reported-by:
Dan Carpenter <dan.carpenter@oracle.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/20150409090805.GG17605@mwandaSigned-off-by:
-
- 11 Apr, 2015 1 commit
-
-
Ingo Molnar authored
Merge tag 'perf-core-for-mingo' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo: New user visible features: - Support multiple probes on different binaries on the same command line (Masami Hiramatsu) User visible changes: - Fix synthesizing fork_event.ppid for non-main thread (David Ahern) - Fix cross-endian analysis (David Ahern) - Fix segfault in 'perf buildid-list' when show DSOs with hits (He Kuang) Infrastructure changes: - Fix type for references to data_head/tail (David Ahern) - Fix error path to do closedir() when synthesizing threads (Arnaldo Carvalho de Melo) Signed-off-by:
-
- 10 Apr, 2015 7 commits
-
-
David Ahern authored
The data_head and data_tail fields are defined as __u64 in linux/perf_event.h, but perf userspace uses int and unsigned int. Convert all references to u64 for consistency. Signed-off-by:
Peter Zijlstra <peterz@infradead.org> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1428420037-26599-1-git-send-email-dsahern@gmail.comSigned-off-by:
-
Masami Hiramatsu authored
To avoid probing in unintended binary, the orphaned -x option must be checked and warned. Without this patch, following command sets up the probe in the kernel. ----- # perf probe -a strcpy -x ./perf Added new event: probe:strcpy (on strcpy) You can now use it in all perf tools, such as: perf record -e probe:strcpy -aR sleep 1 ----- But in this case, it seems that the user may want to probe in the perf binary. With this patch, perf-probe correctly handles the orphaned -x. ----- # perf probe -a strcpy -x ./perf Error: -x/-m must follow the probe definitions. ... ----- Reported-by:Jiri Olsa <jolsa@redhat.com> Acked-by:
-
Masami Hiramatsu authored
Support multiple probes on different binaries with just one command. In the result, this example sets up the probes on icmp_rcv in kernel, on main and set_target in perf, and on pcspkr_event in pcspker.ko driver. ----- # perf probe -a icmp_rcv -x ./perf -a main -a set_target \ -m /lib/modules/4.0.0-rc5+/kernel/drivers/input/misc/pcspkr.ko \ -a pcspkr_event Added new event: probe:icmp_rcv (on icmp_rcv) You can now use it in all perf tools, such as: perf record -e probe:icmp_rcv -aR sleep 1 Added new event: probe_perf:main (on main in /home/mhiramat/ksrc/linux-3/tools/perf/perf) You can now use it in all perf tools, such as: perf record -e probe_perf:main -aR sleep 1 Added new event: probe_perf:set_target (on set_target in /home/mhiramat/ksrc/linux-3/tools/perf/perf) You can now use it in all perf tools, such as: perf record -e probe_perf:set_target -aR sleep 1 Added new event: probe:pcspkr_event (on pcspkr_event in pcspkr) You can now use it in all perf tools, such as: perf record -e probe:pcspkr_event -aR sleep 1 ----- Reported-by:Arnaldo Carvalho de Melo <acme@infradead.org> Signed-off-by:
-
He Kuang authored
commit: f3b623b8 ("perf tools: Reference count struct thread") appends every thread->node to dead_threads in machine__remove_thread() and list_del_init() this node in thread__put(). perf_event__exit_del_thread() releases thread wihout using machine__remove_thread(), and causes a NULL pointer crash when list_del_init(&thread->node) is called. Fix this by using machine_remove_thread() instead of using thread__put() directly. This problem can be reproduced as following: $ perf record ls $ perf buildid-list --with-hits [ 3874.195070] perf[1018]: segfault at 0 ip 00000000004b0b15 sp 00007ffc35b44780 error 6 in perf[400000+166000] Segmentation fault After this patch: $ perf record ls $ perf buildid-list --with-hits bc23e7c3281e542650ba4324421d6acf78f4c23e /proc/kcore 643324cb0e969f30c56d660f167f84a150845511 [vdso] 0000000000000000000000000000000000000000 /bin/busybox ... Signed-off-by:
-
David Ahern authored
Trying to analyze a big endian data file on little endian system fails with the error: 0xa9b40 [0x70]: failed to process type: 9 The problem is that header parsing is not done correctly because the file attributes are not swapped. Make it so. With this patch able to analyze a sparc64 data file on x86_64. Signed-off-by:
-
Arnaldo Carvalho de Melo authored
When traversing /proc to synthesize the PERF_RECORD_FORK et al events we were bailing out on errors without calling closedir(), fix it. Reported-by:
-
David Ahern authored
Commit ca6c41c5 sets the ppid based on what is read from the /proc/pid/status file when synthesizing fork events. This is correct thing to do for new processes but not threads of a process. Fix ppid for threads to be the main thread when synthesizing fork events (ie., assume main thread spawned all sub-threads in a process). Reported-by:
Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> Signed-off-by:
Don Zickus <dzickus@redhat.com> Link: http://lkml.kernel.org/r/1428598107-178999-1-git-send-email-david.ahern@oracle.comSigned-off-by:
-
- 08 Apr, 2015 15 commits
-
-
Ingo Molnar authored
Merge tag 'perf-core-for-mingo' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo: User visible changes: - Teach 'perf record' about perf_event_attr.clockid (Peter Zijlstra) - Improve 'perf sched replay' on high CPU core count machines (Yunlong Song) - Consider PERF_RECORD_ events with cpumode == 0 in 'perf top', removing one cause of long term memory usage buildup, i.e. not processing PERF_RECORD_EXIT events (Arnaldo Carvalho de Melo) - Add 'I' event modifier for perf_event_attr.exclude_idle bit (Jiri Olsa) - Respect -i option 'in perf kmem' (Jiri Olsa) Infrastructure changes: - Honor operator priority in libtraceevent (Namhyung Kim) - Merge all perf_event_attr print functions (Peter Zijlstra) - Check kmaps access to make code more robust (Wang Nan) - Fix inverted logic in perf_mmap__empty() (He Kuang) - Fix ARM 32 'perf probe' building error (Wang Nan) - Fix perf_event_attr tests (Jiri Olsa) Signed-off-by:
-
Jiri Olsa authored
Adding 'I' event modifier to have complete set of modifiers for perf_event_attr:exclude_* bits. Any event specified with 'I' modifier will have the perf_event_attr:exclude_idle bit set. $ perf record -e cycles:I -vv ls 2>&1 | grep exclude_idle exclude_hv 0 exclude_idle 1 Adding automated tests. Signed-off-by:
-
Wang Nan authored
report__warn_kptr_restrict() calls map__kmap(kernel_map) before checking kernel_map againest NULL. Which is dangerous, since map__kmap() will return a invalid and not NULL address. It will trigger a warning message in map__kmap() after the patch "perf: kmaps: enforce usage of kmaps to protect futher bugs." was applied. This patch fixes it by adding the missing checking. Signed-off-by:
Wang Nan <wangnan0@huawei.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Zefan Li <lizefan@huawei.com> Cc: pi3orama@163.com Link: http://lkml.kernel.org/r/1428490772-135393-1-git-send-email-wangnan0@huawei.comSigned-off-by:
-
Jiri Olsa authored
Following commit: 1a594131 perf: Add wakeup watermark control to the AUX area enlarged perf_event_attr, but did not updated attr tests. Reported-by:
-
Wang Nan authored
Commit 9b118aca ("perf probe: Fix to handle aliased symbols in glibc") uses an absolute format '%lx' to print u64 argument, which causes compiling error on ARM 32. This patch replaces it with PRIx64. Signed-off-by:
-
Peter Zijlstra authored
Currently there's 3 (that I found) different and incomplete implementations of printing perf_event_attr. This is quite silly. Merge the lot. While this patch does not retain the exact form all printing that I found is debug output and thus it should not be critical. Also, I cannot find a single print_event_desc() caller. Pre: $ perf record -vv -e cycles -- sleep 1 ------------------------------------------------------------ perf_event_attr: type 0 size 104 config 0 sample_period 4000 sample_freq 4000 sample_type 0x107 read_format 0 disabled 1 inherit 1 pinned 0 exclusive 0 exclude_user 0 exclude_kernel 0 exclude_hv 0 exclude_idle 0 mmap 1 comm 1 mmap2 1 comm_exec 1 freq 1 inherit_stat 0 enable_on_exec 1 task 1 watermark 0 precise_ip 0 mmap_data 0 sample_id_all 1 exclude_host 0 exclude_guest 1 excl.callchain_kern 0 excl.callchain_user 0 wakeup_events 0 wakeup_watermark 0 bp_type 0 bp_addr 0 config1 0 bp_len 0 config2 0 branch_sample_type 0 sample_regs_user 0 sample_stack_user 0 sample_regs_intr 0 ------------------------------------------------------------ $ perf evlist -vv cycles: sample_freq=4000, size: 104, sample_type: IP|TID|TIME|PERIOD, disabled: 1, inherit: 1, mmap: 1, mmap2: 1, comm: 1, comm_exec: 1, freq: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, exclude_guest: 1 Post: $ ./perf record -vv -e cycles -- sleep 1 ------------------------------------------------------------ perf_event_attr: size 112 { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|PERIOD disabled 1 inherit 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 sample_id_all 1 exclude_guest 1 mmap2 1 comm_exec 1 ------------------------------------------------------------ $ ./perf evlist -vv cycles: size: 112, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|PERIOD, disabled: 1, inherit: 1, mmap: 1, comm: 1, freq: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1 Signed-off-by:Adrian Hunter <adrian.hunter@intel.com> Acked-by:
-
Peter Zijlstra authored
Teach perf-record about the new perf_event_attr::{use_clockid, clockid} fields. Add a simple parameter to set the clock (if any) to be used for the events to be recorded into the data file. Since we store the entire perf_event_attr in the EVENT_DESC section we also already store the used clockid in the data file. Signed-off-by: -
Yunlong Song authored
perf sched replay: Use replay_repeat to calculate the runavg of cpu usage instead of the default value 10 Since sched->replay_repeat is set to 10 as default, the sched->run_avg, sched->runavg_cpu_usage, and sched->runavg_parent_cpu_usage all use 10 to calculate their value. However, the replay_repeat can be changed to other value by using -r option, so the calculation above should use replay_repeat to achieve more accurate results instead of the default value 10. Signed-off-by:
Yunlong Song <yunlong.song@huawei.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/r/1427809596-29559-10-git-send-email-yunlong.song@huawei.comSigned-off-by:
-
Yunlong Song authored
Enable to use perf.data when it is not owned by current user or root. Example: $ ls -al perf.data -rw------- 1 Yunlong.Song Yunlong.Song 5321918 Mar 25 15:14 perf.data $ sudo id uid=0(root) gid=0(root) groups=0(root),64(pkcs11) Before this patch: $ sudo perf sched replay -f run measurement overhead: 98 nsecs sleep measurement overhead: 52909 nsecs the run test took 1000015 nsecs the sleep test took 1054253 nsecs File perf.data not owned by current user or root (use -f to override) As shown above, the -f option does not work at all. After this patch: $ sudo perf sched replay -f run measurement overhead: 221 nsecs sleep measurement overhead: 40514 nsecs the run test took 1000003 nsecs the sleep test took 1056098 nsecs nr_run_events: 10 nr_sleep_events: 1562 nr_wakeup_events: 5 task 0 ( :1: 1), nr_events: 1 task 1 ( :2: 2), nr_events: 1 task 2 ( :3: 3), nr_events: 1 ... ... task 1549 ( :163132: 163132), nr_events: 1 task 1550 ( :163540: 163540), nr_events: 1 task 1551 ( <unknown>: 0), nr_events: 10 ------------------------------------------------------------ #1 : 50.198, ravg: 50.20, cpu: 2335.18 / 2335.18 #2 : 219.099, ravg: 67.09, cpu: 2835.11 / 2385.17 #3 : 238.626, ravg: 84.24, cpu: 3278.26 / 2474.48 #4 : 200.364, ravg: 95.85, cpu: 2977.41 / 2524.77 #5 : 176.882, ravg: 103.96, cpu: 2801.35 / 2552.43 #6 : 191.093, ravg: 112.67, cpu: 2813.70 / 2578.56 #7 : 189.448, ravg: 120.35, cpu: 2809.21 / 2601.62 #8 : 200.637, ravg: 128.38, cpu: 2849.91 / 2626.45 #9 : 248.338, ravg: 140.37, cpu: 4380.61 / 2801.87 #10 : 511.139, ravg: 177.45, cpu: 3077.73 / 2829.45 As shown above, the -f option really works now. Besides for replay, -f option can also work for latency and map. Signed-off-by:
-
Yunlong Song authored
The soft maximum number of open files for a calling process is 1024, which is defined as INR_OPEN_CUR in include/uapi/linux/fs.h, and the hard maximum number of open files for a calling process is 4096, which is defined as INR_OPEN_MAX in include/uapi/linux/fs.h. Both INR_OPEN_CUR and INR_OPEN_MAX are used to limit the value of RLIMIT_NOFILE in include/asm-generic/resource.h. And the soft maximum number finally decides the limitation of the maximum files which are allowed to be opened. That is to say a process can use at most 1024 file descriptors for its o pened files, or an EMFILE error will happen. This error can be fixed by increasing the soft maximum number, under the constraint that the soft maximum number can not exceed the hard maximum number, or both soft and hard maximum number should be increased simultaneously with privilege. For perf sched replay, it uses sys_perf_event_open to create the file descriptor for each of the tasks in order to handle information of perf events. That is to say each task needs a unique file descriptor. In x86_64, there may be over 1024 or 4096 tasks correspoinding to the record in perf.data, which causes that no enough file descriptors can be used. As a result, EMFILE error happens and stops the replay process. To solve this problem, we adaptively increase the soft and hard maximum number of open files with a '-f' option. Example: Test environment: x86_64 with 160 cores $ cat /proc/sys/kernel/pid_max 163840 $ cat /proc/sys/fs/file-max 6815744 $ ulimit -Sn 1024 $ ulimit -Hn 4096 Before this patch: $ perf sched replay ... task 1549 ( :163132: 163132), nr_events: 1 task 1550 ( :163540: 163540), nr_events: 1 task 1551 ( <unknown>: 0), nr_events: 10 Error: sys_perf_event_open() syscall returned with -1 (Too many open files) After this patch: $ perf sched replay ... task 1549 ( :163132: 163132), nr_events: 1 task 1550 ( :163540: 163540), nr_events: 1 task 1551 ( <unknown>: 0), nr_events: 10 Error: sys_perf_event_open() syscall returned with -1 (Too many open files) Have a try with -f option $ perf sched replay -f ... task 1549 ( :163132: 163132), nr_events: 1 task 1550 ( :163540: 163540), nr_events: 1 task 1551 ( <unknown>: 0), nr_events: 10 ------------------------------------------------------------ #1 : 54.401, ravg: 54.40, cpu: 3285.21 / 3285.21 #2 : 199.548, ravg: 68.92, cpu: 4999.65 / 3456.66 #3 : 170.483, ravg: 79.07, cpu: 1349.94 / 3245.99 #4 : 192.034, ravg: 90.37, cpu: 1322.88 / 3053.67 #5 : 182.929, ravg: 99.62, cpu: 1406.51 / 2888.96 #6 : 152.974, ravg: 104.96, cpu: 1167.54 / 2716.82 #7 : 155.579, ravg: 110.02, cpu: 2992.53 / 2744.39 #8 : 130.557, ravg: 112.08, cpu: 1126.43 / 2582.59 #9 : 138.520, ravg: 114.72, cpu: 1253.22 / 2449.65 #10 : 134.328, ravg: 116.68, cpu: 1587.95 / 2363.48 Signed-off-by:
-
Yunlong Song authored
Since there is sem_wait for each task in the wait_for_tasks(), e.g. sem_wait(&task->work_done_sem). The sem_wait can continue only when work_done_sem is greater than 0, or it will be blocked. For perf sched replay, one task may sem_post the work_done_sem of another task, which causes the work_done_sem of that task processed in a reasonable sequence, e.g. sem_post, sem_wait, sem_wait, sem_post... This sequence simulates the sched process of the running tasks at the time when perf sched record runs. As a result, all the tasks are required and their threads must be successfully created. If any one (task A) of the tasks fails to create its thread, then another task (task B), whose work_done_sem needs sem_post from that failed task A, may likely block itself due to seg_wait. And this is a dead halt, since task B's thread_func cannot continue at all. To solve this problem, perf sched replay should exit once any task fails to create its thread. Example: Test environment: x86_64 with 160 cores Before this patch: $ perf sched replay ... Error: sys_perf_event_open() syscall returned with -1 (Too many open files) ------------------------------------------------------------ <- dead halt After this patch: $ perf sched replay ... task 1551 ( <unknown>: 0), nr_events: 10 Error: sys_perf_event_open() syscall returned with -1 (Too many open files) $ As shown above, perf sched replay finishes the process after printing an error message and does not block itself. Signed-off-by:
-
Yunlong Song authored
The pr_err in self_open_counters() prints error message to stderr. Unlike stdout, stderr uses memory buffer on the stack of each calling process. The pr_err in self_open_counters() works in a thread called thread_func created in function create_tasks, which concurrently creates sched->nr_tasks threads. If the error happens and pr_err prints the error message in each of these threads, the stack size of the perf process (default is 8192 kbytes) will quickly run out and the segmentation fault will happen then. To solve this problem, pr_err with self_open_counters() should be moved from newly created threads to the old main thread of the perf process. Then the pr_err can work in a stable situation without the strange segmentation fault problem. Example: Test environment: x86_64 with 160 cores Before this patch: $ perf sched replay ... task 1549 ( :163132: 163132), nr_events: 1 task 1550 ( :163540: 163540), nr_events: 1 task 1551 ( <unknown>: 0), nr_events: 10 Segmentation fault After this patch: $ perf sched replay ... task 1549 ( :163132: 163132), nr_events: 1 task 1550 ( :163540: 163540), nr_events: 1 task 1551 ( <unknown>: 0), nr_events: 10 ... As shown above, the result continues without any segmentation fault. Signed-off-by:
-
Yunlong Song authored
perf sched replay: Realloc the memory of pid_to_task stepwise to adapt to the different pid_max configurations Although the memory of pid_to_task can be allocated via calloc according to the value of /proc/sys/kernel/pid_max, it cannot handle the case when pid_max is changed after 'perf sched record' has created its perf.data. If the new pid_max configured in 'perf sched replay' is smaller than the old pid_max configured in 'perf sched record', then it will cause the assertion failure problem. To solve this problem, we realloc the memory of pid_to_task stepwise once the passed-in pid parameter in register_pid is larger than the current pid_max. Example: Test environment: x86_64 with 160 cores $ cat /proc/sys/kernel/pid_max 163840 $ perf sched record ls $ echo 5000 > /proc/sys/kernel/pid_max $ cat /proc/sys/kernel/pid_max 5000 Before this patch: $ perf sched replay run measurement overhead: 221 nsecs sleep measurement overhead: 55356 nsecs the run test took 1000011 nsecs the sleep test took 1060940 nsecs perf: builtin-sched.c:337: register_pid: Assertion `!(pid >= (unsigned long)pid_max)' failed. Aborted After this patch: $ perf sched replay run measurement overhead: 221 nsecs sleep measurement overhead: 55611 nsecs the run test took 1000026 nsecs the sleep test took 1060486 nsecs nr_run_events: 10 nr_sleep_events: 1562 nr_wakeup_events: 5 task 0 ( :1: 1), nr_events: 1 task 1 ( :2: 2), nr_events: 1 task 2 ( :3: 3), nr_events: 1 task 3 ( :5: 5), nr_events: 1 ... Signed-off-by:
-
Yunlong Song authored
perf sched replay: Alloc the memory of pid_to_task dynamically to adapt to the unexpected change of pid_max The current memory allocation of struct task_desc *pid_to_task[MAX_PID] is in a permanent and preset way, and it has two problems: Problem 1: If the pid_max, which is the max number of pids in the system, is much smaller than MAX_PID (1024*1000), then it causes a waste of stack memory. This may happen in the case where the number of cpu cores is much smaller than 1000. Problem 2: If the pid_max is changed from the default value to a value larger than MAX_PID, then it will cause assertion failure problem. The maximum value of pid_max can be set to pid_max_max (see pidmap_init defined in kernel/pid.c), which equals to PID_MAX_LIMIT. In x86_64, PID_MAX_LIMIT is 4*1024*1024 (defined in include/linux/threads.h). This value is much larger than MAX_PID, and will take up 32768 Kbytes (4*1024*1024*8/1024) for memory allocation of pid_to_task, which is much larger than the default 8192 Kbytes of the stack size of calling process. Due to these two problems, we use calloc to allocate the memory of pid_to_task dynamically. Example: Test environment: x86_64 with 160 cores $ cat /proc/sys/kernel/pid_max 163840 $ echo 1025000 > /proc/sys/kernel/pid_max $ cat /proc/sys/kernel/pid_max 1025000 Run some applications until the pid of some process is greater than the value of MAX_PID (1024*1000). Before this patch: $ perf sched replay run measurement overhead: 221 nsecs sleep measurement overhead: 55480 nsecs the run test took 1000008 nsecs the sleep test took 1063151 nsecs perf: builtin-sched.c:330: register_pid: Assertion `!(pid >= 1024000)' failed. Aborted After this patch: $ perf sched replay run measurement overhead: 221 nsecs sleep measurement overhead: 55435 nsecs the run test took 1000004 nsecs the sleep test took 1059312 nsecs nr_run_events: 10 nr_sleep_events: 1562 nr_wakeup_events: 5 task 0 ( :1: 1), nr_events: 1 task 1 ( :2: 2), nr_events: 1 task 2 ( :3: 3), nr_events: 1 task 3 ( :5: 5), nr_events: 1 ... Signed-off-by:
-
Yunlong Song authored
Current MAX_PID is only 65536, which will cause assertion failure problem when CPU cores are more than 64 in x86_64. This is because the pid_max value in x86_64 is at least PIDS_PER_CPU_DEFAULT * num_possible_cpus() (see function pidmap_init defined in kernel/pid.c), where PIDS_PER_CPU_DEFAULT is 1024 (defined in include/linux/threads.h). Thus for MAX_PID = 65536, the correspoinding CPU cores are 65536/1024=64. This is obviously not enough at all for x86_64, and will cause an assertion failure problem due to BUG_ON(pid >= MAX_PID) in the codes. We increase MAX_PID value from 65536 to 1024*1000, which can be used in x86_64 with 1000 cores. This number is finally decided according to the limitation of stack size of calling process. Use 'ulimit -a', the result shows the stack size of any process is 8192 Kbytes, which is defined in include/uapi/linux/resource.h (#define _STK_LIM (8*1024*1024)). Thus we choose a large enough value for MAX_PID, and make it satisfy to the limitation of the stack size, i.e., making the perf process take up a memory space just smaller than 8192 Kbytes. We have calculated and tested that 1024*1000 is OK for MAX_PID. This means perf sched replay can now be used with at most 1000 cores in x86_64 without any assertion failure problem. Example: Test environment: x86_64 with 160 cores $ cat /proc/sys/kernel/pid_max 163840 Before this patch: $ perf sched replay run measurement overhead: 240 nsecs sleep measurement overhead: 55379 nsecs the run test took 1000004 nsecs the sleep test took 1059424 nsecs perf: builtin-sched.c:330: register_pid: Assertion `!(pid >= 65536)' failed. Aborted After this patch: $ perf sched replay run measurement overhead: 221 nsecs sleep measurement overhead: 55397 nsecs the run test took 999920 nsecs the sleep test took 1053313 nsecs nr_run_events: 10 nr_sleep_events: 1562 nr_wakeup_events: 5 task 0 ( :1: 1), nr_events: 1 task 1 ( :2: 2), nr_events: 1 task 2 ( :3: 3), nr_events: 1 task 3 ( :5: 5), nr_events: 1 ... Signed-off-by:
-
