Fix to probe on gcc generated functions on modules. Since
probing on a module is based on its symbol name, it should
be adjusted on actual symbols.
E.g. without this fix, perf probe shows probe definition
on non-exist symbol as below.
$ perf probe -m build-x86_64/net/netfilter/nf_nat.ko -F in_range*
in_range.isra.12
$ perf probe -m build-x86_64/net/netfilter/nf_nat.ko -D in_range
p:probe/in_range nf_nat:in_range+0
With this fix, perf probe correctly shows a probe on
gcc-generated symbol.
$ perf probe -m build-x86_64/net/netfilter/nf_nat.ko -D in_range
p:probe/in_range nf_nat:in_range.isra.12+0
This also fixes same problem on online module as below.
$ perf probe -m i915 -D assert_plane
p:probe/assert_plane i915:assert_plane.constprop.134+0
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/148411450673.9978.14905987549651656075.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix to show correct locations for events on modules by relocating given
address instead of retrying after failure.
This happens when the module text size is big enough, bigger than
sh_addr, because the original code retries with given address + sh_addr
if it failed to find CU DIE at the given address.
Any address smaller than sh_addr always fails and it retries with the
correct address, but addresses bigger than sh_addr will get a CU DIE
which is on the given address (not adjusted by sh_addr).
In my environment(x86-64), the sh_addr of ".text" section is 0x10030.
Since i915 is a huge kernel module, we can see this issue as below.
$ grep "[Tt] .*\[i915\]" /proc/kallsyms | sort | head -n1
ffffffffc0270000 t i915_switcheroo_can_switch [i915]
ffffffffc0270000 + 0x10030 = ffffffffc0280030, so we'll check
symbols cross this boundary.
$ grep "[Tt] .*\[i915\]" /proc/kallsyms | grep -B1 ^ffffffffc028\
| head -n 2
ffffffffc027ff80 t haswell_init_clock_gating [i915]
ffffffffc0280110 t valleyview_init_clock_gating [i915]
So setup probes on both function and see what happen.
$ sudo ./perf probe -m i915 -a haswell_init_clock_gating \
-a valleyview_init_clock_gating
Added new events:
probe:haswell_init_clock_gating (on haswell_init_clock_gating in i915)
probe:valleyview_init_clock_gating (on valleyview_init_clock_gating in i915)
You can now use it in all perf tools, such as:
perf record -e probe:valleyview_init_clock_gating -aR sleep 1
$ sudo ./perf probe -l
probe:haswell_init_clock_gating (on haswell_init_clock_gating@gpu/drm/i915/intel_pm.c in i915)
probe:valleyview_init_clock_gating (on i915_vga_set_decode:4@gpu/drm/i915/i915_drv.c in i915)
As you can see, haswell_init_clock_gating is correctly shown,
but valleyview_init_clock_gating is not.
With this patch, both events are shown correctly.
$ sudo ./perf probe -l
probe:haswell_init_clock_gating (on haswell_init_clock_gating@gpu/drm/i915/intel_pm.c in i915)
probe:valleyview_init_clock_gating (on valleyview_init_clock_gating@gpu/drm/i915/intel_pm.c in i915)
Committer notes:
In my case:
# perf probe -m i915 -a haswell_init_clock_gating -a valleyview_init_clock_gating
Added new events:
probe:haswell_init_clock_gating (on haswell_init_clock_gating in i915)
probe:valleyview_init_clock_gating (on valleyview_init_clock_gating in i915)
You can now use it in all perf tools, such as:
perf record -e probe:valleyview_init_clock_gating -aR sleep 1
# perf probe -l
probe:haswell_init_clock_gating (on i915_getparam+432@gpu/drm/i915/i915_drv.c in i915)
probe:valleyview_init_clock_gating (on __i915_printk+240@gpu/drm/i915/i915_drv.c in i915)
#
# readelf -SW /lib/modules/4.9.0+/build/vmlinux | egrep -w '.text|Name'
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 1] .text PROGBITS ffffffff81000000 200000 822fd3 00 AX 0 0 4096
#
So both are b0rked, now with the fix:
# perf probe -m i915 -a haswell_init_clock_gating -a valleyview_init_clock_gating
Added new events:
probe:haswell_init_clock_gating (on haswell_init_clock_gating in i915)
probe:valleyview_init_clock_gating (on valleyview_init_clock_gating in i915)
You can now use it in all perf tools, such as:
perf record -e probe:valleyview_init_clock_gating -aR sleep 1
# perf probe -l
probe:haswell_init_clock_gating (on haswell_init_clock_gating@gpu/drm/i915/intel_pm.c in i915)
probe:valleyview_init_clock_gating (on valleyview_init_clock_gating@gpu/drm/i915/intel_pm.c in i915)
#
Both looks correct.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/148411436777.9978.1440275861947194930.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Skip probes if the entry address of the target function is 0. This can
happen when we're handling C++ debuginfo files.
E.g. without this fix, below case still fail.
----
$ ./perf probe -x /usr/lib64/libstdc++.so.6 -vD is_open
probe-definition(0): is_open
symbol:is_open file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib/debug/usr/lib64/libstdc++.so.6.0.22.debug
Try to find probe point from debuginfo.
Matched function: is_open [295df]
found inline addr: 0x8ca80
Probe point found: is_open+0
found inline addr: 0x8ca70
Probe point found: is_open+0
found inline addr: 0x8ca60
Probe point found: is_open+0
Matched function: is_open [6527f]
Matched function: is_open [9fe8a]
Probe point found: is_open+0
Matched function: is_open [19710b]
found inline addr: 0xecca9
Probe point found: stdio_filebuf+57
found inline addr: 0x0
Probe point found: swap+0
Matched function: is_open [19fc9d]
Probe point found: is_open+0
Found 7 probe_trace_events.
p:probe_libstdc++/is_open /usr/lib64/libstdc++.so.6.0.22:0x8ca80
p:probe_libstdc++/is_open_1 /usr/lib64/libstdc++.so.6.0.22:0x8ca70
p:probe_libstdc++/is_open_2 /usr/lib64/libstdc++.so.6.0.22:0x8ca60
p:probe_libstdc++/is_open_3 /usr/lib64/libstdc++.so.6.0.22:0xb0ad0
p:probe_libstdc++/is_open_4 /usr/lib64/libstdc++.so.6.0.22:0xecca9
Failed to synthesize probe trace event.
Error: Failed to add events. Reason: Invalid argument (Code: -22)
----
This is because some instances have entry_pc == 0 (see 19710b and
19fc9d). With this fix, those are skipped.
----
$ ./perf probe -x /usr/lib64/libstdc++.so.6 -D is_open
p:probe_libstdc++/is_open /usr/lib64/libstdc++.so.6.0.22:0x8ca80
p:probe_libstdc++/is_open_1 /usr/lib64/libstdc++.so.6.0.22:0x8ca70
p:probe_libstdc++/is_open_2 /usr/lib64/libstdc++.so.6.0.22:0x8ca60
p:probe_libstdc++/is_open_3 /usr/lib64/libstdc++.so.6.0.22:0xb0ad0
p:probe_libstdc++/is_open_4 /usr/lib64/libstdc++.so.6.0.22:0xecca9
----
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/147464490707.29804.14277897643725143867.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Ignore the error when the perf probe failed to find inline function
instances. This can happen when we search a method in C++ debuginfo. If
there is completely no instance in target, perf probe can return an
error.
E.g. without this fix:
----
$ perf probe -x /usr/lib64/libstdc++.so.6 -vD showmanyc
probe-definition(0): showmanyc
symbol:showmanyc file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib/debug/usr/lib64/libstdc++.so.6.0.22.debug
Try to find probe point from debuginfo.
Matched function: showmanyc
An error occurred in debuginfo analysis (-2).
Trying to use symbols.
Failed to find symbol showmanyc in /usr/lib64/libstdc++.so.6.0.22
Error: Failed to add events. Reason: No such file or directory (Code: -2)
----
This is because one of showmanyc is defined as inline but no instance
found. With this fix, it is succeeded to show as below.
----
$ perf probe -x /usr/lib64/libstdc++.so.6 -D showmanyc
p:probe_libstdc++/showmanyc /usr/lib64/libstdc++.so.6.0.22:0xb0e50
p:probe_libstdc++/showmanyc_1 /usr/lib64/libstdc++.so.6.0.22:0xc7c40
p:probe_libstdc++/showmanyc_2 /usr/lib64/libstdc++.so.6.0.22:0xecfa0
p:probe_libstdc++/showmanyc_3 /usr/lib64/libstdc++.so.6.0.22:0x115fc0
p:probe_libstdc++/showmanyc_4 /usr/lib64/libstdc++.so.6.0.22:0x121a90
----
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/147464489775.29804.3190419491209875936.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Move generic dwarf related functions from util/probe-finder.c to
util/dwarf-aux.c. Functions name and their prototype are also changed
accordingly. No functionality changes.
Suggested-and-Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1472546377-25612-1-git-send-email-ravi.bangoria@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The function prologue prepares stack and registers before executing
function logic.
When target program is compiled without optimization, function parameter
information is only valid after the prologue.
When we probe entrypc of the function, and try to record a function
parameter, it contains a garbage value.
For example:
$ vim test.c
#include <stdio.h>
void foo(int i)
{
printf("i: %d\n", i);
}
int main()
{
foo(42);
return 0;
}
$ gcc -g test.c -o test
$ objdump -dl test | less
foo():
/home/ravi/test.c:4
400536: 55 push %rbp
400537: 48 89 e5 mov %rsp,%rbp
40053a: 48 83 ec 10 sub -bashx10,%rsp
40053e: 89 7d fc mov %edi,-0x4(%rbp)
/home/ravi/test.c:5
400541: 8b 45 fc mov -0x4(%rbp),%eax
...
...
main():
/home/ravi/test.c:9
400558: 55 push %rbp
400559: 48 89 e5 mov %rsp,%rbp
/home/ravi/test.c:10
40055c: bf 2a 00 00 00 mov -bashx2a,%edi
400561: e8 d0 ff ff ff callq 400536 <foo>
$ perf probe -x ./test 'foo i'
$ cat /sys/kernel/debug/tracing/uprobe_events
p:probe_test/foo /home/ravi/test:0x0000000000000536 i=-12(%sp):s32
$ perf record -e probe_test:foo ./test
$ perf script
test 5778 [001] 4918.562027: probe_test:foo: (400536) i=0
Here variable 'i' is passed via stack which is pushed on stack at
0x40053e. But we are probing at 0x400536.
To resolve this issues, we need to probe on next instruction after
prologue. gdb and systemtap also does same thing. I've implemented this
patch based on approach systemtap has used.
After applying patch:
$ perf probe -x ./test 'foo i'
$ cat /sys/kernel/debug/tracing/uprobe_events
p:probe_test/foo /home/ravi/test:0x0000000000000541 i=-4(%bp):s32
$ perf record -e probe_test:foo ./test
$ perf script
test 6300 [001] 5877.879327: probe_test:foo: (400541) i=42
No need to skip prologue for optimized case since debug info is correct
for each instructions for -O2 -g. For more details please visit:
https://bugzilla.redhat.com/show_bug.cgi?id=612253#c6
Changes in v2:
- Skipping prologue only when any ARG is either C variable, $params or
$vars.
- Probe on line(:1) may not be always possible. Recommend only address
to force probe on function entry.
Committer notes:
Testing it with 'perf trace':
# perf probe -x ./test foo i
Added new event:
probe_test:foo (on foo in /home/acme/c/test with i)
You can now use it in all perf tools, such as:
perf record -e probe_test:foo -aR sleep 1
# cat /sys/kernel/debug/tracing/uprobe_events
p:probe_test/foo /home/acme/c/test:0x0000000000000526 i=-12(%sp):s32
# trace --no-sys --event probe_*:* ./test
i: 42
0.000 probe_test:foo:(400526) i=0)
#
After the patch:
# perf probe -d *:*
Removed event: probe_test:foo
# perf probe -x ./test foo i
Target program is compiled without optimization. Skipping prologue.
Probe on address 0x400526 to force probing at the function entry.
Added new event:
probe_test:foo (on foo in /home/acme/c/test with i)
You can now use it in all perf tools, such as:
perf record -e probe_test:foo -aR sleep 1
# cat /sys/kernel/debug/tracing/uprobe_events
p:probe_test/foo /home/acme/c/test:0x0000000000000531 i=-4(%bp):s32
# trace --no-sys --event probe_*:* ./test
i: 42
0.000 probe_test:foo:(400531) i=42)
#
Reported-by: Michael Petlan <mpetlan@redhat.com>
Report-Link: https://www.mail-archive.com/linux-perf-users@vger.kernel.org/msg02348.html
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Acked-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Wang Nan <wangnan0@huawei.com>
Cc: Yauheni Kaliuta <yauheni.kaliuta@redhat.com>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1299021
Link: http://lkml.kernel.org/r/1470214725-5023-2-git-send-email-ravi.bangoria@linux.vnet.ibm.com
[ Rename 'die' to 'cu_die' to avoid shadowing a die() definition on at least centos 5, Debian 7 and ubuntu:12.04.5]
[ Use PRIx64 instead of lx to format a Dwarf_Addr, aka long long unsigned int, fixing the build on 32-bit systems ]
[ dwarf_getsrclines() expects a size_t * argument ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Support probing on offline cross-architecture binary by adding getting
the target machine arch from ELF and choose correct register string for
the machine.
Here is an example:
-----
$ perf probe --vmlinux=./vmlinux-arm --definition 'do_sys_open $params'
p:probe/do_sys_open do_sys_open+0 dfd=%r5:s32 filename=%r1:u32 flags=%r6:s32 mode=%r3:u16
-----
Here, we can get probe/do_sys_open from above and append it to to the target
machine's tracing/kprobe_events file in the tracefs mountput, usually
/sys/kernel/debug/tracing/kprobe_events (or /sys/kernel/tracing/kprobe_events).
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/147214229717.23638.6440579792548044658.stgit@devbox
[ Add definition for EM_AARCH64 to fix the build on at least centos 6, debian 7 & ubuntu 12.04.5 ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Use hexadecimal type by default if it is available on current running
kernel.
This keeps the default behavior of perf probe after changing the output
format of 'u8/16/32/64' to unsigned decimal number.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Naohiro Aota <naohiro.aota@hgst.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/147151074685.12957.16415861010796255514.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Support hexadecimal unsigned integer casting by 'x'. This allows user
to explicitly specify the output format of the probe arguments as
hexadecimal.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Naohiro Aota <naohiro.aota@hgst.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/147151072679.12957.4458656416765710753.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add a checking routine what types are supported by the running kernel by
finding the pattern in <debugfs>/tracing/README.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Naohiro Aota <naohiro.aota@hgst.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/147151071172.12957.3340095690753291085.stgit@devbox
[ 'enum probe_type' has no negative entries, so ends up as 'unsigned', remove '< 0'
test to fix the build on at least centos:5, debian:7 & ubuntu:12.04.5 ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The 'perf probe' tool detects a variable's type and use the detected
type to add a new probe. Then, kprobes prints its variable in
hexadecimal format if the variable is unsigned and prints in decimal if
it is signed.
We sometimes want to see unsigned variable in decimal format (i.e.
sector_t or size_t). In that case, we need to investigate the variable's
size manually to specify just signedness.
This patch add signedness casting support. By specifying "s" or "u" as a
type, perf-probe will investigate variable size as usual and use the
specified signedness.
E.g. without this:
$ perf probe -a 'submit_bio bio->bi_iter.bi_sector'
Added new event:
probe:submit_bio (on submit_bio with bi_sector=bio->bi_iter.bi_sector)
You can now use it in all perf tools, such as:
perf record -e probe:submit_bio -aR sleep 1
$ cat trace_pipe|head
dbench-9692 [003] d..1 971.096633: submit_bio: (submit_bio+0x0/0x140) bi_sector=0x3a3d00
dbench-9692 [003] d..1 971.096685: submit_bio: (submit_bio+0x0/0x140) bi_sector=0x1a3d80
dbench-9692 [003] d..1 971.096687: submit_bio: (submit_bio+0x0/0x140) bi_sector=0x3a3d80
...
// need to investigate the variable size
$ perf probe -a 'submit_bio bio->bi_iter.bi_sector:s64'
Added new event:
probe:submit_bio (on submit_bio with bi_sector=bio->bi_iter.bi_sector:s64)
You can now use it in all perf tools, such as:
perf record -e probe:submit_bio -aR sleep 1
With this:
// just use "s" to cast its signedness
$ perf probe -v -a 'submit_bio bio->bi_iter.bi_sector:s'
Added new event:
probe:submit_bio (on submit_bio with bi_sector=bio->bi_iter.bi_sector:s)
You can now use it in all perf tools, such as:
perf record -e probe:submit_bio -aR sleep 1
$ cat trace_pipe|head
dbench-9689 [001] d..1 1212.391237: submit_bio: (submit_bio+0x0/0x140) bi_sector=128
dbench-9689 [001] d..1 1212.391252: submit_bio: (submit_bio+0x0/0x140) bi_sector=131072
dbench-9697 [006] d..1 1212.398611: submit_bio: (submit_bio+0x0/0x140) bi_sector=30208
This commit also update perf-probe.txt to describe "types". Most parts
are based on existing documentation: Documentation/trace/kprobetrace.txt
Committer note:
Testing using 'perf trace':
# perf probe -a 'submit_bio bio->bi_iter.bi_sector'
Added new event:
probe:submit_bio (on submit_bio with bi_sector=bio->bi_iter.bi_sector)
You can now use it in all perf tools, such as:
perf record -e probe:submit_bio -aR sleep 1
# trace --no-syscalls --ev probe:submit_bio
0.000 probe:submit_bio:(ffffffffac3aee00) bi_sector=0xc133c0)
3181.861 probe:submit_bio:(ffffffffac3aee00) bi_sector=0x6cffb8)
3181.881 probe:submit_bio:(ffffffffac3aee00) bi_sector=0x6cffc0)
3184.488 probe:submit_bio:(ffffffffac3aee00) bi_sector=0x6cffc8)
<SNIP>
4717.927 probe:submit_bio:(ffffffffac3aee00) bi_sector=0x4dc7a88)
4717.970 probe:submit_bio:(ffffffffac3aee00) bi_sector=0x4dc7880)
^C[root@jouet ~]#
Now, using this new feature:
[root@jouet ~]# perf probe -a 'submit_bio bio->bi_iter.bi_sector:s'
Added new event:
probe:submit_bio (on submit_bio with bi_sector=bio->bi_iter.bi_sector:s)
You can now use it in all perf tools, such as:
perf record -e probe:submit_bio -aR sleep 1
[root@jouet ~]# trace --no-syscalls --ev probe:submit_bio
0.000 probe:submit_bio:(ffffffffac3aee00) bi_sector=7145704)
0.017 probe:submit_bio:(ffffffffac3aee00) bi_sector=7145712)
0.019 probe:submit_bio:(ffffffffac3aee00) bi_sector=7145720)
2.567 probe:submit_bio:(ffffffffac3aee00) bi_sector=7145728)
5631.919 probe:submit_bio:(ffffffffac3aee00) bi_sector=0)
5631.941 probe:submit_bio:(ffffffffac3aee00) bi_sector=8)
5631.945 probe:submit_bio:(ffffffffac3aee00) bi_sector=16)
5631.948 probe:submit_bio:(ffffffffac3aee00) bi_sector=24)
^C#
With callchains:
# trace --no-syscalls --ev probe:submit_bio/max-stack=10/
0.000 probe:submit_bio:(ffffffffac3aee00) bi_sector=50662544)
submit_bio+0xa8200001 ([kernel.kallsyms])
submit_bh+0xa8200013 ([kernel.kallsyms])
jbd2_journal_commit_transaction+0xa8200691 ([kernel.kallsyms])
kjournald2+0xa82000ca ([kernel.kallsyms])
kthread+0xa82000d8 ([kernel.kallsyms])
ret_from_fork+0xa820001f ([kernel.kallsyms])
0.023 probe:submit_bio:(ffffffffac3aee00) bi_sector=50662552)
submit_bio+0xa8200001 ([kernel.kallsyms])
submit_bh+0xa8200013 ([kernel.kallsyms])
jbd2_journal_commit_transaction+0xa8200691 ([kernel.kallsyms])
kjournald2+0xa82000ca ([kernel.kallsyms])
kthread+0xa82000d8 ([kernel.kallsyms])
ret_from_fork+0xa820001f ([kernel.kallsyms])
0.027 probe:submit_bio:(ffffffffac3aee00) bi_sector=50662560)
submit_bio+0xa8200001 ([kernel.kallsyms])
submit_bh+0xa8200013 ([kernel.kallsyms])
jbd2_journal_commit_transaction+0xa8200691 ([kernel.kallsyms])
kjournald2+0xa82000ca ([kernel.kallsyms])
kthread+0xa82000d8 ([kernel.kallsyms])
ret_from_fork+0xa820001f ([kernel.kallsyms])
2.593 probe:submit_bio:(ffffffffac3aee00) bi_sector=50662568)
submit_bio+0xa8200001 ([kernel.kallsyms])
submit_bh+0xa8200013 ([kernel.kallsyms])
journal_submit_commit_record+0xa82001ac ([kernel.kallsyms])
jbd2_journal_commit_transaction+0xa82012e8 ([kernel.kallsyms])
kjournald2+0xa82000ca ([kernel.kallsyms])
kthread+0xa82000d8 ([kernel.kallsyms])
ret_from_fork+0xa820001f ([kernel.kallsyms])
^C#
Signed-off-by: Naohiro Aota <naohiro.aota@hgst.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1470710408-23515-1-git-send-email-naohiro.aota@hgst.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The tools so far have been using the strerror_r() GNU variant, that
returns a string, be it the buffer passed or something else.
But that, besides being tricky in cases where we expect that the
function using strerror_r() returns the error formatted in a provided
buffer (we have to check if it returned something else and copy that
instead), breaks the build on systems not using glibc, like Alpine
Linux, where musl libc is used.
So, introduce yet another wrapper, str_error_r(), that has the GNU
interface, but uses the portable XSI variant of strerror_r(), so that
users rest asured that the provided buffer is used and it is what is
returned.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-d4t42fnf48ytlk8rjxs822tf@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Check the return value of strbuf APIs in perf-probe
related code, so that it can handle errors in strbuf.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20160510054707.6158.69861.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We have addch() for chars, add() for fixed size data, and addstr() for
variable length strings, use them.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-0ap02fn2xtvpduj2j6b2o1j4@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
'perf probe' through debuginfo__find_probes() in util/probe-finder.c
checks for the functions' frame descriptions in either .eh_frame section
of an ELF or the .debug_frame.
The check is based on whether either one of these sections is present.
Depending on distro, toolchain defaults, architetcutre, build flags,
etc., CFI might be found in either .eh_frame and/or .debug_frame.
Sometimes, it may happen that, .eh_frame, even if present, may not be
complete and may miss some descriptions.
Therefore, to be sure, to find the CFI covering an address we will
always have to investigate both if available.
For e.g., in powerpc, this may happen:
$ gcc -g bin.c -o bin
$ objdump --dwarf ./bin
<1><145>: Abbrev Number: 7 (DW_TAG_subprogram)
<146> DW_AT_external : 1
<146> DW_AT_name : (indirect string, offset: 0x9e): main
<14a> DW_AT_decl_file : 1
<14b> DW_AT_decl_line : 39
<14c> DW_AT_prototyped : 1
<14c> DW_AT_type : <0x57>
<150> DW_AT_low_pc : 0x100007b8
If the .eh_frame and .debug_frame are checked for the same binary, we
will find that, .eh_frame (although present) doesn't contain a
description for "main" function.
But, .debug_frame has a description:
000000d8 00000024 00000000 FDE cie=00000000 pc=100007b8..10000838
DW_CFA_advance_loc: 16 to 100007c8
DW_CFA_def_cfa_offset: 144
DW_CFA_offset_extended_sf: r65 at cfa+16
...
Due to this (since, perf checks whether .eh_frame is present and goes on
searching for that address inside that frame), perf is unable to process
the probes:
# perf probe -x ./bin main
Failed to get call frame on 0x100007b8
Error: Failed to add events.
To avoid this issue, we need to check both the sections (.eh_frame and
.debug_frame), which is done in this patch.
Note that, we can always force everything into both .eh_frame and
.debug_frame by:
$ gcc bin.c -fasynchronous-unwind-tables -fno-dwarf2-cfi-asm -g -o bin
Signed-off-by: Hemant Kumar <hemant@linux.vnet.ibm.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: Mark Wielaard <mjw@redhat.com>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Link: http://lkml.kernel.org/r/1454426806-13974-1-git-send-email-hemant@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The commit 05c8d802fa ("perf probe: Fix to free temporal Dwarf_Frame")
tried to fix the memory leak of Dwarf_Frame, but it released the frame
at wrong point. Since the dwarf_frame_cfa(frame, &pf->fb_ops, &nops) can
return an address inside the frame data structure to pf->fb_ops, we can
not release the frame before using pf->fb_ops.
This reverts the commit and releases the frame afterwards (right before
returning from call_probe_finder) correctly.
Reported-and-Tested-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Reported-by: Michael Petlan <mpetlan@redhat.com>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Fixes: 05c8d802fa ("perf probe: Fix to free temporal Dwarf_Frame")
LPU-Reference: 20151125103432.1473.31009.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since dwarf_cfi_addrframe returns malloc'd Dwarf_Frame object, it has to
be freed after it is used.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/20151118064011.30709.65674.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When probing with a glob, errors in add_probe_trace_event() won't be
passed to debuginfo__find_trace_events() because it would be modified by
probe_point_search_cb(). It causes a segfault if perf fails to find an
argument for a probe point matched by the glob. For example:
# ./perf probe -v -n 'SyS_dup? oldfd'
probe-definition(0): SyS_dup? oldfd
symbol:SyS_dup? file:(null) line:0 offset:0 return:0 lazy:(null)
parsing arg: oldfd into oldfd
1 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/4.3.0-rc4+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.3.0-rc4+/build/vmlinux
Try to find probe point from debuginfo.
Matched function: SyS_dup3
found inline addr: 0xffffffff812095c0
Probe point found: SyS_dup3+0
Searching 'oldfd' variable in context.
Converting variable oldfd into trace event.
oldfd type is long int.
found inline addr: 0xffffffff812096d4
Probe point found: SyS_dup2+36
Searching 'oldfd' variable in context.
Failed to find 'oldfd' in this function.
Matched function: SyS_dup3
Probe point found: SyS_dup3+0
Searching 'oldfd' variable in context.
Converting variable oldfd into trace event.
oldfd type is long int.
Matched function: SyS_dup2
Probe point found: SyS_dup2+0
Searching 'oldfd' variable in context.
Converting variable oldfd into trace event.
oldfd type is long int.
Found 4 probe_trace_events.
Opening /sys/kernel/debug/tracing//kprobe_events write=1
Writing event: p:probe/SyS_dup3 _text+2135488 oldfd=%di:s64
Segmentation fault (core dumped)
#
This patch ensures that add_probe_trace_event() doesn't touches
tf->ntevs and tf->tevs if those functions fail.
After the patch:
# perf probe 'SyS_dup? oldfd'
Failed to find 'oldfd' in this function.
Added new events:
probe:SyS_dup3 (on SyS_dup? with oldfd)
probe:SyS_dup3_1 (on SyS_dup? with oldfd)
probe:SyS_dup2 (on SyS_dup? with oldfd)
You can now use it in all perf tools, such as:
perf record -e probe:SyS_dup2 -aR sleep 1
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1447417761-156094-3-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix memory leaking on the debuginfo__find_trace_events() failure path
which frees an array of probe_trace_events but doesn't clears all the
allocated sub-structures and strings.
So, before doing zfree(tevs), clear all the array elements which may
have allocated resources.
Reported-by: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1447417761-156094-2-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf probe shows more precisely message when it finds given
%return target function is inlined.
Without this fix:
----
# ./perf probe -V getname_flags%return
Return probe must be on the head of a real function.
Debuginfo analysis failed.
Error: Failed to show vars.
----
With this fix:
----
# ./perf probe -V getname_flags%return
Failed to find "getname_flags%return",
because getname_flags is an inlined function and has no return point.
Debuginfo analysis failed.
Error: Failed to show vars.
----
Suggested-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/20150930164137.3733.55055.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf probe always failed to find appropriate line numbers because of
failing to find .text start address offset from debuginfo.
e.g.
----
# ./perf probe -m pcspkr pcspkr_event:5
Added new events:
probe:pcspkr_event (on pcspkr_event:5 in pcspkr)
probe:pcspkr_event_1 (on pcspkr_event:5 in pcspkr)
You can now use it in all perf tools, such as:
perf record -e probe:pcspkr_event_1 -aR sleep 1
# ./perf probe -l
Failed to find debug information for address ffffffffa031f006
Failed to find debug information for address ffffffffa031f016
probe:pcspkr_event (on pcspkr_event+6 in pcspkr)
probe:pcspkr_event_1 (on pcspkr_event+22 in pcspkr)
----
This fixes the above issue as below.
1. Get the relative address of the symbol in .text by using
map->start.
2. Adjust the address by adding the offset of .text section
in the kernel module binary.
With this fix, perf probe -l shows lines correctly.
----
# ./perf probe -l
probe:pcspkr_event (on pcspkr_event:5@drivers/input/misc/pcspkr.c in pcspkr)
probe:pcspkr_event_1 (on pcspkr_event:5@drivers/input/misc/pcspkr.c in pcspkr)
----
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/20150930164132.3733.24643.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix a trival bug about libdwfl usage of the report session, it should
explicitly begin and end a report session around dwfl_report_offline().
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/20150930164128.3733.59876.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It should be useful to allow 'perf probe' probe at absolute offset of a
target. For example, when (u)probing at a instruction of a shared object
in a embedded system where debuginfo is not avaliable but we know the
offset of that instruction by manually digging.
This patch enables following perf probe command syntax:
# perf probe 0xffffffff811e6615
And
# perf probe /lib/x86_64-linux-gnu/libc-2.19.so 0xeb860
In the above example, we don't need a anchor symbol, so it is possible
to compute absolute addresses using other methods and then use 'perf
probe' to create the probing points.
v1 -> v2:
Drop the leading '+' in cmdline;
Allow uprobing at offset 0x0;
Improve 'perf probe -l' result when uprobe at area without debuginfo.
v2 -> v3:
Split bugfix to a separated patch.
Test result:
# perf probe 0xffffffff8119d175 %ax
# perf probe sys_write %ax
# perf probe /lib64/libc-2.18.so 0x0 %ax
# perf probe /lib64/libc-2.18.so 0x5 %ax
# perf probe /lib64/libc-2.18.so 0xd8e40 %ax
# perf probe /lib64/libc-2.18.so __write %ax
# perf probe /lib64/libc-2.18.so 0xd8e49 %ax
# cat /sys/kernel/debug/tracing/uprobe_events
p:probe_libc/abs_0 /lib64/libc-2.18.so:0x (null) arg1=%ax
p:probe_libc/abs_5 /lib64/libc-2.18.so:0x0000000000000005 arg1=%ax
p:probe_libc/abs_d8e40 /lib64/libc-2.18.so:0x00000000000d8e40 arg1=%ax
p:probe_libc/__write /lib64/libc-2.18.so:0x00000000000d8e40 arg1=%ax
p:probe_libc/abs_d8e49 /lib64/libc-2.18.so:0x00000000000d8e49 arg1=%ax
# cat /sys/kernel/debug/tracing/kprobe_events
p:probe/abs_ffffffff8119d175 0xffffffff8119d175 arg1=%ax
p:probe/sys_write _text+1692016 arg1=%ax
# perf probe -l
Failed to find debug information for address 5
probe:abs_ffffffff8119d175 (on sys_write+5 with arg1)
probe:sys_write (on sys_write with arg1)
probe_libc:__write (on @unix/syscall-template.S:81 in /lib64/libc-2.18.so with arg1)
probe_libc:abs_0 (on 0x0 in /lib64/libc-2.18.so with arg1)
probe_libc:abs_5 (on 0x5 in /lib64/libc-2.18.so with arg1)
probe_libc:abs_d8e40 (on @unix/syscall-template.S:81 in /lib64/libc-2.18.so with arg1)
probe_libc:abs_d8e49 (on __GI___libc_write+9 in /lib64/libc-2.18.so with arg1)
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1440586666-235233-7-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
So that we can pass more info to strlist__new() without having to change
its function signature, just adding entries to the strlist_config struct
with sensible defaults for when those fields are not specified.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-5uaaler4931i0s9sedxjquhq@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This has a different model than the 'thread' and 'map' struct lifetimes:
there is not a definitive "don't use this DSO anymore" event, i.e. we may
get many 'struct map' holding references to the '/usr/lib64/libc-2.20.so'
DSO but then at some point some DSO may have no references but we still
don't want to straight away release its resources, because "soon" we may
get a new 'struct map' that needs it and we want to reuse its symtab or
other resources.
So we need some way to garbage collect it when crossing some memory
usage threshold, which is left for anoter patch, for now it is
sufficient to release it when calling dsos__exit(), i.e. when deleting
the whole list as part of deleting the 'struct machine' containing it,
which will leave only referenced objects being used.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/n/tip-majzgz07cm90t2tejrjy4clf@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf probe currently errors out if there are any tail calls to probed
functions:
[root@rhel71be]# perf probe do_fork
Failed to find probe point in any functions.
Error: Failed to add events.
Fix this by teaching perf to ignore tail calls.
Without patch:
[root@rhel71be perf]# ./perf probe -v do_fork
probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0
return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
symsrc__init: build id mismatch for /boot/vmlinux.
Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for symbols
Open Debuginfo file:
/usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux
Try to find probe point from debuginfo.
found inline addr: 0xc0000000000bb9b0
Probe point found: do_fork+0
found inline addr: 0xc0000000000bbe20
Probe point found: kernel_thread+48
found inline addr: 0xc0000000000bbe5c
Probe point found: sys_fork+28
found inline addr: 0xc0000000000bbfac
Probe point found: sys_vfork+44
found inline addr: 0xc0000000000bc27c
Failed to find probe point in any functions.
An error occurred in debuginfo analysis (-2).
Error: Failed to add events. Reason: No such file or directory (Code: -2)
With patch:
[root@rhel71be perf]# ./perf probe -v do_fork
probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0
return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
symsrc__init: build id mismatch for /boot/vmlinux.
Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for symbols
Open Debuginfo file:
/usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux
Try to find probe point from debuginfo.
found inline addr: 0xc0000000000bb9b0
Probe point found: do_fork+0
found inline addr: 0xc0000000000bbe20
Probe point found: kernel_thread+48
found inline addr: 0xc0000000000bbe5c
Probe point found: sys_fork+28
found inline addr: 0xc0000000000bbfac
Probe point found: sys_vfork+44
found inline addr: 0xc0000000000bc27c
Ignoring tail call from SyS_clone
Found 4 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
No kprobe blacklist support, ignored
Added new events:
Writing event: p:probe/do_fork _text+768432
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[Ignore the error about failure to write event - this kernel is missing
a patch to resolve _text properly]
The reason to ignore tail calls is that the address does not belong to
any function frame. In the example above, the address in SyS_clone is
0xc0000000000bc27c, but looking at the debug-info:
<1><830081>: Abbrev Number: 133 (DW_TAG_subprogram)
<830083> DW_AT_external : 1
<830083> DW_AT_name : (indirect string, offset: 0x3cea3): SyS_clone
<830087> DW_AT_decl_file : 7
<830088> DW_AT_decl_line : 1689
<83008a> DW_AT_prototyped : 1
<83008a> DW_AT_type : <0x8110eb>
<83008e> DW_AT_low_pc : 0xc0000000000bc270
<830096> DW_AT_high_pc : 0xc
<83009e> DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa)
<8300a0> DW_AT_GNU_all_call_sites: 1
<8300a0> DW_AT_sibling : <0x830178>
<snip>
<3><830147>: Abbrev Number: 125 (DW_TAG_GNU_call_site)
<830148> DW_AT_low_pc : 0xc0000000000bc27c
<830150> DW_AT_GNU_tail_call: 1
<830150> DW_AT_abstract_origin: <0x82e7e1>
The frame ends at 0xc0000000000bc27c. I suppose this is why this
particular call is a "tail" call. FWIW, systemtap seems to ignore these
as well and requires users to explicitly place probes at these call
sites if necessary. I print out the caller so that users know.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Link: http://lkml.kernel.org/r/1430394151-15928-1-git-send-email-naveen.n.rao@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Indicate to check variable location range in error message when we got
failed to find the variable.
Before this patch:
$ perf probe --add 'generic_perform_write+118 bytes'
Failed to find the location of bytes at this address.
Perhaps, it has been optimized out.
Error: Failed to add events.
After this patch:
$ perf probe --add 'generic_perform_write+118 bytes'
Failed to find the location of the 'bytes' variable at this address.
Perhaps it has been optimized out.
Use -V with the --range option to show 'bytes' location range.
Error: Failed to add events.
Signed-off-by: He Kuang <hekuang@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1431336304-16863-3-git-send-email-hekuang@huawei.com
[ Improve the error message based on lkml thread ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It is not easy for users to get the accurate byte offset or the line
number where a local variable can be probed.
With '--range' option, local variables in the scope of the probe point
are showed with a byte offset range, and can be added according to this
range information.
For example, there are some variables in the function
generic_perform_write():
<generic_perform_write@mm/filemap.c:0>
0 ssize_t generic_perform_write(struct file *file,
1 struct iov_iter *i, loff_t pos)
2 {
3 struct address_space *mapping = file->f_mapping;
4 const struct address_space_operations *a_ops = mapping->a_ops;
...
42 status = a_ops->write_begin(file, mapping, pos, bytes, flags,
&page, &fsdata);
44 if (unlikely(status < 0))
But we fail when we try to probe the variable 'a_ops' at line 42 or 44.
$ perf probe --add 'generic_perform_write:42 a_ops'
Failed to find the location of a_ops at this address.
Perhaps, it has been optimized out.
This is because the source code do not match the assembly, so a variable
may not be available in the source code line where it appears.
After this patch, we can lookup the accurate byte offset range of a
variable, 'INV' indicates that this variable is not valid at the given
point, but available in the scope:
$ perf probe --vars 'generic_perform_write:42' --range
Available variables at generic_perform_write:42
@<generic_perform_write+141>
[INV] ssize_t written @<generic_perform_write+[324-331]>
[INV] struct address_space_operations* a_ops @<generic_perform_write+[55-61,170-176,223-246]>
[VAL] (unknown_type) fsdata @<generic_perform_write+[70-307,346-411]>
[VAL] loff_t pos @<generic_perform_write+[0-286,286-336,346-411]>
[VAL] long int status @<generic_perform_write+[83-342,346-411]>
[VAL] long unsigned int bytes @<generic_perform_write+[122-311,320-338,346-403,403-411]>
[VAL] struct address_space* mapping @<generic_perform_write+[35-344,346-411]>
[VAL] struct iov_iter* i @<generic_perform_write+[0-340,346-411]>
[VAL] struct page* page @<generic_perform_write+[70-307,346-411]>
Then it is more clear for us to add a probe with this variable:
$ perf probe --add 'generic_perform_write+170 a_ops'
Added new event:
probe:generic_perform_write (on generic_perform_write+170 with a_ops)
Signed-off-by: He Kuang <hekuang@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1431336304-16863-2-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Use struct strbuf instead of bare char[] to remove the length limitation
of variables in variable_list, so they will not disappear due to
overlength, and make preparation for adding more description for
variables.
Signed-off-by: He Kuang <hekuang@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1431336304-16863-1-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Support glob wildcards for function name when adding new probes. This
will allow us to build caches of function-entry level information with
$params.
e.g.
----
# perf probe --no-inlines --add 'kmalloc* $params'
Added new events:
probe:kmalloc_slab (on kmalloc* with $params)
probe:kmalloc_large_node (on kmalloc* with $params)
probe:kmalloc_order_trace (on kmalloc* with $params)
You can now use it in all perf tools, such as:
perf record -e probe:kmalloc_order_trace -aR sleep 1
# perf probe --list
probe:kmalloc_large_node (on kmalloc_large_node@mm/slub.c with size flags node)
probe:kmalloc_order_trace (on kmalloc_order_trace@mm/slub.c with size flags order)
probe:kmalloc_slab (on kmalloc_slab@mm/slab_common.c with size flags)
----
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150508010335.24812.19972.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add --no-inlines(--inlines) option to avoid searching inline functions.
Searching all functions which matches glob pattern can take a long time
and find a lot of inline functions.
With this option perf-probe searches target on the non-inlined
functions.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150508010333.24812.86568.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Introduce probe_conf global configuration parameters for probe-event and
probe-finder, and removes related parameters from APIs.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150508010330.24812.21095.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Pull perf fixes from Ingo Molnar:
"Mostly tooling fixes, but also an uncore PMU driver fix and an uncore
PMU driver hardware-enablement addition"
* 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
perf probe: Fix segfault if passed with ''.
perf report: Fix -T/--threads option to work again
perf bench numa: Fix immediate meeting of convergence condition
perf bench numa: Fixes of --quiet argument
perf bench futex: Fix hung wakeup tasks after requeueing
perf probe: Fix bug with global variables handling
perf top: Fix a segfault when kernel map is restricted.
tools lib traceevent: Fix build failure on 32-bit arch
perf kmem: Fix compiles on RHEL6/OL6
tools lib api: Undefine _FORTIFY_SOURCE before setting it
perf kmem: Consistently use PRIu64 for printing u64 values
perf trace: Disable events and drain events when forked workload ends
perf trace: Enable events when doing system wide tracing and starting a workload
perf/x86/intel/uncore: Move PCI IDs for IMC to uncore driver
perf/x86/intel/uncore: Add support for Intel Haswell ULT (lower power Mobile Processor) IMC uncore PMUs
perf/x86/intel: Add cpu_(prepare|starting|dying) for core_pmu
There are missing curly braces which causes find_variable() return wrong
value when probing with global variables.
This problem can be reproduced as following:
$ perf probe -v --add='generic_perform_write global_variable_for_test'
...
Try to find probe point from debuginfo.
Probe point found: generic_perform_write+0
Searching 'global_variable_for_test' variable in context.
An error occurred in debuginfo analysis (-2).
Error: Failed to add events. Reason: No such file or directory (Code: -2)
After this patch:
$ perf probe -v --add='generic_perform_write global_variable_for_test'
...
Converting variable global_variable_for_test into trace event.
global_variable_for_test type is int.
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: p:probe/generic_perform_write _stext+1237464
global_variable_for_test=@global_variable_for_test+0:s32
probe:generic_perform_write (on generic_perform_write with
global_variable_for_test)
You can now use it in all perf tools, such as:
perf record -e probe:generic_perform_write -aR sleep 1
Signed-off-by: He Kuang <hekuang@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1429949338-18678-1-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Pull perf updates from Ingo Molnar:
"This update has mostly fixes, but also other bits:
- perf tooling fixes
- PMU driver fixes
- Intel Broadwell PMU driver HW-enablement for LBR callstacks
- a late coming 'perf kmem' tool update that enables it to also
analyze page allocation data. Note, this comes with MM tracepoint
changes that we believe to not break anything: because it changes
the formerly opaque 'struct page *' field that uniquely identifies
pages to 'pfn' which identifies pages uniquely too, but isn't as
opaque and can be used for other purposes as well"
* 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
perf/x86/intel/pt: Fix and clean up error handling in pt_event_add()
perf/x86/intel: Add Broadwell support for the LBR callstack
perf/x86/intel/rapl: Fix energy counter measurements but supporing per domain energy units
perf/x86/intel: Fix Core2,Atom,NHM,WSM cycles:pp events
perf/x86: Fix hw_perf_event::flags collision
perf probe: Fix segfault when probe with lazy_line to file
perf probe: Find compilation directory path for lazy matching
perf probe: Set retprobe flag when probe in address-based alternative mode
perf kmem: Analyze page allocator events also
tracing, mm: Record pfn instead of pointer to struct page
Pull perf changes from Ingo Molnar:
"Core kernel changes:
- One of the more interesting features in this cycle is the ability
to attach eBPF programs (user-defined, sandboxed bytecode executed
by the kernel) to kprobes.
This allows user-defined instrumentation on a live kernel image
that can never crash, hang or interfere with the kernel negatively.
(Right now it's limited to root-only, but in the future we might
allow unprivileged use as well.)
(Alexei Starovoitov)
- Another non-trivial feature is per event clockid support: this
allows, amongst other things, the selection of different clock
sources for event timestamps traced via perf.
This feature is sought by people who'd like to merge perf generated
events with external events that were measured with different
clocks:
- cluster wide profiling
- for system wide tracing with user-space events,
- JIT profiling events
etc. Matching perf tooling support is added as well, available via
the -k, --clockid <clockid> parameter to perf record et al.
(Peter Zijlstra)
Hardware enablement kernel changes:
- x86 Intel Processor Trace (PT) support: which is a hardware tracer
on steroids, available on Broadwell CPUs.
The hardware trace stream is directly output into the user-space
ring-buffer, using the 'AUX' data format extension that was added
to the perf core to support hardware constraints such as the
necessity to have the tracing buffer physically contiguous.
This patch-set was developed for two years and this is the result.
A simple way to make use of this is to use BTS tracing, the PT
driver emulates BTS output - available via the 'intel_bts' PMU.
More explicit PT specific tooling support is in the works as well -
will probably be ready by 4.2.
(Alexander Shishkin, Peter Zijlstra)
- x86 Intel Cache QoS Monitoring (CQM) support: this is a hardware
feature of Intel Xeon CPUs that allows the measurement and
allocation/partitioning of caches to individual workloads.
These kernel changes expose the measurement side as a new PMU
driver, which exposes various QoS related PMU events. (The
partitioning change is work in progress and is planned to be merged
as a cgroup extension.)
(Matt Fleming, Peter Zijlstra; CPU feature detection by Peter P
Waskiewicz Jr)
- x86 Intel Haswell LBR call stack support: this is a new Haswell
feature that allows the hardware recording of call chains, plus
tooling support. To activate this feature you have to enable it
via the new 'lbr' call-graph recording option:
perf record --call-graph lbr
perf report
or:
perf top --call-graph lbr
This hardware feature is a lot faster than stack walk or dwarf
based unwinding, but has some limitations:
- It reuses the current LBR facility, so LBR call stack and
branch record can not be enabled at the same time.
- It is only available for user-space callchains.
(Yan, Zheng)
- x86 Intel Broadwell CPU support and various event constraints and
event table fixes for earlier models.
(Andi Kleen)
- x86 Intel HT CPUs event scheduling workarounds. This is a complex
CPU bug affecting the SNB,IVB,HSW families that results in counter
value corruption. The mitigation code is automatically enabled and
is transparent.
(Maria Dimakopoulou, Stephane Eranian)
The perf tooling side had a ton of changes in this cycle as well, so
I'm only able to list the user visible changes here, in addition to
the tooling changes outlined above:
User visible changes affecting all tools:
- Improve support of compressed kernel modules (Jiri Olsa)
- Save DSO loading errno to better report errors (Arnaldo Carvalho de Melo)
- Bash completion for subcommands (Yunlong Song)
- Add 'I' event modifier for perf_event_attr.exclude_idle bit (Jiri Olsa)
- Support missing -f to override perf.data file ownership. (Yunlong Song)
- Show the first event with an invalid filter (David Ahern, Arnaldo Carvalho de Melo)
User visible changes in individual tools:
'perf data':
New tool for converting perf.data to other formats, initially
for the CTF (Common Trace Format) from LTTng (Jiri Olsa,
Sebastian Siewior)
'perf diff':
Add --kallsyms option (David Ahern)
'perf list':
Allow listing events with 'tracepoint' prefix (Yunlong Song)
Sort the output of the command (Yunlong Song)
'perf kmem':
Respect -i option (Jiri Olsa)
Print big numbers using thousands' group (Namhyung Kim)
Allow -v option (Namhyung Kim)
Fix alignment of slab result table (Namhyung Kim)
'perf probe':
Support multiple probes on different binaries on the same command line (Masami Hiramatsu)
Support unnamed union/structure members data collection. (Masami Hiramatsu)
Check kprobes blacklist when adding new events. (Masami Hiramatsu)
'perf record':
Teach 'perf record' about perf_event_attr.clockid (Peter Zijlstra)
Support recording running/enabled time (Andi Kleen)
'perf sched':
Improve the performance of 'perf sched replay' on high CPU core count machines (Yunlong Song)
'perf report' and 'perf top':
Allow annotating entries in callchains in the hists browser (Arnaldo Carvalho de Melo)
Indicate which callchain entries are annotated in the
TUI hists browser (Arnaldo Carvalho de Melo)
Add pid/tid filtering to 'report' and 'script' commands (David Ahern)
Consider PERF_RECORD_ events with cpumode == 0 in 'perf top', removing one
cause of long term memory usage buildup, i.e. not processing PERF_RECORD_EXIT
events (Arnaldo Carvalho de Melo)
'perf stat':
Report unsupported events properly (Suzuki K. Poulose)
Output running time and run/enabled ratio in CSV mode (Andi Kleen)
'perf trace':
Handle legacy syscalls tracepoints (David Ahern, Arnaldo Carvalho de Melo)
Only insert blank duration bracket when tracing syscalls (Arnaldo Carvalho de Melo)
Filter out the trace pid when no threads are specified (Arnaldo Carvalho de Melo)
Dump stack on segfaults (Arnaldo Carvalho de Melo)
No need to explicitely enable evsels for workload started from perf, let it
be enabled via perf_event_attr.enable_on_exec, removing some events that take
place in the 'perf trace' before a workload is really started by it.
(Arnaldo Carvalho de Melo)
Allow mixing with tracepoints and suppressing plain syscalls. (Arnaldo Carvalho de Melo)
There's also been a ton of infrastructure work done, such as the
split-out of perf's build system into tools/build/ and other changes -
see the shortlog and changelog for details"
* 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (358 commits)
perf/x86/intel/pt: Clean up the control flow in pt_pmu_hw_init()
perf evlist: Fix type for references to data_head/tail
perf probe: Check the orphaned -x option
perf probe: Support multiple probes on different binaries
perf buildid-list: Fix segfault when show DSOs with hits
perf tools: Fix cross-endian analysis
perf tools: Fix error path to do closedir() when synthesizing threads
perf tools: Fix synthesizing fork_event.ppid for non-main thread
perf tools: Add 'I' event modifier for exclude_idle bit
perf report: Don't call map__kmap if map is NULL.
perf tests: Fix attr tests
perf probe: Fix ARM 32 building error
perf tools: Merge all perf_event_attr print functions
perf record: Add clockid parameter
perf sched replay: Use replay_repeat to calculate the runavg of cpu usage instead of the default value 10
perf sched replay: Support using -f to override perf.data file ownership
perf sched replay: Fix the EMFILE error caused by the limitation of the maximum open files
perf sched replay: Handle the dead halt of sem_wait when create_tasks() fails for any task
perf sched replay: Fix the segmentation fault problem caused by pr_err in threads
perf sched replay: Realloc the memory of pid_to_task stepwise to adapt to the different pid_max configurations
...
Pull trivial tree from Jiri Kosina:
"Usual trivial tree updates. Nothing outstanding -- mostly printk()
and comment fixes and unused identifier removals"
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial:
goldfish: goldfish_tty_probe() is not using 'i' any more
powerpc: Fix comment in smu.h
qla2xxx: Fix printks in ql_log message
lib: correct link to the original source for div64_u64
si2168, tda10071, m88ds3103: Fix firmware wording
usb: storage: Fix printk in isd200_log_config()
qla2xxx: Fix printk in qla25xx_setup_mode
init/main: fix reset_device comment
ipwireless: missing assignment
goldfish: remove unreachable line of code
coredump: Fix do_coredump() comment
stacktrace.h: remove duplicate declaration task_struct
smpboot.h: Remove unused function prototype
treewide: Fix typo in printk messages
treewide: Fix typo in printk messages
mod_devicetable: fix comment for match_flags
The first argument passed to find_probe_point_lazy() should be CU die,
which will be passed to die_walk_lines() when lazy_line matches.
Currently, when we probe with lazy_line pattern to file without function
name, NULL pointer is passed and causes a segment fault.
Can be reproduced as following:
$ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
[ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp
00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000]
Segmentation fault
After this patch:
$ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
Added new event:
probe:_stext (on @fs/super.c)
You can now use it in all perf tools, such as:
perf record -e probe:_stext -aR sleep 1
Signed-off-by: He Kuang <hekuang@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1428925290-5623-3-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
If we use lazy matching, it failed to open a souce file if perf command
is invoked outside of compilation directory:
$ perf probe -a '__schedule;clear_*'
Failed to open kernel/sched/core.c: No such file or directory
Error: Failed to add events. (-2)
OTOH, other commands like "probe -L" can solve the souce directory by
themselves. Let's make it possible for lazy matching too!
Signed-off-by: Naohiro Aota <naota@elisp.net>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1426223923-1493-1-git-send-email-naota@elisp.net
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix 'perf probe' to track down unnamed union/structure members.
perf probe did not track down the tree of unnamed union/structure
members, since it just failed to find given "name" in a parent
structure/union. To solve this issue, I've introduced 2 changes.
- Fix die_find_member() to track down the type-DIE if it is
unnamed, and if it contains the specified member, returns the
unnamed member.
(note that we don't return found member, since unnamed member
has the offset in the parent structure)
- Fix convert_variable_fields() to track down the unnamed union/
structure (one-by-one).
With this patch, perf probe can access unnamed fields:
-----
#./perf probe -nfx ./perf lock__delete ops 'locked_ops=ops->locked.ops'
Added new event:
probe_perf:lock__delete (on lock__delete in /home/mhiramat/ksrc/linux-3/tools/perf/perf with ops locked_ops=ops->locked.ops)
You can now use it in all perf tools, such as:
perf record -e probe_perf:lock__delete -aR sleep 1
-----
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Report-Link: https://lkml.org/lkml/2015/3/5/431
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150402073312.14482.37942.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Remove bias offset to find probe point by address.
Without this patch, probe points on kernel and executables are shown
correctly, but do not work with libraries:
# ./perf probe -l
probe:do_fork (on do_fork@kernel/fork.c)
probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so)
probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
Removing bias allows it to show it as real place:
# ./perf probe -l
probe:do_fork (on do_fork@kernel/fork.c)
probe_libc:malloc (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so)
probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150302124946.9191.64085.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix to handle optimized no-inline functions which have only function
definition but no actual instance at that point.
To fix this problem, we need to find actual instance of the function.
Without this patch:
----
# perf probe -a __up
Failed to get entry address of __up.
Error: Failed to add events.
# perf probe -L __up
Specified source line is not found.
Error: Failed to show lines.
----
With this patch:
----
# perf probe -a __up
Added new event:
probe:__up (on __up)
You can now use it in all perf tools, such as:
perf record -e probe:__up -aR sleep 1
# perf probe -L __up
<__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
0 static noinline void __sched __up(struct semaphore *sem)
{
struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
struct semaphore_waite
4 list_del(&waiter->list);
5 waiter->up = true;
6 wake_up_process(waiter->task);
7 }
----
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150130093744.30575.43290.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
David reported that perf can segfault when adding an uprobe event like
this:
$ perf probe -x /lib64/libc-2.14.90.so -a 'malloc size=%di'
(gdb) bt
#0 parse_eh_frame_hdr (hdr=0x0, hdr_size=2596, hdr_vaddr=71788,
ehdr=0x7fffffffd390, eh_frame_vaddr=
0x7fffffffd378, table_entries=0x8808d8, table_encoding=0x8808e0 "") at
dwarf_getcfi_elf.c:79
#1 0x000000385f81615a in getcfi_scn_eh_frame (hdr_vaddr=71788,
hdr_scn=0x8839b0, shdr=0x7fffffffd2f0, scn=<optimized out>,
ehdr=0x7fffffffd390, elf=0x882b30) at dwarf_getcfi_elf.c:231
#2 getcfi_shdr (ehdr=0x7fffffffd390, elf=0x882b30) at dwarf_getcfi_elf.c:283
#3 dwarf_getcfi_elf (elf=0x882b30) at dwarf_getcfi_elf.c:309
#4 0x00000000004d5bac in debuginfo__find_probes (pf=0x7fffffffd4f0,
dbg=Unhandled dwarf expression opcode 0xfa) at util/probe-finder.c:993
#5 0x00000000004d634a in debuginfo__find_trace_events (dbg=0x880840,
pev=<optimized out>, tevs=0x880f88, max_tevs=<optimized out>) at
util/probe-finder.c:1200
#6 0x00000000004aed6b in try_to_find_probe_trace_events (target=0x881b20
"/lib64/libpthread-2.14.90.so",
max_tevs=128, tevs=0x880f88, pev=0x859b30) at util/probe-event.c:482
#7 convert_to_probe_trace_events (target=0x881b20
"/lib64/libpthread-2.14.90.so", max_tevs=128, tevs=0x880f88,
pev=0x859b30) at util/probe-event.c:2356
#8 add_perf_probe_events (pevs=<optimized out>, npevs=1, max_tevs=128,
target=0x881b20 "/lib64/libpthread-2.14.90.so", force_add=false) at
util/probe-event.c:2391
#9 0x000000000044014f in __cmd_probe (argc=<optimized out>,
argv=0x7fffffffe2f0, prefix=Unhandled dwarf expression opcode 0xfa) at
at builtin-probe.c:488
#10 0x0000000000440313 in cmd_probe (argc=5, argv=0x7fffffffe2f0,
prefix=<optimized out>) at builtin-probe.c:506
#11 0x000000000041d133 in run_builtin (p=0x805680, argc=5,
argv=0x7fffffffe2f0) at perf.c:341
#12 0x000000000041c8b2 in handle_internal_command (argv=<optimized out>,
argc=<optimized out>) at perf.c:400
#13 run_argv (argv=<optimized out>, argcp=<optimized out>) at perf.c:444
#14 main (argc=5, argv=0x7fffffffe2f0) at perf.c:559
And I found a related commit (5704c8c4fa71 "getcfi_scn_eh_frame: Don't
crash and burn when .eh_frame bits aren't there.") in elfutils that can
lead to a unexpected crash like this. To safely use the function, it
needs to check the .eh_frame section is a PROGBITS type.
Reported-by: David Ahern <dsahern@gmail.com>
Tested-by: David Ahern <dsahern@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Mark Wielaard <mjw@redhat.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Link: http://lkml.kernel.org/r/20141230090533.GH6081@sejong
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Do not use dwfl_module_addrsym if dwarf_diename can find the symbol
name, since dwfl_module_addrsym can be failed on shared libraries.
Without this patch
----
$ perf probe -x ../lib/traceevent/libtraceevent.so -V create_arg_op
Failed to find symbol at 0x11df1
Failed to find the address of create_arg_op
Error: Failed to show vars.
----
With this patch
----
$ perf probe -x ../lib/traceevent/libtraceevent.so -V create_arg_op
Available variables at create_arg_op
@<create_arg_op+0>
enum filter_op_type btype
struct filter_arg* arg
----
This bug was reported on linux-perf-users@vger.kernel.org.
Reported-by: david lerner <dlernerdroid@gmail.com>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: david lerner <dlernerdroid@gmail.com>
Cc: linux-perf-user@vger.kernel.org
Cc: yrl.pp-manager.tt@hitachi.com
Link: http://permalink.gmane.org/gmane.linux.kernel.perf.user/1691
Link: http://lkml.kernel.org/r/20140917084101.3722.25299.stgit@kbuild-f20.novalocal
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To make error messages thread-safe, this replaces strerror with
strerror_r for warnings, and just shows the return value instead of
using strerror for debug messages.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/20140814022234.3545.22199.stgit@kbuild-fedora.novalocal
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Masami Hiramatsu