s390/docs: Remove sections that are not related to s390
Information how to use the GCC pre-processor, objdump, strace, top, etc. are generic and not specific to the S390 architecture, so we do not need this information in Debugging390.txt Signed-off-by: Thomas Huth <thuth@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
This commit is contained in:
parent
b195562311
commit
a6b42afa3f
|
@ -26,11 +26,6 @@ The Linux for s/390 & z/Architecture Kernel Task Structure
|
||||||
Register Usage & Stackframes on Linux for s/390 & z/Architecture
|
Register Usage & Stackframes on Linux for s/390 & z/Architecture
|
||||||
A sample program with comments
|
A sample program with comments
|
||||||
Compiling programs for debugging on Linux for s/390 & z/Architecture
|
Compiling programs for debugging on Linux for s/390 & z/Architecture
|
||||||
Figuring out gcc compile errors
|
|
||||||
Debugging Tools
|
|
||||||
objdump
|
|
||||||
strace
|
|
||||||
Performance Debugging
|
|
||||||
Debugging under VM
|
Debugging under VM
|
||||||
s/390 & z/Architecture IO Overview
|
s/390 & z/Architecture IO Overview
|
||||||
Debugging IO on s/390 & z/Architecture under VM
|
Debugging IO on s/390 & z/Architecture under VM
|
||||||
|
@ -740,376 +735,7 @@ Debugging with optimisation has since much improved after fixing
|
||||||
some bugs, please make sure you are using gdb-5.0 or later developed
|
some bugs, please make sure you are using gdb-5.0 or later developed
|
||||||
after Nov'2000.
|
after Nov'2000.
|
||||||
|
|
||||||
Figuring out gcc compile errors
|
|
||||||
===============================
|
|
||||||
If you are getting a lot of syntax errors compiling a program & the problem
|
|
||||||
isn't blatantly obvious from the source.
|
|
||||||
It often helps to just preprocess the file, this is done with the -E
|
|
||||||
option in gcc.
|
|
||||||
What this does is that it runs through the very first phase of compilation
|
|
||||||
( compilation in gcc is done in several stages & gcc calls many programs to
|
|
||||||
achieve its end result ) with the -E option gcc just calls the gcc preprocessor (cpp).
|
|
||||||
The c preprocessor does the following, it joins all the files #included together
|
|
||||||
recursively ( #include files can #include other files ) & also the c file you wish to compile.
|
|
||||||
It puts a fully qualified path of the #included files in a comment & it
|
|
||||||
does macro expansion.
|
|
||||||
This is useful for debugging because
|
|
||||||
1) You can double check whether the files you expect to be included are the ones
|
|
||||||
that are being included ( e.g. double check that you aren't going to the i386 asm directory ).
|
|
||||||
2) Check that macro definitions aren't clashing with typedefs,
|
|
||||||
3) Check that definitions aren't being used before they are being included.
|
|
||||||
4) Helps put the line emitting the error under the microscope if it contains macros.
|
|
||||||
|
|
||||||
For convenience the Linux kernel's makefile will do preprocessing automatically for you
|
|
||||||
by suffixing the file you want built with .i ( instead of .o )
|
|
||||||
|
|
||||||
e.g.
|
|
||||||
from the linux directory type
|
|
||||||
make arch/s390/kernel/signal.i
|
|
||||||
this will build
|
|
||||||
|
|
||||||
s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
|
|
||||||
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -E arch/s390/kernel/signal.c
|
|
||||||
> arch/s390/kernel/signal.i
|
|
||||||
|
|
||||||
Now look at signal.i you should see something like.
|
|
||||||
|
|
||||||
|
|
||||||
# 1 "/home1/barrow/linux/include/asm/types.h" 1
|
|
||||||
typedef unsigned short umode_t;
|
|
||||||
typedef __signed__ char __s8;
|
|
||||||
typedef unsigned char __u8;
|
|
||||||
typedef __signed__ short __s16;
|
|
||||||
typedef unsigned short __u16;
|
|
||||||
|
|
||||||
If instead you are getting errors further down e.g.
|
|
||||||
unknown instruction:2515 "move.l" or better still unknown instruction:2515
|
|
||||||
"Fixme not implemented yet, call Martin" you are probably are attempting to compile some code
|
|
||||||
meant for another architecture or code that is simply not implemented, with a fixme statement
|
|
||||||
stuck into the inline assembly code so that the author of the file now knows he has work to do.
|
|
||||||
To look at the assembly emitted by gcc just before it is about to call gas ( the gnu assembler )
|
|
||||||
use the -S option.
|
|
||||||
Again for your convenience the Linux kernel's Makefile will hold your hand &
|
|
||||||
do all this donkey work for you also by building the file with the .s suffix.
|
|
||||||
e.g.
|
|
||||||
from the Linux directory type
|
|
||||||
make arch/s390/kernel/signal.s
|
|
||||||
|
|
||||||
s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
|
|
||||||
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -S arch/s390/kernel/signal.c
|
|
||||||
-o arch/s390/kernel/signal.s
|
|
||||||
|
|
||||||
|
|
||||||
This will output something like, ( please note the constant pool & the useful comments
|
|
||||||
in the prologue to give you a hand at interpreting it ).
|
|
||||||
|
|
||||||
.LC54:
|
|
||||||
.string "misaligned (__u16 *) in __xchg\n"
|
|
||||||
.LC57:
|
|
||||||
.string "misaligned (__u32 *) in __xchg\n"
|
|
||||||
.L$PG1: # Pool sys_sigsuspend
|
|
||||||
.LC192:
|
|
||||||
.long -262401
|
|
||||||
.LC193:
|
|
||||||
.long -1
|
|
||||||
.LC194:
|
|
||||||
.long schedule-.L$PG1
|
|
||||||
.LC195:
|
|
||||||
.long do_signal-.L$PG1
|
|
||||||
.align 4
|
|
||||||
.globl sys_sigsuspend
|
|
||||||
.type sys_sigsuspend,@function
|
|
||||||
sys_sigsuspend:
|
|
||||||
# leaf function 0
|
|
||||||
# automatics 16
|
|
||||||
# outgoing args 0
|
|
||||||
# need frame pointer 0
|
|
||||||
# call alloca 0
|
|
||||||
# has varargs 0
|
|
||||||
# incoming args (stack) 0
|
|
||||||
# function length 168
|
|
||||||
STM 8,15,32(15)
|
|
||||||
LR 0,15
|
|
||||||
AHI 15,-112
|
|
||||||
BASR 13,0
|
|
||||||
.L$CO1: AHI 13,.L$PG1-.L$CO1
|
|
||||||
ST 0,0(15)
|
|
||||||
LR 8,2
|
|
||||||
N 5,.LC192-.L$PG1(13)
|
|
||||||
|
|
||||||
Adding -g to the above output makes the output even more useful
|
|
||||||
e.g. typing
|
|
||||||
make CC:="s390-gcc -g" kernel/sched.s
|
|
||||||
|
|
||||||
which compiles.
|
|
||||||
s390-gcc -g -D__KERNEL__ -I/home/barrow/linux-2.3/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing -pipe -fno-strength-reduce -S kernel/sched.c -o kernel/sched.s
|
|
||||||
|
|
||||||
also outputs stabs ( debugger ) info, from this info you can find out the
|
|
||||||
offsets & sizes of various elements in structures.
|
|
||||||
e.g. the stab for the structure
|
|
||||||
struct rlimit {
|
|
||||||
unsigned long rlim_cur;
|
|
||||||
unsigned long rlim_max;
|
|
||||||
};
|
|
||||||
is
|
|
||||||
.stabs "rlimit:T(151,2)=s8rlim_cur:(0,5),0,32;rlim_max:(0,5),32,32;;",128,0,0,0
|
|
||||||
from this stab you can see that
|
|
||||||
rlimit_cur starts at bit offset 0 & is 32 bits in size
|
|
||||||
rlimit_max starts at bit offset 32 & is 32 bits in size.
|
|
||||||
|
|
||||||
|
|
||||||
Debugging Tools:
|
|
||||||
================
|
|
||||||
|
|
||||||
objdump
|
|
||||||
=======
|
|
||||||
This is a tool with many options the most useful being ( if compiled with -g).
|
|
||||||
objdump --source <victim program or object file> > <victims debug listing >
|
|
||||||
|
|
||||||
|
|
||||||
The whole kernel can be compiled like this ( Doing this will make a 17MB kernel
|
|
||||||
& a 200 MB listing ) however you have to strip it before building the image
|
|
||||||
using the strip command to make it a more reasonable size to boot it.
|
|
||||||
|
|
||||||
A source/assembly mixed dump of the kernel can be done with the line
|
|
||||||
objdump --source vmlinux > vmlinux.lst
|
|
||||||
Also, if the file isn't compiled -g, this will output as much debugging information
|
|
||||||
as it can (e.g. function names). This is very slow as it spends lots
|
|
||||||
of time searching for debugging info. The following self explanatory line should be used
|
|
||||||
instead if the code isn't compiled -g, as it is much faster:
|
|
||||||
objdump --disassemble-all --syms vmlinux > vmlinux.lst
|
|
||||||
|
|
||||||
As hard drive space is valuable most of us use the following approach.
|
|
||||||
1) Look at the emitted psw on the console to find the crash address in the kernel.
|
|
||||||
2) Look at the file System.map ( in the linux directory ) produced when building
|
|
||||||
the kernel to find the closest address less than the current PSW to find the
|
|
||||||
offending function.
|
|
||||||
3) use grep or similar to search the source tree looking for the source file
|
|
||||||
with this function if you don't know where it is.
|
|
||||||
4) rebuild this object file with -g on, as an example suppose the file was
|
|
||||||
( /arch/s390/kernel/signal.o )
|
|
||||||
5) Assuming the file with the erroneous function is signal.c Move to the base of the
|
|
||||||
Linux source tree.
|
|
||||||
6) rm /arch/s390/kernel/signal.o
|
|
||||||
7) make /arch/s390/kernel/signal.o
|
|
||||||
8) watch the gcc command line emitted
|
|
||||||
9) type it in again or alternatively cut & paste it on the console adding the -g option.
|
|
||||||
10) objdump --source arch/s390/kernel/signal.o > signal.lst
|
|
||||||
This will output the source & the assembly intermixed, as the snippet below shows
|
|
||||||
This will unfortunately output addresses which aren't the same
|
|
||||||
as the kernel ones you should be able to get around the mental arithmetic
|
|
||||||
by playing with the --adjust-vma parameter to objdump.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
static inline void spin_lock(spinlock_t *lp)
|
|
||||||
{
|
|
||||||
a0: 18 34 lr %r3,%r4
|
|
||||||
a2: a7 3a 03 bc ahi %r3,956
|
|
||||||
__asm__ __volatile(" lhi 1,-1\n"
|
|
||||||
a6: a7 18 ff ff lhi %r1,-1
|
|
||||||
aa: 1f 00 slr %r0,%r0
|
|
||||||
ac: ba 01 30 00 cs %r0,%r1,0(%r3)
|
|
||||||
b0: a7 44 ff fd jm aa <sys_sigsuspend+0x2e>
|
|
||||||
saveset = current->blocked;
|
|
||||||
b4: d2 07 f0 68 mvc 104(8,%r15),972(%r4)
|
|
||||||
b8: 43 cc
|
|
||||||
return (set->sig[0] & mask) != 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
6) If debugging under VM go down to that section in the document for more info.
|
|
||||||
|
|
||||||
|
|
||||||
I now have a tool which takes the pain out of --adjust-vma
|
|
||||||
& you are able to do something like
|
|
||||||
make /arch/s390/kernel/traps.lst
|
|
||||||
& it automatically generates the correctly relocated entries for
|
|
||||||
the text segment in traps.lst.
|
|
||||||
This tool is now standard in linux distro's in scripts/makelst
|
|
||||||
|
|
||||||
strace:
|
|
||||||
-------
|
|
||||||
Q. What is it ?
|
|
||||||
A. It is a tool for intercepting calls to the kernel & logging them
|
|
||||||
to a file & on the screen.
|
|
||||||
|
|
||||||
Q. What use is it ?
|
|
||||||
A. You can use it to find out what files a particular program opens.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Example 1
|
|
||||||
---------
|
|
||||||
If you wanted to know does ping work but didn't have the source
|
|
||||||
strace ping -c 1 127.0.0.1
|
|
||||||
& then look at the man pages for each of the syscalls below,
|
|
||||||
( In fact this is sometimes easier than looking at some spaghetti
|
|
||||||
source which conditionally compiles for several architectures ).
|
|
||||||
Not everything that it throws out needs to make sense immediately.
|
|
||||||
|
|
||||||
Just looking quickly you can see that it is making up a RAW socket
|
|
||||||
for the ICMP protocol.
|
|
||||||
Doing an alarm(10) for a 10 second timeout
|
|
||||||
& doing a gettimeofday call before & after each read to see
|
|
||||||
how long the replies took, & writing some text to stdout so the user
|
|
||||||
has an idea what is going on.
|
|
||||||
|
|
||||||
socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
|
|
||||||
getuid() = 0
|
|
||||||
setuid(0) = 0
|
|
||||||
stat("/usr/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
|
|
||||||
stat("/usr/share/locale/libc/C", 0xbffff134) = -1 ENOENT (No such file or directory)
|
|
||||||
stat("/usr/local/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
|
|
||||||
getpid() = 353
|
|
||||||
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
|
|
||||||
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [49152], 4) = 0
|
|
||||||
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(3, 1), ...}) = 0
|
|
||||||
mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40008000
|
|
||||||
ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0
|
|
||||||
write(1, "PING 127.0.0.1 (127.0.0.1): 56 d"..., 42PING 127.0.0.1 (127.0.0.1): 56 data bytes
|
|
||||||
) = 42
|
|
||||||
sigaction(SIGINT, {0x8049ba0, [], SA_RESTART}, {SIG_DFL}) = 0
|
|
||||||
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {SIG_DFL}) = 0
|
|
||||||
gettimeofday({948904719, 138951}, NULL) = 0
|
|
||||||
sendto(3, "\10\0D\201a\1\0\0\17#\2178\307\36"..., 64, 0, {sin_family=AF_INET,
|
|
||||||
sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 64
|
|
||||||
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
|
|
||||||
sigaction(SIGALRM, {0x8049ba0, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
|
|
||||||
alarm(10) = 0
|
|
||||||
recvfrom(3, "E\0\0T\0005\0\0@\1|r\177\0\0\1\177"..., 192, 0,
|
|
||||||
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
|
|
||||||
gettimeofday({948904719, 160224}, NULL) = 0
|
|
||||||
recvfrom(3, "E\0\0T\0006\0\0\377\1\275p\177\0"..., 192, 0,
|
|
||||||
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
|
|
||||||
gettimeofday({948904719, 166952}, NULL) = 0
|
|
||||||
write(1, "64 bytes from 127.0.0.1: icmp_se"...,
|
|
||||||
5764 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=28.0 ms
|
|
||||||
|
|
||||||
Example 2
|
|
||||||
---------
|
|
||||||
strace passwd 2>&1 | grep open
|
|
||||||
produces the following output
|
|
||||||
open("/etc/ld.so.cache", O_RDONLY) = 3
|
|
||||||
open("/opt/kde/lib/libc.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
|
|
||||||
open("/lib/libc.so.5", O_RDONLY) = 3
|
|
||||||
open("/dev", O_RDONLY) = 3
|
|
||||||
open("/var/run/utmp", O_RDONLY) = 3
|
|
||||||
open("/etc/passwd", O_RDONLY) = 3
|
|
||||||
open("/etc/shadow", O_RDONLY) = 3
|
|
||||||
open("/etc/login.defs", O_RDONLY) = 4
|
|
||||||
open("/dev/tty", O_RDONLY) = 4
|
|
||||||
|
|
||||||
The 2>&1 is done to redirect stderr to stdout & grep is then filtering this input
|
|
||||||
through the pipe for each line containing the string open.
|
|
||||||
|
|
||||||
|
|
||||||
Example 3
|
|
||||||
---------
|
|
||||||
Getting sophisticated
|
|
||||||
telnetd crashes & I don't know why
|
|
||||||
|
|
||||||
Steps
|
|
||||||
-----
|
|
||||||
1) Replace the following line in /etc/inetd.conf
|
|
||||||
telnet stream tcp nowait root /usr/sbin/in.telnetd -h
|
|
||||||
with
|
|
||||||
telnet stream tcp nowait root /blah
|
|
||||||
|
|
||||||
2) Create the file /blah with the following contents to start tracing telnetd
|
|
||||||
#!/bin/bash
|
|
||||||
/usr/bin/strace -o/t1 -f /usr/sbin/in.telnetd -h
|
|
||||||
3) chmod 700 /blah to make it executable only to root
|
|
||||||
4)
|
|
||||||
killall -HUP inetd
|
|
||||||
or ps aux | grep inetd
|
|
||||||
get inetd's process id
|
|
||||||
& kill -HUP inetd to restart it.
|
|
||||||
|
|
||||||
Important options
|
|
||||||
-----------------
|
|
||||||
-o is used to tell strace to output to a file in our case t1 in the root directory
|
|
||||||
-f is to follow children i.e.
|
|
||||||
e.g in our case above telnetd will start the login process & subsequently a shell like bash.
|
|
||||||
You will be able to tell which is which from the process ID's listed on the left hand side
|
|
||||||
of the strace output.
|
|
||||||
-p<pid> will tell strace to attach to a running process, yup this can be done provided
|
|
||||||
it isn't being traced or debugged already & you have enough privileges,
|
|
||||||
the reason 2 processes cannot trace or debug the same program is that strace
|
|
||||||
becomes the parent process of the one being debugged & processes ( unlike people )
|
|
||||||
can have only one parent.
|
|
||||||
|
|
||||||
|
|
||||||
However the file /t1 will get big quite quickly
|
|
||||||
to test it telnet 127.0.0.1
|
|
||||||
|
|
||||||
now look at what files in.telnetd execve'd
|
|
||||||
413 execve("/usr/sbin/in.telnetd", ["/usr/sbin/in.telnetd", "-h"], [/* 17 vars */]) = 0
|
|
||||||
414 execve("/bin/login", ["/bin/login", "-h", "localhost", "-p"], [/* 2 vars */]) = 0
|
|
||||||
|
|
||||||
Whey it worked!.
|
|
||||||
|
|
||||||
|
|
||||||
Other hints:
|
|
||||||
------------
|
|
||||||
If the program is not very interactive ( i.e. not much keyboard input )
|
|
||||||
& is crashing in one architecture but not in another you can do
|
|
||||||
an strace of both programs under as identical a scenario as you can
|
|
||||||
on both architectures outputting to a file then.
|
|
||||||
do a diff of the two traces using the diff program
|
|
||||||
i.e.
|
|
||||||
diff output1 output2
|
|
||||||
& maybe you'll be able to see where the call paths differed, this
|
|
||||||
is possibly near the cause of the crash.
|
|
||||||
|
|
||||||
More info
|
|
||||||
---------
|
|
||||||
Look at man pages for strace & the various syscalls
|
|
||||||
e.g. man strace, man alarm, man socket.
|
|
||||||
|
|
||||||
|
|
||||||
Performance Debugging
|
|
||||||
=====================
|
|
||||||
gcc is capable of compiling in profiling code just add the -p option
|
|
||||||
to the CFLAGS, this obviously affects program size & performance.
|
|
||||||
This can be used by the gprof gnu profiling tool or the
|
|
||||||
gcov the gnu code coverage tool ( code coverage is a means of testing
|
|
||||||
code quality by checking if all the code in an executable in exercised by
|
|
||||||
a tester ).
|
|
||||||
|
|
||||||
|
|
||||||
Using top to find out where processes are sleeping in the kernel
|
|
||||||
----------------------------------------------------------------
|
|
||||||
To do this copy the System.map from the root directory where
|
|
||||||
the linux kernel was built to the /boot directory on your
|
|
||||||
linux machine.
|
|
||||||
Start top
|
|
||||||
Now type fU<return>
|
|
||||||
You should see a new field called WCHAN which
|
|
||||||
tells you where each process is sleeping here is a typical output.
|
|
||||||
|
|
||||||
6:59pm up 41 min, 1 user, load average: 0.00, 0.00, 0.00
|
|
||||||
28 processes: 27 sleeping, 1 running, 0 zombie, 0 stopped
|
|
||||||
CPU states: 0.0% user, 0.1% system, 0.0% nice, 99.8% idle
|
|
||||||
Mem: 254900K av, 45976K used, 208924K free, 0K shrd, 28636K buff
|
|
||||||
Swap: 0K av, 0K used, 0K free 8620K cached
|
|
||||||
|
|
||||||
PID USER PRI NI SIZE RSS SHARE WCHAN STAT LIB %CPU %MEM TIME COMMAND
|
|
||||||
750 root 12 0 848 848 700 do_select S 0 0.1 0.3 0:00 in.telnetd
|
|
||||||
767 root 16 0 1140 1140 964 R 0 0.1 0.4 0:00 top
|
|
||||||
1 root 8 0 212 212 180 do_select S 0 0.0 0.0 0:00 init
|
|
||||||
2 root 9 0 0 0 0 down_inte SW 0 0.0 0.0 0:00 kmcheck
|
|
||||||
|
|
||||||
The time command
|
|
||||||
----------------
|
|
||||||
Another related command is the time command which gives you an indication
|
|
||||||
of where a process is spending the majority of its time.
|
|
||||||
e.g.
|
|
||||||
time ping -c 5 nc
|
|
||||||
outputs
|
|
||||||
real 0m4.054s
|
|
||||||
user 0m0.010s
|
|
||||||
sys 0m0.010s
|
|
||||||
|
|
||||||
Debugging under VM
|
Debugging under VM
|
||||||
==================
|
==================
|
||||||
|
|
Loading…
Reference in New Issue