System tap
One stap to know
T-Dose 6 November 2010
Marcel Nijenhof
marcel.nijenhof@proxy.nl
http://pion.xs4all.nl/lezingen/system-tap2010.pdf
Index
• Introduction
• What is system tap
• Small examples
• System tap examples
Marcel Nijenhof
• Proxy
– Employee
– Unix administrator
• LPI Nederland
– Board member – Proctor
• NLLGG
– Board member 10 Years
Why only one stap?
The command line interface to system tap
stap
Compare system tap with other tools
• Placing printf statements in the code
– But without recompilation
• strace
– But then on all processes
• iostat, vmstat, top
– But more details available
• dtrace
– But that is for solaris
What is system tap
• A interface to collect data from the kernel
– Creates kernel probes
– Collects information during the probes
• A scripting language
– AWK like language
• probe
• code
– Processes data
• Filter
• Summarize
• Insert these statements in the running code!
Probes
• System calls
– syscall.open – vm.brk
– kprocess.create
– kernel.function("<function>")
• Timed events
– begin – end
– timer.s, timer.ms, timer.us
• Trace points
Probe code
• Functions
– uid()
– execname() – printf()
• Variables
– Created and typed automatically – Hashes
– global/local – Operators
• c like syntax
Systemtap script
• Generates kernel module
– C code compiled to module – Module loaded in the kernel – Creates the kernel probes – Creates the statistics
• Results of the probes
– Processed by user process – Creates output
hello.stp
• The script:
#!/usr/bin/stap probe begin {
printf ("Hello world\n") exit ()
}
• The execution:
# ./hello.stp Hello world
#
open.stp
• Print file open events with the time:
#!/usr/bin/stap global start_time probe begin {
start_time=gettimeofday_ns() }
probe syscall.open {
printf ("%16u\t%s\t%s(%d) open (%s)\n",
gettimeofday_ns()start_time, ctime(gettimeofday_s()), execname(), pid(), argstr)
}
• 53398704690 Sun Oct 31 21:45:01 2010 crond(1105)
open ("/etc/passwd", O_RDONLY|O_CLOEXEC|O_CLOEXEC)
Statistics
• Count events during a period
– Create hash tables for types of events – Creates hash tables for processes
– Summarizes sizes
• Print results after a period
– Clear hashes – Start again
time.stp
• Just print the time every second probe timer.s(1) {
printf ("%20s\n", ctime(gettimeofday_s())) }
• Tue Nov 2 22:46:16 2010 Tue Nov 2 22:46:17 2010 Tue Nov 2 22:46:18 2010
Statistics of file open
• global open;
probe timer.s(10) {
printf ("%20s: %i\n", ctime(gettimeofday_s()), open) open = 0
}
probe syscall.open { open++
}
• Tue Nov 2 22:52:42 2010: 0 Tue Nov 2 22:52:52 2010: 0 Tue Nov 2 22:53:02 2010: 523 Tue Nov 2 22:53:12 2010: 672 Tue Nov 2 22:53:22 2010: 504
Lets use hashes for processes
• global open_s;
probe timer.s(10) {
printf ("%20s:\n", ctime(gettimeofday_s())) foreach (execname in open_s limit 10) {
printf ("\t%i: %s\n", open_s[execname], execname)
}
delete open_s }
probe syscall.open {
open_s[execname()]++
}
Result
15: sleep
Tue Nov 2 23:21:20 2010:
2457: ps 510: top 126: df 112: file 108: du
Tue Nov 2 23:21:30 2010:
2184: ps 680: top
But what about the: Pid, Opened file
global open_s;
probe timer.s(10) {
foreach ([pid, exec, file] in open_s limit 10) { printf ("\t%i: %i\\%s > %s\n",
open_s[pid, exec, file], pid, exec, file) }
delete open_s }
probe syscall.open {
open_s[pid(), execname(), user_string($filename)]++
}
System tap examples (documentation)
• iotop
– Gives the io per process every 5 seconds
• topsys
– Gives a count of system calls of the last 5 seconds
iotop
Thu Nov 4 21:15:38 2010 , Average: 2Kb/sec, Read: 13Kb, Write: 0Kb
UID PID PPID CMD DEVICE T BYTES 500 1441 1419 ls vda1 R 13657 500 1441 1419 bash vda1 R 531
Thu Nov 4 21:15:43 2010 , Average: 333Kb/sec, Read: 835Kb, Write: 830Kb
UID PID PPID CMD DEVICE T BYTES 500 1442 1419 cp dm1 R 854522 500 1442 1419 cp dm3 W 850714 500 1442 1419 bash vda1 R 531
systap
SYSCALL COUNT
brk 129464
read 47
fcntl 40
ppoll 25
nanosleep 20
rt_sigprocmask 4
write 2
select 2
clock_gettime 2
gettimeofday 1
epoll_wait 1
Questions
Presentation
• Documentation
– http://sourceware.org/systemtap/documentation.html
• Tutorial
• Beginner's Guide
• Language Reference
• Tapset Reference
– http://pion.xs4all.nl/lezingen/System-tap2010.pdf
• Copyright: CC Some rights reserved
– The proxy logo and the presentation template – The lpi logo
– The nllgg logo
Note: Clipart from http://www.openclipart.org
