From 4125a78ae5fc99abb8606bf0eda2a08b3e2d300c Mon Sep 17 00:00:00 2001 From: ddomingo Date: Thu, 2 Oct 2008 23:07:01 +1000 Subject: new section, iotop.stp --- .../en-US/CrossInstrumenting.xml | 12 ++- .../en-US/Installation.xml | 37 ++++---- .../en-US/Introduction.xml | 2 +- doc/SystemTap_Beginners_Guide/en-US/Scripts.xml | 56 ++++++----- .../en-US/SystemTap_Beginners_Guide.xml | 3 +- .../en-US/Tips_Tricks.xml | 7 +- .../en-US/Understanding_How_SystemTap_Works.xml | 12 ++- .../en-US/Useful_Scripts-disktop.xml | 22 ++--- .../en-US/Useful_Scripts-iotop.xml | 76 +++++++++++++++ .../en-US/Useful_SystemTap_Scripts.xml | 4 + .../en-US/Using_SystemTap.xml | 105 +++------------------ 11 files changed, 175 insertions(+), 161 deletions(-) create mode 100644 doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-iotop.xml (limited to 'doc/SystemTap_Beginners_Guide') diff --git a/doc/SystemTap_Beginners_Guide/en-US/CrossInstrumenting.xml b/doc/SystemTap_Beginners_Guide/en-US/CrossInstrumenting.xml index 07280100..cc451ec4 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/CrossInstrumenting.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/CrossInstrumenting.xml @@ -77,7 +77,7 @@ Configuring a Host System and Target Systems - Configure yum on the host system to point to a repository containing the necessary debug RPMs for the target kernels. The following yum repository file (which you can add to /etc/yum.repos.d/ points to a popular debug RPM repository for Red Hat Enterprise Linux 5: + Configure yum on the host system to point to a repository containing the necessary debug RPMs for the target kernels. The following yum repository file (which you can add to /etc/yum.repos.d/ points to a popular debug RPM repository for i386 systems running Red Hat Enterprise Linux 5: [rhel-debuginfo] name=Red Hat Enterprise Linux $releasever - $basearch - Debug @@ -119,10 +119,11 @@ enabled=1 Note - To determine the version of a running kernel, run uname -r. To determine the architecture notation of a running kernel, run uname -m. + To determine the architecture notation of a running kernel, run uname -m. + -Once the the instrumentation moduleis compiled, copy it to the target system and load it using: +Once the the instrumentation module is compiled, copy it to the target system and load it using: staprun instrumentation @@ -142,6 +143,9 @@ enabled=1 Note the version of the target system's kernel on which you wish to use SystemTap. You can do this by logging onto the target system and running uname -r (assuming the system is running the kernel on which you wish to use SystemTap), or by inspecting /boot. --> - + + Important + The host system must be the same architecture as the target system in order for the instrumentation module to work. + \ No newline at end of file diff --git a/doc/SystemTap_Beginners_Guide/en-US/Installation.xml b/doc/SystemTap_Beginners_Guide/en-US/Installation.xml index 92d40bc0..0ccefa3a 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Installation.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Installation.xml @@ -1,9 +1,9 @@ - - +
Setup and Installation required packages, installation thru yum, repos (?); possibly, a script to install all required packages @@ -14,7 +14,7 @@ - To deploy SystemTap, you need to install the SystemTap packages along with the corresponding set of debug RPMs of your kernel. This means that if your system has multiple kernels installed, and you wish to use SystemTap on more than one kernel, you will need to install the debug RPMs for each of those kernels. + To deploy SystemTap, you need to install the SystemTap packages along with the corresponding set of -devel and -debuginfo packages for your kernel. This means that if your system has multiple kernels installed, and you wish to use SystemTap on more than one kernel kernel, you will need to install the -devel and -debuginfo packages for each of those kernel versions.
Preparing For Installation @@ -29,14 +29,21 @@ uname -r + - You will also need to configure yum to point to a repository that houses the necessary debug RPMs. Most debugging RPMs for Red Hat Enterprise Linux 5 can be found at the following repository: + Most -debuginfo packages for Red Hat Enterprise Linux 5 can be found at the following link (under arch/Debuginfo, where arch is the appropriate architecture for your system: - ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Client/en/os/i386/Debuginfo/ + ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Client/en/os/ find any other such repository, if only for RHEL + + +
@@ -55,7 +62,7 @@ uname -r - Next, you'll need to download and install the necessary debug RPMs for your kernel. The necessary debugging RPMs for the ordinary "vanilla" kernel are as follows: + Next, you'll need to download and install the necessary -devel and -debuginfo packages for your kernel. The necessary -devel and -debuginfo packages for the ordinary "vanilla" kernel are as follows: @@ -65,7 +72,7 @@ uname -r kernel-devel -For example, if you wish to use SystemTap on kernel version 2.6.18-53.el5, then you need to download the following debugging RPMs: +For example, if you wish to use SystemTap on kernel version 2.6.18-53.el5, then you need to download and install the following RPMs: Sample List of Debugging RPMs @@ -76,23 +83,13 @@ uname -r - Install the debugging RPMs using rpm -ivh RPM or yum localinstall RPM. - -
- - \ No newline at end of file +
+ \ No newline at end of file diff --git a/doc/SystemTap_Beginners_Guide/en-US/Introduction.xml b/doc/SystemTap_Beginners_Guide/en-US/Introduction.xml index b19f5e48..28abf439 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Introduction.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Introduction.xml @@ -30,7 +30,7 @@ To introduce users to SystemTap, familiarize them with its architecture, and provide setup instructions for all kernel types. - To provide pre-written SystemTap scripts for monitoring and forensic tasks, along with instructions on how to analyze their output. + To provide pre-written SystemTap scripts for monitoring detailed activity in different components of the system, along with instructions on how to analyze their output. above, Short description on the underlying goals of SystemTap_Beginners_Guide, what we want to teach users. diff --git a/doc/SystemTap_Beginners_Guide/en-US/Scripts.xml b/doc/SystemTap_Beginners_Guide/en-US/Scripts.xml index fec9aee7..b928372b 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Scripts.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Scripts.xml @@ -6,7 +6,7 @@ SystemTap Scripts - For the most part, SystemTap scripts are the foundation of each SystemTap session. SystemTap scripts instruct SystemTap on what type of information to trap, and what to do once that information is trapped. + For the most part, SystemTap scripts are the foundation of each SystemTap session. SystemTap scripts instruct SystemTap on what type of information to collect, and what to do once that information is collected. @@ -21,9 +21,12 @@ A probe's handler is also commonly referred to as a probe body. - - In terms of application development, using events and handlers is similar to inserting print statements in a program's sequence of commands. These print statements allow you to view a history of commands executed once the program is run. + + In terms of application development, using events and handlers is similar to inserting diagnostic print statements in a program's sequence of commands. These diagnostic print statements allow you to view a history of commands executed once the program is run. + SystemTap scripts go one step further by allowing you more flexibility with regard to handlers. Events serve as the triggers for handlers to run; handlers can be specified to trap specified data and print it in a certain manner. @@ -37,16 +40,14 @@ probe event, -another event { handler - - exit() } - The exit() condition is optional; this condition safely terminates the session once the script successfully traps the required information the first time. - + Important @@ -59,9 +60,13 @@ probe event, SystemTap events can be broadly classified into two types: synchronous and asynchronous. Synchronous Events - A synchronous event occurs when any processor executes an instruction matched by the specification. This gives other events a reference point (or instruction address) from which more contextual data may be available. +A synchronous event occurs when any process executes an instruction that references a particular location in kernel code. This gives other events a reference point from which more contextual data may be available. + + + + Examples of synchronous events include: @@ -70,7 +75,7 @@ probe event, The entry to the kernel function function. For example, kernel.function("sys_open") refers to the "event" that occurs when the kernel function sys_open is called by any thread in the system. To specify the return of the kernel function sys_open, append the return string to the event statement; i.e. kernel.function("sys_open").return. - When defining functions, you can use asterisk (*) for wildcards. You can also trace the entry/exit of a function in a kernel source file. Consider the following example: + When defining functions, you can use asterisk (*) for wildcards. You can also trace the entry or exit of a function in a kernel source file. Consider the following example: Wildcards and Kernel Source Files in an Event probe kernel.function("*@net/socket.c") { } @@ -118,7 +123,8 @@ probe kernel.function("*@net/socket.c").return { } Asynchronous Events - Asynchronous events, on the other hand, do not point to any reference point. This family of probe points consists mainly of counters, timers, and similar constructs. + Asynchronous events, on the other hand, occur as instructed in the probe itself, rather than waiting for a particular instruction in kernel code to be executed by a process. This family of probe points consists mainly of counters, timers, and similar constructs. + Examples of asynchronous events include: @@ -144,19 +150,19 @@ probe kernel.function("*@net/socket.c").return { } timer events - An event that specifies a handler to be executed "after X number of milliseconds". For example: + An event that specifies a handler to be executed every specified period of time. For example: Using timer.ms probe timer.ms(4000) { - exit() + printf("hello world\n") } - is an example of a probe that allows you to terminate the script after 4000 milliseconds. Note that you can also use the following timer events: + is an example of a probe that prints hello world every 4000 milliseconds. Note that you can also use the following timer events: @@ -172,8 +178,12 @@ probe timer.ms(4000) - When used in conjunction with another probe that traps a large quantity of data, timer events allow you to limit the information your script is collecting (and printing out). + When used in conjunction with another probe that collects information that updates periodically, timer events allows you to see how that information changes over time. + @@ -231,7 +241,7 @@ probe begin printf ( ) Statements - The printf () statement is one of the simplest functions for printing data. printf () can also be used to trap data using a wide variety of SystemTap handler functions using the following format: + The printf () statement is one of the simplest functions for printing data. printf () can also be used to display data using a wide variety of SystemTap functions in the following format: @@ -288,13 +298,13 @@ hald(2360) open - Handler Functions - SystemTap supports a wide variety of handler functions that can be used as printf () arguments. uses the handler functions execname() (current process name) and pid() (current process ID). + SystemTap Functions + SystemTap supports a wide variety of functions that can be used as printf () arguments. uses the SystemTap functions execname() (current process name) and pid() (current process ID). -is "handler function" an appropriate term? +is "handler function" an appropriate term? wcohen: use "SystemTap functions" to match up language in man pages - The following is a list of commonly-used handler functions: + The following is a list of commonly-used SystemTap functions: @@ -349,7 +359,7 @@ hald(2360) open thread_indent() - This particular handler function is quite useful, providing you with a way to better organize your print results. When used with an indentation parameter (for example, -1), it allows the probe to internally store an "indentation counter" for each thread (identified by ID, as in tid). It then returns a string with some generic trace data along with an appropriate number of indentation spaces. + This particular function is quite useful, providing you with a way to better organize your print results. When used with an indentation parameter (for example, -1), it allows the probe to internally store an "indentation counter" for each thread (identified by ID, as in tid). It then returns a string with some generic trace data along with an appropriate number of indentation spaces. The generic data included in the returned string includes a timestamp (number of microseconds since the most recent initial indentation), a process name, and the thread ID. This allows you to identify what functions were called, who called them, and the duration of each function call. @@ -405,9 +415,9 @@ probe kernel.function("*@net/socket.c").return --> -For more information about supported handler functions, refer to man stapfuncs. +For more information about supported SystemTap functions, refer to man stapfuncs. -will need a complete listing of supported handler functions? also, handler function descriptions seem ambiguous, please advise. +will need a complete listing of supported handler functions? also, SystemTap function descriptions seem ambiguous, please advise. + diff --git a/doc/SystemTap_Beginners_Guide/en-US/Tips_Tricks.xml b/doc/SystemTap_Beginners_Guide/en-US/Tips_Tricks.xml index 0377c499..0327962b 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Tips_Tricks.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Tips_Tricks.xml @@ -4,11 +4,12 @@ Tips and Tricks - This chapter covers miscellaneous tips/tricks - - This is a tentative section, and will only be included if content can be provided + - This chapter covers miscellaneous tips/tricks + - This is a tentative section, and will only be included if content can be provided + - add use of exit() here + - add "basic constructs" here; i.e. using if/else, while loops, for loops, examples; if done, consider moving up to before Useful_SystemTap_Scripts.xm, after Understanding_How_SystemTap_Works.xml diff --git a/doc/SystemTap_Beginners_Guide/en-US/Understanding_How_SystemTap_Works.xml b/doc/SystemTap_Beginners_Guide/en-US/Understanding_How_SystemTap_Works.xml index 2794901c..1fb8cfb4 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Understanding_How_SystemTap_Works.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Understanding_How_SystemTap_Works.xml @@ -35,16 +35,20 @@ First, SystemTap checks the script against the existing tapset library (normally in /usr/share/systemtap/tapset/ for any tapsets used. - SystemTap then translates the script to C, running the system C compiler to create a kernel module from it. + SystemTap then translates the script to C, running the system C compiler to create a kernel module from it. The tools that perform this step are contained in the systemtap package (refer to for more information). - SystemTap loads the module, then enables all the probed events by "hooking" those events into the kernel. + SystemTap loads the module, then enables all the probes (events and handlers) in the script. The tools that enable this function are deployed by the systemtap-runtime package (refer to for more information). + As the events occur, their corresponding handlers are executed. - Once the SystemTap session is terminated, the hooked events are disconnected from the kernel; afterwards, the kernel module is unloaded. + Once the SystemTap session is terminated, the probes are disconnected from the kernel; afterwards, the kernel module is unloaded. + -This sequence is driven from a single command-line program: stap. This program is SystemTap's main front-end tool. For more information about stap, refer to man stap (once SystemTap is set up on your machine). +This sequence is driven from a single command-line program: stap. This program is SystemTap's main front-end tool. For more information about stap, refer to man stap (once SystemTap is properly installed on your machine). diff --git a/doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-disktop.xml b/doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-disktop.xml index a3e3d8a7..0c191cff 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-disktop.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-disktop.xml @@ -3,7 +3,7 @@ ]> -
+
Summarizing Disk Read/Write Traffic @@ -73,7 +73,7 @@ probe timer.ms(5000) { } /* print top ten I/O */ foreach ([process,cmd,userid,parent,action] in io_stat- limit 10) - printf("%8d %8d %8d %25s %8s %4s %12d\n",userid,process,parent,cmd,device[process,cmd,userid,parent,action],action,io_stat[process,cmd,userid,parent,action]) + printf("%8d %8d %8d %25s %8s %4s %12d\n",userid,process,parent,cmd,device[process,cmd,userid,parent,action], action,io_stat[process,cmd,userid,parent,action]) /* clear data */ delete io_stat @@ -115,18 +115,18 @@ probe end{ <xref linkend="scriptdisktop"/> Sample Output [...] -Mon Sep 29 03:38:28 2008 , Average: 19Kb/sec, Read: 7Kb, Write: 89Kb +Mon Sep 29 03:38:28 2008 , Average: 19Kb/sec, Read: 7Kb, Write: 89Kb -UID PID PPID CMD DEVICE T BYTES -0 26319 26294 firefox sda5 W 90229 -0 2758 2757 pam_timestamp_c sda5 R 8064 -0 2885 1 cupsd sda5 W 1678 +UID PID PPID CMD DEVICE T BYTES +0 26319 26294 firefox sda5 W 90229 +0 2758 2757 pam_timestamp_c sda5 R 8064 +0 2885 1 cupsd sda5 W 1678 -Mon Sep 29 03:38:38 2008 , Average: 1Kb/sec, Read: 7Kb, Write: 1Kb +Mon Sep 29 03:38:38 2008 , Average: 1Kb/sec, Read: 7Kb, Write: 1Kb -UID PID PPID CMD DEVICE T BYTES -0 2758 2757 pam_timestamp_c sda5 R 8064 -0 2885 1 cupsd sda5 W 1678 +UID PID PPID CMD DEVICE T BYTES +0 2758 2757 pam_timestamp_c sda5 R 8064 +0 2885 1 cupsd sda5 W 1678 diff --git a/doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-iotop.xml b/doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-iotop.xml new file mode 100644 index 00000000..fe249428 --- /dev/null +++ b/doc/SystemTap_Beginners_Guide/en-US/Useful_Scripts-iotop.xml @@ -0,0 +1,76 @@ + + + +
+ Periodically Print I/O Activity + + This section describes how to monitor I/O activity on the system. + + + iotop.stp + + +global reads, writes, total_io + +probe kernel.function("vfs_read") { + reads[execname()] += $count +} + +probe kernel.function("vfs_write") { + writes[execname()] += $count +} + +# print top 10 IO processes every 5 seconds +probe timer.s(5) { + foreach (name in writes) + total_io[name] += writes[name] + foreach (name in reads) + total_io[name] += reads[name] + printf ("%16s\t%10s\t%10s\n", "Process", "KB Read", "KB Written") + foreach (name in total_io- limit 10) + printf("%16s\t%10d\t%10d\n", name, + reads[name]/1024, writes[name]/1024) + delete reads + delete writes + delete total_io + print("\n") +} + + + + + prints out the top ten executables generating I/O traffic every 5-second interval, in descending order. Its output contains the process name and the amount of data read or written by the process, in KB. For example: + + + <xref linkend="iotop"/> Sample Output + +[...] + Process KB Read KB Written + Xorg 50287 0 + staprun 3328 0 + multiload-apple 3039 23 + sshd 208 1 + floaters 14 62 + NetworkManager 15 0 + gnome-vfs-daemo 8 0 + cupsd 3 3 + sendmail 4 0 + mixer_applet2 3 0 + + Process KB Read KB Written + Xorg 51886 0 + staprun 3328 0 + multiload-apple 3039 23 + sshd 1344 4 + snmpd 90 0 + floaters 15 66 + NetworkManager 23 0 + irqbalance 16 0 + pam_timestamp_c 9 0 + sendmail 4 0 + + + + displays top I/O writes and reads within a random 10-second interval. Note that is recursive, as it also detects I/O resulting from SystemTap activity — also displays reads done by staprun. +
\ No newline at end of file diff --git a/doc/SystemTap_Beginners_Guide/en-US/Useful_SystemTap_Scripts.xml b/doc/SystemTap_Beginners_Guide/en-US/Useful_SystemTap_Scripts.xml index 51fef276..03e515ff 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Useful_SystemTap_Scripts.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Useful_SystemTap_Scripts.xml @@ -21,10 +21,14 @@ + + diff --git a/doc/SystemTap_Beginners_Guide/en-US/Using_SystemTap.xml b/doc/SystemTap_Beginners_Guide/en-US/Using_SystemTap.xml index 8b851a49..ebf71276 100644 --- a/doc/SystemTap_Beginners_Guide/en-US/Using_SystemTap.xml +++ b/doc/SystemTap_Beginners_Guide/en-US/Using_SystemTap.xml @@ -11,99 +11,7 @@ This chapter instructs users how to install SystemTap, and provides an introduction on how to run SystemTap scripts. - -
- Setup and Installation - - required packages, installation thru yum, repos (?); possibly, a script to install all required packages - - - - notes in ~/Desktop/SystemTap/aug21chatlog and ~/Desktop/SystemTap/noted_wcohenmeeting - - - - To deploy SystemTap, you need to install the SystemTap packages along with the corresponding set of debug RPMs of your kernel. This means that if your system has multiple kernels installed, and you wish to use SystemTap on more than one kernel, you will need to install the debug RPMs for each of those kernels. - - - Preparing For Installation - - To view what kernels and kernel versions are installed on your system, check the contents of /boot. Each installed kernel/kernel version has a corresponding vmlinuz-kernel version there. - - - - To determine what kernel your system is currently using, use: - - - -uname -r - - - - You will also need to configure yum to point to a repository that houses the necessary debug RPMs. One such repository is: - - - ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Client/en/os/i386/Debuginfo/ - - find any other such repository, if only for RHEL - - - - Deploying SystemTap - - - Once you've decided which kernels you need to use SystemTap with, install the following packages: - - - systemtap - systemtap-runtime - - - This will install the SystemTap suite of tools. - - - - Next, you'll need to download and install the necessary debug RPMs for your kernel. Most debugging RPMs for Red Hat Enterprise Linux 5 can be found at the following link: - - The necessary debugging RPMs are as follows: - - - kernel-debuginfo - kernel-debuginfo-common - kernel-devel - - -For example, if you wish to use SystemTap on kernel version 2.6.18-53.el5, then you need to download the following debugging RPMs: - - - Sample List of Debugging RPMs - - kernel-debuginfo-2.6.18-53.1.13.el5.i686.rpm - kernel-debuginfo-common-2.6.18-53.1.13.el5.i686.rpm - kernel-devel-2.6.18-53.1.13.el5.i686.rpm - - - - - - Install the debugging RPMs using rpm -ivh RPM or yum localinstall RPM. - - - - - Restart the system, loading the appropriate kernel at the grub screen. - - - - - - -
+
Usage @@ -118,9 +26,18 @@ uname -r - Tapsets: short intro on usage --> - SystemTap scripts are run through the command stap. stap can run SystemTap scripts from standard input or from file. Below is a list of common options available to you: + SystemTap scripts are run through the command stap. stap can run SystemTap scripts from standard input or from file. + + Important +Running SystemTap requires root privileges. As such, you need to either log in as root or configure sudo accordingly for specific users who need to run SystemTap (refer to man sudo or man visudo for more information. + + + + + Below is a list of commonly used stap options: + -- cgit