SYSTEMTAP EXAMPLES INDEX BY KEYWORD (see also index.txt) = BACKTRACE = interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt (IPI) keywords: interrupt backtrace The Linux kernel function smp_call_function causes expensive inter-processor interrupts (IPIs). The scf.stp script tallies the processes and backtraces causing the interprocessor interrupts to identify the cause of the expensive IPI. On exit the script prints the tallies in descending frequency. io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call keywords: io backtrace When a reschedule occurs during an AIO io_submit call, accumulate the traceback in a histogram. When the script exits prints out a sorted list from most common to least common backtrace. process/sleepingBeauties.stp - Generating Backtraces of Threads Waiting for IO Operations keywords: io scheduler backtrace The script monitors the time that threads spend waiting for IO operations (in "D" state) in the wait_for_completion function. If a thread spends over 10ms, its name and backtrace is printed, and later so is the total delay. = CALLGRAPH = general/para-callgraph.stp - Callgraph tracing with arguments keywords: trace callgraph Print a timed per-thread callgraph, complete with function parameters and return values. The first parameter names the function probe points to trace. The optional second parameter names the probe points for trigger functions, which acts to enable tracing for only those functions that occur while the current thread is nested within the trigger. = CPU = general/graphs.stp - Graphing Disk and CPU Utilization keywords: disk cpu use graph The script tracks the disk and CPU utilization. The resulting output of the script can be piped into gnuplot to generate a graph of disk and CPU USE. = DISK = general/graphs.stp - Graphing Disk and CPU Utilization keywords: disk cpu use graph The script tracks the disk and CPU utilization. The resulting output of the script can be piped into gnuplot to generate a graph of disk and CPU USE. io/disktop.stp - Summarize Disk Read/Write Traffic keywords: disk Get the status of reading/writing disk every 5 seconds, output top ten entries during that period. = FUNCTIONS = profiling/functioncallcount.stp - Count Times Functions Called keywords: profiling functions The functioncallcount.stp script takes one argument, a list of functions to probe. The script will run and count the number of times that each of the functions on the list is called. On exit the script will print a sorted list from most frequently to least frequently called function. = FUTEX = process/futexes.stp - System-Wide Futex Contention keywords: syscall locking futex The script watches the futex syscall on the system. On exit the futexes address, the number of contentions, and the average time for each contention on the futex are printed from lowest pid number to highest. = GRAPH = general/graphs.stp - Graphing Disk and CPU Utilization keywords: disk cpu use graph The script tracks the disk and CPU utilization. The resulting output of the script can be piped into gnuplot to generate a graph of disk and CPU USE. = INTERRUPT = interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt (IPI) keywords: interrupt backtrace The Linux kernel function smp_call_function causes expensive inter-processor interrupts (IPIs). The scf.stp script tallies the processes and backtraces causing the interprocessor interrupts to identify the cause of the expensive IPI. On exit the script prints the tallies in descending frequency. = IO = io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call keywords: io backtrace When a reschedule occurs during an AIO io_submit call, accumulate the traceback in a histogram. When the script exits prints out a sorted list from most common to least common backtrace. io/iostats.stp - List Executables Reading and Writing the Most Data keywords: io profiling The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written. io/iotime.stp - Trace Time Spent in Read and Write for Files keywords: syscall read write time io The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spend in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parenthesese. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls. io/iotop.stp - Periodically Print I/O Activity by Process Name keywords: io Every five seconds print out the top ten executables generating I/O traffic during that interval sorted in descending order. io/traceio.stp - Track Cumulative I/O Activity by Process Name keywords: io Every second print out the top ten executables sorted in descending order based on cumulative I/O traffic observed. io/traceio2.stp - Watch I/O Activity on a Particular Device keywords: io Print out the executable name and process number as reads and writes to the specified device occur. process/sleepingBeauties.stp - Generating Backtraces of Threads Waiting for IO Operations keywords: io scheduler backtrace The script monitors the time that threads spend waiting for IO operations (in "D" state) in the wait_for_completion function. If a thread spends over 10ms, its name and backtrace is printed, and later so is the total delay. = LOCKING = process/futexes.stp - System-Wide Futex Contention keywords: syscall locking futex The script watches the futex syscall on the system. On exit the futexes address, the number of contentions, and the average time for each contention on the futex are printed from lowest pid number to highest. = NETWORK = network/nettop.stp - Periodic Listing of Processes Using Network Interfaces keywords: network traffic per-process Every five seconds the nettop.stp script prints out a list of processed (PID and command) with the number of packets sent/received and the amount of data sent/received by the process during that interval. network/socket-trace.stp - Trace Functions called in Network Socket Code keywords: network socket The script instrument each of the functions inn the Linux kernel's net/socket.c file. The script prints out trace. The first element of a line is time delta in microseconds from the previous entry. This is followed by the command name and the PID. The "->" and "<-" indicates function entry and function exit, respectively. The last element of the line is the function name. network/tcp_connections.stp - Track Creation of Incoming TCP Connections keywords: network tcp socket The tcp_connections.stp script prints information for each new incoming TCP connection accepted by the computer. The information includes the UID, the command accepting the connection, the PID of the command, the port the connection is on, and the IP address of the originator of the request. = PER-PROCESS = network/nettop.stp - Periodic Listing of Processes Using Network Interfaces keywords: network traffic per-process Every five seconds the nettop.stp script prints out a list of processed (PID and command) with the number of packets sent/received and the amount of data sent/received by the process during that interval. = PROFILING = io/iostats.stp - List Executables Reading and Writing the Most Data keywords: io profiling The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written. process/pf2.stp - Profile kernel functions keywords: profiling The pf2.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top ten kernel functions with samples. profiling/functioncallcount.stp - Count Times Functions Called keywords: profiling functions The functioncallcount.stp script takes one argument, a list of functions to probe. The script will run and count the number of times that each of the functions on the list is called. On exit the script will print a sorted list from most frequently to least frequently called function. profiling/thread-times.stp - Profile kernel functions keywords: profiling The thread-times.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top twenty processes with samples broken down into percentage total time spent in user-space and kernel-space. profiling/timeout.stp - Show Processes Doing Polling Operations keywords: profiling The timeout.stp script is based on a blog entry (http://udrepper.livejournal.com/19041.html) mentioning a need for a tool to help developers find applications that are polling. The timeout.stp script monitors systemcall used for polling and records the systemcalls that timed out rather than returned because some action occurred. The script updates the screen once a second with the top twenty processes. profiling/topsys.stp - Show Processes Doing Polling Operations keywords: profiling The topsys.stp script lists out the top twenty systemcalls for the previous 5 seconds. The output is sorted from most frequent to least frequent. = READ = io/iotime.stp - Trace Time Spent in Read and Write for Files keywords: syscall read write time io The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spend in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parenthesese. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls. = SCHEDULER = process/sleepingBeauties.stp - Generating Backtraces of Threads Waiting for IO Operations keywords: io scheduler backtrace The script monitors the time that threads spend waiting for IO operations (in "D" state) in the wait_for_completion function. If a thread spends over 10ms, its name and backtrace is printed, and later so is the total delay. = SIGNALS = process/sig_by_pid.stp - Signal Counts by Process ID keywords: signals Print signal counts by process ID in descending order. process/sig_by_proc.stp - Signal Counts by Process Name keywords: signals Print signal counts by process name in descending order. process/sigkill.stp - Track SIGKILL Signals keywords: signals The script traces any SIGKILL signals. When that SIGKILL signal is sent to a process, the script prints out the signal name, the desination executable and process ID, the executable name user ID that sent the signal. process/syscalls_by_pid.stp - System-Wide Count of Syscalls by PID keywords: signals The script watches for a particular signal sent to a specific process. When that signal is sent to the specified process, the script prints out the PID and executable of the process sending the signal, the PID and executable name of the process receiving the signal, and the signal number and name. = SIMPLE = general/helloworld.stp - SystemTap "Hello World" Program keywords: simple A basic "Hello World" program implemented in SystemTap script. It prints out "hello world" message and then immediately exits. = SLEEP = process/sleeptime.stp - Trace Time Spent in nanosleep Syscalls keywords: syscall sleep The script watches each nanosleep syscall on the system. At the end of each nanosleep syscall the script prints out a line with a timestamp in microseconds, the pid, the executable name in paretheses, the "nanosleep:" key, and the duration of the sleep in microseconds. = SOCKET = network/socket-trace.stp - Trace Functions called in Network Socket Code keywords: network socket The script instrument each of the functions inn the Linux kernel's net/socket.c file. The script prints out trace. The first element of a line is time delta in microseconds from the previous entry. This is followed by the command name and the PID. The "->" and "<-" indicates function entry and function exit, respectively. The last element of the line is the function name. network/tcp_connections.stp - Track Creation of Incoming TCP Connections keywords: network tcp socket The tcp_connections.stp script prints information for each new incoming TCP connection accepted by the computer. The information includes the UID, the command accepting the connection, the PID of the command, the port the connection is on, and the IP address of the originator of the request. = SYSCALL = io/iotime.stp - Trace Time Spent in Read and Write for Files keywords: syscall read write time io The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spend in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parenthesese. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls. process/futexes.stp - System-Wide Futex Contention keywords: syscall locking futex The script watches the futex syscall on the system. On exit the futexes address, the number of contentions, and the average time for each contention on the futex are printed from lowest pid number to highest. process/sleeptime.stp - Trace Time Spent in nanosleep Syscalls keywords: syscall sleep The script watches each nanosleep syscall on the system. At the end of each nanosleep syscall the script prints out a line with a timestamp in microseconds, the pid, the executable name in paretheses, the "nanosleep:" key, and the duration of the sleep in microseconds. process/syscalls_by_pid.stp - System-Wide Count of Syscalls by PID keywords: syscall The script watches all syscall on the system. On exit the script prints a list showing the number of systemcalls executed by each PID ordered from greatest to least number of syscalls. process/syscalls_by_proc.stp - System-Wide Count of Syscalls by Executable keywords: syscall The script watches all syscall on the system. On exit the script prints a list showing the number of systemcalls executed by each executable ordered from greates to least number of syscalls. process/wait4time.stp - Trace Time Spent in wait4 Syscalls keywords: syscall wait4 The script watches each wait4 syscall on the system. At the end of each wait4 syscall the script prints out a line with a timestamp in microseconds, the pid, the executable name in paretheses, the "wait4:" key, the duration of the wait and the PID that the wait4 was waiting for. If the waited for PID is not specified , it is "-1". = TCP = network/tcp_connections.stp - Track Creation of Incoming TCP Connections keywords: network tcp socket The tcp_connections.stp script prints information for each new incoming TCP connection accepted by the computer. The information includes the UID, the command accepting the connection, the PID of the command, the port the connection is on, and the IP address of the originator of the request. = TIME = io/iotime.stp - Trace Time Spent in Read and Write for Files keywords: syscall read write time io The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spend in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parenthesese. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls. = TRACE = general/para-callgraph.stp - Callgraph tracing with arguments keywords: trace callgraph Print a timed per-thread callgraph, complete with function parameters and return values. The first parameter names the function probe points to trace. The optional second parameter names the probe points for trigger functions, which acts to enable tracing for only those functions that occur while the current thread is nested within the trigger. = TRAFFIC = network/nettop.stp - Periodic Listing of Processes Using Network Interfaces keywords: network traffic per-process Every five seconds the nettop.stp script prints out a list of processed (PID and command) with the number of packets sent/received and the amount of data sent/received by the process during that interval. = USE = general/graphs.stp - Graphing Disk and CPU Utilization keywords: disk cpu use graph The script tracks the disk and CPU utilization. The resulting output of the script can be piped into gnuplot to generate a graph of disk and CPU USE. = WAIT4 = process/wait4time.stp - Trace Time Spent in wait4 Syscalls keywords: syscall wait4 The script watches each wait4 syscall on the system. At the end of each wait4 syscall the script prints out a line with a timestamp in microseconds, the pid, the executable name in paretheses, the "wait4:" key, the duration of the wait and the PID that the wait4 was waiting for. If the waited for PID is not specified , it is "-1". = WRITE = io/iotime.stp - Trace Time Spent in Read and Write for Files keywords: syscall read write time io The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spend in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parenthesese. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls.