* What's new in version 0.6

- A copy of the systemtap tutorial and language reference guide
  are now included.

- There is a new format specifier, %m, for the printf family of
  functions.  It functions like %s, except that it does not stop when
  a nul ('\0') byte is encountered.  The number of bytes output is
  determined by the precision specifier.  The default precision is 1.
  For example:

      printf ("%m", "My String") // prints one character: M
      printf ("%.5", myString)   // prints 5 bytes beginning at the start
      	     	     		 // of myString

- The %b format specifier for the printf family of functions has been enhanced
  as follows:

  1) When the width and precision are both unspecified, the default is %8.8b.
  2) When only one of the width or precision is specified, the other defaults
     to the same value.  For example, %4b == %.4b == %4.4b
  3) Nul ('\0') bytes are used for field width padding.  For example,

     printf ("%b", 0x1111deadbeef2222) // prints all eight bytes
     printf ("%4.2b", 0xdeadbeef)      // prints  \0\0\xbe\xef

- Dynamic width and precision are now supported for all printf family format
  specifiers.  For example:

     four = 4
     two = 2
     printf ("%*.*b", four, two, 0xdeadbbeef) // prints  \0\0\xbe\xef
     printf ("%*d", four, two)                // prints  <space><space><space>2

- Preprocessor conditional expressions can now include wildcard style
  matches on kernel versions. 
  %( kernel_vr != "*xen" %? foo %: bar %)

- Prototype support for user-space probing is showing some progress.
  No symbolic notations are supported yet (so no probing by function names,
  file names, process names, and no access to $context variables), but at
  least it's something:

    probe process(PID).statement(ADDRESS).absolute { }

  This will set a uprobe on the given process-id and given virtual address.
  The proble handler runs in kernel-space as usual, and can generally use
  existing tapset functions.

- Crash utility can retrieve systemtap's relay buffer from a kernel dump
  image by using staplog which is a crash extension module. To use this
  feature, type commands as below from crash(8)'s command line:

    crash> extend staplog.so
    crash> help systemtaplog

  Then, you can see more precise help message.

- You can share a relay buffer amoung several scripts and merge outputs from
  several scripts by using "-DRELAY_HOST" and "-DRELAY_GUEST" options.
  For example:

    # run a host script
    % stap -ve 'probe begin{}' -o merged.out -DRELAY_HOST &
    # wait until starting the host.
    % stap -ve 'probe begin{print("hello ");exit()}' -DRELAY_GUEST
    % stap -ve 'probe begin{print("world\n");exit()}' -DRELAY_GUEST

  Then, you'll see "hello world" in merged.out.

- You can add a conditional statement for each probe point or aliase, which
  is evaluated when the probe point is hit. If the condition is false, the
  whole probe body(including aliases) is skipped. For example:

    global switch = 0;
    probe syscall.* if (switch) { ... }
    probe procfs.write {switch = strtol($value,10)} /* enable/disable ctrl */

- Systemtap will warn you if your script contains unused variables or
  functions.  This is helpful in case of misspelled variables.  If it
  doth protest too much, turn it off with "stap -w ...".

- You can add error-handling probes to a script, which are run if a
  script was stopped due to errors.  In such a case, "end" probes are
  not run, but "error" ones are.

    probe error { println ("oops, errors encountered; here's a report anyway")
                  foreach (coin in mint) { println (coin) } }

- In a related twist, one may list probe points in order of preference,
  and mark any of them as "sufficient" beyond just "optional".  Probe
  point sequence expansion stops if a sufficient-marked probe point has a hit.
  This is useful for probes on functions that may be in a module (CONFIG_FOO=m)
  or may have been compiled into the kernel (CONFIG_FOO=y), but we don't know
  which.  Instead of

    probe module("sd").function("sd_init_command") ? ,
          kernel.function("sd_init_command") ? { ... }

  which might match neither, now one can write this:

    probe module("sd").function("sd_init_command") ! , /* <-- note excl. mark */
          kernel.function("sd_init_command")  { ... }

- New security model.  To install a systemtap kernel module, a user
  must be one of the following: the root user; a member of the
  'stapdev' group; or a member of the 'stapusr' group.  Members of the
  stapusr group can only use modules located in the
  /lib/modules/VERSION/systemtap directory (where VERSION is the
  output of "uname -r").

- .statement("...@file:line") probes now apply heuristics to allow an
  approximate match for the line number.  This works similarly to gdb,
  where a breakpoint placed on an empty source line is automatically
  moved to the next statement.  A silly bug that made many $target
  variables inaccessible to .statement() probes was also fixed.

- LKET has been retired.  Please let us know on <systemtap@sourceware.org>
  if you have been a user of the tapset/tools, so we can help you find
  another way.

- New families of printing functions println() and printd() have been added.
  println() is like print() but adds a newline at the end;
  printd() is like a sequence of print()s, with a specified field delimiter.

* What's new since version 0.5.14?

- The way in which command line arguments for scripts are substituted has
  changed.  Previously, $1 etc. would interpret the corresponding command
  line argument as an numeric literal, and @1 as a string literal.  Now,
  the command line arguments are pasted uninterpreted wherever $1 etc.
  appears at the beginning of a token.  @1 is similar, but is quoted as
  a string.  This change does not modify old scripts, but has the effect
  of permitting substitution of arbitrary token sequences.

  # This worked before, and still does:
  % stap -e 'probe timer.s($1) {}'        5
  # Now this also works:
  % stap -e 'probe syscall.$1 {log(@1)}'  open
  # This won't crash, just signal a recursion error:
  % stap -e '$1'                          '$1'
  # As before, $1... is recognized only at the beginning of a token
  % stap -e 'probe begin {foo$1=5}'

* What's new since version 0.5.13?

- The way in which systemtap resolves function/inline probes has changed:
   .function(...) - now refers to all functions, inlined or not
   .inline(...)   - is deprecated, use instead:
   .function(...).inline - filters function() to only inlined instances
   .function(...).call - filters function() to only non-inlined instances
   .function(...).return - as before, but now pairs best with .function().call
   .statement() is unchanged.

* What's new since version 0.5.12?

- When running in -p4 (compile-only) mode, the compiled .ko file name
  is printed on standard output.

- An array element with a null value such as zero or an empty string
  is now preserved, and will show up in a "foreach" loop or "in" test.
  To delete such an element, the scripts needs to use an explicit
  "delete array[idx]" statement rather than something like "array[idx]=0".

- The new "-P" option controls whether prologue searching heuristics
  will be activated for function probes.  This was needed to get correct
  debugging information (dwarf location list) data for $target variables.
  Modern compilers (gcc 4.1+) tend not to need this heuristic, so it is
  no longer default.  A new configure flag (--enable-prologues) restores
  it as a default setting, and is appropriate for older compilers (gcc 3.*).

- Each systemtap module prints a one-line message to the kernel informational
  log when it starts.  This line identifies the translator version, base
  address of the probe module, a broken-down memory consumption estimate, and
  the total number of probes.  This is meant as a debugging / auditing aid.

- Begin/end probes are run with interrupts enabled (but with
  preemption disabled).  This will allow begin/end probes to be
  longer, to support generating longer reports.

- The numeric forms of kernel.statement() and kernel.function() probe points
  are now interpreted as relocatable values - treated as relative to the
  _stext symbol in that kernel binary.  Since some modern kernel images
  are relocated to a different virtual address at startup, such addresses
  may shift up or down when actually inserted into a running kernel.

     kernel.statement(0xdeadbeef): validated, interpreted relative to _stext,
                                   may map to 0xceadbeef at run time.

  In order to specify unrelocated addresses, use the new ".absolute"
  probe point suffix for such numeric addresses.  These are only
  allowed in guru mode, and provide access to no $target variables.
  They don't use debugging information at all, actually.

     kernel.statement(0xfeedface).absolute: raw, unvalidated, guru mode only

* What's new since version 0.5.10?

- Offline processing of debugging information, enabling general
  cross-compilation of probe scripts to remote hosts, without
  requiring identical module/memory layout.  This slows down
  compilation/translation somewhat.

- Kernel symbol table data is loaded by staprun at startup time
  rather than compiled into the module.

- Support the "limit" keyword for foreach iterations:
    foreach ([x,y] in ary limit 5) { ... }
  This implicitly exits after the fifth iteration.  It also enables
  more efficient key/value sorting.

- Support the "maxactive" keyword for return probes:
    probe kernel.function("sdfsdf").maxactive(848) { ... }
  This allows up to 848 concurrently outstanding entries to
  the sdfsdf function before one returns.  The default maxactive
  number is smaller, and can result in missed return probes.

- Support accessing of saved function arguments from within
  return probes.  These values are saved by a synthesized
  function-entry probe.

- Add substantial version/architecture checking in compiled probes to
  assert correct installation of debugging information and correct
  execution on a compatible kernel.

- Add probe-time checking for sufficient free stack space when probe
  handlers are invoked, as a safety improvement.

- Add an optional numeric parameter for begin/end probe specifications,
  to order their execution.  
     probe begin(10) { } /* comes after */ probe begin(-10) {}

- Add an optional array size declaration, which is handy for very small
  or very large ones.
     global little[5], big[20000] 

- Include some example scripts along with the documentation.

- Change the start-time allocation of probe memory to avoid causing OOM
  situations, and to abort cleanly if free kernel memory is short.

- Automatically use the kernel DWARF unwinder, if present, for stack
  tracebacks.

- Many minor bug fixes, performance, tapset, and error message
  improvements.