* What's new in version 1.6

- The systemtap compile-server no longer uses the -k option when calling the
  translator (stap). As a result, the server will now take advantage of the
  module cache when compiling the same script more than once. You may observe
  an improvement in the performance of the server in this situation.

- The systemtap compile-server and client now each check the version of the
  other, allowing both to adapt when communicating with a down-level
  counterpart. As a result, all version of the client can communicate
  with all versions of the server and vice-versa. Client will prefer newer
  servers when selecting a server automatically.

- SystemTap has improved support for the ARM architecture.  The
  kread() and kwrite() operations for ARM were corrected allowing many
  of the tapsets probes and function to work properly on the ARM
  architecture.

- Staprun can now rename the module to a unique name with the '-R' option before 
  inserting it. Systemtap itself will also call staprun with '-R' by default. 
  This allows the same module to be inserted more than once, without conflicting 
  duplicate names.

- Systemtap error messages now provide feedback when staprun or any other
  process fails to launch.  This also specifically covers when the user 
  doesn't have the proper permissions to launch staprun.

- Systemtap will now map - to _ in module names.  Previously, 
  stap -L 'module("i2c-core").function("*")' would be empty.  It now returns
  a list had stap -L 'module("i2c_core").function("*") been specified.

- Systemtap now fills in missing process names to probe points, to
  avoid having to name them twice twice:
  % stap -e 'probe process("a.out").function("*") {}' -c 'a.out ...'
  Now the probed process name is inferred from the -c CMD argument.
  % stap -e 'probe process.function("*") {}' -c 'a.out ...'

- stap -L 'process("PATH").syscall' will now list context variables

- Depends on elfutils 0.142+.

- Deprecated task_backtrace:string (task:long). This function will go
  away after 1.6. Please run your scripts with stap --check-version.

* What's new in version 1.5, 2011-05-23

- The compile server and its related tools (stap-gen-ert, stap-authorize-cert,
   stap-sign-module) have been re-implemented in C++. Previously, these
   components were a mix of bash scripts and C code. These changes should be
   transparent to the end user with the exception of NSS certificate database
   password prompting (see below). The old implementation would prompt more
   than once for the same password in some situations.
  
- eventcount.stp now allows for event counting in the format of
  'stap eventcount.stp process.end syscall.* ...', and also reports
  corresponding event tid's.

- Systemtap checks that the build-id of the module being probed matches the
  build-id saved in the systemtap module.  Invoking systemtap with
  -DSTP_NO_BUILDID_CHECK will bypass this build-id runtime verification.  See
  man ld(1) for info on --build-id.

- stapio will now report if a child process has an abnormal exit along with
  the associated status or signal.

- Compiler optimization may sometimes result in systemtap not being able to
  access a user-space probe argument.  Compiling the application with
  -DSTAP_SDT_ARG_CONSTRAINT=nr will force the argument to be an immediate or
  register value which should enable systemtap to access the argument.

- GNU Gettext has now been intergrated with systemtap.  Our translation
  page can be found at http://www.transifex.net/projects/p/systemtap/ .
  "make update-po" will generate the necessary files to use translated
  messages.  Please refer to the po/README file for more info and 
  please consider contributing to this I18N effort!

- The new addr() function returns the probe's instruction pointer.

- process("...").library("...") probes are now supported.  Wildcards
  are supported in the library-name part, to refer to any shared
  library that is required by process-name, which matches the glob
  pattern and the rest of the probe point.

- The "--remote USER@HOST" functionality can now be specified multiple times
  to fan out on multiple targets.  If the targets have distinct kernel and
  architecture configurations, stap will automatically build the script
  appropriately for each one.  This option is also no longer considered
  experimental.

- The NSS certificate database generated for use by the compile server is now
  generated with no password. Previously, a random password was generated and
  used to access the database. This change should be transparent to most users.
  However, if you are prompted for a password when using systemtap, then
  running $libexecdir/stap-gen-cert should correct the problem.

- The timestamp tapset includes jiffies() and HZ() for lightweight approximate
  timekeeping.

- A powerful new command line option --version has been added.

- process.mark now supports $$parms for reading probe parameters.

- A new command line option, --use-server-on-error[=yes|no] is available
  for stap.  It instructs stap to retry compilation of a script using a
  compile server if it fails on the local host.  The default setting
  is 'no'.

- The following deprecated tools have been removed:
      stap-client
      stap-authorize-server-cert
      stap-authorize-signing-cert
      stap-find-or-start-server
      stap-find-servers
  Use the --use-server, --trust-server and --list-servers options of stap
  instead.

* What's new in version 1.4, 2011-01-17

- A new /* myproc-unprivileged */ marker is now available for embedded C
  code and and expressions. Like the /* unprivileged */ marker, it makes
  the code or expression available for use in unprivileged mode (see
  --unprivileged). However, it also automatically adds a call to
  assert_is_myproc() to the code or expression, thus, making it available
  to the unprivileged user only if the target of the current probe is within
  the user's own process.

- The experimental "--remote USER@HOST" option will run pass 5 on a given
  ssh host, after building locally (or with --use-server) for that target.

- Warning messages from the script may now be suppressed with the stap
  and/or staprun -w option.  By default, duplicate warning messages are
  suppressed (up to a certain limit).  With stap --vp 00002 and above,
  the duplicate elimination is defeated.

- The print_ubacktrace and usym* functions attempt to print the full
  path of the user-space binaries' paths, instead of just the basename.
  The maximum saved path length is set by -DTASK_FINDER_VMA_ENTRY_PATHLEN,
  default 64.  Warning messages are produced if unwinding fails due to
  a missing 'stap -d MODULE' option, providing preloaded unwind data.

- The new tz_ctime() tapset function prints times in the local time zone.

- More kernel tracepoints are accessible to the kernel.trace("...") mechanism,
  if kernel source trees or debuginfo are available.  These formerly "hidden"
  tracepoints are those that are declared somewhere other than the usual
  include/linux/trace/ headers, such as xfs and kvm.

- debuginfo-based process("...").function/.statement/.mark probes support
  wildcards in the process-name part, to refer to any executable files that
  match the glob pattern and the rest of the probe point.

- The -t option now displays information per probe-point rather than a summary
  for each probe.  It also now shows the derivation chain for each probe-point.

- A rewrite of the sys/sdt.h header file provides zero-cost startup (few or
  no ELF relocations) for the debuginfo-less near-zero-cost runtime probes.
  Binaries compiled with earlier sdt.h versions remain supported.  The
  stap -L (listing) option now lists parameters for sys/sdt.h markers.

- The implementation of the integrated compile-server client has been
  extended.
  o --use-server now accepts an argument representing a particular server and
    may be specified more than once.
  o --list-servers now accepts an expanded range of arguments.
  o a new --trust-servers option has been added to stap to replace several
    old certificate-management scripts.
  o The following tools are now deprecated and will be removed in release 1.5:
      stap-client
      stap-authorize-server-cert
      stap-authorize-signing-cert
      stap-find-or-start-server
      stap-find-servers
  See man stap(1) for complete details.

- The compile-server now returns the uprobes.ko to the client when it is
  required by the script being compiled. The integrated compile-server client
  now makes it available to be loaded by staprun. The old (deprecated)
  stap-client does not do this.

- process probes with scripts as the target are recognized by stap and the
  interpreter would be selected for probing.

- Starting in release 1.5, these old variables/functions will be deprecated
  and will only be available when the '--compatible=1.4' flag is used:

  - In the 'syscall.add_key' probe, the 'description_auddr' variable
    has been deprecated in favor of the new 'description_uaddr'
    variable.
  - In the 'syscall.fgetxattr', 'syscall.fsetxattr',
    'syscall.getxattr', 'syscall.lgetxattr', and
    'syscall.lremovexattr' probes, the 'name2' variable has been
    deprecated in favor of the new 'name_str' variable.
  - In the 'nd_syscall.accept' probe the 'flag_str' variable
    has been deprecated in favor of the new 'flags_str' variable.
  - In the 'nd_syscall.dup' probe the 'old_fd' variable has been
    deprecated in favor of the new 'oldfd' variable.
  - In the 'nd_syscall.fgetxattr', 'nd_syscall.fremovexattr',
    'nd_syscall.fsetxattr', 'nd_syscall.getxattr', and
    'nd_syscall.lremovexattr' probes, the 'name2' variable has been 
    deprecated in favor of the new 'name_str' variable.
  - The tapset alias 'nd_syscall.compat_pselect7a' was misnamed.  It should
    have been 'nd_syscall.compat_pselect7' (without the trailing 'a').
  - The tapset function 'cpuid' is deprecated in favor of the better known 
    'cpu'.
  - In the i386 'syscall.sigaltstack' probe, the 'ussp' variable has
    been deprecated in favor of the new 'uss_uaddr' variable.
  - In the ia64 'syscall.sigaltstack' probe, the 'ss_uaddr' and
    'oss_uaddr' variables have been deprecated in favor of the new
    'uss_uaddr' and 'uoss_uaddr' variables.
  - The powerpc tapset alias 'syscall.compat_sysctl' was deprecated
    and renamed 'syscall.sysctl32'.
  - In the x86_64 'syscall.sigaltstack' probe, the 'regs_uaddr'
    variable has been deprecated in favor of the new 'regs' variable.

* What's new in version 1.3, 2010-07-21

- The uprobes kernel module now has about half the overhead when probing
  NOPs, which is particularly relevant for sdt.h markers.

- New stap option -G VAR=VALUE allows overriding global variables
  by passing the settings to staprun as module options.

- The tapset alias 'syscall.compat_pselect7a' was misnamed.  It should
  have been 'syscall.compat_pselect7' (without the trailing 'a').
  Starting in release 1.4, the old name will be deprecated and
  will only be available when the '--compatible=1.3' flag is used.

- A new procfs parameter .umask(UMASK) which provides modification of
  file permissions using the proper umask value.  Default file
  permissions for a read probe are 0400, 0200 for a write probe, and
  0600 for a file with a read and write probe.

- It is now possible in some situations to use print_ubacktrace() to
  get a user space stack trace from a kernel probe point. e.g. for
  user backtraces when there is a pagefault:
  $ stap -d /bin/sort --ldd -e 'probe vm.pagefault {
      if (pid() == target()) {
         printf("pagefault @0x%x\n", address); print_ubacktrace();
      } }' -c /bin/sort
  [...]
  pagefault @0x7fea0595fa70
   0x000000384f07f958 : __GI_strcmp+0x12b8/0x1440 [libc-2.12.so]
   0x000000384f02824e : __gconv_lookup_cache+0xee/0x5a0 [libc-2.12.so]
   0x000000384f021092 : __gconv_find_transform+0x92/0x2cf [libc-2.12.so]
   0x000000384f094896 : __wcsmbs_load_conv+0x106/0x2b0 [libc-2.12.so]
   0x000000384f08bd90 : mbrtowc+0x1b0/0x1c0 [libc-2.12.so]
   0x0000000000404199 : ismbblank+0x39/0x90 [sort]
   0x0000000000404a4f : inittables_mb+0xef/0x290 [sort]
   0x0000000000406934 : main+0x174/0x2510 [sort]
   0x000000384f01ec5d : __libc_start_main+0xfd/0x1d0 [libc-2.12.so]
   0x0000000000402509 : _start+0x29/0x2c [sort]
  [...]

- New tapset functions to get a string representation of a stack trace:
  sprint_[u]backtrace() and sprint_[u]stack().

- New tapset function to get the module (shared library) name for a
  user space address umodname:string(long). The module name will now
  also be in the output of usymdata() and in backtrace addresses even
  when they were not given with -d at the command line.

- Kernel backtraces are now much faster (replaced a linear search
  with a binary search).

- A new integrated compile-server client is now available as part of stap.

  o 'stap --use-server ...' is equivalent to 'stap-client ...'
  o 'stap --list-servers' is equivalent to 'stap-find-servers'
  o 'stap --list-servers=online' is equivalent to 'stap-find-servers --all'
  o stap-client and its related tools will soon be deprecated.
  o the nss-devel and avahi-devel packages are required for building stap with
    the integrated client (checked during configuration).
  o nss and avahi are required to run the integrated client.

- A new operator @entry is available for automatically saving an expression
  at entry time for use in a .return probe.
   probe foo.return { println(get_cycles() - @entry(get_cycles())) }

- Probe $target variables and @cast() can now use a suffix to print complex
  data types as strings.  Use a single '$' for a shallow view, or '$$' for a
  deeper view that includes nested types.  For example, with fs_struct:
   $fs$ : "{.users=%i, .lock={...}, .umask=%i,
            .in_exec=%i, .root={...}, .pwd={...}}"
   $fs$$ : "{.users=%i, .lock={.raw_lock={.lock=%u}}, .umask=%i, .in_exec=%i,
             .root={.mnt=%p, .dentry=%p}, .pwd={.mnt=%p, .dentry=%p}}"

- The <sys/sdt.h> user-space markers no longer default to an implicit
  MARKER_NAME_ENABLED() semaphore check for each marker.  To check for
  enabled markers use a .d declaration file, then:
     if (MARKER_NAME_ENABLED()) MARKER_NAME()

- Hyphenated <sys/sdt.h> marker names such as process(...).mark("foo-bar")
  are now accepted in scripts.  They are mapped to the double-underscore
  form ("foo__bar").

- More robust <sys/sdt.h> user-space markers support is included.  For
  some platforms (x86*, ppc*), this can let systemtap probe the markers
  without debuginfo.  This implementation also supports preserving
  the "provider" name associated with a marker:
    probe process("foo").provider("bar").mark("baz") to match
    STAP_PROBE<n>(bar, baz <...>)
  (Compile with -DSTAP_SDT_V1 to revert to the previous implementation.
  Systemtap supports pre-existing or new binaries using them.)

- Embedded-C may be used within expressions as values, when in guru mode:
     num = %{ LINUX_VERSION_CODE %}               // int64_t
     name = %{ /* string */ THIS_MODULE->name %}  // const char*
     printf ("%s %x\n", name, num)
  The usual /* pure */, /* unprivileged */, and /* guru */ markers may be used 
  as with embedded-C functions.

- By default the systemtap-runtime RPM builds now include a shared
  library, staplog.so, that allows crash to extract systemtap data from
  a vmcore image.

- Iterating with "foreach" can now explicitly save the value for the loop.
     foreach(v = [i,j] in array)
       printf("array[%d,%s] = %d\n", i, j, v /* array[i,j] */)

- The new "--ldd" option automatically adds any additional shared
  libraries needed by probed or -d-listed userspace binaries to the -d
  list, to enable symbolic backtracing through them.  Similarly, the
  new "--all-modules" option automatically adds any currently loaded
  kernel modules (listed in /proc/modules) to the -d list.

- A new family of set_kernel_* functions make it easier for gurus to write
  new values at arbitrary memory addresses.

- Probe wildcards can now use '**' to cross the '.' separator.
     $ stap -l 'sys**open'
     syscall.mq_open
     syscall.open

- Backward compatibility flags (--compatible=VERSION, and matching
  script preprocessing predicate %( systemtap_v CMP "version" %)
  and a deprecation policy are being introduced, in case future
  tapset/language changes break valid scripts.

* What's new in version 1.2, 2010-03-22

- Prototype support for "perf events", where the kernel supports the
  2.6.33 in-kernel API.  Probe points may refer to low-level 
  perf_event_attr type/config numbers, or to a number of aliases
  defined in the new perf.stp tapset:
     probe perf.sw.cpu_clock, perf.type(0).config(4) { }

- Type-casting can now use multiple headers to resolve codependencies.
     @cast(task, "task_struct",
           "kernel<linux/sched.h><linux/fs_struct.h>")->fs->umask

- Tapset-related man pages have been renamed.  'man -k 3stap' should show
  the installed list, which due to prefixing should no longer collide over
  ordinary system functions.

- User space marker arguments no longer use volatile if the version of gcc,
  which must be at least 4.5.0, supports richer DWARF debuginfo.  Use cflags
  -DSTAP_SDT_VOLATILE=volatile or -DSTAP_SDT_VOLATILE= when building
  the sys/sdt.h application to override this one way or another.

- A new construct for error handling is available.  It is similar to c++
  exception catching, using try and catch as new keywords.  Within a handler
  or function, the following is valid and may be nested:
     try { /* arbitrary statemen                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                