     [file version.texi not included]

Retrace server
**************

This manual is for retrace server for ABRT version VERSION, UPDATED.  The retrace server provides coredump analysis and backtrace generation service over a network using HTTP protocol.

Overview
********

Analyzing a program crash from a coredump is a difficult task. The GNU Debugger (GDB), that is commonly used to analyze coredumps on free operating systems, expects that the system analyzing the coredump is identical to the system where the program crashed. Software updates often break this assumption even on the system where the crash occured, making the coredump analyzable only with significant effort.

Retrace server solves this problem for Fedora 14+ and RHEL 6+ operating systems, and allows developers to analyze coredumps without having access to the machine where the crash occurred.

Retrace server is usually run as a service on a local network, or on Internet. A user sends a coredump together with some additional information to a retrace server. The server reads the coredump and depending on its contents it installs necessary software dependencies to create a software environment which is, from the GDB point of view, identical to the environment where the crash happened. Then the server runs GDB to generate a backtrace from the coredump and provides it back to the user.

Core dumps generated on i386 and x86_64 architectures are supported within a single x86_64 retrace server instance.

The retrace server consists of the following major parts:

 1. a HTTP interface, consisting of a set of scripts handling communication with clients

 2. a retrace worker, doing the coredump processing, environment preparation, and running the debugger to generate a backtrace

 3. a cleanup script, handling stalled retracing tasks and removing old data

 4. a package repository, providing the application binaries, libraries, and debuginfo necessary for generating backtraces from coredumps

HTTP interface
**************

The client-server communication proceeds as follows:

 1. Client uploads a coredump to a retrace server. Retrace server creates a task for processing the coredump, and sends the task ID and task password in response to the client.

 2. Client asks server for the task status using the task ID and password. Server responds with the status information (task finished successfully, task failed, task is still running).

 3. Client asks server for the backtrace from a successfully finished task using the task ID and password. Server sends the backtrace in response.

 4. Client asks server for a log from the finished task using the task ID and password, and server sends the log in response.

The HTTP interface application is a set of script written in Python, using the Python Web Server Gateway Interface (http://www.python.org/dev/peps/pep-0333/) (WSGI) to interact with a web server. The only supported and tested configuration is the Apache HTTPD Server with mod_wsgi (http://code.google.com/p/modwsgi/).

Only secure (HTTPS) communication is allowed for communicating with a public instance of retrace server, because coredumps and backtraces are private data. Users may decide to publish their backtraces in a bug tracker after reviewing them, but the retrace server doesn't do that. The server is supposed to use HTTP persistent connections to to avoid frequent SSL renegotiations.

Creating a new task
===================

A client might create a new task by sending a HTTP request to the https://server/create URL, and providing an archive as the request content. The archive contains crash data files. The crash data files are a subset of some local /var/spool/abrt/ccpp-time-pid directory contents, so the client must only pack and upload them.

The server supports uncompressed tar archives, and tar archives compressed with gzip and xz. Uncompressed archives are the most efficient way for local network delivery, and gzip can be used there as well because of its good compression speed.

The xz compression file format is well suited for public server setup (slow network), as it provides good compression ratio, which is important for compressing large coredumps, and it provides reasonable compress/decompress speed and memory consumption. See Traffic and load estimation for the measurements. The XZ Utils (http://tukaani.org/xz/) implementation with the compression level 2 is used to compress the data.

The HTTP request for a new task must use the POST method. It must contain a proper CONTENT-LENGTH and CONTENT-TYPE fields. If the method is not POST, the server returns the 405 Method Not Allowed HTTP error code. If the CONTENT-LENGTH field is missing, the server returns the 411 Length Required HTTP error code. If an CONTENT-TYPE other than application/x-tar, application/x-gzip, application/x-xz is used, the server returns the 415 unsupported Media Type HTTP error code. If the CONTENT-LENGTH value is greater than a limit set in the server configuration file (50 MB by default), or the real HTTP request size gets larger than the limit + 10 KB for headers, then the server returns the 413 Request Entity Too Large HTTP error code, and provides an explanation, including the limit, in the response body. The limit is changeable from the server configuration file.

If there is less than 20 GB of free disk space in the /var/spool/abrt-retrace directory, the server returns the 507 Insufficient Storage HTTP error code. The server returns the same HTTP error code if decompressing the received archive would cause the free disk space to become less than 20 GB. The 20 GB limit is changeable from the server configuration file.

If the data from the received archive would take more than 500 MB of disk space when uncompressed, the server returns the 413 Request Entity Too Large HTTP error code, and provides an explanation, including the limit, in the response body. The size limit is changeable from the server configuration file. It can be set pretty high because coredumps, that take most disk space, are stored on the server only temporarily until the backtrace is generated. When the backtrace is generated the coredump is deleted by the abrt-retrace-worker, so most disk space is released.

The uncompressed data size for xz archives is obtained by calling `xz --list file.tar.xz`. The --list option has been implemented only recently, so updating xz on your server might be necessary. Likewise, the uncompressed data size for gzip archives is obtained by calling `gzip --list file.tar.gz`.

If an upload from a client succeeds, the server creates a new directory /var/spool/abrt-retrace/ID and extracts the received archive into it. Then it checks that the directory contains all the required files, checks their sizes, and then sends a HTTP response. After that it spawns a subprocess with abrt-retrace-worker on that directory.

The following files from the local crash directory are required to be present in the archive: coredump, architecture, release, packages (this one does not exist yet). If one or more files are not present in the archive, or some other file is present in the archive, the server returns the 403 Forbidden HTTP error code. If the size of any file except the coredump exceeds 100 KB, the server returns the 413 Request Entity Too Large HTTP error code, and provides an explanation, including the limit, in the response body. The 100 KB limit is changeable from the server configuration file.

If the file check succeeds, the server HTTP response has the 201 Created HTTP code. The response includes the following HTTP header fields:

  * X-TASK-ID containing a new server-unique numerical task id

  * X-TASK-PASSWORD containing a newly generated password, required to access the result

The X-TASK-PASSWORD is a random alphanumeric ([a-zA-Z0-9]) sequence 22 characters long. The password is stored in the /var/spool/abrt-retrace/ID/password file, and passwords sent by a client in subsequent requests are verified by comparing with this file.

The task id is intentionally not used as a password, because it is desirable to keep the id readable and memorable for humans. Password-like ids would be a loss when an user authentication mechanism is added, and server-generated password will no longer be necessary.

Task status
===========

A client might request a task status by sending a HTTP GET request to the https://someserver/ID URL, where ID is the numerical task id returned in the X-TASK-ID field by https://someserver/create. If the ID is not in the valid format, or the task ID does not exist, the server must return the 404 Not Found HTTP error code.

The client request must contain the X-TASK-PASSWORD field, and its content must match the password stored in the /var/spool/abrt-retrace/ID/password file. If the password is not valid, the server must return the 403 Forbidden HTTP error code.

If the checks pass, the server returns the 200 OK HTTP code, and includes a field X-TASK-STATUS containing one of the following values: FINISHED_SUCCESS, FINISHED_FAILURE, PENDING.

The field contains FINISHED_SUCCESS if the file /var/spool/abrt-retrace/ID/backtrace exists. The client might get the backtrace on the https://someserver/ID/backtrace URL. The log might be obtained on the https://someserver/ID/log URL, and it might contain warnings about some missing debuginfos etc.

The field contains FINISHED_FAILURE if the file /var/spool/abrt-retrace/ID/backtrace does not exist, and file /var/spool/abrt-retrace/ID/retrace-log exists. The retrace-log file containing error messages can be downloaded by the client from the https://someserver/ID/log URL.

The field contains PENDING if neither file exists. The client should ask again after 10 seconds or later.

Requesting a backtrace
======================

A client might request a backtrace by sending a HTTP GET request to the https://someserver/ID/backtrace URL, where ID is the numerical task id returned in the X-TASK-ID field by https://someserver/create. If the ID is not in the valid format, or the task ID does not exist, the server returns the 404 Not Found HTTP error code.

The client request must contain the X-TASK-PASSWORD field, and its content must match the password stored in the /var/spool/abrt-retrace/ID/password file. If the password is not valid, the server returns the 403 Forbidden HTTP error code.

If the file /var/spool/abrt-retrace/ID/backtrace does not exist, the server returns the 404 Not Found HTTP error code. Otherwise it returns the file contents, and the CONTENT-TYPE field contains text/plain.

Requesting a log
================

A client might request a task log by sending a HTTP GET request to the https://someserver/ID/log URL, where ID is the numerical task id returned in the X-TASK-ID field by https://someserver/create. If the ID is not in the valid format, or the task ID does not exist, the server returns the 404 Not Found HTTP error code.

The client request must contain the X-TASK-PASSWORD field, and its content must match the password stored in the /var/spool/abrt-retrace/ID/password file. If the password is not valid, the server returns the 403 Forbidden HTTP error code.

If the file /var/spool/abrt-retrace/ID/retrace-log does not exist, the server returns the 404 Not Found HTTP error code. Otherwise it returns the file contents, and the CONTENT-TYPE contains text/plain.

Limiting traffic
================

The maximum number of simultaneously running tasks is limited to 20 by the server. The limit is changeable from the server configuration file. If a new request comes when the server is fully occupied, the server returns the 503 Service Unavailable HTTP error code.

The archive extraction, chroot preparation, and gdb analysis is mostly limited by the hard drive size and speed.

Retrace worker
**************

Retrace worker is a program (usually residing in /usr/bin/abrt-retrace-worker), which:

 1. takes a task id as a parameter, and turns it into a directory containing a coredump

 2. determines which packages need to be installed from the coredump

 3. installs the packages in a newly created chroot environment together with gdb

 4. copies the coredump to the chroot environment

 5. runs gdb from inside the environment to generate a backtrace from the coredump

 6. copies the resulting backtrace from the environment to the directory

The tasks reside in /var/spool/abrt-retrace/TASKID directories.

To determine which packages need to be installed, abrt-retrace-worker runs the coredump2packages tool. The tool reads build-ids from the coredump, and tries to find the best set of packages (epoch, name, version, release) matching the build-ids. Local yum repositories are used as the source of packages. GDB requirements are strict, and this is the reason why proper backtraces cannot be directly and reliably generated on systems whose software is updated:

  * The exact binary which crashed needs to be available to GDB.

  * All libraries which are linked to the binary need to be available in the same exact versions from the time of the crash.

  * The binary plugins loaded by the binary or libraries via dlopen need to be present in proper versions.

  * The files containing the debugging symbols for the binary and libraries (build-ids are used to find the pairs) need to be available to GDB.

The chroot environments are created and managed by mock, and they reside in /var/lib/mock/TASKID. The retrace worker generates a mock configuration file and then invokes mock to create the chroot, and to run programs from inside the chroot.

The chroot environment is populated by installing packages using yum. Package installation cannot be avoided, as GDB expects to operate on an installed system, and on crashes from that system. GDB uses plugins written in Python, that are shipped with packages (for example see rpm -ql libstdc++).

Coredumps might be affected by prelink, which is used on Fedora to speed up dynamic linking by caching its results directly in binaries. The system installed by mock for the purpose of retracing doesn't use prelink, so the binaries differ between the system of origin and the mock environment. It has been tested that this is not an issue, but in the case some issue occurs (http://sourceware.org/ml/gdb/2009-05/msg00175.html) (GDB fails to work with a binary even if it's the right one), a bug should be filed on prelink, as its operation should not affect the area GDB operates on.

No special care is taken to avoid the possibility that GDB will not run with the set of packages (fixed versions) as provided by coredump. It is expected that any combination of packages user might use in a released system should satisfy the needs of some version of GDB. Yum selects the newest possible version which has its requirements satisfied.

Task cleanup
************

Tasks that were created more than 5 days ago are deleted, because tasks occupy disk space (not so much space, as the coredumps are deleted after the retrace, and only backtraces and configuration remain). A shell script abrt-retrace-clean must check the creation time and delete the directories in /var/spool/abrt-retrace/. It is supposed that the server administrator sets cron to call the script once a day. This assumption must be mentioned in the abrt-retrace-clean manual page.

The database containing packages and processing times should also be regularly pruned to remain small and provide data quickly. The cleanup script should delete some rows for packages with too many entries:

 1. get a list of packages from the database: SELECT DISTINCT package, release FROM package_time

 2. for every package, get the row count: SELECT COUNT(*) FROM package_time WHERE package == '??' AND release == '??'

 3. for every package with the row count larger than 100, some rows most be removed so that only the newest 100 rows remain in the database:

       * to get highest row id which should be deleted, execute SELECT id FROM package_time WHERE package == '??' AND release == '??' ORDER BY id LIMIT 1 OFFSET ??, where the OFFSET is the total number of rows for that single package minus 100

       * then all the old rows can be deleted by executing DELETE FROM package_time WHERE package == '??' AND release == '??' AND id <= ??

Package repository
******************

We should support every Fedora release with all packages that ever made it to the updates and updates-testing repositories. In order to provide all that packages, a local repository is maintained for every supported operating system. The debuginfos might be provided by a debuginfo server in future (it will save the server disk space). We should support the usage of local debuginfo first, and add the debuginfofs support later.

A repository with Fedora packages must be maintained locally on the server to provide good performance and to provide data from older packages already removed from the official repositories. We need a package downloader, which scans Fedora servers for new packages, and downloads them so they are immediately available.

Older versions of packages are regularly deleted from the updates and updates-testing repositories. We must support older versions of packages, because that is one of two major pain-points that the retrace server is supposed to solve (the other one is the slowness of debuginfo download and debuginfo disk space requirements).

A script abrt-reposync must download packages from Fedora repositories, but it must not delete older versions of the packages. The retrace server administrator is supposed to call this script using cron every ~6 hours. This expectation must be documented in the abrt-reposync manual page. The script can use use wget, rsync, or reposync tool to get the packages. The remote yum source repositories must be configured from a configuration file or files (/etc/yum.repos.d might be used).

When the abrt-reposync is used to sync with the Rawhide repository, unneeded packages (where a newer version exists) must be removed after residing one week with the newer package in the same repository.

All the unneeded content from the newly downloaded packages should be removed to save disk space and speed up chroot creation. We need just the binaries and dynamic libraries, and that is a tiny part of package contents.

The packages should be downloaded to a local repository in /var/cache/abrt-repo/{fedora12,fedora12-debuginfo,...}.

Traffic and load estimation
***************************

2500 bugs are reported from ABRT every month. Approximately 7.3% from that are Python exceptions, which don't need a retrace server. That means that 2315 bugs need a retrace server. That is 77 bugs per day, or 3.3 bugs every hour on average. Occasional spikes might be much higher (imagine a user that decided to report all his 8 crashes from last month).

We should probably not try to predict if the monthly bug count goes up or down. New, untested versions of software are added to Fedora, but on the other side most software matures and becomes less crashy.  So let's assume that the bug count stays approximately the same.

Test crashes (see that we should probably use `xz -2` to compress coredumps):

  * firefox with 7 tabs (random pages opened), coredump size 172 MB

       * xz compression

            * compression level 6 (default): compression took 32.5 sec, compressed size 5.4 MB, decompression took 2.7 sec

            * compression level 3: compression took 23.4 sec, compressed size 5.6 MB, decompression took 1.6 sec

            * compression level 2: compression took 6.8 sec, compressed size 6.1 MB, decompression took 3.7 sec

            * compression level 1: compression took 5.1 sec, compressed size 6.4 MB, decompression took 2.4 sec

       * gzip compression

            * compression level 9 (highest): compression took 7.6 sec, compressed size 7.9 MB, decompression took 1.5 sec

            * compression level 6 (default): compression took 2.6 sec, compressed size 8 MB, decompression took 2.3 sec

            * compression level 3: compression took 1.7 sec, compressed size 8.9 MB, decompression took 1.7 sec

  * thunderbird with thousands of emails opened, coredump size 218 MB

       * xz compression

            * compression level 6 (default): compression took 60 sec, compressed size 12 MB, decompression took 3.6 sec

            * compression level 3: compression took 42 sec, compressed size 13 MB, decompression took 3.0 sec

            * compression level 2: compression took 10 sec, compressed size 14 MB, decompression took 3.0 sec

            * compression level 1: compression took 8.3 sec, compressed size 15 MB, decompression took 3.2 sec

       * gzip compression

            * compression level 9 (highest): compression took 14.9 sec, compressed size 18 MB, decompression took 2.4 sec

            * compression level 6 (default): compression took 4.4 sec, compressed size 18 MB, decompression took 2.2 sec

            * compression level 3: compression took 2.7 sec, compressed size 20 MB, decompression took 3 sec

  * evince with 2 pdfs (1 and 42 pages) opened, coredump size 73 MB

       * xz compression

            * compression level 2: compression took 2.9 sec, compressed size 3.6 MB, decompression took 0.7 sec

            * compression level 1: compression took 2.5 sec, compressed size 3.9 MB, decompression took 0.7 sec

  * OpenOffice.org Impress with 25 pages presentation, coredump size 116 MB

       * xz compression

            * compression level 2: compression took 7.1 sec, compressed size 12 MB, decompression took 2.3 sec

So let's imagine there are some users that want to report their crashes approximately at the same time. Here is what the retrace server must handle:

 1. 2 OpenOffice crashes

 2. 2 evince crashes

 3. 2 thunderbird crashes

 4. 2 firefox crashes

We will use the xz archiver with the compression level 2 on the ABRT's side to compress the coredumps. So the users spend 53.6 seconds in total packaging the coredumps.

The packaged coredumps have 71.4 MB, and the retrace server must receive that data.

The server unpacks the coredumps (perhaps in the same time), so they need 1158 MB of disk space on the server. The decompression will take 19.4 seconds.

Several hundred megabytes will be needed to install all the required packages and debuginfos for every chroot (8 chroots 1 GB each = 8 GB, but this seems like an extreme, maximal case). Some space will be saved by using a debuginfofs.

Note that most applications are not as heavyweight as OpenOffice and Firefox.

Security
********

The retrace server communicates with two other entities: it accepts coredumps form users, and it downloads debuginfos and packages from distribution repositories.

General security from GDB flaws and malicious data is provided by chroot. The GDB accesses the debuginfos, packages, and the coredump from within the chroot, unable to access the retrace server's environment. We should consider setting a disk quota to every chroot directory, and limit the GDB access to resources using cgroups.

SELinux policy should be written for both the retrace server's HTTP interface, and for the retrace worker.

Clients
=======

The clients, which are using the retrace server and sending coredumps to it, must fully trust the retrace server administrator.  The server administrator must not try to get sensitive data from client coredumps.  That seems to be a major bottleneck of the retrace server idea.  However, users of an operating system already trust the OS provider in various important matters. So when the retrace server is operated by the operating system provider, that might be acceptable by users.

We cannot avoid sending clients' coredumps to the retrace server, if we want to generate quality backtraces containing the values of variables. Minidumps lower the quality of the resulting backtraces, while not improving user security.

Can the retrace server trust clients? We must know what can a malicious client achieve by crafting a nonstandard coredump, which will be processed by server's GDB.  We should ask GDB experts about this.

Another question is whether we can allow users providing some packages and debuginfo together with a coredump. That might be useful for users, who run the operating system only with some minor modifications, and they still want to use the retrace server. So they send a coredump together with a few nonstandard packages. The retrace server uses the nonstandard packages together with the OS packages to generate the backtrace. Is it safe? We must know what can a malicious client achieve by crafting a special binary and debuginfo, which will be processed by server's GDB.

As for attacker trying to steal users' backtraces from the retrace server, the passwords protecting the backtraces in the X-TASK-PASSWORD header are random alphanumeric ([a-zA-Z0-9]) sequences 22 characters long. 22 alphanumeric characters corresponds to 128 bit password, because [a-zA-Z0-9] is 62 characters, and 2^128 < 62^22. The source of randomness is /dev/urandom.

Packages and debuginfo
======================

We can safely download packages and debuginfo from the distribution, as the packages are signed by the distribution, and the package origin can be verified.

When the debuginfo server is done, the retrace server can safely use it, as the data will also be signed.

Future work
***********

Coredump stripping
==================

Jan Kratochvil: With my test of OpenOffice.org presentation kernel core file has 181MB, xz -2 of it has 65MB.  According to `set target debug 1' GDB reads only 131406 bytes of it (incl. the NOTE segment).

Supporting other architectures
==============================

Three approaches:

  * Use GDB builds with various target architectures: gdb-i386, gdb-ppc64, gdb-s390.

  * Run QEMU user space emulation (http://wiki.qemu.org/download/qemu-doc.html#QEMU-User-space-emulator) on the server

  * Run abrt-retrace-worker on a machine with right architecture. Introduce worker machines and tasks, similarly to Koji.

Use gdbserver instead of uploading whole coredump
=================================================

GDB's gdbserver cannot process coredumps, but Jan Kratochvil's can:
git://git.fedorahosted.org/git/elfutils.git
branch: jankratochvil/gdbserver
  src/gdbserver.c
   * Currently threading is not supported.
   * Currently only x86_64 is supported (the NOTE registers layout).

User management for the HTTP interface
======================================

Multiple authentication sources (x509 for RHEL).

Make all files except coredump optional on the input
====================================================

Make architecture, release, packages files, which must be included in the package when creating a task, optional. Allow uploading a coredump without involving tar: just coredump, coredump.gz, or coredump.xz.

Handle non-standard packages (provided by user)
===============================================

This would make retrace server very vulnerable to attacks, it never can be enabled in a public instance.

Support vmcores
===============

See Core analysis system (https://fedorahosted.org/cas/), its features etc.

Do not refuse new tasks on a fully loaded server
================================================

Consider using kobo (http://git.fedorahosted.org/git/?p=kobo.git) for task management and worker handling (master/slaves arch).

Support synchronous operation
=============================

Client sends a coredump, and keeps receiving the server response message. The server response HTTP body is generated and sent gradually as the task is performed. Client can choose to stop receiving the response body after getting all headers and ask the server for status and backtrace asynchronously.

The server re-sends the output of abrt-retrace-worker (its stdout and stderr) to the response the body. In addition, a line with the task status is added in the form X-Task-Status: PENDING to the body every 5 seconds. When the worker process ends, either FINISHED_SUCCESS or FINISHED_FAILURE status line is sent. If it's FINISHED_SUCCESS, the backtrace is attached after this line. Then the response body is closed.

Provide task estimation time
============================

The response to the /create action should contain a header X-TASK-EST-TIME, that contains a number of seconds the server estimates it will take to generate the backtrace

The algorithm for the X-TASK-EST-TIME time estimation should take the previous analyses of coredumps with the same corresponding package name into account. The server should store simple history in a SQLite database to know how long it takes to generate a backtrace for certain package. It could be as simple as this:

  * initialization step one: CREATE TABLE package_time (id INTEGER PRIMARY KEY AUTOINCREMENT, package, release, time); we need the ID for the database cleanup - to know the insertion order of rows, so the AUTOINCREMENT is important here; the PACKAGE is the package name without the version and release numbers, the RELEASE column stores the operating system, and the TIME is the number of seconds it took to generate the backtrace

  * initialization step two: CREATE INDEX package_release ON package_time (package, release); we compute the time only for single package on single supported OS release per query, so it makes sense to create an index to speed it up

  * when a task is finished: INSERT INTO package_time (package, release, time) VALUES ('??', '??', '??')

  * to get the average time: SELECT AVG(time) FROM package_time WHERE package == '??' AND release == '??'; the arithmetic mean seems to be sufficient here

So the server knows that crashes from an OpenOffice.org package take 5 minutes to process in average, and it can return the value 300 (seconds) in the field. The client does not waste time asking about that task every 20 seconds, but the first status request comes after 300 seconds. And even when the package changes (rebases etc.), the database provides good estimations after some time anyway (Task cleanup chapter describes how the data are pruned).