Dummy (or Destructive) Testsuite Framework, DTF
===============================================

Testsuite supported by this framework is expected¹ to be destructive, may
require root permissions or basically whatever.  So do not run such testsuite on
your machine unless you *really* know what you are doing.  It is always better
to create temporary virtual machine to run them there (we have some tooling for
OpenStack VM provisioning).


WORKFLOW
--------

In general, we expect these steps for running the testsuite:

    * generate self-standing testsuite tarball
    * start (temporary) virtual machine
    * copy the test tarball on the new machine and perform testing
    * collect the results from VM
    * terminate the VM

However, the testsuite is designed so it may be developed (and thus run)
locally.

OVERVIEW
--------

The framework consists of basically two parts, 'tester' and 'controller'.

The 'tester' component is something like tool-set to help users with writing
theirs 'testsuites'.  That is (a) convenient (Bash) library available for
for testsuite, (b) main running script (named 'run', it is also part of DTF API)
and (c) tooling which helps with testsuite initialization.

The 'controller' part is tool-set which helps you to automatize running of the
testsuite on remote boxes (currently on OpenStack VMs only).


PORTABILITY
-----------

Currently, the tools here are mostly written in Bash (not shell) and Perl and
thats the reason why you need² to have these languages working on your system.
Also, it was tested only on GNU/Linux operating system.


CREATE TESTSUITE
----------------

Each testsuite must be self-standing, distributable and ideally non-destructive.
Creating testsuite should follow this cook-book:

* Create the directory structure:

            $ mkdir YOURPKG_TEST_DIR
            $ dtf-prepare-testsuite YOURPKG_TEST_DIR

  Then prepares the 'dtflib', 'run' script and basic dummy task for you.  The
  dummy test-case created is in 'YOURPKG_TEST_DIR/tasks/sample/runtest.sh'.

  Be aware of the fact that 'dtf-prepare-testsuite' is expected to be always run
  against latest GIT version of that script.  The 'dtflib' and 'run' files (and
  probably other) will then be copied from GIT.

* Test the sample:

            $ ./YOURPKG_TEST_DIR/run
            RUN dummy-sample ...                    [  OK  ]

  The 'runtest.sh' file must have the 'run()' function defined and should follow
  the exit-status API defined in 'run' script.

* To distribute the tarball, just run:

            $ ./YOURPKG_TEST_DIR/run --dist | gzip > testsuite.tar.gz

  Also, please look at the `YOURPKG_TEST_DIR/run --help` output.


REMOTE RUN
----------

To perform this task easily, we expect you to have access to some OpenStack
instance.  You need to have OpenStack credentials file in
'~/.dtf/private/os/OSID.yml' (note the OSID identifier).  The file should look
like (be sure that only *you* can read that file):

    $ cat ~/.dtf/private/os/OSID.yml
    ---
    os_auth_url: http://control.example.com:5000/v2.0
    os_tenant_id: 9df30fc192f5xxxxxxxxxxxxxxxx0110
    os_tenant_name: Your OS Project
    os_username: os1username
    os_nova_password: your_os_password

Also, you must have your distribution images prepared available in OpenStack.
This needs to be configured for DTF in '~/.dtf/os/OSID.sh'.  See the
'OSID.sh.template' doc file for example configuration.

Then it should be enough to run:

    $ dtf-run-remote --distro=fedora --distro-version=20 --taskdir YOUR_TEST_DIR
    ...


REQUIREMENTS
------------

Packages needed to successful run
    * ansible (for remote runs)
    * Perl
    * Bash
    * perl(Text::Xslate)
    * perl(YAML::Syck)

----
¹ However, making tests it non-destructive is highly recommended.
² So far there was no need to make it more portable, however it should be
  possible.