Copyright (c) 2011 Opera Software Australia Pty. Ltd.  All rights reserved.

This document describes how to run the Cassandane tests.

Prerequisites
-------------

Before running any Cassandane tests, you need to set up Cassandane,
Cyrus and your system.  Read the file setting_up.txt and follow the
instructions there.

Running Tests
-------------

Cassandane tests are run out of the Cassandane directory itself, without
installing Cassandane anywhere.  This is not the result of deliberate policy so
much as implementation laziness.

All runtime state is created under the cassandane rootdir configured in
cassandane.ini (by default: /var/tmp/cass).

Internally, Cassandane (or more precisely, the Cyrus code it exercises) needs
to be run either as the superuser or as the "cyrus" user.  But you should
generally invoke Cassandane as yourself, not as "cyrus" or "root".  It will
try to re-run itself using sudo, which you already configured during setup
(didn't you?)

The script 'testrunner.pl' is your interface for running Cassandane tests.
There are several other Perl scripts in the directory, but they are utilities
which were helpful during manual testing rather than part of the test suite
itself.

With no arguments, testrunner.pl runs all the tests that come with Cassandane
and reports the results to the terminal in the 'prettier' test report format.
The testrunner.pl exit code will be 0 if all tests passed, non-zero otherwise.

    $ ./testrunner.pl
    [  OK  ] Cyrus::ACL.reconstruct
    [  OK  ] Cyrus::ACL.move
    [  OK  ] Cyrus::ACL.delete
    ...

There are several test report formats to choose from, by invoking testrunner.pl
with the -f 'format' option.

-f pretty:
    Human readable output to the terminal, showing the ok/failed/error status
    and name for each test, as well as the error reports from any not-ok tests.
    This gets noisy in the case of failures!  It's mostly useful when debugging
    single tests, especially in conjunction with -vvv.

-f prettier (the default):
    As for pretty, but without the noise when problems occur.  This is most
    useful when running many (or all) tests at once.  A list of failed tests
    is written to $rootdir/failed, and the full error reports for any failed
    tests are written to $rootdir/reports, so you can still access these details
    if you find yourself needing them after the fact.

-f xml:
    This writes reports in jUnit format.  The reports will be xml files in a
    subdirectory "reports" of the current directory at the time testrunner.pl
    was invoked.  Note that this is NOT the same "reports" file as used by
    -f prettier.  This format is apparently useful for integration with various
    CI systems, though it's not used by our Github CI.

-f tap:
    TAP is a common format which originated with Perl and is now widely used,
    see http://en.wikipedia.org/wiki/Test_Anything_Protocol for more
    information.  This seems to prints a single character for each test, or
    something.  I'm not sure what it's useful for, since if a test fails you
    don't know which one or why.

You can run just a subset of tests by giving arguments to testrunner.pl.
Tests to run are most commonly specified as:

 * a test suite without the leading Cassandane::Cyrus

    $ ./testrunner.pl Quota

 * a single test in a single test suite

    $ ./testrunner.pl Quota.quotarename

Multiple test suites or tests can be specified as well:

    $ ./testrunner.pl Admin Quota.quotarename

Arguments can be negated by using a leading exclamation mark (!) or tilde (~)
character.  Note that you may need to escape the ! from the shell, so ~ is
generally preferable:

    $ ./testrunner.pl ~Quota

will run all the tests from all the suites except the Quota suite.
Arguments accumulate from left to right, so e.g.

    $ ./testrunner.pl Quota ~Quota.quotarename

will run all the tests in the Quota suite except the quotarename test.

The -v (or --verbose) option to testrunner.pl causes both Cassandane and
several Cyrus programs run by Cassandane to emit a lot of information to
stderr.  You can specify this option multiple times for increased verbosity,
and the single-character version can be stacked, like -vvv.

The --valgrind option to testrunner.pl runs all the Cyrus executables
using Valgrind.  This is of course much slower but is recommended
because it finds many subtle bugs.  The Valgrind logs are saved in
the files $rootdir/$instance/vglogs/$name.$pid.  Cassandane will
examine these logs after each test finishes, and will fail the test
if there are any errors (including memory leaks) reported.

The --cleanup option causes Cassandane to do two things.  Firstly, it
immediately cleans up any files left over in $rootdir.  Secondly,
it cleans up any such files after each test, unless the test fails.
This should be helpful when the filesystem in use does not have much room,
such as when running on a tmpfs filesystem.  You'll probably find this useful,
so enable cassandane.cleanup in your cassandane.ini rather than typing it
all the time.  Then use --no-cleanup to override it when you don't want that.

testrunner.pl also accepts a bunch of other options that are not documented
here.  Consult the script itself for the full and most up-to-date set.
