blob: 4ae043692dddba634327a732c47c25b4d164cd5b (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
# vim: ft=rst
This directory contains test scripts that are useful for running a
bunch of tests all at once.
There are two parts to this:
* The test runner (selftest/selftest.pl)
* The test formatter
selftest.pl simply outputs subunit, which can then be formatted or analyzed
by tools that understand the subunit protocol. One of these tools is
format-subunit.pl, which is used by default as part of "make test".
Available testsuites
====================
The available testsuites are obtained from a script, usually
selftest/samba{3,4}_tests.sh. This script should for each testsuite output
the name of the test, the command to run and the environment that should be
provided. Use the included "plantest" function to generate the required output.
Testsuite behaviour
===================
Exit code
------------
The testsuites should exit with a non-zero exit code if at least one
test failed. Skipped tests should not influence the exit code.
Output format
-------------
Testsuites can simply use the exit code to indicate whether all of their
tests have succeeded or one or more have failed. It is also possible to
provide more granular information using the Subunit protocol.
This protocol works by writing simple messages to standard output. Any
messages that can not be interpreted by this protocol are considered comments
for the last announced test.
Accepted commands are:
test
~~~~
test: <NAME>
Announce that a new test with the specified name is starting
success
~~~~~~~
success: <NAME>
Announce that the test with the specified name is done and ran successfully.
failure
~~~~~~~
failure: <NAME>
failure: <NAME> [ REASON ]
Announce that the test with the specified name failed. Optionally, it is
possible to specify a reason it failed.
The alias "fail" will also work.
xfail
~~~~~
xfail: <NAME>
xfail: <NAME> [ REASON ]
Announce that the test with the specified name failed but that the failure
was expected, e.g. it's a test for a known bug that hasn't been fixed yet.
Alternatively it is also possible to simply return "failure:" here but
specify in the samba4-knownfailures file that it is failing.
skip
~~~~
skip: <NAME>
skip: <NAME> [ REASON ]
Announce that the test with the specified name was skipped. Optionally a
reason can be specified.
time
~~~~
time: YYYY-MM-DD HH:mm:ssZ
Announce the current time. This may be used to calculate the duration of
various tests.
The following are Samba extensions to Subunit:
testsuite-count
~~~~~~~~~~~~~~~
testsuite-count: number
Announce the number of tests that is going to be run.
start-testsuite
~~~~~~~~~
start-testsuite: name
The testsuite name is used as prefix for all containing tests.
skip-testsuite
~~~~~~~~~~~~~~
skip-testsuite: name
Mark the testsuite with the specified name as skipped.
testsuite-success
~~~~~~~~~~~~~~~~~
testsuite-success: name
Indicate that the testsuite has succeeded successfully.
testsuite-fail
~~~~~~~~~~~~~~
testsuite-fail: name
Indicate that a testsuite has failed.
Environments
============
Tests often need to run against a server with particular things set up,
a "environment". This environment is provided by the test "target": Samba 3,
Samba 4 or Windows.
The following environments are currently available:
- none: No server set up, no variables set.
- dc: Domain controller set up. The following environment variables will
be set:
* USERNAME: Administrator user name
* PASSWORD: Administrator password
* DOMAIN: Domain name
* REALM: Realm name
* SERVER: DC host name
* SERVER_IP: DC IPv4 address
* NETBIOSNAME: DC NetBIOS name
* NETIOSALIAS: DC NetBIOS alias
- member: Domain controller and member server that is joined to it set up. The
following environment variables will be set:
* USERNAME: Domain administrator user name
* PASSWORD: Domain administrator password
* DOMAIN: Domain name
* REALM: Realm name
* SERVER: Name of the member server
Running tests
=============
To run all the tests use::
make test
To run a quick subset (aiming for about 1 minute of testing) run::
make quicktest
To run a specific test, use this syntax::
make test TESTS=testname
for example::
make test TESTS=samba4.BASE-DELETE
|
