Differential D253

Document the TAP13 structure required for resultsdb and bodhi comment reporting
ClosedPublic
Actions

Authored by jskladan on Oct 16 2014, 10:03 AM.

Details

Reviewers
kparal
adamwill
tflink
Maniphest Tasks
T183: Document the TAP13 structure required for resultsdb and bodhi comment reporting
Commits
rLTRNfff55dc614c2: Document the TAP13 structure required for resultsdb and bodhi comment reporting
Summary

This is an initial version of the documentation describing the TAP13 output and its (special) structure in Taskotron.

Now is probably the time for someone with sleet-english-skillz to take over and polish the text.

Test Plan

make docs

Diff Detail

Repository
rLTRN libtaskotron
Lint
Lint Skipped
Unit
Unit Tests Skipped
jskladan retitled this revision from to Document the TAP13 structure required for resultsdb and bodhi comment reporting.Oct 16 2014, 10:03 AM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added reviewers: tflink, adamwill, kparal.
kparal added inline comments.Oct 16 2014, 10:53 AM
docs/source/tapformat.rst
13

Hyperlink for "TAP13 specification"?

33

Since the second example is also not ok, we could have ok for the first one.

42

If you want this to look like code, use double backticks. Single backtick is interpreted text:
http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#interpreted-text

kparal added inline comments.Oct 16 2014, 10:55 AM
docs/source/tapformat.rst
13

/me headdesk
Of course it's mentioned in the previous paragraph.

jskladan updated this revision to Diff 709.Oct 16 2014, 10:56 AM
  • Fixed according to review
This revision is now accepted and ready to land.Oct 16 2014, 11:15 AM
jskladan closed this revision.Oct 17 2014, 9:11 AM
jskladan updated this revision to Diff 718.

Closed by commit rLTRNfff55dc614c2 (authored by @jskladan).

Revision Contents

Diff 718

View Options

docs/source/index.rst

Show All 23 Lines23.. toctree::
24 :maxdepth: 1 24​ :maxdepth: 1
25 25​
26 quickstart 26​ quickstart
27 writingtasks 27​ writingtasks
28 taskyaml 28​ taskyaml
29 library 29​ library
30 devguide 30​ devguide
31 directives/list_of_directives_modules 31​ directives/list_of_directives_modules
32​ tapformat
32 33​
33 34​
34 35​
35.. _what-can-i-do-with-taskotron: 36​.. _what-can-i-do-with-taskotron:
36 37​
37What Can I Do With Taskotron? 38​What Can I Do With Taskotron?
38============================= 39​=============================
39 40​
▲ Show 20 Lines • Show All 85 Lines • Show Last 20 Lines
View Options

docs/source/tapformat.rst

  • This file was added.
1​.. _tap13-format:
2​
3​================
4​TAP in Taskotron
5​================
6​
7​The required output format of the checks in Taskotron is the `Test Anything Protocol (TAP) v13
8​<http://podwiki.hexten.net/TAP/TAP13.html?page=TAP13>`_.
9​
10​We recommend using the provided :py:class:`libtaskotron.check.CheckDetail` and the
11​helper method :py:meth:`libtaskotron.check.export_TAP` to create the TAP output.
12​If you can not use it, then please adhere to the TAP13 specification, and add
13​the required additional information (described below) to the YAML block.
kparalUnsubmitted
Not Done

Hyperlink for "TAP13 specification"?

kparalUnsubmitted
Not Done

/me headdesk
Of course it's mentioned in the previous paragraph.

14​
15​
16​TAP and reporting
17​=================
18​
19​First up, we will provide you with two examples of the valid TAP output - the minimal
20​version, which just covers requirements, and then the extended version, which uses the
21​TAP format (and additional data in the YAML block) to its full potential Taskotron-vise.
22​
23​
24​Minimal version
25​---------------
26​
27​If you don't need anything fancy, then this is the minimalistic version of the TAP output
28​which, at the same time, is valid for results reporting in Taskotron::
29​
30​ TAP version 13
31​ 1..1
32​ ok
33​ ---
kparalUnsubmitted
Not Done

Since the second example is also not ok, we could have ok for the first one.

34​ item: tzdata-2014f-1.fc20
35​ type: bodhi_update
36​ ...
37​
38​Apart of the standard TAP format, you just need to provide the ``item`` and ``type`` values in
39​the YAML block.
40​
41​The TAP state ``ok`` is interpreted as ``PASSED``, the ``not ok`` as ``FAILED``.
42​
kparalUnsubmitted
Not Done

If you want this to look like code, use double backticks. Single backtick is interpreted text:
http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#interpreted-text

43​``type`` describes "what" was on a high level - whether it was a build from Koji, DVD image,
44​update in Bodhi, or something completely different. At the moment, we have internally
45​standardized three ``type`` values for the most common items we test:
46​
47​ * ``koji_build`` for testing new builds in Koji
48​ * ``bodhi_update`` to represent results tied to a specific Bodhi update
49​ * ``yum_repository`` for repo-sanity checks, and so on
50​
51​You can, of course, use any other value for the ``type`` parameter, but if you have a check
52​for repositories, Koji builds or Bodhi updates, please stick to the already defined ones,
53​so the tools next in the chain will recognize your results properly.
54​
55​``item`` is a string representation of the specific piece, that was tested - NVR for the ``koji_build``
56​type, update-name or update-id for the ``bodhi_update``, etc. Once again - you can set whatever
57​value you think is right, but sticking to the reasonable/obvious choices will ensure seamless
58​processing.
59​
60​.. Note::
61​
62​ ``item``, and ``type`` are just some of the "reserved" key-names in the YAML block. Refer to the
63​ next section for the full list.
64​
65​
66​Full version
67​------------
68​
69​Although the minimal version of the TAP output is sufficient, you can use additional fields
70​to provide more information::
71​
72​ TAP version 13
73​ 1..1
74​ not ok
75​ ---
76​ item: tzdata-2014f-1.fc20
77​ type: bodhi_update
78​ outcome: ABORTED
79​ summary: mytask ABORTED for tzdata-2014f-1.fc20
80​ details:
81​ output: |-
82​ tzdata-2014f-1.fc20.x86_64.rpm: ABORTED
83​ tzdata-java-2014f-1.fc20.x86_64.rpm: PASSED
84​ arch: x86_64
85​ package: tzdata
86​ ...
87​
88​Going top-down, the new fields are ``outcome``, ``summary``, and ``details/output``.
89​
90​The ``outcome`` field overrides the basic TAP state (``ok``/``not ok``). Values allowed at the moment, are
91​``PASSED``, ``INFO``, ``FAILED``, ``ERROR``, ``WAIVED``, and ``NEEDS_INSPECTION``. Using other values will cause
92​the ResultsDB reporting to fail.
93​
94​``summary`` field allows you to add a comprehensive brief text describing the check's outcome.
95​
96​The ``output`` key in the ``details`` is usefull for cherry-picking the important sections of your
97​check's output. Although it is not stored in the ResultsDB, it allows the people reading logs, to find
98​the important information in an easy fashion, without the need to go through the whole of it.
99​
100​Note on the reserved field names
101​--------------------------------
102​
103​``item``, ``type``, ``outcome``, ``summary``, ``details`` are the only "reserved" fields. We also decided to use
104​``arch`` as a field with "recommended meaning" (to represent the architecture). You can then add any
105​number of custom key-value pairs (like the ``package`` in the above example), which will get stored
106​in ResultsDB, and can be then used for searching.
View Options

docs/source/writingtasks.rst

Show First 20 Lines • Show All 203 Lines • ▼ Show 20 Line(s)200.. code-block:: python
204 def run_mytask(rpmfiles): 204​ def run_mytask(rpmfiles):
205 for rpmfile in rpmfiles['downloaded_rpms']: 205​ for rpmfile in rpmfiles['downloaded_rpms']:
206 print "Running mytask on %s" % rpmfile 206​ print "Running mytask on %s" % rpmfile
207 207​
208To create valid TAP, we want to populate a :py:class:`libtaskotron.check.CheckDetail` 208​To create valid TAP, we want to populate a :py:class:`libtaskotron.check.CheckDetail`
209object and use :py:meth:`libtaskotron.check.export_TAP` to generate valid TAP13 209​object and use :py:meth:`libtaskotron.check.export_TAP` to generate valid TAP13
210with the data required for reporting. 210​with the data required for reporting.
211 211​
212​.. note::
213​
214​ If you can not use the :py:class:`libtaskotron.check.CheckDetail`
215​ directly, then create the TAP output according to :ref:`tap13-format`.
216​
212.. code-block:: python 217​.. code-block:: python
213 218​
214 from libtaskotron import check 219​ from libtaskotron import check
215 220​
216 def run_mytask(rpmfiles, bodhi_id): 221​ def run_mytask(rpmfiles, bodhi_id):
217 """run through all passed in rpmfiles and emit TAP13 saying they all passed""" 222​ """run through all passed in rpmfiles and emit TAP13 saying they all passed"""
218 223​
219 print "Running mytask on %s" % bodhi_id 224​ print "Running mytask on %s" % bodhi_id
▲ Show 20 Lines • Show All 105 Lines • Show Last 20 Lines