Switched from TAP to YAML
ClosedPublic
Actions

Authored by jskladan on Sep 3 2015, 8:13 PM.

Details

Reviewers

lbrabec
kparal
garretraziel
tflink
mkrizek

Commits

rLTRN0172820d148d: Switched from TAP to YAML

Summary

This is a WIP of the TAP to YAML conversion for the purposes of results-reporting.
The code is in the state where all the unittests work, but I'm yet to try it on a
real-life libtaskotron run. The documentation is practically untouched so far.

But I guess this might be a good start for discussions.

Test Plan

Unittests work

Diff Detail

Repository

rLTRN libtaskotron

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

jskladan retitled this revision from to Switched from TAP to YAML.Sep 3 2015, 8:13 PM

jskladan updated this object.

jskladan edited the test plan for this revision. (Show Details)

jskladan added reviewers: kparal, mkrizek, tflink, lbrabec, garretraziel.

I think it would be nice to include a ResultYAML sample to showcase what is going to be supported and what is not (we had a lot of ideas when brainstorming this).

I went through the main code changes (skipping docstrings and tests), even though not fully closely, and the changes look fine to me.

libtaskotron/check.py
270–271	Are we going to support multiple documents (document roots) in a single ResultYAML? If not, is there a purpose for using the triple-dash and triple-dot syntax?

There will be ResultYAML samples in the docs, which I'm working on now. Do you think it's worth going into the "not yet implemented" details anywhere else than there?

libtaskotron/check.py
270–271	meh, probably left it there accidentally...

Documentation

typo

fixed lint warns
moar tests

coverage

A few nitpicks. Looks good overall.

docs/source/writingtasks.rst
244	We concluded we don't want `note` to be used like this, it should present some additional information that should be highly visible. Either improve or remove the line?
291–292	Again sub-par note usage.
302	Again sub-par note usage.
docs/source/yamlformat.rst
3–5 ↗	(On Diff #1461)	Maybe a title like "Task result format" might be more concrete here? We use YAML in many places, in configuration for example, and this chapter seems to talk just about the result format. It might be nice to fit YAML word into the title somehow, but I don't have any good ideas now.
9–11 ↗	(On Diff #1461)	In case you didn't, please render the docs and make sure all the hyperlinks like these work.
36 ↗	(On Diff #1461)	I would use quotes instead of backticks around "ok" and "not ok", so that it doesn't look like values. When I first read it, it seemed like `ok` and `not ok` were allowed values for `outcome`. Or make it into a longer sentence: The ``outcome`` field defines whether the check result was OK or not OK.
40 ↗	(On Diff #1461)	A verb seems to be missing. Maybe "what" was tested on a high level?
42–46 ↗	(On Diff #1461)	We also have `compose`.
61–62 ↗	(On Diff #1461)	It would be nice to see an example with multiple (2-3) results in a single YAML document, to show that you can easily report results of multiple items. Either here in "minimal version" section, or in/after the "full version" section. Simply like this (with a short description): results: - item: xchat-0.5-1.fc20 type: koji_build outcome: PASSED - item: pidgin-1.4-2.fc20 type: koji_build outcome: FAILED
85 ↗	(On Diff #1461)	typo: useful
90 ↗	(On Diff #1461)	Maybe "allows you to add a short valuable information that should be easily visible when looking at the result overview"?
libtaskotron/check.py
74–77	I would get rid of this, it serve no more purpose now that we have artifact support. Might be part of a future diff.
libtaskotron/directives/resultsdb_directive.py
206–207	This is no longer true, we used to strip values in TAP, but your patch doesn't do it. If we intend to keep `CheckDetail.output`, I'd probably keep the message and introduce output stripping again. If we plan to get rid of it, let's delete this message and then remove `CheckDetail.output` soon.
254–255	This method has no further purpose (it calls a single method and does nothing else), let's either implement it according to the docstring, or delete it completely.

This revision now requires changes to proceed.Sep 8 2015, 2:30 PM

docs
removed report_summary from resultsdb directive

LGTM

docs/source/yamlformat.rst
47 ↗	(On Diff #1475)	The word seems to be so bare and lonely on that line... ;-)

This revision is now accepted and ready to land.Sep 9 2015, 11:03 AM

Switched from TAP to YAML
Docs wrangling

un-lonely the compose

ACK

I like it - looks good overall. A couple of thoughts outside the inline comments:

The example output is not very easily readable (a side effect of moving from TAP to YAML, not a criticism) and I wonder if we might want to write a pretty-printer for the resultyaml output instead of just relying on the bare YAML
Once this lands, we'll need to check on the existing tasks to make sure that they're compatible - I think that depcheck will need tweaking, at least

Is there a situation where we might reasonably want nested results? I think that'd be a "deal with that later, if we need it" kind of thing but figured that I'd ask the question.

docs/source/index.rst
105	Sometimes the docs/comments refer to a resultyaml format and other times, like here, it's just called YAML. I think we'd be better off being consistent about naming and sticking with the resultyaml moniker to reduce the potential for confusion with all the other places we're using YAML
docs/source/yamlformat.rst
4–6 ↗	(On Diff #1478)	Following on this - I'd like to see consistent naming of the resultyaml (or whatever we end up calling it) from titles in documentation to filenames and comments.

In D555#10909, @tflink wrote:

The example output is not very easily readable (a side effect of moving from TAP to YAML, not a criticism) and I wonder if we might want to write a pretty-printer for the resultyaml output instead of just relying on the bare YAML

I'm a bit confused, do you consider this less readable than TAP?

results:
   - item: xchat-0.5-1.fc20
     type: koji_build
     outcome: PASSED

Or are you talking about some other example output?

In D555#10918, @kparal wrote:
In D555#10909, @tflink wrote:

The example output is not very easily readable (a side effect of moving from TAP to YAML, not a criticism) and I wonder if we might want to write a pretty-printer for the resultyaml output instead of just relying on the bare YAML

I'm a bit confused, do you consider this less readable than TAP?
results:
   - item: xchat-0.5-1.fc20
     type: koji_build
     outcome: PASSED
Or are you talking about some other example output?

Yeah, that's what I meant. I don't think it's a huge difference but I do think it shows up more with multiple results

1..41
ok - depcheck for Koji build COPASI-4.16-0.13.20150817git3bc4e9.fc22
  ---
  arch: i386
  artifact: /var/lib/taskotron/artifacts/d3ebc790-5ae7-11e5-9c54-525400062113/COPASI-4.16-0.13.20150817git3bc4e9.fc22.i386.log
  item: COPASI-4.16-0.13.20150817git3bc4e9.fc22
  outcome: PASSED
  type: koji_build
  ...
ok - depcheck for Koji build nrpe-2.15-6.fc22
  ---
  arch: i386
  artifact: /var/lib/taskotron/artifacts/d3ebc790-5ae7-11e5-9c54-525400062113/nrpe-2.15-6.fc22.i386.log
  item: nrpe-2.15-6.fc22
  outcome: PASSED
  type: koji_build
  ...

I think that TAP13 is a lot easier to scan and see where the failure is. It's not as big of a deal now that we're doing artifacts but I still think it'd be worth printing out a summary or a pretty-printed form for the times when people are reading through the raw output

In D555#10930, @tflink wrote:

I think that TAP13 is a lot easier to scan and see where the failure is. It's not as big of a deal now that we're doing artifacts but I still think it'd be worth printing out a summary or a pretty-printed form for the times when people are reading through the raw output

While I do agree, I have not found a reliable (and easy) way to add comments "to the right places", so the output could look like:


results:
   #ok
   - item: xchat-0.5-1.fc20
     type: koji_build
     outcome: PASSED
   #not ok
   - item: bugemos-0.2-1.fc20
     type: koji_build
     outcome: ABORTED

I'm not at all against us adding the comments, so one can easily scan through the results, but I would not even mention it in the documentation (of the format), as it's just eye-candy in the end (and requiring the eye-candy comments would effectively make it "not yaml", and not that easy to produce while not using libtaskotron tooling).

The good thing about yaml (without the required comments) is that this:

{results: [{item: xchat, type: koji_build, outcome: PASSED}, {item: bugemos, type: koji_build, outcome: ABORTED}]}

is also effectively a valid format of the resultyaml, and can be easier for some check-developers to produce, while exactly the same for us to consume.

It seems to me that @tflink didn't intend to have the comments part of the specification, just wanted to have it pretty printed when we dump debug output into stdout (in resultsdb directive and during variable export process).

It seems to be impossible to dump comments with PyYAML (we would have to use other libraries). So as long as we don't want this debug output to be valid yaml, we can dump it in any structure we want, and make it well readable. It won't be yaml, but it can be really nice and succinct, like this:

[DEBUG] The following would get reported:
OK:
xchat-0.5-1.fc20:    depcheck PASSED
firefox-44.1-2.fc20: depcheck PASSED
NOT OK:
bugemos-0.2-1.fc20:  depcheck ABORTED

Or introduce another third-party library for yaml-with-comments dumping (which I must say I'm not really enthusiastic about, just for some debug prints).

Personally I wouldn't mind dumping bare yaml structure directly in the logs, because it's easy and they are just debug logs. We have other places to present nicely structured and readable results (we can even generate some nice html overview by default and attach it as an artifact). And it has the benefit of developers seeing the exact structure that we pass to reporting directives. But I wouldn't mind some simplified non-yaml structure as the one shown above either.

In D555#10987, @jskladan wrote:
In D555#10930, @tflink wrote:

I think that TAP13 is a lot easier to scan and see where the failure is. It's not as big of a deal now that we're doing artifacts but I still think it'd be worth printing out a summary or a pretty-printed form for the times when people are reading through the raw output

While I do agree, I have not found a reliable (and easy) way to add comments "to the right places", so the output could look like:
results:
   #ok
   - item: xchat-0.5-1.fc20
     type: koji_build
     outcome: PASSED
   #not ok
   - item: bugemos-0.2-1.fc20
     type: koji_build
     outcome: ABORTED
I'm not at all against us adding the comments, so one can easily scan through the results, but I would not even mention it in the documentation (of the format), as it's just eye-candy in the end (and requiring the eye-candy comments would effectively make it "not yaml", and not that easy to produce while not using libtaskotron tooling).

I saw that example output and was thinking "yes, that's pretty much exactly what I was looking for" until I read the text :)

While I think it's a bit more than just eye candy, I also think that there is a limit to how much that extra understandability is worth. In my mind, one of the big things that TAP has going for it is high human understandability and if that was the only factor, we probably wouldn't be moving away from that.

The good thing about yaml (without the required comments) is that this:
{results: [{item: xchat, type: koji_build, outcome: PASSED}, {item: bugemos, type: koji_build, outcome: ABORTED}]}
is also effectively a valid format of the resultyaml, and can be easier for some check-developers to produce, while exactly the same for us to consume.

Yeah, I think that one of the other wins for us switching to a results format that is YAML is that we don't have to worry about libraries that may or may not output the correct version of TAP and may not support the yamlish extension.

In D555#10988, @kparal wrote:

It seems to me that @tflink didn't intend to have the comments part of the specification, just wanted to have it pretty printed when we dump debug output into stdout (in resultsdb directive and during variable export process).

Honestly, I'd be all for adding comments if I thought we could do that sanely. However, I don't think that's going to happen unless we start writing our own parser/generator code and that sounds like a bad idea.

It seems to be impossible to dump comments with PyYAML (we would have to use other libraries). So as long as we don't want this debug output to be valid yaml, we can dump it in any structure we want, and make it well readable. It won't be yaml, but it can be really nice and succinct, like this:
[DEBUG] The following would get reported:
OK:
xchat-0.5-1.fc20:    depcheck PASSED
firefox-44.1-2.fc20: depcheck PASSED
NOT OK:
bugemos-0.2-1.fc20:  depcheck ABORTED

Yeah, that would satisfy what I was looking for - a way to quickly scan the output and see what did or didn't pass. I think that indenting the items so that they're not inline with OK/NOT OK helps make it more readable but the concept seems good.

Or introduce another third-party library for yaml-with-comments dumping (which I must say I'm not really enthusiastic about, just for some debug prints).

Yeah, that doesn't sound worth it to me. I thought about trying to print out something like what @jskladan listed (even if it wasn't part of the format) but I suspect that wouldn't be trivial and we might risk confusion over what the actual format is.

Personally I wouldn't mind dumping bare yaml structure directly in the logs, because it's easy and they are just debug logs. We have other places to present nicely structured and readable results (we can even generate some nice html overview by default and attach it as an artifact). And it has the benefit of developers seeing the exact structure that we pass to reporting directives. But I wouldn't mind some simplified non-yaml structure as the one shown above either.

I don't think it's critical to have a pretty-printed representation in the stdout but I do think that it would make it easier to read command line output for folks running tasks locally on the command line. It'd also make it easier to scan log output in one of the public instances - I usually end up scanning logs for the TAP13 when looking through logs.

In D555#11037, @tflink wrote:

While I think it's a bit more than just eye candy, I also think that there is a limit to how much that extra understandability is worth. In my mind, one of the big things that TAP has going for it is high human understandability and if that was the only factor, we probably wouldn't be moving away from that.

Yeah, that makes sense. I'm all up for an pretty-printer, would you be ok with it having a separate Diff?

In D555#10909, @tflink wrote:

Is there a situation where we might reasonably want nested results? I think that'd be a "deal with that later, if we need it" kind of thing but figured that I'd ask the question.

I was thinking about tests like rpmgril, which might want to nest the results per-tier. So there is a "deal with it later" way in the format, which allows you to add results: to the result, thus creating a nested structure (which will then get represented in the ResultsDB, but... later :))

Sometimes the docs/comments refer to a resultyaml format and other times, like here, it's just called YAML. I think we'd be better off being consistent about naming and sticking with the resultyaml moniker to reduce the potential for confusion with all the other places we're using YAML
Following on this - I'd like to see consistent naming of the resultyaml (or whatever we end up calling it) from titles in documentation to filenames and comments.

Good catches - I probably introduced the inconsistencies while discussing random stuff with @kparal. If there are no objections, I'll switch to "resultyaml" for the sake of consistency and reducing the confusion.

In D555#11039, @jskladan wrote:

In D555#11037, @tflink wrote:

While I think it's a bit more than just eye candy, I also think that there is a limit to how much that extra understandability is worth. In my mind, one of the big things that TAP has going for it is high human understandability and if that was the only factor, we probably wouldn't be moving away from that.

Yeah, that makes sense. I'm all up for an pretty-printer, would you be ok with it having a separate Diff?

Yeah, another ticket/diff is fine by me.

In D555#10909, @tflink wrote:

Is there a situation where we might reasonably want nested results? I think that'd be a "deal with that later, if we need it" kind of thing but figured that I'd ask the question.

I was thinking about tests like rpmgril, which might want to nest the results per-tier. So there is a "deal with it later" way in the format, which allows you to add results: to the result, thus creating a nested structure (which will then get represented in the ResultsDB, but... later :))

Just to make sure I understand - there is something like a plan for resultsdb to handle nested results but that's to be added later when we need it?

Sometimes the docs/comments refer to a resultyaml format and other times, like here, it's just called YAML. I think we'd be better off being consistent about naming and sticking with the resultyaml moniker to reduce the potential for confusion with all the other places we're using YAML
Following on this - I'd like to see consistent naming of the resultyaml (or whatever we end up calling it) from titles in documentation to filenames and comments.

Good catches - I probably introduced the inconsistencies while discussing random stuff with @kparal. If there are no objections, I'll switch to "resultyaml" for the sake of consistency and reducing the confusion.

No objections here.

In D555#11040, @tflink wrote:

Just to make sure I understand - there is something like a plan for resultsdb to handle nested results but that's to be added later when we need it?

Yes

jskladan mentioned this in D590: Move from TAP to YAML in Depcheck.Sep 16 2015, 12:27 PM

Closed by commit rLTRN0172820d148d: Switched from TAP to YAML (authored by Josef Skladanka <jskladan@redhat.com>). · Explain WhySep 16 2015, 1:08 PM

This revision was automatically updated to reflect the committed changes.

kparal mentioned this in T117: Merge bayeux & pytap13.Sep 16 2015, 2:41 PM

kparal mentioned this in T123: Unify Bayeux's and PyTAP13's APIs.

kparal mentioned this in T794: change Summary to Note, change rendering a bit.May 13 2016, 11:10 AM

Revision Contents

		Path
M		docs/source/index.rst (4 lines)
A	M	docs/source/resultyaml.rst (120 lines)
D	M	docs/source/tapformat.rst (106 lines)
M		docs/source/writingtasks.rst (72 lines)
M		libtaskotron.spec (10 lines)
M		libtaskotron/check.py (158 lines)
M		libtaskotron/directives/bodhi_comment_directive.py (13 lines)
M		libtaskotron/directives/exitcode_directive.py (72 lines)
M		libtaskotron/directives/python_directive.py (7 lines)
M		libtaskotron/directives/resultsdb_directive.py (92 lines)
M		readme.rst (2 lines)
M		requirements.txt (3 lines)
M		testing/test_bodhi_comment_directive.py (33 lines)
M		testing/test_check.py (288 lines)
M		testing/test_exitcode_directive.py (65 lines)
M		testing/test_resultsdb_directive.py (65 lines)
M		testing/test_runner.py (35 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	1446	a8bec90		Sep 3 2015, 8:12 PM	★	★
Diff 2	1458	a8bec90	- Documentation	Sep 8 2015, 10:18 AM	★	★
Diff 3	1459	a8bec90	- typo	Sep 8 2015, 10:29 AM	★	★
Diff 4	1460	a8bec90	- fixed lint warns	Sep 8 2015, 10:43 AM	★	★
Diff 5	1461	a8bec90	- coverage	Sep 8 2015, 10:57 AM	★	★
Diff 6	1475	a8bec90	- docs	Sep 9 2015, 8:03 AM	★	★
Diff 7	1477	a8bec90	- Switched from TAP to YAML	Sep 9 2015, 11:24 AM	★	★
Diff 8	1478	a8bec90	- un-lonely the compose	Sep 9 2015, 11:28 AM	★	★
Diff 9	1525	a8bec90	- .	Sep 16 2015, 10:22 AM	★	★
Diff 10	1534	1939727	Commit rLTRN0172820d148dc4e01f1d8805592e7477e367aeca	Sep 16 2015, 1:08 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
ec882461ccce	f371573132dc	444c9baebb44	Josef Skladanka	Refer to resultYaml as result-YAML instead of plain YAML (Show More…)	Sep 16 2015, 10:21 AM
444c9baebb44	4add01968e6b	1af58006cd05	Josef Skladanka	un-lonely the compose	Sep 9 2015, 11:27 AM
1af58006cd05	dcfc7c6e74f3	fc60b6deedce	Josef Skladanka	moar linting of comments...	Sep 9 2015, 11:22 AM
fc60b6deedce	46dcca43f64f	cbccc6c5d2c6	Josef Skladanka	Docs wrangling	Sep 9 2015, 11:21 AM
cbccc6c5d2c6	f0af458533d3	a8bec903d1bd	Josef Skladanka	Switched from TAP to YAML (Show More…)	Sep 3 2015, 2:04 PM

Diff 1534

View Options

docs/source/index.rst

Show All 23 Lines		23	.. toctree::
24	:maxdepth: 1	24		:maxdepth: 1
25		25
26	quickstart	26		quickstart
27	writingtasks	27		writingtasks
28	taskyaml	28		taskyaml
29	library	29		library
30	devguide	30		devguide
31	directives/list_of_directives_modules	31		directives/list_of_directives_modules
32	tapformat	32		resultyaml
33		33
34		34
35		35
36	.. _what-can-i-do-with-taskotron:	36		.. _what-can-i-do-with-taskotron:
37		37
38	What Can I Do With Taskotron?	38		What Can I Do With Taskotron?
39	=============================	39		=============================
40		40
▲ Show 20 Lines • Show All 56 Lines • ▼ Show 20 Line(s)		94	task-rpmlint/
97	rpmlint.yml	97		rpmlint.yml
98		98
99	Rpmlint runs off of koji builds, so to run it against the x86_64 build of	99		Rpmlint runs off of koji builds, so to run it against the x86_64 build of
100	``foo-1.2-3.fc99`` (substitute a real build if you're going to run this)::	100		``foo-1.2-3.fc99`` (substitute a real build if you're going to run this)::
101		101
102	runtask -i foo-1.2-3.fc99 -t koji_build -a x86_64 rpmlint.yml	102		runtask -i foo-1.2-3.fc99 -t koji_build -a x86_64 rpmlint.yml
103		103
104	The build will be downloaded to a local tempdir, rpmlint will be run against	104		The build will be downloaded to a local tempdir, rpmlint will be run against
105	that build and the results will be printed to the console in TAP13 format.	105		that build and the results will be printed to the console in :ref:`resultyaml-format`.
			tflinkUnsubmitted Not Done Sometimes the docs/comments refer to a resultyaml format and other times, like here, it's just called YAML. I think we'd be better off being consistent about naming and sticking with the resultyaml moniker to reduce the potential for confusion with all the other places we're using YAML
106		106
107		107
108	.. _contact-us:	108		.. _contact-us:
109		109
110	Contact Us	110		Contact Us
111	==========	111		==========
112	Please direct questions and comments to either the `Fedora QA Devel List	112		Please direct questions and comments to either the `Fedora QA Devel List
113	<https://admin.fedoraproject.org/mailman/listinfo/qa-devel>`_ or the #fedora-qa	113		<https://admin.fedoraproject.org/mailman/listinfo/qa-devel>`_ or the #fedora-qa
Show All 13 Lines

View Options

docs/source/resultyaml.rst

This file was added.

		1		.. _resultyaml-format:
		2
		3		==================
		4		Task result format
		5		==================
		6
		7		The required output format of the checks in Taskotron is YAML.
		8
		9		We recommend using the provided :py:class:`libtaskotron.check.CheckDetail` and the
		10		helper method :py:meth:`libtaskotron.check.export_YAML` to create the YAML output.
		11		If you can not use it, then please adhere to the specification described below.
		12
		13
		14		YAML and reporting
		15		==================
		16
		17		First up, we will provide you with two examples of the valid YAML output - the minimal
		18		version, which just covers requirements, and then the extended version, which uses the
		19		format to its full potential Taskotron-vise.
		20
		21
		22		Minimal version
		23		---------------
		24
		25		If you don't need anything fancy, then this is the minimalistic version of the output
		26		which, at the same time, is valid for results reporting in Taskotron::
		27
		28		results:
		29		- item: xchat-0.5-1.fc20
		30		type: koji_build
		31		outcome: PASSED
		32
		33		You just need to provide the ``item``, ``type`` and ``outcome`` values in
		34		the YAML block.
		35
		36		The ``outcome`` field defines whether the check result was OK or not OK.
		37		Values allowed at the moment, are ``PASSED``, ``INFO``, ``FAILED``, ``ERROR``, ``WAIVED``,
		38		and ``NEEDS_INSPECTION``. Using other values will cause the ResultsDB reporting to fail.
		39
		40		``type`` describes "what" was tested on a high level - whether it was a build from Koji, DVD image,
		41		update in Bodhi, or something completely different. At the moment, we have internally
		42		standardized three ``type`` values for the most common items we test:
		43
		44		* ``koji_build`` for testing new builds in Koji
		45		* ``bodhi_update`` to represent results tied to a specific Bodhi update
		46		* ``yum_repository`` for repo-sanity checks, and so on
		47		* ``compose`` for compose-related testing
		48
		49		You can, of course, use any other value for the ``type`` parameter, but if you have a check
		50		for repositories, Koji builds or Bodhi updates, please stick to the already defined ones,
		51		so the tools next in the chain will recognize your results properly.
		52
		53		``item`` is a string representation of the specific piece, that was tested - NVR for the ``koji_build``
		54		type, update-name or update-id for the ``bodhi_update``, etc. Once again - you can set whatever
		55		value you think is right, but sticking to the reasonable/obvious choices will ensure seamless
		56		processing.
		57
		58		.. Note::
		59
		60		``item``, ``type`` and ``outcome`` are just some of the "reserved" key-names in the YAML block. Refer to the
		61		next section for the full list.
		62
		63
		64		Full version
		65		------------
		66
		67		Although the minimal version of the output is sufficient, you can use additional fields
		68		to provide more information::
		69
		70		results:
		71		- item: tzdata-2014f-1.fc20
		72		type: bodhi_update
		73		outcome: ABORTED
		74		artifact: upgradepath_details.html
		75		checkname: upgradepath
		76		note: from testing to stable
		77		arch: x86_64
		78		package: tzdata
		79		- item: bluez-5.34-1.fc23
		80		type: bodhi_update
		81		outcome: PASSED
		82		artifact: upgradepath_details.html
		83		checkname: upgradepath
		84		note: from testing to stable
		85		arch: x86_64
		86		package: bluez
		87		- item: tzdata-2014f-1.fc20
		88		type: bodhi_update
		89		outcome: PASSED
		90		artifact: depcheck_tzdata-2014f-1.fc20.txt
		91		checkname: depcheck
		92		note: for stable
		93		arch: x86_64
		94
		95		Going top-down, the new fields are ``artifact``, ``checkname``, ``note``, ``arch`` and ``package``.
		96
		97		``artifact`` contains the name of a file in the ``$artifactsdir``, that contains detailed
		98		information about the check run. In this example the upgradepath check produced an user-friendly
		99		report in HTML format, and the author decided to link it as an artifact.
		100
		101		``checkname`` allows overriding the value devised from the Taskotron recipe. It is useful
		102		when your task implements multiple checks, and you want to report the per-check results
		103		separately (e.g. ``repo-sanity`` could output ``depcheck``, ``upgradepath`` and ``repoclosure``
		104		results in one run).
		105
		106		``note`` adds a short valuable information that should be easily visible in the result overview.
		107
		108		``arch`` and ``package`` are custom fields that do not have any special meaning for the Taskotron
		109		tooling, but provide additional information about the check results as such.
		110
		111
		112		Note on the reserved field names
		113		--------------------------------
		114
		115		``item``, ``type``, ``outcome``, ``note`` are the commonly used "reserved" fields. We also decided to use
		116		``arch`` as a field with "recommended meaning" (to represent the architecture). You can then add any
		117		number of custom key-value pairs (like the ``package`` in the above example), which will get stored
		118		in ResultsDB, and can be then used for searching.
		119
		120		There are two more reserved fields - ``_internal`` and ``results``._

View Options

docs/source/tapformat.rst

This file was deleted.

This file was completely deleted. Show File Contents

View Options

docs/source/writingtasks.rst

Show All 39 Lines
40	purposes.	40		purposes.
41		41
42	rpmlint	42		rpmlint
43	-------	43		-------
44		44
45	`rpmlint task git repository <https://bitbucket.org/fedoraqa/task-rpmlint>`_	45		`rpmlint task git repository <https://bitbucket.org/fedoraqa/task-rpmlint>`_
46		46
47	rpmlint is the simplest of the production tasks. It takes a koji build and runs	47		rpmlint is the simplest of the production tasks. It takes a koji build and runs
48	rpmlint on it, reporting results as TAP13.	48		rpmlint on it, reporting results as result-YAML.
49		49
50		50
51	Creating a New Task	51		Creating a New Task
52	===================	52		===================
53		53
54	While running rpmlint is an easy and trivial example, that doesn't help with	54		While running rpmlint is an easy and trivial example, that doesn't help with
55	creating something new. To write a new task, you'll need some information:	55		creating something new. To write a new task, you'll need some information:
56		56
▲ Show 20 Lines • Show All 87 Lines • ▼ Show 20 Line(s)		143	- name: run mytask
144	python:	144		python:
145	file: mytask.py	145		file: mytask.py
146	callable: run_mytask	146		callable: run_mytask
147	rpmfiles: ${bodhi_downloads}	147		rpmfiles: ${bodhi_downloads}
148	bodhi_id: ${bodhi_id}	148		bodhi_id: ${bodhi_id}
149	export: mytask_output	149		export: mytask_output
150		150
151		151
152	# this assumes that the output from mytask is valid TAP	152		# this assumes that the output from mytask is valid result-YAML
153	- name: report mytask results to resultsdb	153		- name: report mytask results to resultsdb
154	resultsdb:	154		resultsdb:
155	results: ${mytask_output}	155		results: ${mytask_output}
156		156
157	.. this needs to be fixed, this output is doctored since it doesn't actually	157		.. this needs to be fixed, this output is doctored since it doesn't actually
158	work.	158		work.
159		159
160	We can execute this new task using::	160		We can execute this new task using::
Show All 13 Lines		169	[libtaskotron:logger.py:34] 2014-05-14 21:30:30 CRITICAL Traceback (most recent call last):
174	File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 31, in run	174		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 31, in run
175	self.do_actions()	175		self.do_actions()
176	File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 105, in do_actions	176		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 105, in do_actions
177	self.do_single_action(action)	177		self.do_single_action(action)
178	File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 94, in do_single_action	178		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/runner.py", line 94, in do_single_action
179	self.envdata)	179		self.envdata)
180	File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/directives/resultsdb_directive.py", line 136, in process	180		File "/home/tflink/code/taskotron/libtaskotron/libtaskotron/directives/resultsdb_directive.py", line 136, in process
181	raise TaskotronDirectiveError("Failed to load 'results': %s" % e.message)	181		raise TaskotronDirectiveError("Failed to load 'results': %s" % e.message)
182	TaskotronDirectiveError: Failed to load 'results': Failed to parse TAP contents: Missing plan in the TAP source	182		TaskotronDirectiveError: Failed to load 'results': Failed to parse YAML contents: Missing plan in the YAML source
183		183
184	We're not actually passing TAP output from our task, so let's get that fixed.	184		We're not actually passing result-YAML output from our task, so let's get that fixed.
185		185
186	Actually Doing Something	186		Actually Doing Something
187	------------------------	187		------------------------
188		188
189	The Python code for ``mytask.py`` above doesn't really do anything at all, so let's	189		The Python code for ``mytask.py`` above doesn't really do anything at all, so let's
190	make it do something:	190		make it do something:
191		191
192	* Change the task to create a result from the input bodhi update id and list	192		* Change the task to create a result from the input bodhi update id and list
193	all of the rpms contained in that update as details.	193		all of the rpms contained in that update as details.
194		194
195	There is a bit of complexity to the `TAP13 specification <http://podwiki.hexten.net/TAP/TAP13.html?page=TAP13>`_	195		Generating result-YAML from python tasks in Taskotron isn't very difficult.
196	but generating TAP from python tasks in Taskotron isn't very difficult.
197		196
198	Our original task looked like:	197		Our original task looked like:
199		198
200	.. code-block:: python	199		.. code-block:: python
201		200
202	# this is a trivial example that doesn't do everything we want it to do, more	201		# this is a trivial example that doesn't do everything we want it to do, more
203	# details will be added later in the example	202		# details will be added later in the example
204	def run_mytask(rpmfiles):	203		def run_mytask(rpmfiles):
205	for rpmfile in rpmfiles['downloaded_rpms']:	204		for rpmfile in rpmfiles['downloaded_rpms']:
206	print "Running mytask on %s" % rpmfile	205		print "Running mytask on %s" % rpmfile
207		206
208	To create valid TAP, we want to populate a :py:class:`libtaskotron.check.CheckDetail`	207		To create valid result-YAML, we want to populate a :py:class:`libtaskotron.check.CheckDetail`
209	object and use :py:meth:`libtaskotron.check.export_TAP` to generate valid TAP13	208		object and use :py:meth:`libtaskotron.check.export_YAML` to generate valid result-YAML
210	with the data required for reporting.	209		with the data required for reporting.
211		210
212	.. note::	211		.. note::
213		212
214	If you can not use the :py:class:`libtaskotron.check.CheckDetail`	213		If you can not use the :py:class:`libtaskotron.check.CheckDetail`
215	directly, then create the TAP output according to :ref:`tap13-format`.	214		directly, then create the YAML output according to :ref:`resultyaml-format`.
216		215
217	.. code-block:: python	216		.. code-block:: python
218		217
219	from libtaskotron import check	218		from libtaskotron import check
220		219
221	def run_mytask(rpmfiles, bodhi_id):	220		def run_mytask(rpmfiles, bodhi_id):
222	"""run through all passed in rpmfiles and emit TAP13 saying they all passed"""	221		"""run through all passed in rpmfiles and emit result-YAML saying they all passed"""
223		222
224	print "Running mytask on %s" % bodhi_id	223		print "Running mytask on %s" % bodhi_id
225		224
226	details = []	225		details = []
227	result = 'PASSED'	226		result = 'PASSED'
228	summary = 'mycheck %s for %s' % (result, bodhi_id)	227		note = 'from testing-pending to testing'
229	detail = check.CheckDetail(bodhi_id, check.ReportType.BODHI_UPDATE,	228		detail = check.CheckDetail(bodhi_id, check.ReportType.BODHI_UPDATE,
230	result, summary)	229		result, note)
231	for rpmfile in rpmfiles['downloaded_rpms']:	230		for rpmfile in rpmfiles['downloaded_rpms']:
232	detail.store(rpmfile, printout=False)	231		detail.store(rpmfile, printout=False)
233		232
234	return check.export_TAP(detail)	233		return check.export_YAML(detail)
235		234
236	Now if we run the task we get output that ends with::	235		Now if we run the task we get output that ends with::
237		236
238	[libtaskotron:resultsdb_directive.py:123] 2014-05-15 09:17:44 INFO Reporting to ResultsDB is disabled.	237		[libtaskotron:resultsdb_directive.py:123] 2014-05-15 09:17:44 INFO Reporting to ResultsDB is disabled.
239	INFO:libtaskotron:Reporting to ResultsDB is disabled.	238		INFO:libtaskotron:Reporting to ResultsDB is disabled.
240	[libtaskotron:runner.py:198] 2014-05-15 09:17:44 INFO	239		[libtaskotron:runner.py:198] 2014-05-15 09:17:44 INFO
241	TAP version 13	240		results:
242	1..1	241		- item: foo-1.2-3.fc99
243	ok - $CHECKNAME for Bodhi update openvswitch-2.1.2-1.fc19
244	---
245	details:
246	output: "foo-1.2-3.fc99.x86_64.rpm"
247	item: foo-1.2-3.fc99
248	outcome: PASSED
249	summary: mycheck PASSED for foo-1.2-3.fc99
250	type: bodhi_update	242		type: bodhi_update
251	...	243		outcome: PASSED
		244		note: from testing-pending to testing
			kparalUnsubmitted Not Done We concluded we don't want `note` to be used like this, it should present some additional information that should be highly visible. Either improve or remove the line?
252		245
253	Working with CheckDetail Objects	246		Working with CheckDetail Objects
254	--------------------------------	247		--------------------------------
255		248
256	Check can have one of these outcomes (with increasing priority): ``PASSED``,	249		Check can have one of these outcomes (with increasing priority): ``PASSED``,
257	``INFO``, ``FAILED``, ``NEEDS_INSPECTION``, ``ABORTED`` and ``CRASHED``. These	250		``INFO``, ``FAILED``, ``NEEDS_INSPECTION``, ``ABORTED`` and ``CRASHED``. These
258	are available as a list (where higher index means bigger priority) in	251		are available as a list (where higher index means bigger priority) in
259	:py:attr:`CheckDetail.outcome_priority <libtaskotron.check.CheckDetail.outcome_priority>`.	252		:py:attr:`CheckDetail.outcome_priority <libtaskotron.check.CheckDetail.outcome_priority>`.
Show All 13 Lines		263	.. code-block:: python
273	import random	266		import random
274		267
275	from libtaskotron import check	268		from libtaskotron import check
276		269
277	def random_choice(_):	270		def random_choice(_):
278	return random.choice(['PASSED', 'FAILED', 'ABORTED'])	271		return random.choice(['PASSED', 'FAILED', 'ABORTED'])
279		272
280	def run_mytask(rpmfiles, bodhi_id):	273		def run_mytask(rpmfiles, bodhi_id):
281	"""run through all passed in rpmfiles and emit TAP13. randomly report	274		"""run through all passed in rpmfiles and emit result-YAML. randomly report
282	failure or pass"""	275		failure or pass"""
283		276
284	print "Running mytask on %s" % bodhi_id	277		print "Running mytask on %s" % bodhi_id
285		278
		279		seed = random.random()
		280		random.seed(seed)
		281
286	results = [(rpmfile, random_choice(rpmfile)) for rpmfile in rpmfiles['downloaded_rpms']]	282		results = [(rpmfile, random_choice(rpmfile)) for rpmfile in rpmfiles['downloaded_rpms']]
287		283
288	detail = check.CheckDetail(bodhi_id, check.ReportType.BODHI_UPDATE)	284		detail = check.CheckDetail(bodhi_id, check.ReportType.BODHI_UPDATE)
289		285
290	for rpmfile, outcome in results:	286		for rpmfile, outcome in results:
291	# e.g. "xchat-1.2-3.fc20.x86_64.rpm: FAILED"	287		# e.g. "xchat-1.2-3.fc20.x86_64.rpm: FAILED"
292	detail.store('%s: %s' % (os.path.basename(rpmfile), outcome))	288		detail.store('%s: %s' % (os.path.basename(rpmfile), outcome))
293	detail.update_outcome(outcome)	289		detail.update_outcome(outcome)
294		290
295	detail.summary = "mytask %s for %s" % (detail.outcome, bodhi_id)	291		detail.note = "seed was %s" % seed
		292		detail.checkname = "mytask"
			kparalUnsubmitted Not Done Again sub-par note usage.
296		293
297	return check.export_TAP(detail, checkname="mytask")	294		return check.export_YAML(detail)
298		295
299	Running this code as ``python runtask.py -i tzdata-2014f-1.fc20 -t bodhi_id -a x86_64 ../task-mytask/mytask.yml``	296		Running this code as ``python runtask.py -i tzdata-2014f-1.fc20 -t bodhi_id -a x86_64 ../task-mytask/mytask.yml``
300	ends with::	297		ends with::
301		298
302	TAP version 13	299		results:
303	1..1	300		- item: tzdata-2014f-1.fc20
304	not ok - mytask for Bodhi update tzdata-2014f-1.fc20 # FAIL
305	---
306	details:
307	output: \|-
308	tzdata-2014f-1.fc20.noarch.rpm: ABORTED
309	tzdata-java-2014f-1.fc20.noarch.rpm: PASSED
310	item: tzdata-2014f-1.fc20
311	outcome: ABORTED	301		outcome: ABORTED
312	summary: mytask ABORTED for tzdata-2014f-1.fc20	302		note: seed was 0.45189
			kparalUnsubmitted Not Done Again sub-par note usage.
313	type: bodhi_update	303		type: bodhi_update
314	...	304		checkname: mytask
315		305
316	For more information about APIs and ``CheckDetail`` usage, see	306		For more information about APIs and ``CheckDetail`` usage, see
317	:py:class:`CheckDetail class <libtaskotron.check.CheckDetail>`.	307		:py:class:`CheckDetail class <libtaskotron.check.CheckDetail>`.
318		308
319	.. note::	309		.. note::
320	Reporting is disabled in the default configuration for Taskotron. While it is	310		Reporting is disabled in the default configuration for Taskotron. While it is
321	possible to set up a local resultsdb instance to check reporting, we want	311		possible to set up a local resultsdb instance to check reporting, we want
322	to make it easier than that and will update these docs when that easier method	312		to make it easier than that and will update these docs when that easier method
323	is available.	313		is available.
324		314
325	For now, if the resultsdb directive doesn't throw exceptions, it's reasonable	315		For now, if the resultsdb directive doesn't throw exceptions, it's reasonable
326	to assume that the TAP output was constructed correctly and is reasonably valid.	316		to assume that the result-YAML output was constructed correctly and is reasonably valid.
327		317
328		318
329		319

View Options

libtaskotron.spec

1	# sitelib for noarch packages, sitearch for others (remove the unneeded one)	1	# sitelib for noarch packages, sitearch for others (remove the unneeded one)
2	%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}	2	%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
3	%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}	3	%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
4		4
5	Name: libtaskotron	5	Name: libtaskotron
6	Version: 0.3.23	6	Version: 0.3.24
7	Release: 1%{?dist}	7	Release: 1%{?dist}
8	Summary: Taskotron Support Library	8	Summary: Taskotron Support Library
9		9
10	License: GPLv3	10	License: GPLv3
11	URL: https://bitbucket.org/fedoraqa/libtaskotron	11	URL: https://bitbucket.org/fedoraqa/libtaskotron
12	Source0: http://qadevel.fedoraproject.org/releases/%{name}/%{name}-%{version}.tar.gz	12	Source0: http://qadevel.fedoraproject.org/releases/%{name}/%{name}-%{version}.tar.gz
13		13
14	BuildArch: noarch	14	BuildArch: noarch
15		15
16	Requires: createrepo	16	Requires: createrepo
17	Requires: koji	17	Requires: koji
18	Requires: libtaskotron-config	18	Requires: libtaskotron-config
19	Requires: mash	19	Requires: mash
20	Requires: pyOpenSSL	20	Requires: pyOpenSSL
21	Requires: pytap13 >= 0.3.0
22	Requires: python-bayeux >= 0.9
23	Requires: python-bunch >= 1.0.1	21	Requires: python-bunch >= 1.0.1
24	Requires: python-fedora >= 0.5.5	22	Requires: python-fedora >= 0.5.5
25	Requires: python-hawkey >= 0.4.13-1	23	Requires: python-hawkey >= 0.4.13-1
26	Requires: python-pycurl	24	Requires: python-pycurl
27	Requires: python-setuptools	25	Requires: python-setuptools
28	Requires: python-urlgrabber	26	Requires: python-urlgrabber
29	Requires: python-yamlish >= 0.18
30	Requires: PyYAML >= 3.11	27	Requires: PyYAML >= 3.11
31	Requires: resultsdb_api >= 1.2.1	28	Requires: resultsdb_api >= 1.2.1
32	Requires: rpm-python	29	Requires: rpm-python
33	BuildRequires: koji	30	BuildRequires: koji
34	BuildRequires: mash	31	BuildRequires: mash
35	BuildRequires: pytap13 >= 0.3.0
36	BuildRequires: pytest >= 2.6.4	32	BuildRequires: pytest >= 2.6.4
37	BuildRequires: python-bayeux >= 0.9
38	BuildRequires: python-bunch >= 1.0.1	33	BuildRequires: python-bunch >= 1.0.1
39	BuildRequires: python-devel	34	BuildRequires: python-devel
40	BuildRequires: python-dingus >= 0.3.4	35	BuildRequires: python-dingus >= 0.3.4
41	BuildRequires: python-fedora >= 0.5.5	36	BuildRequires: python-fedora >= 0.5.5
42	BuildRequires: python-hawkey >= 0.4.13-1	37	BuildRequires: python-hawkey >= 0.4.13-1
43	BuildRequires: python-setuptools	38	BuildRequires: python-setuptools
44	BuildRequires: python-urlgrabber	39	BuildRequires: python-urlgrabber
45	BuildRequires: PyYAML >= 3.11	40	BuildRequires: PyYAML >= 3.11
▲ Show 20 Lines • Show All 56 Lines • ▼ Show 20 Line(s)
102	%files -n libtaskotron-config	97	%files -n libtaskotron-config
103	%dir %{_sysconfdir}/taskotron	98	%dir %{_sysconfdir}/taskotron
104	%config(noreplace) %{_sysconfdir}/taskotron/taskotron.yaml	99	%config(noreplace) %{_sysconfdir}/taskotron/taskotron.yaml
105	%config %{_sysconfdir}/taskotron/yumrepoinfo.conf	100	%config %{_sysconfdir}/taskotron/yumrepoinfo.conf
106		101
107		102
108		103
109	%changelog	104	%changelog
		105	* Wed Sep 02 2015 Josef Skladanka <jskaladan@fedoraproject.org> - 0.3.24-1
		106	- remove pytap, bayeux and yamlish dependencies
		107
110	* Fri Aug 28 2015 Tim Flink <tflink@fedoraproject.org> - 0.3.23-1	108	* Fri Aug 28 2015 Tim Flink <tflink@fedoraproject.org> - 0.3.23-1
111	- only submit bodhi comments on failure (D542)	109	- only submit bodhi comments on failure (D542)
112		110
113	* Wed Aug 26 2015 Tim Flink <tflink@fedoraproject.org> - 0.3.22-1	111	* Wed Aug 26 2015 Tim Flink <tflink@fedoraproject.org> - 0.3.22-1
114	- Re-enabling bodhi comment directive (D527)	112	- Re-enabling bodhi comment directive (D527)
115	- Include artifacts dir at end of run (D533)	113	- Include artifacts dir at end of run (D533)
116		114
117	* Wed Aug 19 2015 Tim Flink <tflink@fedoraproject.org> - 0.3.21-1	115	* Wed Aug 19 2015 Tim Flink <tflink@fedoraproject.org> - 0.3.21-1
▲ Show 20 Lines • Show All 120 Lines • Show Last 20 Lines

View Options

libtaskotron/check.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	'''Helper tools for managing check status, outcome, and output.'''	6		'''Helper tools for managing check status, outcome, and output.'''
7		7
8	from __future__ import absolute_import	8		from __future__ import absolute_import
9	import pprint	9		import pprint
10	import collections	10		import collections
11		11		import yaml
12	import tap
13		12
14	from . import python_utils	13		from . import python_utils
15	from . import exceptions as exc	14		from . import exceptions as exc
16	from .logger import log	15		from .logger import log
17		16
18	from pytap13 import TAP13	17		# a list of reserved keywords for ResultYAML output, which can't be overridden in
19
20	# a list of reserved keywords for TAP output, which can't be overridden in
21	# keyvals	18		# keyvals
22	RESERVED_KEYS = ('item', 'type', 'outcome',	19		RESERVED_KEYS = ('item', 'type', 'outcome',
23	'summary', 'details', 'checkname',	20		'note', 'results', 'checkname',
24	'artifact', '_internal')	21		'artifact', '_internal')
25		22
26	class CheckDetail(object):	23		class CheckDetail(object):
27	'''Class encompassing everything related to the outcome of a check run.	24		'''Class encompassing everything related to the outcome of a check run.
28	It contains convenience methods for tracking, evaluating and reporting	25		It contains convenience methods for tracking, evaluating and reporting
29	check results.	26		check results.
30		27
31	For some checks, it's trivial to parse its output at the end of its	28		For some checks, it's trivial to parse its output at the end of its
Show All 33 Lines		49	:ivar str outcome: a keyword specifying the final outcome of the check.
65	* CRASHED - the check crashed and did not provide usable	62		* CRASHED - the check crashed and did not provide usable
66	results	63		results
67		64
68	If no outcome has been set, this attribute returns	65		If no outcome has been set, this attribute returns
69	NEEDS_INSPECTION.	66		NEEDS_INSPECTION.
70		67
71	Raises :class:`TaskotronValueError` if you try to assign	68		Raises :class:`TaskotronValueError` if you try to assign
72	an unknown keyword.	69		an unknown keyword.
73	:ivar str summary: a few words or one-sentence summary about the result of	70		:ivar str note: a few words or one-sentence note about the result of
74	the check run (e.g. ``5 WARNINGS, 1 ERROR`` for an	71		the check run (e.g. ``5 WARNINGS, 1 ERROR`` for an
75	rpmlint result). Should not unnecessarily duplicate	72		rpmlint result). Should not unnecessarily duplicate
76	:attr:`item` or :attr:`outcome`, if possible.	73		:attr:`item` or :attr:`outcome`, if possible.
77	:ivar output: output from the check run (a list of strings). You can easily	74		:ivar output: output from the check run (a list of strings). You can easily
78	populate this by using @method store(), or you can modify it	75		populate this by using @method store(), or you can modify it
79	directly (for example filter out some messages you don't want	76		directly (for example filter out some messages you don't want
80	to see in the check output).	77		to see in the check output).
			kparalUnsubmitted Not Done I would get rid of this, it serve no more purpose now that we have artifact support. Might be part of a future diff.
81	:ivar dict keyvals: all key-value pairs in this dictionary are stored in	78		:ivar dict keyvals: all key-value pairs in this dictionary are stored in
82	ResultsDB as 'extra data'. This can be used to store	79		ResultsDB as 'extra data'. This can be used to store
83	e.g. a compose id or a kernel version, if that	80		e.g. a compose id or a kernel version, if that
84	information is important for querying the results. Keep	81		information is important for querying the results. Keep
85	it as short as possible.	82		it as short as possible.
86	:ivar str checkname: name of the check, which the CheckDetail belongs to. This	83		:ivar str checkname: name of the check, which the CheckDetail belongs to. This
87	is usually not needed, as the check name is devised from	84		is usually not needed, as the check name is devised from
88	the task metadata, but if a single task produces	85		the task metadata, but if a single task produces
89	results for multiple checks, this is the way to override	86		results for multiple checks, this is the way to override
90	the default behavior.	87		the default behavior.
91	:ivar str artifact: path to a file or directory placed in the artifacts directory,	88		:ivar str artifact: path to a file or directory placed in the artifacts directory,
92	either absolute ``$artifactsdir/report.html`` or relative	89		either absolute ``$artifactsdir/report.html`` or relative
93	``report.html``. It will represent task output specific for	90		``report.html``. It will represent task output specific for
94	this particular :attr:``item``.	91		this particular :attr:``item``.
95	'''	92		'''
96		93
97	outcome_priority = ('PASSED', 'INFO', 'FAILED', 'NEEDS_INSPECTION',	94		outcome_priority = ('PASSED', 'INFO', 'FAILED', 'NEEDS_INSPECTION',
98	'ABORTED', 'CRASHED')	95		'ABORTED', 'CRASHED')
99		96
100		97		def __init__(self, item, report_type=None, outcome=None, note='',
101	def __init__(self, item, report_type=None, outcome=None, summary='',
102	output=None, keyvals=None, checkname=None, artifact=None):	98		output=None, keyvals=None, checkname=None, artifact=None):
103	# validate input	99		# validate input
104	if (output is not None and	100		if (output is not None and
105	not python_utils.sequence(output, basestring, mutable=True)):	101		not python_utils.sequence(output, basestring, mutable=True)):
106	raise exc.TaskotronValueError("'output' parameter must be a "	102		raise exc.TaskotronValueError("'output' parameter must be a "
107	"mutable sequence of strings. Yours was: %s" % type(output))	103		"mutable sequence of strings. Yours was: %s" % type(output))
108	if keyvals is not None and not isinstance(keyvals, collections.Mapping):	104		if keyvals is not None and not isinstance(keyvals, collections.Mapping):
109	raise exc.TaskotronValueError("'keyvals' parameter must be a "	105		raise exc.TaskotronValueError("'keyvals' parameter must be a "
110	"mapping. Yours was: %s" % type(keyvals))	106		"mapping. Yours was: %s" % type(keyvals))
111		107
112	self.item = item	108		self.item = item
113	self.report_type = report_type	109		self.report_type = report_type
114	self._outcome = None	110		self._outcome = None
115	if outcome:	111		if outcome:
116	self.outcome = outcome	112		self.outcome = outcome
117	self.checkname = checkname	113		self.checkname = checkname
118	self.summary = summary	114		self.note = note
119	self.output = output or []	115		self.output = output or []
120	self.keyvals = keyvals or {}	116		self.keyvals = keyvals or {}
121	self.artifact = artifact	117		self.artifact = artifact
122	# this dictionary will hold implementation-specific data, that needs	118		# this dictionary will hold implementation-specific data, that needs
123	# to be transfered in between directives (like resultsdb_result_id)	119		# to be transfered in between directives (like resultsdb_result_id)
124	self._internal = {}	120		self._internal = {}
125		121
126		122
▲ Show 20 Lines • Show All 116 Lines • ▼ Show 20 Line(s)		237	def __str__(self):
243	# But don't show self.output, because that can be huuge	239		# But don't show self.output, because that can be huuge
244	attrs = vars(self)	240		attrs = vars(self)
245	attrs['output'] = '<stripped out>'	241		attrs['output'] = '<stripped out>'
246	return '<%s: %s>' % (self.__class__.__name__, pprint.pformat(attrs))	242		return '<%s: %s>' % (self.__class__.__name__, pprint.pformat(attrs))
247		243
248		244
249	class ReportType(object):	245		class ReportType(object):
250	''' Enum for different types of :attr:`CheckDetail.report_type`'''	246		''' Enum for different types of :attr:`CheckDetail.report_type`'''
251	# the values are used as identifiers in a TAP export	247		# the values are used as identifiers in a ResultYAML export
252	KOJI_BUILD = 'koji_build' #:	248		KOJI_BUILD = 'koji_build' #:
253	BODHI_UPDATE = 'bodhi_update' #:	249		BODHI_UPDATE = 'bodhi_update' #:
254	YUM_REPOSITORY = 'yum_repository' #:	250		YUM_REPOSITORY = 'yum_repository' #:
255	COMPOSE = 'compose' #:	251		COMPOSE = 'compose' #:
256		252
257		253
258	def export_TAP(check_details, checkname="$CHECKNAME"):	254		def export_YAML(check_details):
259	'''Generate TAP output used for reporting to ResultsDB.	255		'''Generate YAML output used for reporting to ResultsDB.
260		256
261	Note: You need to provide all your :class:`CheckDetail`\s in a single pass	257		Note: You need to provide all your :class:`CheckDetail`\s in a single pass
262	in order to generate a valid TAP output. You can't call this method several	258		in order to generate a valid YAML output. You can't call this method several
263	times and then simply join the outputs simply as strings.	259		times and then simply join the outputs simply as strings.
264		260
265	:param check_details: iterable of :class:`CheckDetail` instances or single	261		:param check_details: iterable of :class:`CheckDetail` instances or single
266	instance of :class:`CheckDetail`	262		instance of :class:`CheckDetail`
267	:param str checkname: name of the executed check	263		:return: YAML output with results for every :class:`CheckDetail` instance
268	:return: TAP output with results for every :class:`CheckDetail` instance
269	provided	264		provided
270	:rtype: str	265		:rtype: str
271	:raise TaskotronValueError: if :attr:`CheckDetail.item` is empty for any	266		:raise TaskotronValueError: if :attr:`CheckDetail.item` or :attr:`CheckDetail.outcome`
272	parameter provided	267		is empty for any parameter provided
273		268
274	Example output::	269		Example output::
275		270
276	TAP Version 13	271		results:
			kparalUnsubmitted Not Done Are we going to support multiple documents (document roots) in a single ResultYAML? If not, is there a purpose for using the triple-dash and triple-dot syntax?
			jskladanAuthorUnsubmitted Not Done meh, probably left it there accidentally...
277	1..1	272		- item: xchat-0.5-1.fc20
278	ok - $CHECKNAME for Koji build xchat-0.5-1.fc20	273		type: koji_build
279	---	274		outcome: PASSED
280	details:	275		note: 5 ERRORS, 10 WARNINGS
281	output: \|-	276		artifact: xchat-0.5-1.fc20.x86_64.log
282	xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
283	xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps.schemas
284	xchat.x86_64: W: no-manual-page-for-binary xchat
285	item: xchat-0.5-1.fc20
286	outcome: PASSED
287	summary: 5 ERRORS, 10 WARNINGS
288	type: koji_build
289	artifact: xchat-0.5-1.fc20.x86_64.log
290	...
291	'''	277		'''
292		278
293	# if check_details is single CheckDetail create a list with one element
294	if isinstance(check_details, CheckDetail):	279		if isinstance(check_details, CheckDetail):
295	check_details = [check_details]	280		check_details = [check_details]
296		281
297		282
298	# validate input	283		# validate input
299	for detail in check_details:	284		for detail in check_details:
300	if not detail.item:	285		if not detail.item:
301	raise exc.TaskotronValueError('CheckDetail.item is empty for: %s' %	286		raise exc.TaskotronValueError('CheckDetail.item is empty for: %s' %
302	detail)	287		detail)
303		288
304	# TAP test line will be more readable for recognized report types	289		results = []
305	pretty_type_name = {
306	ReportType.KOJI_BUILD: 'Koji build',
307	ReportType.BODHI_UPDATE: 'Bodhi update',
308	ReportType.YUM_REPOSITORY: 'Yum repository',
309	}
310
311	tapgen = tap.TAPGenerator()
312	tapout = ['TAP version 13', '1..%d' % len(check_details)]
313		290
314	for detail in check_details:	291		for detail in check_details:
315	# TAP test line	292		# Result YAML block
316	tap_outcome = tap.FAIL
317	if detail.outcome in ['PASSED', 'INFO']:
318	tap_outcome = tap.PASS
319
320	name = '%s for ' % checkname
321	if detail.report_type:
322	name += '%s ' % pretty_type_name.get(detail.report_type,
323	detail.report_type)
324	name += detail.item
325
326	# TAP YAML block
327	data = {}	293		data = {}
328	data['outcome'] = detail.outcome
329	data['item'] = detail.item	294		data['item'] = detail.item
		295		data['outcome'] = detail.outcome
330	if detail.artifact:	296		if detail.artifact:
331	data['artifact'] = detail.artifact	297		data['artifact'] = detail.artifact
332	if detail.report_type:	298		if detail.report_type:
333	data['type'] = detail.report_type	299		data['type'] = detail.report_type
334	if detail.summary:	300		if detail.note:
335	data['summary'] = detail.summary	301		data['note'] = detail.note
336	if detail.checkname:	302		if detail.checkname:
337	data['checkname'] = detail.checkname	303		data['checkname'] = detail.checkname
338	data['details'] = {}
339	if detail._internal:	304		if detail._internal:
340	data['_internal'] = detail._internal	305		data['_internal'] = detail._internal
341	if detail.output:
342	data['details']['output'] = '\n'.join(detail.output)
343	if not data['details']:
344	del data['details']
345
346	for key, value in detail.keyvals.iteritems():	306		for key, value in detail.keyvals.iteritems():
347	if key in RESERVED_KEYS:	307		if key in RESERVED_KEYS:
348	log.warn("Reserved key '%s' found in keyvals. Ignoring for "	308		log.warn("Reserved key '%s' found in keyvals. Ignoring for "
349	"export purposes.", key)	309		"export purposes.", key)
350	continue	310		continue
351		311
352	data[key] = value	312		data[key] = value
353		313
354	# generate	314		results.append(data)
355	output = tapgen.format_TAP_msg(result=tap_outcome, name=name, data=data)
356	tapout.append(output)
357		315
358	return '\n'.join(tapout)	316		return yaml.safe_dump(
		317		{"results": results},
		318		indent=2,
		319		allow_unicode=True,
		320		default_flow_style=False
		321		)
359		322
360	def import_TAP(source):
361	'''Parses TAP and returns a list of :class:`CheckDetails <CheckDetail>`.
362		323
363	:param str source: TAP-formatted text	324		def import_YAML(source):
364	:raise TaskotronValueError: if TAP syntax is incorrect	325		'''Parses YAML and returns a list of :class:`CheckDetails <CheckDetail>`.
		326
		327		:param str source: YAML-formatted text
		328		:raise TaskotronValueError: if YAML syntax is incorrect
365	'''	329		'''
366		330
367	tap = TAP13()
368	try:	331		try:
369	tap.parse(source)	332		data = yaml.safe_load(source)
370	except ValueError as e:	333		except yaml.scanner.ScannerError as e:
371	raise exc.TaskotronValueError('Failed to parse TAP contents: %s' % e)	334		raise exc.TaskotronValueError('Failed to parse YAML contents: %s' % e)
		335
		336		# check basic structure of the loaded object"
		337		if type(data) is not dict:
		338		raise exc.TaskotronValueError("Invalid type of the YAML root element, %s instead of dict" %
		339		type(data))
		340		try:
		341		if type(data['results']) not in (list, type(None)):
		342		raise exc.TaskotronValueError("Invalid type of 'results', %s instead of list" %
		343		type(data['results']))
		344		except KeyError:
		345		raise exc.TaskotronValueError("Invalid format of the YAML data -"
		346		"missing `results` key")
		347
		348		if data['results'] is None:
		349		return []
372		350
373	check_details = []	351		check_details = []
374	for test in tap.tests:	352		for result in data['results']:
375	y = test.yaml or {}	353		item = result.get('item', None)
376	item = y.get('item', None)	354		report_type = result.get('type', None)
377	report_type = y.get('type', None)	355		artifact = result.get('artifact', None)
378	artifact = y.get('artifact', None)	356		outcome = result.get('outcome', None)
379	outcome = 'PASSED' if test.result == 'ok' else 'FAILED'	357		note = result.get('note', '')
380	outcome = y.get('outcome', outcome)	358		checkname = result.get('checkname', None)
381	summary = y.get('summary', '')	359		_internal = result.get('_internal', {})
382	checkname = y.get('checkname', None)
383	details = y.get('details', {})
384	output = details.get('output', None)
385	_internal = y.get('_internal', {})
386	if output is not None:
387	output = [output]
388		360
389	# TODO: is there a better way to do this?	361		# TODO: is there a better way to do this?
390	other_keys = set(y.keys()) - set(RESERVED_KEYS)	362		other_keys = set(result.keys()) - set(RESERVED_KEYS)
391	keyvals = dict([(k,y[k]) for k in other_keys])	363		keyvals = dict([(k, result[k]) for k in other_keys])
392		364
393	cd = CheckDetail(item, report_type, outcome, summary, output, checkname=checkname, artifact=artifact)	365		cd = CheckDetail(item, report_type, outcome, note, checkname=checkname, artifact=artifact)
394		366
395	cd.keyvals = keyvals	367		cd.keyvals = keyvals
396	cd._internal = _internal	368		cd._internal = _internal
397	check_details.append(cd)	369		check_details.append(cd)
398		370
399	return check_details	371		return check_details
400		372

View Options

libtaskotron/directives/bodhi_comment_directive.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5	from __future__ import absolute_import	5		from __future__ import absolute_import
6		6
7	DOCUMENTATION = """	7		DOCUMENTATION = """
8	module: bodhi_comment_directive	8		module: bodhi_comment_directive
9	short_description: create a comment for a Bodhi update	9		short_description: create a comment for a Bodhi update
10	description: \|	10		description: \|
11	Take TAP formmated check results, find all `Bodhi`_ update results in it,	11		Take YAML formmated check results, find all `Bodhi`_ update results in it,
12	and post a comment under each of these updates stating the result of the	12		and post a comment under each of these updates stating the result of the
13	check.	13		check.
14		14
15	.. _Bodhi: https://bodhi.fedoraproject.org	15		.. _Bodhi: https://bodhi.fedoraproject.org
16		16
17	The comment will roughly look like this::	17		The comment will roughly look like this::
18		18
19	Taskotron: depcheck test PASSED on x86_64. Result log: <url>	19		Taskotron: depcheck test PASSED on x86_64. Result log: <url>
20		20
21	.. note:: You have to have ``fas_username`` and ``fas_password`` configured	21		.. note:: You have to have ``fas_username`` and ``fas_password`` configured
22	in your config file, otherwise the comments won't be posted.	22		in your config file, otherwise the comments won't be posted.
23		23
24	.. note:: The <url> will point to the Buildbot's full stdio log.	24		.. note:: The <url> will point to the Buildbot's full stdio log.
25	It is possible to make the <url> point to the respective ResultsDB's	25		It is possible to make the <url> point to the respective ResultsDB's
26	result page, if you use the ``resultsdb_directive``'s exported output	26		result page, if you use the ``resultsdb_directive``'s exported output
27	as an input for this directive (see Examples section).	27		as an input for this directive (see Examples section).
28	parameters:	28		parameters:
29	results:	29		results:
30	required: true	30		required: true
31	description: single or multiple check results in TAP13 format. Only results	31		description: \|
32	with ``type: bodhi_update`` will be processed. The ``item: `` value has	32		single or multiple check results in YAML format. Only results
		33		with ``type: bodhi_update`` will be processed. The ``item:`` value has
33	to be a Bodhi update ID.	34		to be a Bodhi update ID.
34	type: str	35		type: str
35	doreport:	36		doreport:
36	required: true	37		required: true
37	description: \|	38		description: \|
38	select ``all`` to always post the check result, regardless of	39		select ``all`` to always post the check result, regardless of
39	previoust Taskotron comments for the same update. Select ``onchange`` to	40		previoust Taskotron comments for the same update. Select ``onchange`` to
40	only post comments if the check result status changed (e.g. from ``FAILED``	41		only post comments if the check result status changed (e.g. from ``FAILED``
41	to ``PASSED``), or if the ``FAILED`` status repeats itself and long	42		to ``PASSED``), or if the ``FAILED`` status repeats itself and long
42	enough time has passed between these two failures (configurable as	43		enough time has passed between these two failures (configurable as
43	``bodhi_posting_comments_span`` in the config file).	44		``bodhi_posting_comments_span`` in the config file).
44		45
45	For production purposes you usually want to use ``onchange``, for	46		For production purposes you usually want to use ``onchange``, for
46	development purposes you might want to use ``all``.	47		development purposes you might want to use ``all``.
47	type: str	48		type: str
48	choice: [all, onchange]	49		choice: [all, onchange]
49	returns: \|	50		returns: \|
50	A list of bool values. The number and order of the bool values matches the	51		A list of bool values. The number and order of the bool values matches the
51	number and order of Bodhi update check results in ``results``. The bool value	52		number and order of Bodhi update check results in ``results``. The bool value
52	is ``True`` if comment was posted successfully or comment wasn't meant to be	53		is ``True`` if comment was posted successfully or comment wasn't meant to be
53	posted (either posting is turned off or comment was already posted), ``False``	54		posted (either posting is turned off or comment was already posted), ``False``
54	if some problem was encountered.	55		if some problem was encountered.
55	raises: \|	56		raises: \|
56	* :class:`.TaskotronDirectiveError`: if ``results`` is not a string in valid	57		* :class:`.TaskotronDirectiveError`: if ``results`` is not a string in valid
57	TAP13 format	58		YAML format
58	* :class:`.TaskotronValueError`: if ``results`` refer to a Bodhi update which	59		* :class:`.TaskotronValueError`: if ``results`` refer to a Bodhi update which
59	can't be found in Bodhi	60		can't be found in Bodhi
60	version_added: 0.4	61		version_added: 0.4
61	"""	62		"""
62		63
63	EXAMPLES = """	64		EXAMPLES = """
64	Run ``upgradepath`` check and then report all results to Bodhi (using	65		Run ``upgradepath`` check and then report all results to Bodhi (using
65	``onchange`` approach)::	66		``onchange`` approach)::
▲ Show 20 Lines • Show All 88 Lines • ▼ Show 20 Line(s)		154	raise TaskotronDirectiveError("The argument 'doreport' is"\
154	"set to invalid value '%s'. Valid values are: "\	155		"set to invalid value '%s'. Valid values are: "\
155	"'all', 'onchange'.")	156		"'all', 'onchange'.")
156		157
157	if 'checkname' not in env_data:	158		if 'checkname' not in env_data:
158	raise TaskotronDirectiveError("The bodhi_comment directive "\	159		raise TaskotronDirectiveError("The bodhi_comment directive "\
159	"requires checkname.")	160		"requires checkname.")
160		161
161	try:	162		try:
162	check_details = check.import_TAP(input_data['results'])	163		check_details = check.import_YAML(input_data['results'])
163	except TaskotronValueError as e:	164		except TaskotronValueError as e:
164	log.exception(e)	165		log.exception(e)
165	raise TaskotronDirectiveError("Failed to load the 'results':"\	166		raise TaskotronDirectiveError("Failed to load the 'results':"\
166	"%s" % e)	167		"%s" % e)
167		168
168	# Filter the results of "BODHI_UPDATE" type	169		# Filter the results of "BODHI_UPDATE" type
169	check_details = [r for r in check_details	170		check_details = [r for r in check_details
170	if r.report_type == check.ReportType.BODHI_UPDATE]	171		if r.report_type == check.ReportType.BODHI_UPDATE]
▲ Show 20 Lines • Show All 315 Lines • ▼ Show 20 Line(s)		453	def _post_testresult(bodhi_api, updateid, testname, result, url,
486	log.debug('Posting a result to Bodhi for: %s', updateid)	487		log.debug('Posting a result to Bodhi for: %s', updateid)
487		488
488	comment = 'Taskotron: %s test %s on %s. Result log: %s ' \	489		comment = 'Taskotron: %s test %s on %s. Result log: %s ' \
489	'(results are informative only)' % (testname, result, arch, url)	490		'(results are informative only)' % (testname, result, arch, url)
490	try:	491		try:
491	bodhi_update = bodhi_api.get_update(updateid)	492		bodhi_update = bodhi_api.get_update(updateid)
492	if not bodhi_update:	493		if not bodhi_update:
493	# this might happen for old Bodhi1 updates still circulating in the system,	494		# this might happen for old Bodhi1 updates still circulating in the system,
494	# the TAP might not contain `item: updateid` but `item: update_title`	495		# the YAML might not contain `item: updateid` but `item: update_title`
495	log.warning("No update with ID '%s' found, skipping...")	496		log.warning("No update with ID '%s' found, skipping...")
496	return False	497		return False
497		498
498	(old_result, comment_time) = _already_commented(bodhi_update, testname, arch)	499		(old_result, comment_time) = _already_commented(bodhi_update, testname, arch)
499	comment_needed = _is_comment_needed(old_result, comment_time, result)	500		comment_needed = _is_comment_needed(old_result, comment_time, result)
500		501
501	if not comment_needed and doreport != 'all':	502		if not comment_needed and doreport != 'all':
502	log.debug('Bodhi comment is not needed for: %s %s',	503		log.debug('Bodhi comment is not needed for: %s %s',
Show All 13 Lines

View Options

libtaskotron/directives/exitcode_directive.py

1	# -- coding: utf-8 --	1	# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2	# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4	# See the LICENSE file for more details on Licensing
5		5
6	from libtaskotron.directives import BaseDirective	6	from libtaskotron.directives import BaseDirective
7	from libtaskotron.exceptions import TaskotronDirectiveError	7	from libtaskotron.exceptions import TaskotronDirectiveError
8	from libtaskotron.logger import log	8	from libtaskotron.logger import log
9	from libtaskotron import check	9	from libtaskotron import check
10		10
11		11
12	DOCUMENTATION = """	12	DOCUMENTATION = """
13	module: exitcode_directive	13	module: exitcode_directive
14	short_description: set runtask exit code based on last or worst TAP outcome	14	short_description: set runtask exit code based on last or worst YAML outcome
15	description: \|	15	description: \|
16	This directives takes TAP specified by key ``tap_last`` or ``tap_worst`` (keys are mutually	16	This directives takes YAML specified by key ``result_last`` or ``result_worst`` (keys are
17	exclusive), and generates returncode based on last or worst TAP outcome. If TAP is empty,	17	mutually exclusive), and generates returncode based on last or worst YAML outcome.
18	exitcode is set to SUCCESS.	18	If YAML is empty, exitcode is set to SUCCESS.
19		19
20	If task formula contains multiple usages of exitcode directive, worst exitcode is returned by	20	If task formula contains multiple usages of exitcode directive, worst exitcode is returned by
21	runtask.	21	runtask.
22		22
23	Input for directive is supposed to be in a `TAP13 format`_. The easiest way to create it is to	23	Input for directive is supposed to be in a :ref:`resultyaml-format`. The easiest way to create it
24	use :class:`.CheckDetail` objects to construct your result (or results), and then generate	24	is to use :class:`.CheckDetail` objects to construct your result (or results), and then generate
25	the TAP output with :func:`~.check.export_TAP`. Read more in :ref:`writing-tasks-for-taskotron`.	25	the YAML output with :func:`~.check.export_YAML`.
		26	Read more in :ref:`writing-tasks-for-taskotron`.
26		27
27	.. _TAP13 format: http://testanything.org/tap-version-13-specification.html
28	parameters:	28	parameters:
29	tap_last:	29	result_last:
30	required: true	30	required: true
31	description: TAP output, last outcome is used. ``tap_worst`` cannot be specified together with	31	description: YAML output, last outcome is used. ``result_worst`` cannot be specified together
32	this one.	32	with this one.
33	type: str	33	type: str
34	tap_worst:	34	result_worst:
35	required: true	35	required: true
36	description: TAP output, worst outcome is used. ``tap_last`` cannot be specified together with	36	description: YAML output, worst outcome is used. ``result_last`` cannot be specified together
37	this one.	37	with this one.
38	type: str	38	type: str
39	returns: \|	39	returns: \|
40	:int: Returncode based on TAP last or worst outcome. Success is ``0``, failure is ``100``.	40	:int: Returncode based on YAML last or worst outcome. Success is ``0``, failure is ``100``.
41	raises: \|	41	raises: \|
42	* :class:`.TaskotronDirectiveError`: when there's not exactly one of parameters ``tap_last`` or	42	* :class:`.TaskotronDirectiveError`: when there's not exactly one of parameters ``result_last``
43	``tap_last`` present	43	or ``result_last`` present
44	* :class:`.TaskotronValueError`: when TAP input cannot be parsed	44	* :class:`.TaskotronValueError`: when YAML input cannot be parsed
45	version_added: 0.3.18	45	version_added: 0.3.18
46	"""	46	"""
47		47
48	EXAMPLES = """	48	EXAMPLES = """
49	Run a check returning multiple results and make ``runtask`` fail if any of the results is failed::	49	Run a check returning multiple results and make ``runtask`` fail if any of the results is failed::
50		50
51	- name: run my check and return TAP	51	- name: run my check and return YAML
52	python:	52	python:
53	file: my_check.py	53	file: my_check.py
54	callable: run	54	callable: run
55	export: tap	55	export: results
56		56
57	- name: set runtask exit code according to the worst result in TAP	57	- name: set runtask exit code according to the worst result in YAML
58	exitcode:	58	exitcode:
59	tap_worst: ${tap}	59	result_worst: ${results}
60		60
61	Run a check that returns multiple results and one "overall" result (at the end) which is computed	61	Run a check that returns multiple results and one "overall" result (at the end) which is computed
62	according check's internal logic. Then make ``runtask`` fail if the overall result failed::	62	according check's internal logic. Then make ``runtask`` fail if the overall result failed::
63		63
64	- name: run my check and return TAP	64	- name: run my check and return YAML
65	python:	65	python:
66	file: my_check.py	66	file: my_check.py
67	callable: run	67	callable: run
68	export: tap	68	export: results
69		69
70	- name: set runtask exit code according to the last (overall) result in TAP	70	- name: set runtask exit code according to the last (overall) result in YAML
71	exitcode:	71	exitcode:
72	tap_last: ${tap}	72	result_last: ${results}
73	"""	73	"""
74		74
75	# exitcode for PASSED and INFO outcome	75	# exitcode for PASSED and INFO outcome
76	SUCCESS = 0	76	SUCCESS = 0
77		77
78	# exitcode for other outcomes	78	# exitcode for other outcomes
79	FAILURE = 100	79	FAILURE = 100
80		80
81	directive_class = 'ExitcodeDirective'	81	directive_class = 'ExitcodeDirective'
82		82
83		83
84	class ExitcodeDirective(BaseDirective):	84	class ExitcodeDirective(BaseDirective):
85	def process(self, input_data, env_data):	85	def process(self, input_data, env_data):
86	"""Process this directive.	86	"""Process this directive.
87	:param dict input_data: dictionary of all kwargs from yaml declaration	87	:param dict input_data: dictionary of all kwargs from yaml declaration
88	:param dict env_data: information on the environment in which we're running	88	:param dict env_data: information on the environment in which we're running
89	:returns: returncode based on TAP last or worst outcome	89	:returns: returncode based on YAML last or worst outcome
90	:raise TaskotronDirectiveError: ``input data`` doesn't contain exactly one of keys	90	:raise TaskotronDirectiveError: ``input data`` doesn't contain exactly one of keys
91	``tap_last`` or ``tap_last``	91	``result_last`` or ``result_last``
92	:raise TaskotronValueError: when TAP input cannot be parsed	92	:raise TaskotronValueError: when YAML input cannot be parsed
93	"""	93	"""
94		94
95	# keys 'tap_worst' and 'tap_last' must be mutually exclusive	95	# keys 'result_worst' and 'result_last' must be mutually exclusive
96	if ('tap_worst' in input_data) == ('tap_last' in input_data):	96	if ('result_worst' in input_data) == ('result_last' in input_data):
97	raise TaskotronDirectiveError("The exitcode directive requires exactly one of keys "	97	raise TaskotronDirectiveError("The exitcode directive requires exactly one of keys "
98	"'tap_worst' or 'tap_last'.")	98	"'result_worst' or 'result_last'.")
99		99
100	if 'tap_worst' in input_data:	100	if 'result_worst' in input_data:
101	details = check.import_TAP(input_data['tap_worst'])	101	details = check.import_YAML(input_data['result_worst'])
102	code = SUCCESS	102	code = SUCCESS
103	for detail in details:	103	for detail in details:
104	if detail.outcome not in ['PASSED', 'INFO']:	104	if detail.outcome not in ['PASSED', 'INFO']:
105	code = FAILURE	105	code = FAILURE
106		106
107	log.debug("Returning exitcode %d" % code)	107	log.debug("Returning exitcode %d" % code)
108	return code	108	return code
109		109
110	elif 'tap_last' in input_data:	110	elif 'result_last' in input_data:
111	details = check.import_TAP(input_data['tap_last'])	111	details = check.import_YAML(input_data['result_last'])
112	if details and details[-1].outcome not in ['PASSED', 'INFO']:	112	if details and details[-1].outcome not in ['PASSED', 'INFO']:
113		113
114	log.debug("Returning exitcode %d" % FAILURE)	114	log.debug("Returning exitcode %d" % FAILURE)
115	return FAILURE	115	return FAILURE
116	else:	116	else:
117	log.debug("Returning exitcode %d" % SUCCESS)	117	log.debug("Returning exitcode %d" % SUCCESS)
118	return SUCCESS	118	return SUCCESS
119		119
120	else:	120	else:
121	assert False, "This should never occur"	121	assert False, "This should never occur"

View Options

libtaskotron/directives/python_directive.py

Show All 27 Lines		26	<kwarg>:
28	description: arbitrarily named parameter with a value. All these ``key=value``	28		description: arbitrarily named parameter with a value. All these ``key=value``
29	pairs will be passed to the python script as ``callable`` parameters.	29		pairs will be passed to the python script as ``callable`` parameters.
30	type: anything	30		type: anything
31	returns: \|	31		returns: \|
32	The output of the python script, which must be either ``None`` or a	32		The output of the python script, which must be either ``None`` or a
33	``basestring``.	33		``basestring``.
34		34
35	If this python script is supposed to be the main check of the task, it is	35		If this python script is supposed to be the main check of the task, it is
36	supposed to provide output in a `TAP13 format`_. This format is then used in	36		supposed to provide output in a :ref:`resultyaml-format`. This format is then used in
37	the :ref:`resultsdb directive <resultsdb_directive>`. The easiest way to	37		the :ref:`resultsdb directive <resultsdb_directive>`. The easiest way to
38	create it is to use :class:`.CheckDetail` objects to construct your result (or	38		create it is to use :class:`.CheckDetail` objects to construct your result (or
39	results), and then generate the TAP output with :func:`~.check.export_TAP`.	39		results), and then generate the YAML output with :func:`~.check.export_YAML`.
40	Read more in :ref:`writing-tasks-for-taskotron`.	40		Read more in :ref:`writing-tasks-for-taskotron`.
41		41
42	.. _TAP13 format: http://testanything.org/tap-version-13-specification.html
43	raises: \|	42		raises: \|
44	* :class:`.TaskotronDirectiveError`: if ``file`` can't be read, or if	43		* :class:`.TaskotronDirectiveError`: if ``file`` can't be read, or if
45	``callable`` returns something else than ``None`` or ``basestring``	44		``callable`` returns something else than ``None`` or ``basestring``
46	* anything that the python script raises itself	45		* anything that the python script raises itself
47	version_added: 0.4	46		version_added: 0.4
48	"""	47		"""
49		48
50	EXAMPLES = """	49		EXAMPLES = """
▲ Show 20 Lines • Show All 134 Lines • ▼ Show 20 Line(s)		183	raise TaskotronDirectiveError("The file %s does not exist!"
185	% filename)	184		% filename)
186		185
187	def process(self, input_data, env_data):	186		def process(self, input_data, env_data):
188	"""Process this directive.	187		"""Process this directive.
189	:param dict input_data: dictionary of all kwargs from yaml declaration,	188		:param dict input_data: dictionary of all kwargs from yaml declaration,
190	including callable and file	189		including callable and file
191	:param dict env_data: information on the environment in which we're	190		:param dict env_data: information on the environment in which we're
192	running	191		running
193	:returns: output from the callable specified, this is expected to be TAP	192		:returns: output from the callable specified, this is expected to be YAML
194	output for checks	193		output for checks
195	:raise TaskotronDirectiveError: if the callable returns something else	194		:raise TaskotronDirectiveError: if the callable returns something else
196	than ``None`` or ``basestring``	195		than ``None`` or ``basestring``
197	"""	196		"""
198		197
199	if 'callable' not in input_data or 'file' not in input_data:	198		if 'callable' not in input_data or 'file' not in input_data:
200	detected_args = ', '.join(input_data.keys())	199		detected_args = ', '.join(input_data.keys())
201	raise TaskotronDirectiveError(	200		raise TaskotronDirectiveError(
Show All 15 Lines

View Options

libtaskotron/directives/resultsdb_directive.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5	from __future__ import absolute_import	5		from __future__ import absolute_import
6		6
7	DOCUMENTATION = """	7		DOCUMENTATION = """
8	module: resultsdb_directive	8		module: resultsdb_directive
9	short_description: send task results to ResultsDB or check TAP correctness	9		short_description: send task results to ResultsDB or check YAML correctness
10	description: \|	10		description: \|
11	Send check output to `ResultsDB <https://fedoraproject.org/wiki/ResultsDB>`_.	11		Send check output to `ResultsDB <https://fedoraproject.org/wiki/ResultsDB>`_.
12		12
13	If reporting to ResultsDB is disabled in the config (`report_to_resultsdb`	13		If reporting to ResultsDB is disabled in the config (`report_to_resultsdb`
14	option, disabled by default for the development profile), the directive at	14		option, disabled by default for the development profile), the directive at
15	least checks whether the check output is in valid `TAP13 format`_ and logs the	15		least checks whether the check output is in valid :ref:`resultyaml-format` and logs the
16	details that would have been reported. For this reason, it's useful to use	16		details that would have been reported. For this reason, it's useful to use
17	resultsdb directive always, even when you don't have any ResultsDB server	17		resultsdb directive always, even when you don't have any ResultsDB server
18	configured.	18		configured.
19		19
20	.. _TAP13 format: http://testanything.org/tap-version-13-specification.html
21	parameters:	20		parameters:
22	results:	21		results:
23	required: true	22		required: true
24	description: The output of the check in TAP13 format	23		description: The output of the check in the ResultYAML format
25	type: str	24		type: str
26	returns: \|	25		returns: \|
27	TAP13 formatted output as taken from input, enhanced with result-id's for each	26		YAML formatted output as taken from input, enhanced with result-id's for each
28	successfully stored result. The result-id information then can be used by	27		successfully stored result. The result-id information then can be used by
29	other directives - e.g. ``bodhi_comment_directive``. If reporting to ResultsDB	28		other directives - e.g. ``bodhi_comment_directive``. If reporting to ResultsDB
30	is disabled, the returned TAP does not contain this extra field.	29		is disabled, the returned YAML does not contain this extra field.
31	raises: \|	30		raises: \|
32	* :class:`.TaskotronDirectiveError`: if ``results`` is not in a valid TAP13	31		* :class:`.TaskotronDirectiveError`: if ``results`` is not in a valid YAML
33	format, or when ``item`` or ``type`` are missing in the TAP's result data	32		format, or when ``item`` or ``type`` are missing in the YAML's result data
34	* :class:`resultsdb_api.ResultsDBapiException`: for any errors coming from	33		* :class:`resultsdb_api.ResultsDBapiException`: for any errors coming from
35	the ResultsDB server	34		the ResultsDB server
36	version_added: 0.4	35		version_added: 0.4
37	"""	36		"""
38		37
39	EXAMPLES = """	38		EXAMPLES = """
40	These two actions first run ``run_rpmlint.py`` and export its TAP13 output to	39		These two actions first run ``run_rpmlint.py`` and export its YAML output to
41	``${rpmlint_output}``, and then feed this TAP13 output to ``resultsdb``	40		``${rpmlint_output}``, and then feed this YAML output to ``resultsdb``
42	directive::	41		directive::
43		42
44	- name: run rpmlint on downloaded rpms	43		- name: run rpmlint on downloaded rpms
45	python:	44		python:
46	file: run_rpmlint.py	45		file: run_rpmlint.py
47	callable: run	46		callable: run
48	workdir: ${workdir}	47		workdir: ${workdir}
49	export: rpmlint_output	48		export: rpmlint_output
50		49
51	- name: report results to resultsdb	50		- name: report results to resultsdb
52	resultsdb:	51		resultsdb:
53	results: ${rpmlint_output}	52		results: ${rpmlint_output}
54	export: resultsdb_output	53		export: resultsdb_output
55		54
56	If ResultsDB reporting is configured in the config file, it will get saved on	55		If ResultsDB reporting is configured in the config file, it will get saved on
57	the ResultsDB server, otherwise only TAP13 compliance will get checked and some	56		the ResultsDB server, otherwise only YAML compliance will get checked and some
58	summary will be printed out into the log, like this::	57		summary will be printed out into the log, like this::
59		58
60	[libtaskotron:resultsdb_directive.py:143] 2014-06-24 13:55:27 INFO TAP is OK.	59		[libtaskotron:resultsdb_directive.py:143] 2014-06-24 13:55:27 INFO YAML is OK.
61	[libtaskotron:resultsdb_directive.py:144] 2014-06-24 13:55:27 INFO Reporting to ResultsDB is disabled.	60		[libtaskotron:resultsdb_directive.py:144] 2014-06-24 13:55:27 INFO Reporting to ResultsDB is disabled.
62	[libtaskotron:resultsdb_directive.py:145] 2014-06-24 13:55:27 INFO Once enabled, the following would be reported:	61		[libtaskotron:resultsdb_directive.py:145] 2014-06-24 13:55:27 INFO Once enabled, the following would be reported:
63	[libtaskotron:resultsdb_directive.py:146] 2014-06-24 13:55:27 INFO <CheckDetail: {'_outcome': 'PASSED',	62		[libtaskotron:resultsdb_directive.py:146] 2014-06-24 13:55:27 INFO <CheckDetail: {'_outcome': 'PASSED',
64	'item': 'xchat-2.8.8-21.fc20.x86_64.rpm',	63		'item': 'xchat-2.8.8-21.fc20.x86_64.rpm',
65	'keyvals': {},	64		'keyvals': {},
66	'output': '<stripped out>',	65		'output': '<stripped out>',
67	'report_type': 'koji_build',	66		'report_type': 'koji_build',
68	'summary': 'RPMLINT PASSED for xchat-2.8.8-21.fc20.x86_64.rpm'}>	67		'summary': 'RPMLINT PASSED for xchat-2.8.8-21.fc20.x86_64.rpm'}>
▲ Show 20 Lines • Show All 112 Lines • ▼ Show 20 Line(s)		160	def get_artifact_path(self, artifactsdir, artifact):
181		180
182	return os.path.relpath(artifact_path, start=artifactsdir)	181		return os.path.relpath(artifact_path, start=artifactsdir)
183		182
184	def process(self, input_data, env_data):	183		def process(self, input_data, env_data):
185	#TODO: Replace with proper location and adjust exc. text	184		#TODO: Replace with proper location and adjust exc. text
186	# we're creating the jobid in the directive for now, so adjust the check	185		# we're creating the jobid in the directive for now, so adjust the check
187		186
188	if 'checkname' not in env_data:	187		if 'checkname' not in env_data:
189	raise TaskotronDirectiveError("The resultsdb directive requires "\	188		raise TaskotronDirectiveError("The resultsdb directive requires checkname.")
190	"resultsdb_job_id and checkname.")
191		189
192	# checking if reporting is enabled is done after importing tap which	190		# checking if reporting is enabled is done after importing yaml which
193	# serves as validation of input results	191		# serves as validation of input results
194	try:	192		try:
195	check_details = check.import_TAP(input_data['results'])	193		check_details = check.import_YAML(input_data['results'])
196	log.debug("TAP output parsed OK.")	194		log.debug("YAML output parsed OK.")
197	except TaskotronValueError as e:	195		except TaskotronValueError as e:
198	raise TaskotronDirectiveError("Failed to load 'results': %s" % e.message)	196		raise TaskotronDirectiveError("Failed to load 'results': %s" % e.message)
199		197
200	for detail in check_details:	198		for detail in check_details:
201	if not (detail.item and detail.report_type):	199		if not (detail.item and detail.report_type):
202	raise TaskotronDirectiveError("The resultsdb directive requires "	200		raise TaskotronDirectiveError(
203	"'item' and 'type' to be present in the TAP data.")	201		"The resultsdb directive requires "
		202		"'item' and 'type' to be present in the YAML data.")
204		203
205	conf = config.get_config()	204		conf = config.get_config()
206	if not (conf.reporting_enabled and conf.report_to_resultsdb):	205		if not (conf.reporting_enabled and conf.report_to_resultsdb):
207	log.info("Reporting to ResultsDB is disabled. Once enabled, the "	206		log.info("Reporting to ResultsDB is disabled. Once enabled, the "
208	"following would get reported:\n%s" %	207		"following would get reported:\n%s" % input_data['results'])
			kparalUnsubmitted Not Done This is no longer true, we used to strip values in TAP, but your patch doesn't do it. If we intend to keep `CheckDetail.output`, I'd probably keep the message and introduce output stripping again. If we plan to get rid of it, let's delete this message and then remove `CheckDetail.output` soon.
209	report_summary(input_data['results'], env_data['checkname']))	208		return check.export_YAML(check_details)
210	log.info('Hint: Enabling debug output allows you to see unstripped '
211	'values during variable export.')
212	return check.export_TAP(check_details)
213		209
214	# for now, we're creating the resultsdb job at reporting time	210		# for now, we're creating the resultsdb job at reporting time
215	# the job needs to be 'RUNNING' in order to append any results	211		# the job needs to be 'RUNNING' in order to append any results
216	job_url, log_url = buildbot_utils.get_urls(env_data['jobid'],	212		job_url, log_url = buildbot_utils.get_urls(env_data['jobid'],
217	self.masterurl,	213		self.masterurl,
218	self.task_stepname)	214		self.task_stepname)
219	job_data = self.create_resultsdb_job(env_data['checkname'],	215		job_data = self.create_resultsdb_job(env_data['checkname'],
220	uuid=env_data['uuid'],	216		uuid=env_data['uuid'],
Show All 12 Lines		224	if detail.artifact:
233	if artifact_path:	229		if artifact_path:
234	result_log_url = "%s/all/%s/task_output/%s" % (	230		result_log_url = "%s/all/%s/task_output/%s" % (
235	self.artifacts_baseurl,	231		self.artifacts_baseurl,
236	env_data['uuid'],	232		env_data['uuid'],
237	artifact_path,	233		artifact_path,
238	)	234		)
239	try:	235		try:
240	result = self.resultsdb.create_result(	236		result = self.resultsdb.create_result(
241	job_id = job_data['id'],	237		job_id=job_data['id'],
242	testcase_name = checkname,	238		testcase_name=checkname,
243	outcome = detail.outcome,	239		outcome=detail.outcome,
244	summary = detail.summary or None,	240		summary=detail.note or None,
245	log_url=result_log_url,	241		log_url=result_log_url,
246	item = detail.item,	242		item=detail.item,
247	type = detail.report_type,	243		type=detail.report_type,
248	**detail.keyvals)	244		**detail.keyvals)
249	detail._internal['resultsdb_result_id'] = result['id']	245		detail._internal['resultsdb_result_id'] = result['id']
250		246
251	except resultsdb_api.ResultsDBapiException, e:	247		except resultsdb_api.ResultsDBapiException, e:
252	log.error(e)	248		log.error(e)
253	log.error("Failed to store ResulsDB: `%s` `%s` `%s`",	249		log.error("Failed to store ResulsDB: `%s` `%s` `%s`",
254	detail.item, checkname, detail.outcome)	250		detail.item, checkname, detail.outcome)
255		251
256	# move job to completed	252		# move job to completed
257	self.complete_resultsdb_job(job_data['id'])	253		self.complete_resultsdb_job(job_data['id'])
258		254
259	return check.export_TAP(check_details)	255		return check.export_YAML(check_details)
			kparalUnsubmitted Not Done This method has no further purpose (it calls a single method and does nothing else), let's either implement it according to the docstring, or delete it completely.
260
261
262	def report_summary(tap, checkname):
263	'''Create a pretty summary of what will/would be reported into ResultsDB.
264	The summary is based on TAP output, with the TAP header stripped out and
265	most of the check output stripped out.
266
267	:param str tap: TAP-formatted results
268	:param str checkname: the name of the check
269	:return: summary in plain text
270	:rtype: str
271	:raise TaskotronValueError: if `tap` has invalid format
272	'''
273
274	check_details = check.import_TAP(tap)
275	summary = []
276
277	for detail in check_details:
278	# keep just the first and the last line of output
279	lines = ('\n'.join(detail.output)).splitlines()
280	if len(lines) > 3:
281	detail.output = [lines[0],
282	'<... %s lines stripped out ...>' % (len(lines) - 2),
283	lines[-1]]
284	out = check.export_TAP(detail, checkname=checkname)
285	# strip "TAP version 13\n1..1" (first two lines)
286	out = '\n'.join(out.splitlines()[2:])
287	summary.append(out)
288
289	return '\n'.join(summary)

View Options

readme.rst

Show First 20 Lines • Show All 118 Lines • ▼ Show 20 Line(s)
119		119
120	To run that task against a koji build with NEVR ``<nevr>`` for some arch	120	To run that task against a koji build with NEVR ``<nevr>`` for some arch
121	``<arch>``, do the following::	121	``<arch>``, do the following::
122		122
123	git clone https://bitbucket.org/fedoraqa/task-rpmlint.git	123	git clone https://bitbucket.org/fedoraqa/task-rpmlint.git
124	runtask -i <nevr> -t koji_build -a <arch> task-rpmlint/rpmlint.yml	124	runtask -i <nevr> -t koji_build -a <arch> task-rpmlint/rpmlint.yml
125		125
126	This will download the ``<nevr>`` from koji into a temp directory under	126	This will download the ``<nevr>`` from koji into a temp directory under
127	``/var/tmp/``, run rpmlint on the downloaded rpms and print output in TAP format	127	``/var/tmp/``, run rpmlint on the downloaded rpms and print output in YAML format
128	to stdout.	128	to stdout.
129		129
130	Example::	130	Example::
131		131
132	runtask -i xchat-2.8.8-21.fc20 -t koji_build -a x86_64 task-rpmlint/rpmlint.yml	132	runtask -i xchat-2.8.8-21.fc20 -t koji_build -a x86_64 task-rpmlint/rpmlint.yml
133		133
134		134
135	Running the Test Suite	135	Running the Test Suite
▲ Show 20 Lines • Show All 67 Lines • Show Last 20 Lines

View Options

requirements.txt

1	# This is a list of pypi packages to be installed into virtualenv. Alternatively,	1	# This is a list of pypi packages to be installed into virtualenv. Alternatively,
2	# you can install these as RPMs instead of pypi packages. Use	2	# you can install these as RPMs instead of pypi packages. Use
3	# `rpmspec -q --requires libtaskotron.spec` and	3	# `rpmspec -q --requires libtaskotron.spec` and
4	# `rpmspec -q --buildrequires libtaskotron.spec` to get a list of RPM dependencies.	4	# `rpmspec -q --buildrequires libtaskotron.spec` to get a list of RPM dependencies.
5	#	5	#
6	# A note for maintainers: Please keep this list in sync with libtaskotron.spec.	6	# A note for maintainers: Please keep this list in sync with libtaskotron.spec.
7	# You should require exact versions here (in order to define an exact working	7	# You should require exact versions here (in order to define an exact working
8	# environment), and require a minimal version in the spec file (in order to allow	8	# environment), and require a minimal version in the spec file (in order to allow
9	# the libraries to have some minor version bumps during the Fedora release	9	# the libraries to have some minor version bumps during the Fedora release
10	# lifecycle). If the Fedora libraries span multiple versions across releases,	10	# lifecycle). If the Fedora libraries span multiple versions across releases,
11	# you can define a dependency range here instead of an exact version.	11	# you can define a dependency range here instead of an exact version.
12		12
13	bayeux == 0.9
14	bunch == 1.0.1	13	bunch == 1.0.1
15	dingus == 0.3.4	14	dingus == 0.3.4
16	pytap13 == 0.3.0
17	pytest == 2.6.4	15	pytest == 2.6.4
18	pytest-cov == 1.8.1	16	pytest-cov == 1.8.1
19	python-fedora == 0.5.5	17	python-fedora == 0.5.5
20	PyYAML == 3.11	18	PyYAML == 3.11
21	resultsdb_api == 1.2.1	19	resultsdb_api == 1.2.1
22	Sphinx == 1.2.3	20	Sphinx == 1.2.3
23	yamlish == 0.18

View Options

testing/test_bodhi_comment_directive.py

Show First 20 Lines • Show All 49 Lines • ▼ Show 20 Line(s)		25	def setup(self, monkeypatch):
50	self.directive = bodhi_comment_directive.BodhiCommentDirective(self.stub_bodhi)	50		self.directive = bodhi_comment_directive.BodhiCommentDirective(self.stub_bodhi)
51		51
52	self.cd = check.CheckDetail(	52		self.cd = check.CheckDetail(
53	item='update_id',	53		item='update_id',
54	report_type=check.ReportType.BODHI_UPDATE,	54		report_type=check.ReportType.BODHI_UPDATE,
55	outcome='PASSED',	55		outcome='PASSED',
56	)	56		)
57		57
58		58		yaml = check.export_YAML(self.cd)
59	tap = check.export_TAP(self.cd)
60		59
61	self.ref_input = {	60		self.ref_input = {
62	'results': tap,	61		'results': yaml,
63	'doreport': 'onchange',	62		'doreport': 'onchange',
64	}	63		}
65	self.ref_envdata = {	64		self.ref_envdata = {
66	'checkname': 'depcheck',	65		'checkname': 'depcheck',
67	'jobid': 'all/%d' % self.ref_job_id,	66		'jobid': 'all/%d' % self.ref_job_id,
68	}	67		}
69		68
70	def test_config_reporting_disabled(self):	69		def test_config_reporting_disabled(self):
71	"""Checks config options can disable reporting."""	70		"""Checks config options can disable reporting."""
72	Config = config.get_config()	71		Config = config.get_config()
73		72
74	for r_e, r_t_b in ((False, False), (False, True), (True, False)):	73		for r_e, r_t_b in ((False, False), (False, True), (True, False)):
75	Config.reporting_enabled = r_e	74		Config.reporting_enabled = r_e
76	Config.report_to_bodhi = r_t_b	75		Config.report_to_bodhi = r_t_b
Show All 22 Lines
99	def test_missing_checkname(self, monkeypatch):	98		def test_missing_checkname(self, monkeypatch):
100	"""Checks if missing checkname raises exception"""	99		"""Checks if missing checkname raises exception"""
101	with pytest.raises(TaskotronDirectiveError):	100		with pytest.raises(TaskotronDirectiveError):
102	envdata = deepcopy(self.ref_envdata)	101		envdata = deepcopy(self.ref_envdata)
103	del(envdata['checkname'])	102		del(envdata['checkname'])
104		103
105	self.directive.process(self.ref_input, envdata)	104		self.directive.process(self.ref_input, envdata)
106		105
107	def test_failed_tap_import(self, monkeypatch):	106		def test_failed_yaml_import(self, monkeypatch):
108	"""Checks if failed TAP import raises exception"""	107		"""Checks if failed YAML import raises exception"""
109	def mock_raise(self, args, *kwargs):	108		def mock_raise(self, args, *kwargs):
110	raise TaskotronValueError("Testing Error")	109		raise TaskotronValueError("Testing Error")
111		110
112	monkeypatch.setattr(check, 'import_TAP', mock_raise)	111		monkeypatch.setattr(check, 'import_YAML', mock_raise)
113		112
114	with pytest.raises(TaskotronDirectiveError):	113		with pytest.raises(TaskotronDirectiveError):
115	self.directive.process(self.ref_input, self.ref_envdata)	114		self.directive.process(self.ref_input, self.ref_envdata)
116		115
117		116
118	def test_parse_result_from_comment(self):	117		def test_parse_result_from_comment(self):
119	"""Checks parsing of the bodhi comments"""	118		"""Checks parsing of the bodhi comments"""
120	output = bodhi_comment_directive._parse_result_from_comment(self.comment)	119		output = bodhi_comment_directive._parse_result_from_comment(self.comment)
▲ Show 20 Lines • Show All 151 Lines • ▼ Show 20 Line(s)		267	def test_post_testresult_crashed(self, prepare_for_bodhi_comment):
272		271
273	bodhi_comment_directive._post_testresult(	272		bodhi_comment_directive._post_testresult(
274	self.stub_bodhi, updateid="update_id", testname="depcheck", result="CRASHED",	273		self.stub_bodhi, updateid="update_id", testname="depcheck", result="CRASHED",
275	url=self.ref_log_url)	274		url=self.ref_log_url)
276		275
277	# if the result was properly ignored, this should be empty	276		# if the result was properly ignored, this should be empty
278	assert len(self.stub_bodhi.client.calls) == 0	277		assert len(self.stub_bodhi.client.calls) == 0
279		278
280	def test_tap_mapping(self, prepare_for_bodhi_comment):	279		def test_yaml_mapping(self, prepare_for_bodhi_comment):
281	"""Checks whether TAP output is mapped correctly to	280		"""Checks whether YAML output is mapped correctly to
282	the bodhi comment call."""	281		the bodhi comment call."""
283	ref_comment = "Taskotron: depcheck test PASSED on noarch. "\	282		ref_comment = "Taskotron: depcheck test PASSED on noarch. "\
284	"Result log: %s (results are informative only)" % self.ref_log_url	283		"Result log: %s (results are informative only)" % self.ref_log_url
285	self.stub_bodhi.client = Dingus('bodhi_api.bodhi')	284		self.stub_bodhi.client = Dingus('bodhi_api.bodhi')
286		285
287	self.directive.process(self.ref_input, self.ref_envdata)	286		self.directive.process(self.ref_input, self.ref_envdata)
288		287
289	# Select the first call of "comment" method.	288		# Select the first call of "comment" method.
290	call = [call for call in self.stub_bodhi.client.calls() if call[0] == 'comment'][0]	289		call = [call for call in self.stub_bodhi.client.calls() if call[0] == 'comment'][0]
291	# Select the positional arguments of that call	290		# Select the positional arguments of that call
292	call_data = call[1]	291		call_data = call[1]
293		292
294	assert call_data[0] == "update title"	293		assert call_data[0] == "update title"
295	assert call_data[1] == ref_comment	294		assert call_data[1] == ref_comment
296	assert call_data[2] == 0	295		assert call_data[2] == 0
297		296
298	def test_tap_mapping_resultsdb(self, prepare_for_bodhi_comment):	297		def test_yaml_mapping_resultsdb(self, prepare_for_bodhi_comment):
299	"""Checks whether TAP output is mapped correctly to	298		"""Checks whether YAML output is mapped correctly to
300	the bodhi comment call when resultsdb_result_id is provided"""	299		the bodhi comment call when resultsdb_result_id is provided"""
301		300
302	Config = config.get_config()	301		Config = config.get_config()
303		302
304	cd = check.CheckDetail(	303		cd = check.CheckDetail(
305	item='update_id',	304		item='update_id',
306	report_type=check.ReportType.BODHI_UPDATE,	305		report_type=check.ReportType.BODHI_UPDATE,
307	outcome='PASSED',	306		outcome='PASSED',
308	)	307		)
309	cd._internal = {'resultsdb_result_id': 1234}	308		cd._internal = {'resultsdb_result_id': 1234}
310	log_url = "%s/results/1234" % Config.resultsdb_frontend	309		log_url = "%s/results/1234" % Config.resultsdb_frontend
311		310
312	tap = check.export_TAP(cd)	311		yaml = check.export_YAML(cd)
313		312
314	ref_input = {	313		ref_input = {
315	'results': tap,	314		'results': yaml,
316	'doreport': 'onchange',	315		'doreport': 'onchange',
317	}	316		}
318		317
319	ref_comment = "Taskotron: depcheck test PASSED on noarch. "\	318		ref_comment = "Taskotron: depcheck test PASSED on noarch. "\
320	"Result log: %s (results are informative only)" % log_url	319		"Result log: %s (results are informative only)" % log_url
321	self.stub_bodhi.client = Dingus('bodhi_api.bodhi')	320		self.stub_bodhi.client = Dingus('bodhi_api.bodhi')
322		321
323	self.directive.process(ref_input, self.ref_envdata)	322		self.directive.process(ref_input, self.ref_envdata)
▲ Show 20 Lines • Show All 60 Lines • Show Last 20 Lines

View Options

testing/test_check.py

Show All 18 Lines		17	def setup_method(self, method):
19	# for most methods, we want a default empty CheckDetail instance	19		# for most methods, we want a default empty CheckDetail instance
20	self.cd = check.CheckDetail(None)	20		self.cd = check.CheckDetail(None)
21		21
22	def test_init_empty(self):	22		def test_init_empty(self):
23	'''Test instance creation with empty parameters'''	23		'''Test instance creation with empty parameters'''
24	# by default, outcome should be 'NEEDS_INSPECTION', not empty	24		# by default, outcome should be 'NEEDS_INSPECTION', not empty
25	assert self.cd.outcome == 'NEEDS_INSPECTION'	25		assert self.cd.outcome == 'NEEDS_INSPECTION'
26	assert not self.cd.item	26		assert not self.cd.item
27	assert not self.cd.summary	27		assert not self.cd.note
28	assert not self.cd.output	28		assert not self.cd.output
29	assert not self.cd.keyvals	29		assert not self.cd.keyvals
30	assert not self.cd.checkname	30		assert not self.cd.checkname
31		31
32	def test_init_params(self):	32		def test_init_params(self):
33	'''Test instance creation with non-empty parameters'''	33		'''Test instance creation with non-empty parameters'''
34	cd = check.CheckDetail(item='foo-1.2-3.fc20', outcome='FAILED',	34		cd = check.CheckDetail(item='foo-1.2-3.fc20', outcome='FAILED',
35	summary='untested', output=['line1', 'line2'],	35		note='untested', output=['line1', 'line2'],
36	checkname='testcheck', artifact='logfile.log')	36		checkname='testcheck', artifact='logfile.log')
37	assert cd.item == 'foo-1.2-3.fc20'	37		assert cd.item == 'foo-1.2-3.fc20'
38	assert cd.outcome == 'FAILED'	38		assert cd.outcome == 'FAILED'
39	assert cd.summary == 'untested'	39		assert cd.note == 'untested'
40	assert cd.output == ['line1', 'line2']
41	assert cd.checkname == 'testcheck'	40		assert cd.checkname == 'testcheck'
42	assert cd.artifact == 'logfile.log'	41		assert cd.artifact == 'logfile.log'
43		42
44	def test_init_raise(self):	43		def test_init_raise(self):
45	'''Test instance creation that raises an error - invalid parameters'''	44		'''Test instance creation that raises an error - invalid parameters'''
46	with pytest.raises(exc.TaskotronValueError):	45		with pytest.raises(exc.TaskotronValueError):
47	check.CheckDetail(None, outcome='INVALID TYPE')	46		check.CheckDetail(None, outcome='INVALID TYPE')
48		47
49	with pytest.raises(exc.TaskotronValueError):	48		with pytest.raises(exc.TaskotronValueError):
50	check.CheckDetail(None, output='foobar')	49		check.CheckDetail(None, output='foobar')
51		50
		51		with pytest.raises(exc.TaskotronValueError):
		52		check.CheckDetail(None, keyvals='foobar')
		53
52	def test_outcome(self):	54		def test_outcome(self):
53	'''Test 'outcome' attribute'''	55		'''Test 'outcome' attribute'''
54	# default value test	56		# default value test
55	# (already tested in constructor test, but it logically belongs here,	57		# (already tested in constructor test, but it logically belongs here,
56	# so let's include it again, in case constructor test changes)	58		# so let's include it again, in case constructor test changes)
57	assert self.cd.outcome == 'NEEDS_INSPECTION'	59		assert self.cd.outcome == 'NEEDS_INSPECTION'
58		60
59	# test setter method with correct values	61		# test setter method with correct values
▲ Show 20 Lines • Show All 118 Lines • ▼ Show 20 Line(s)
178	def test_create_multi_item_summary_raise(self):	180		def test_create_multi_item_summary_raise(self):
179	'''Test create_multi_item_summary method when it should raise an	181		'''Test create_multi_item_summary method when it should raise an
180	error'''	182		error'''
181	# a single string input is not allowed, only a list of them	183		# a single string input is not allowed, only a list of them
182	with pytest.raises(exc.TaskotronValueError):	184		with pytest.raises(exc.TaskotronValueError):
183	assert check.CheckDetail.create_multi_item_summary('PASSED')	185		assert check.CheckDetail.create_multi_item_summary('PASSED')
184		186
185		187
186	class TestExportTAP(object):	188		class TestExportYAML(object):
187	'''Test @method export_TAP'''	189		'''Test @method export_YAML'''
188	item = 'xchat-0.5-1.fc20'	190		item = 'xchat-0.5-1.fc20'
189	outcome = 'PASSED'	191		outcome = 'PASSED'
190	summary = '5 ERRORS, 10 WARNINGS'	192		note = '5 ERRORS, 10 WARNINGS'
191	report_type = check.ReportType.KOJI_BUILD	193		report_type = check.ReportType.KOJI_BUILD
192	output = '''\
193	xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
194	xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
195	xchat.x86_64: W: no-manual-page-for-binary xchat'''
196	keyvals = {"foo": "bar", "moo": 11}	194		keyvals = {"foo": "bar", "moo": 11}
197	checkname = 'testcheck'	195		checkname = 'testcheck'
198	artifact = 'logfile.log'	196		artifact = 'logfile.log'
199	_internal = {'resultsdb_result_id': 1234}	197		_internal = {'resultsdb_result_id': 1234}
200		198
201	def setup_method(self, method):	199		def setup_method(self, method):
202	self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,	200		self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,
203	summary=self.summary,	201		note=self.note,
204	report_type=self.report_type,	202		report_type=self.report_type,
205	keyvals=self.keyvals,	203		keyvals=self.keyvals,
206	checkname=self.checkname,	204		checkname=self.checkname,
207	artifact=self.artifact)	205		artifact=self.artifact)
208	self.cd._internal = self._internal	206		self.cd._internal = self._internal
209	self.cd.output.append(self.output)
210
211	def _tap_to_yaml(self, rendered_tap):
212	'''Remove all TAP-related contents and return only its YAML block(s)'''
213	lines = rendered_tap.splitlines()[2:]
214
215	# split the output to TAP result lines and YAML blocks
216	tap_lines = [line for line in lines if line.startswith('ok') or
217	line.startswith('not ok')]
218	# remove left padding - 2 spaces
219	yaml_lines = [line[2:] for line in lines if line not in tap_lines]
220		207
221	return '\n'.join(yaml_lines)
222
223	def test_single_tap(self):
224	'''Test export with a single item section. Check generic TAP lines.'''
225	tap_output = check.export_TAP(self.cd)
226	lines = tap_output.splitlines()
227
228	# generic lines must exist
229	assert lines[0].startswith('TAP version 13')
230	assert lines[1].startswith('1..')
231	assert lines[2].startswith('ok ')
232	assert lines[3].strip() == '---'
233	assert lines[-1].strip() == '...'
234		208
235	def test_single_yaml(self):	209		def test_single_yaml(self):
236	'''Test export with a single item section. Check specific YAML lines.'''	210		'''Test export with a single item section.'''
237	tap_output = check.export_TAP(self.cd)	211		yaml_output = check.export_YAML(self.cd)
		212		yaml_obj = yaml.safe_load(yaml_output)
238		213
239	# specific lines must exist	214		assert type(yaml_obj) is dict
240	yaml_block = self._tap_to_yaml(tap_output)	215		assert type(yaml_obj['results']) is list
241	yaml_obj = yaml.safe_load(yaml_block)	216
		217		yaml_obj = yaml_obj['results'][0]
242		218
243	assert yaml_obj['item'] == self.item	219		assert yaml_obj['item'] == self.item
244	assert yaml_obj['outcome'] == self.outcome	220		assert yaml_obj['outcome'] == self.outcome
245	assert yaml_obj['summary'] == self.summary	221		assert yaml_obj['note'] == self.note
246	assert yaml_obj['type'] == self.report_type	222		assert yaml_obj['type'] == self.report_type
247	assert yaml_obj['details']
248	assert yaml_obj['details']['output'] == self.output
249	assert yaml_obj['foo'] == self.keyvals['foo']	223		assert yaml_obj['foo'] == self.keyvals['foo']
250	assert yaml_obj['moo'] == self.keyvals['moo']	224		assert yaml_obj['moo'] == self.keyvals['moo']
251	assert yaml_obj['checkname'] == self.checkname	225		assert yaml_obj['checkname'] == self.checkname
252	assert yaml_obj['_internal'] == self._internal	226		assert yaml_obj['_internal'] == self._internal
253		227
254	def test_invalid_keyvals(self):	228		def test_invalid_keyvals(self):
255	'''Test export with keyvals containing reserved keys.'''	229		'''Test export with keyvals containing reserved keys.'''
256	cd = deepcopy(self.cd)	230		cd = deepcopy(self.cd)
257		231
258	for key in check.RESERVED_KEYS:	232		for key in check.RESERVED_KEYS:
259	cd.keyvals[key] = 'foo'	233		cd.keyvals[key] = 'foo'
260		234
261	tap_output = check.export_TAP(cd)	235		yaml_output = check.export_YAML(cd)
262		236		yaml_obj = yaml.load(yaml_output)['results'][0]
263	# specific lines must exist
264	yaml_block = self._tap_to_yaml(tap_output)
265	yaml_obj = yaml.load(yaml_block)
266		237
267	assert yaml_obj['item'] == self.item	238		assert yaml_obj['item'] == self.item
268	assert yaml_obj['outcome'] == self.outcome	239		assert yaml_obj['outcome'] == self.outcome
269	assert yaml_obj['summary'] == self.summary	240		assert yaml_obj['note'] == self.note
270	assert yaml_obj['type'] == self.report_type	241		assert yaml_obj['type'] == self.report_type
271	assert yaml_obj['details']
272	assert yaml_obj['details']['output'] == self.output
273	assert yaml_obj['foo'] == self.keyvals['foo']	242		assert yaml_obj['foo'] == self.keyvals['foo']
274	assert yaml_obj['moo'] == self.keyvals['moo']	243		assert yaml_obj['moo'] == self.keyvals['moo']
275	assert yaml_obj['checkname'] == self.checkname	244		assert yaml_obj['checkname'] == self.checkname
276	assert yaml_obj['_internal'] == self._internal	245		assert yaml_obj['_internal'] == self._internal
277		246
278	def test_multi(self):	247		def test_multi(self):
279	'''Test export with multiple item sections'''	248		'''Test export with multiple item sections'''
280	cd2 = check.CheckDetail(item='foobar-1.2-3.fc20', outcome='FAILED',	249		cd2 = check.CheckDetail(item='foobar-1.2-3.fc20', outcome='FAILED',
281	summary='dependency error',	250		note='dependency error',
282	report_type=check.ReportType.BODHI_UPDATE)	251		report_type=check.ReportType.BODHI_UPDATE)
283	cd3 = check.CheckDetail(item='f20-updates', outcome='INFO',	252		cd3 = check.CheckDetail(item='f20-updates', outcome='INFO',
284	summary='2 stale updates',	253		note='2 stale updates',
285	report_type=check.ReportType.YUM_REPOSITORY)	254		report_type=check.ReportType.YUM_REPOSITORY)
286	tap_output = check.export_TAP([self.cd, cd2, cd3])	255		yaml_output = check.export_YAML([self.cd, cd2, cd3])
287	lines = tap_output.splitlines()	256		yaml_obj = yaml.safe_load(yaml_output)
288		257
289	assert lines[1].strip() == '1..3'	258		assert len(yaml_obj['results']) == 3
		259		assert yaml_obj['results'][0]['item'] == self.cd.item
		260		assert yaml_obj['results'][1]['item'] == cd2.item
		261		assert yaml_obj['results'][2]['item'] == cd3.item
290		262
291	yaml_blocks = self._tap_to_yaml(tap_output)	263		def test_invalid_missing_item(self):
292
293	# test that we have a correct number of TAP lines
294	tap_lines = [line for line in lines if line.startswith('ok') or
295	line.startswith('not ok')]
296	assert len(tap_lines) == 3
297
298	# test that we have a correct number (and rough contents) of YAML blocks
299	yaml_gen = yaml.safe_load_all(yaml_blocks)
300	for index, yaml_obj in enumerate(yaml_gen):
301	assert 'item' in yaml_obj
302	assert index == 2
303
304	def test_invalid(self):
305	'''Test invalid input parameters'''	264		'''Test invalid input parameters'''
306	with pytest.raises(exc.TaskotronValueError):	265		with pytest.raises(exc.TaskotronValueError):
307	self.cd.item = None	266		self.cd.item = None
308	check.export_TAP(self.cd)	267		check.export_YAML(self.cd)
309		268
310	def test_minimal(self):	269		def test_minimal(self):
311	'''Test that empty CheckDetail values don't produce empty TAP lines,	270		'''Test that empty CheckDetail values don't produce empty YAML lines,
312	for example 'summary:' should not be present if there's no	271		for example 'note:' should not be present if there's no
313	CheckDetail.summary'''	272		CheckDetail.note'''
314	cd = check.CheckDetail('foo')	273		cd = check.CheckDetail('foo')
315	tap_output = check.export_TAP(cd)	274		yaml_output = check.export_YAML(cd)
316	yaml_block = self._tap_to_yaml(tap_output)	275		yaml_obj = yaml.safe_load(yaml_output)['results'][0]
317	yaml_obj = yaml.safe_load(yaml_block)
318		276
319	# the output should look like this:	277		# the output should look like this:
320	#	278		#
321	# TAP Version 13	279		# results:
322	# 1..1	280		# - item: XXX
323	# ok - $CHECKNAME for XXX	281		# outcome: XXX
324	# ---
325	# item: XXX
326	# outcome: XXX
327	# ...
328	#	282		#
329	# ('outcome:' is technically not a mandatory line, but with CheckDetail	283		# ('outcome:' is technically not a mandatory line, but with CheckDetail
330	# export we produce it every time)	284		# export we produce it every time)
331	assert len(yaml_obj) == 2	285		assert len(yaml_obj) == 2
332		286
333	def test_tap_ok(self):
334	'''Test that TAP header contains 'ok' for appropriate outcomes'''
335	# by default the outcome is PASSED
336	assert check.export_TAP(self.cd).splitlines()[2].startswith('ok')
337	self.cd.outcome = 'INFO'
338	assert check.export_TAP(self.cd).splitlines()[2].startswith('ok')
339
340	def test_tap_not_ok(self):
341	'''Test that TAP header contains 'not ok' for appropriate outcomes'''
342	self.cd.outcome = 'FAILED'
343	assert check.export_TAP(self.cd).splitlines()[2].startswith('not ok')
344	self.cd.outcome = 'NEEDS_INSPECTION'
345	assert check.export_TAP(self.cd).splitlines()[2].startswith('not ok')
346	self.cd.outcome = 'ABORTED'
347	assert check.export_TAP(self.cd).splitlines()[2].startswith('not ok')
348	self.cd.outcome = 'CRASHED'
349	assert check.export_TAP(self.cd).splitlines()[2].startswith('not ok')
350		287
351	def test_custom_report_type(self):	288		def test_custom_report_type(self):
352	'''Test that user can specify a custom CheckDetail.report_type'''	289		'''Test that user can specify a custom CheckDetail.report_type'''
353	self.cd.report_type = 'My Custom Type'	290		self.cd.report_type = 'My Custom Type'
354	tap_output = check.export_TAP(self.cd)	291		yaml_output = check.export_YAML(self.cd)
355	yaml_block = self._tap_to_yaml(tap_output)	292		yaml_obj = yaml.safe_load(yaml_output)['results'][0]
356	yaml_obj = yaml.safe_load(yaml_block)
357		293
358	assert yaml_obj['type'] == self.cd.report_type	294		assert yaml_obj['type'] == self.cd.report_type
359		295
360	def test_checkname(self):
361	'''Test that replacing $CHECKNAME works'''
362	# by default the outcome is PASSED
363	output = check.export_TAP(self.cd)
364	assert "ok - $CHECKNAME for" in output
365		296
366	output = check.export_TAP(self.cd, "my_beautifull_check_name")	297		class TestImportYAML(object):
367	assert "ok - $CHECKNAME for" not in output	298		'''Test @method import_YAML'''
368	assert "ok - my_beautifull_check_name for" in output
369
370
371	class TestImportTAP(object):
372	'''Test @method import_TAP'''
373	item = 'xchat-0.5-1.fc20'	299		item = 'xchat-0.5-1.fc20'
374	outcome = 'PASSED'	300		outcome = 'PASSED'
375	summary = '5 ERRORS, 10 WARNINGS'	301		note = '5 ERRORS, 10 WARNINGS'
376	report_type = check.ReportType.KOJI_BUILD	302		report_type = check.ReportType.KOJI_BUILD
377	output = '''\
378	xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
379	xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
380	xchat.x86_64: W: no-manual-page-for-binary xchat'''
381	keyvals = {"foo": "bar", "moo": 11}	303		keyvals = {"foo": "bar", "moo": 11}
382	checkname = 'testcheck'	304		checkname = 'testcheck'
383	artifact = 'logfile.log'	305		artifact = 'logfile.log'
384	_internal = {'resultsdb_result_id': 1234}	306		_internal = {'resultsdb_result_id': 1234}
385		307
386	tap_output = """	308		yaml_output = """
387	TAP version 13	309		results:
388	1..1	310		- item: xchat-0.5-1.fc20
389	ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
390	---
391	details:
392	output: \|-
393	xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
394	xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
395	xchat.x86_64: W: no-manual-page-for-binary xchat
396	item: xchat-0.5-1.fc20
397	outcome: PASSED	311		outcome: PASSED
398	artifact: logfile.log	312		artifact: logfile.log
399	summary: 5 ERRORS, 10 WARNINGS	313		note: 5 ERRORS, 10 WARNINGS
400	type: koji_build	314		type: koji_build
401	foo: bar	315		foo: bar
402	moo: 11	316		moo: 11
403	checkname: testcheck	317		checkname: testcheck
404	_internal:	318		_internal:
405	resultsdb_result_id: 1234	319		resultsdb_result_id: 1234
406	..."""	320		"""
407		321
408	def setup_method(self, method):	322		def setup_method(self, method):
409	self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,	323		self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,
410	summary=self.summary,	324		note=self.note,
411	report_type=self.report_type,	325		report_type=self.report_type,
412	keyvals=self.keyvals,	326		keyvals=self.keyvals,
413	checkname=self.checkname,	327		checkname=self.checkname,
414	artifact=self.artifact)	328		artifact=self.artifact)
415	self.cd._internal = self._internal	329		self.cd._internal = self._internal
416	self.cd.output.append(self.output)
417		330
418	def test_import(self):	331		def test_import(self):
419	cds = check.import_TAP(self.tap_output)	332		cds = check.import_YAML(self.yaml_output)
420		333
421	assert len(cds) == 1	334		assert len(cds) == 1
422	assert isinstance(cds[0], check.CheckDetail)	335		assert isinstance(cds[0], check.CheckDetail)
423		336
424	cd = cds[0]	337		cd = cds[0]
425	assert cd.item == self.cd.item	338		assert cd.item == self.cd.item
426	assert cd.outcome == self.cd.outcome	339		assert cd.outcome == self.cd.outcome
427	assert cd.summary == self.cd.summary	340		assert cd.note == self.cd.note
428	assert cd.report_type == self.cd.report_type	341		assert cd.report_type == self.cd.report_type
429	assert cd.output == self.cd.output
430	assert cd.keyvals == self.keyvals	342		assert cd.keyvals == self.keyvals
431	assert cd.checkname == self.checkname	343		assert cd.checkname == self.checkname
432	assert cd.artifact == self.artifact	344		assert cd.artifact == self.artifact
433	assert cd._internal == self._internal	345		assert cd._internal == self._internal
434		346
435	def test_import_exported(self):	347		def test_import_exported(self):
436	tap_output = check.export_TAP(self.cd)	348		yaml_output = check.export_YAML(self.cd)
437		349
438	cds = check.import_TAP(tap_output)	350		cds = check.import_YAML(yaml_output)
439		351
440	assert len(cds) == 1	352		assert len(cds) == 1
441	assert isinstance(cds[0], check.CheckDetail)	353		assert isinstance(cds[0], check.CheckDetail)
442		354
443	cd = cds[0]	355		cd = cds[0]
444	assert cd.item == self.cd.item	356		assert cd.item == self.cd.item
445	assert cd.outcome == self.cd.outcome	357		assert cd.outcome == self.cd.outcome
446	assert cd.summary == self.cd.summary	358		assert cd.note == self.cd.note
447	assert cd.report_type == self.cd.report_type	359		assert cd.report_type == self.cd.report_type
448	assert cd.output == self.cd.output
449	assert cd.keyvals == self.keyvals	360		assert cd.keyvals == self.keyvals
450	assert cd.checkname == self.checkname	361		assert cd.checkname == self.checkname
451	assert cd.artifact == self.artifact	362		assert cd.artifact == self.artifact
452	assert cd._internal == self._internal	363		assert cd._internal == self._internal
453		364
		365		def test_invalid_yaml(self):
		366		yaml_output = "key:value\nkey: value"
		367		with pytest.raises(exc.TaskotronValueError):
		368		check.import_YAML(yaml_output)
454		369
455	def test_invalid_tap(self):	370		def test_invalid_yaml_root_element_type(self):
456	tap_output = """	371		yaml_output = "[]"
457	TAP version 13
458	ok 2 foo
459	ok 1 bar"""
460	with pytest.raises(exc.TaskotronValueError):	372		with pytest.raises(exc.TaskotronValueError):
461	check.import_TAP(tap_output)	373		check.import_YAML(yaml_output)
462
463	def test_invalid_tap_multi_header(self):
464	#simplified test data from phab/T205
465	tap_output_multi_header = """
466	TAP version 13
467	1..1
468	ok - $CHECKNAME for XXX
469	---
470	item: XXX
471	...
472	TAP version 13
473	1..1
474	ok - $CHECKNAME for YYY
475	---
476	item: YYY
477	...
478	"""
479	with pytest.raises(ValueError):
480	check.import_TAP(tap_output_multi_header)
481
482	def test_no_yaml(self):
483	tap_output = '''
484	TAP version 13
485	1..1
486	ok'''
487	cds = check.import_TAP(tap_output)
488	assert len(cds) == 1
489	assert isinstance(cds[0], check.CheckDetail)
490		374
491	cd = cds[0]	375		def test_invalid_yaml_missing_results_key(self):
492	assert not cd.item	376		yaml_output = "{}"
493	assert cd.outcome == 'PASSED'	377		with pytest.raises(exc.TaskotronValueError):
494	assert not cd.summary	378		check.import_YAML(yaml_output)
495	assert not cd.report_type
496	assert not cd.output
497	assert not cd.keyvals
498	assert not cd.artifact
499	assert not cd._internal
500
501
502	def test_empty_yaml(self):
503	tap_output = '''
504	TAP version 13
505	1..1
506	not ok
507	---
508	...'''
509	cds = check.import_TAP(tap_output)
510	assert len(cds) == 1
511	assert isinstance(cds[0], check.CheckDetail)
512		379
513	cd = cds[0]	380		def test_invalid_yaml_wrong_results_type(self):
514	assert not cd.item	381		yaml_output = "{'results': {}}"
515	assert cd.outcome == 'FAILED'	382		with pytest.raises(exc.TaskotronValueError):
516	assert not cd.summary	383		check.import_YAML(yaml_output)
517	assert not cd.report_type	384
518	assert not cd.keyvals	385		def test_no_results(self):
519	assert not cd.output	386		yaml_output = "{'results': []}"
520	assert not cd.checkname	387		cds = check.import_YAML(yaml_output)
521	assert not cd.artifact	388		assert len(cds) == 0
522	assert not cd._internal	389
		390		yaml_output = "{'results': null}"
		391		cds = check.import_YAML(yaml_output)
		392		assert len(cds) == 0

View Options

testing/test_exitcode_directive.py

Show All 9 Lines
10		10
11	@pytest.mark.usefixtures("setup")	11	@pytest.mark.usefixtures("setup")
12	class TestReturncode(object):	12	class TestReturncode(object):
13		13
14	@pytest.fixture	14	@pytest.fixture
15	def setup(self):	15	def setup(self):
16	self.test_env_data = {}	16	self.test_env_data = {}
17	self.test_input_data = {}	17	self.test_input_data = {}
18	self.test_tap = """	18	self.test_yaml = """
19	TAP version 13	19	results:
20	1..4	20	- item: Hello World P
21	ok - $CHECKNAME for Hello World P	21	outcome: PASSED
22	---	22	- item: Hello World I
23	item: Hello World P	23	outcome: INFO
24	outcome: PASSED	24	- item: Hello World F
25	...	25	outcome: FAILED
26	ok - $CHECKNAME for Hello World I	26	- item: Hello World P
27	---	27	outcome: PASSED
28	item: Hello World I
29	outcome: INFO
30	...
31	not ok - $CHECKNAME for Hello World F # FAIL
32	---
33	item: Hello World F
34	outcome: FAILED
35	...
36	ok - $CHECKNAME for Hello World P
37	---
38	item: Hello World P
39	outcome: PASSED
40	...
41	"""	28	"""
42		29
43	def test_tap_last(self):	30	def test_result_last(self):
44	self.test_input_data['tap_last'] = self.test_tap	31	self.test_input_data['result_last'] = self.test_yaml
45	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,	32	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,
46	self.test_env_data)	33	self.test_env_data)
47	assert exitcode == exitcode_directive.SUCCESS	34	assert exitcode == exitcode_directive.SUCCESS
48		35
49	def test_tap_worst(self):	36	def test_result_worst(self):
50	self.test_input_data['tap_worst'] = self.test_tap	37	self.test_input_data['result_worst'] = self.test_yaml
51	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,	38	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,
52	self.test_env_data)	39	self.test_env_data)
53	assert exitcode == exitcode_directive.FAILURE	40	assert exitcode == exitcode_directive.FAILURE
54		41
55	def test_bad_input(self):	42	def test_bad_input(self):
56	with pytest.raises(TaskotronDirectiveError):	43	with pytest.raises(TaskotronDirectiveError):
57	exitcode_directive.ExitcodeDirective().process(self.test_input_data,	44	exitcode_directive.ExitcodeDirective().process(self.test_input_data,
58	self.test_env_data)	45	self.test_env_data)
59		46
60	def test_bad_tap(self):	47	def test_bad_yaml(self):
61	self.test_tap = "noworky"	48	self.test_yaml = "key:value\nkey2: value"
62	self.test_input_data['tap_worst'] = self.test_tap	49	self.test_input_data['result_worst'] = self.test_yaml
63	with pytest.raises(TaskotronValueError):	50	with pytest.raises(TaskotronValueError):
64	exitcode_directive.ExitcodeDirective().process(self.test_input_data,	51	exitcode_directive.ExitcodeDirective().process(self.test_input_data,
65	self.test_env_data)	52	self.test_env_data)
66		53
67	def test_empty_tap_worst(self):	54	def test_empty_result_worst(self):
68	self.test_tap = """	55	self.test_yaml = "{results:[]}"
69	TAP version 13	56	self.test_input_data['result_worst'] = self.test_yaml
70	1..0
71	"""
72	self.test_input_data['tap_worst'] = self.test_tap
73	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,	57	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,
74	self.test_env_data)	58	self.test_env_data)
75	assert exitcode == exitcode_directive.SUCCESS	59	assert exitcode == exitcode_directive.SUCCESS
76		60
77	def test_empty_tap_last(self):	61	def test_empty_result_last(self):
78	self.test_tap = """	62	self.test_yaml = "{results:[]}"
79	TAP version 13	63	self.test_input_data['result_last'] = self.test_yaml
80	1..0
81	"""
82	self.test_input_data['tap_last'] = self.test_tap
83	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,	64	exitcode = exitcode_directive.ExitcodeDirective().process(self.test_input_data,
84	self.test_env_data)	65	self.test_env_data)
85	assert exitcode == exitcode_directive.SUCCESS	66	assert exitcode == exitcode_directive.SUCCESS

View Options

testing/test_resultsdb_directive.py

Show All 21 Lines
22	@pytest.mark.usefixtures('setup')	22		@pytest.mark.usefixtures('setup')
23	class TestResultsdbReport():	23		class TestResultsdbReport():
24		24
25	@pytest.fixture	25		@pytest.fixture
26	def setup(self, monkeypatch):	26		def setup(self, monkeypatch):
27	'''Run this before every test invocation'''	27		'''Run this before every test invocation'''
28		28
29	self.cd = check.CheckDetail(	29		self.cd = check.CheckDetail(
30	item = 'foo_bar',	30		item='foo_bar',
31	report_type = check.ReportType.KOJI_BUILD,	31		report_type=check.ReportType.KOJI_BUILD,
32	outcome = 'NEEDS_INSPECTION',	32		outcome='NEEDS_INSPECTION',
33	summary = 'foo_bar summary',	33		note='foo_bar note',
34	output = ["foo\nbar"],	34		output=["foo\nbar"],
35	keyvals = {"foo": "moo1", "bar": "moo2"},	35		keyvals={"foo": "moo1", "bar": "moo2"},
36	)	36		)
37		37
38	self.tap = check.export_TAP(self.cd)	38		self.yaml = check.export_YAML(self.cd)
39		39
40	self.ref_input = {'results': self.tap}	40		self.ref_input = {'results': self.yaml}
41	self.ref_envdata = {	41		self.ref_envdata = {
42	'resultsdb_job_id': 1,	42		'resultsdb_job_id': 1,
43	'checkname': 'test_resultsdb_report',	43		'checkname': 'test_resultsdb_report',
44	'jobid': 'all/123',	44		'jobid': 'all/123',
45	'uuid': 'c25237a4-b6b3-11e4-b98a-3c970e018701',	45		'uuid': 'c25237a4-b6b3-11e4-b98a-3c970e018701',
46	'artifactsdir': '/some/directory/',	46		'artifactsdir': '/some/directory/',
47	}	47		}
48		48
Show All 27 Lines
76	def test_config_reporting_disabled(self):	76		def test_config_reporting_disabled(self):
77	"""Checks config options can disable reporting."""	77		"""Checks config options can disable reporting."""
78	conf = config.get_config()	78		conf = config.get_config()
79		79
80	for r_e, r_t_r in ((False, False), (False, True), (True, False)):	80		for r_e, r_t_r in ((False, False), (False, True), (True, False)):
81	conf.reporting_enabled = r_e	81		conf.reporting_enabled = r_e
82	conf.report_to_resultsdb = r_t_r	82		conf.report_to_resultsdb = r_t_r
83		83
84	tap = self.test_rdb.process(self.ref_input, self.ref_envdata)	84		yaml = self.test_rdb.process(self.ref_input, self.ref_envdata)
85	cds = check.import_TAP(tap)	85		cds = check.import_YAML(yaml)
86	# return value should be the same TAP	86		my_cd = check.import_YAML(check.export_YAML(self.cd))
		87		# return value should be the same YAML
87	assert len(cds) == 1	88		assert len(cds) == 1
88	assert cds[0].__dict__ == self.cd.__dict__	89		assert cds[0].__dict__ == my_cd[0].__dict__
89		90
90	# no call should have been made	91		# no call should have been made
91	assert len(self.stub_rdb.calls()) == 0	92		assert len(self.stub_rdb.calls()) == 0
92		93
93	config._config = None	94		config._config = None
94		95
95	def test_missing_checkname(self):	96		def test_missing_checkname(self):
96	"""Checks if missing checkname raises exception"""	97		"""Checks if missing checkname raises exception"""
97	with pytest.raises(TaskotronDirectiveError):	98		with pytest.raises(TaskotronDirectiveError):
98	envdata = deepcopy(self.ref_envdata)	99		envdata = deepcopy(self.ref_envdata)
99	del(envdata['checkname'])	100		del(envdata['checkname'])
100		101
101	self.test_rdb.process(self.ref_input, envdata)	102		self.test_rdb.process(self.ref_input, envdata)
102		103
103	def test_failed_tap_import(self, monkeypatch):	104		def test_failed_yaml_import(self, monkeypatch):
104	"""Checks if failed TAP import raises exception"""	105		"""Checks if failed YAML import raises exception"""
105	monkeypatch.setattr(check, 'import_TAP',	106		monkeypatch.setattr(check, 'import_YAML',
106	exception_raiser(TaskotronValueError("Testing Error")))	107		exception_raiser(TaskotronValueError("Testing Error")))
107		108
108	with pytest.raises(TaskotronDirectiveError):	109		with pytest.raises(TaskotronDirectiveError):
109	self.test_rdb.process(self.ref_input, self.ref_envdata)	110		self.test_rdb.process(self.ref_input, self.ref_envdata)
110		111
111	def test_tap_missing_item(self):	112		def test_yaml_missing_item(self):
112	"""Checks if missing item raises exception"""	113		"""Checks if missing item raises exception"""
113	tap = ""	114		yaml = ""
114	for line in self.ref_input['results'].split('\n'):	115		for line in self.ref_input['results'].split('\n'):
115	if 'item:' in line:	116		if 'item:' in line:
116	continue	117		continue
117	tap += line + '\n'	118		yaml += line + '\n'
118		119
119	with pytest.raises(TaskotronDirectiveError):	120		with pytest.raises(TaskotronDirectiveError):
120	self.test_rdb.process({"results": tap}, self.ref_envdata)	121		self.test_rdb.process({"results": yaml}, self.ref_envdata)
121		122
122	def test_tap_missing_type(self):	123		def test_yaml_missing_type(self):
123	"""Checks if missing type raises exception"""	124		"""Checks if missing type raises exception"""
124	tap = ""	125		yaml = ""
125	for line in self.ref_input['results'].split('\n'):	126		for line in self.ref_input['results'].split('\n'):
126	if 'type:' in line:	127		if 'type:' in line:
127	continue	128		continue
128	tap += line + '\n'	129		yaml += line + '\n'
129		130
130	with pytest.raises(TaskotronDirectiveError):	131		with pytest.raises(TaskotronDirectiveError):
131	self.test_rdb.process({"results": tap}, self.ref_envdata)	132		self.test_rdb.process({"results": yaml}, self.ref_envdata)
132		133
133	def test_report(self):	134		def test_report(self):
134	"""Checks whether TAP is correctly mapped to the reporting method's	135		"""Checks whether YAML is correctly mapped to the reporting method's
135	arguments."""	136		arguments."""
136		137
137	self.test_rdb.process(self.ref_input, self.ref_envdata)	138		self.test_rdb.process(self.ref_input, self.ref_envdata)
138		139
139	# Given the input data, the resultsdb should be called once, and only	140		# Given the input data, the resultsdb should be called once, and only
140	# once, calling "create_result".	141		# once, calling "create_result".
141	# This assert failing either means that more calls were added in the	142		# This assert failing either means that more calls were added in the
142	# source code, or that a bug is present, and "create_result" is	143		# source code, or that a bug is present, and "create_result" is
Show All 15 Lines
158	ref_builder, ref_jobid = self.ref_envdata['jobid'].split('/')	159		ref_builder, ref_jobid = self.ref_envdata['jobid'].split('/')
159	ref_log_url = '%s/builders/%s/builds/%s/steps/%s/logs/stdio' %\	160		ref_log_url = '%s/builders/%s/builds/%s/steps/%s/logs/stdio' %\
160	(self.conf.taskotron_master, ref_builder, ref_jobid,	161		(self.conf.taskotron_master, ref_builder, ref_jobid,
161	self.conf.buildbot_task_step)	162		self.conf.buildbot_task_step)
162		163
163	assert call_data['job_id'] == self.ref_jobid	164		assert call_data['job_id'] == self.ref_jobid
164	assert call_data['testcase_name'] == self.ref_envdata['checkname']	165		assert call_data['testcase_name'] == self.ref_envdata['checkname']
165	assert call_data['outcome'] == self.cd.outcome	166		assert call_data['outcome'] == self.cd.outcome
166	assert call_data['summary'] == self.cd.summary	167		assert call_data['summary'] == self.cd.note
167	assert call_data['log_url'] == ref_log_url	168		assert call_data['log_url'] == ref_log_url
168	assert call_data['item'] == self.cd.item	169		assert call_data['item'] == self.cd.item
169	assert call_data['type'] == self.cd.report_type	170		assert call_data['type'] == self.cd.report_type
170		171
171	assert 'output' not in call_data.keys()	172		assert 'output' not in call_data.keys()
172		173
173	for key in self.cd.keyvals.keys():	174		for key in self.cd.keyvals.keys():
174	assert call_data[key] == self.cd.keyvals[key]	175		assert call_data[key] == self.cd.keyvals[key]
Show All 20 Lines		186	def test_get_artifact_path_errors(self, monkeypatch):
195	assert self.test_rdb.get_artifact_path(artifactsdir, artifact) is None	196		assert self.test_rdb.get_artifact_path(artifactsdir, artifact) is None
196		197
197	def test_report_artifact_in_log_url(self, monkeypatch):	198		def test_report_artifact_in_log_url(self, monkeypatch):
198	"""Checks whether artifact is correctly mapped to log_url"""	199		"""Checks whether artifact is correctly mapped to log_url"""
199	cd = check.CheckDetail(	200		cd = check.CheckDetail(
200	item='foo_bar',	201		item='foo_bar',
201	report_type=check.ReportType.KOJI_BUILD,	202		report_type=check.ReportType.KOJI_BUILD,
202	outcome='NEEDS_INSPECTION',	203		outcome='NEEDS_INSPECTION',
203	summary='foo_bar summary',	204		note='foo_bar note',
204	output=["foo\nbar"],	205		output=["foo\nbar"],
205	keyvals={"foo": "moo1", "bar": "moo2"},	206		keyvals={"foo": "moo1", "bar": "moo2"},
206	artifact='digest/logs/logfile.log'	207		artifact='digest/logs/logfile.log'
207	)	208		)
208	monkeypatch.setattr(self.test_rdb, 'get_artifact_path', lambda *x: 'digest/logs/logfile.log')	209		monkeypatch.setattr(self.test_rdb, 'get_artifact_path', lambda *x: 'digest/logs/logfile.log')
209		210
210	tap = check.export_TAP(cd)	211		yaml = check.export_YAML(cd)
211	ref_input = {'results': tap}	212		ref_input = {'results': yaml}
212		213
213	self.test_rdb.process(ref_input, self.ref_envdata)	214		self.test_rdb.process(ref_input, self.ref_envdata)
214		215
215	# Given the input data, the resultsdb should be called once, and only	216		# Given the input data, the resultsdb should be called once, and only
216	# once, calling "create_result".	217		# once, calling "create_result".
217	# This assert failing either means that more calls were added in the	218		# This assert failing either means that more calls were added in the
218	# source code, or that a bug is present, and "create_result" is	219		# source code, or that a bug is present, and "create_result" is
219	# called multiple times.	220		# called multiple times.
▲ Show 20 Lines • Show All 70 Lines • ▼ Show 20 Line(s)		288	def test_ensure_testcase_cached(self):
290	self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])	291		self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
291	self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])	292		self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
292		293
293	rdb_calls = self.stub_rdb.calls()	294		rdb_calls = self.stub_rdb.calls()
294		295
295	assert len(rdb_calls) == 1	296		assert len(rdb_calls) == 1
296	assert rdb_calls[0][0] == 'get_testcase'	297		assert rdb_calls[0][0] == 'get_testcase'
297		298
298	def test_tap_output(self):	299		def test_yaml_output(self):
299	"""Checks whether TAP is correctly mapped to the reporting method's	300		"""Checks whether YAML is correctly mapped to the reporting method's
300	arguments."""	301		arguments."""
301		302
302	output = self.test_rdb.process(self.ref_input, self.ref_envdata)	303		output = self.test_rdb.process(self.ref_input, self.ref_envdata)
303	data = check.import_TAP(output)	304		data = check.import_YAML(output)
304		305
305	assert data[0]._internal['resultsdb_result_id'] == 1234	306		assert data[0]._internal['resultsdb_result_id'] == 1234

View Options

testing/test_runner.py

Show First 20 Lines • Show All 150 Lines • ▼ Show 20 Line(s)		150	with pytest.raises(exc.TaskotronYamlError):
151	test_runner._load_directive(self.ref_directivename, self.ref_dir)	151		test_runner._load_directive(self.ref_directivename, self.ref_dir)
152		152
153	class TestRunnerSingleAction():	153		class TestRunnerSingleAction():
154		154
155	def setup_method(self, method):	155		def setup_method(self, method):
156	self.ref_taskdata = {}	156		self.ref_taskdata = {}
157	self.ref_inputdata = {}	157		self.ref_inputdata = {}
158		158
159	self.ref_tap_passed = """	159		self.ref_yaml_passed = """
160	TAP version 13	160		results:
161	1..1	161		- item: Hello World P
162	ok - foo for Hello World P	162		outcome: PASSED
163	---
164	item: Hello World P
165	outcome: PASSED
166	...
167	"""	163		"""
168	self.ref_tap_failed = """	164		self.ref_yaml_failed = """
169	TAP version 13	165		results:
170	1..1	166		- item: Hello World P
171	not ok - foo for Hello World P # FAIL	167		outcome: FAILED
172	---
173	item: Hello World P
174	outcome: FAILED
175	...
176
177	"""	168		"""
178		169
179		170
180	def test_runner_extract_directive_name(self):	171		def test_runner_extract_directive_name(self):
181	ref_directivename = 'dummy'	172		ref_directivename = 'dummy'
182	ref_action = {'name': 'test action',	173		ref_action = {'name': 'test action',
183	ref_directivename: {'result': 'pass',	174		ref_directivename: {'result': 'pass',
184	'msg': 'dummy message'}}	175		'msg': 'dummy message'}}
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Line(s)		216	def test_runner_do_single_dummy_action_pass_export_rendered_msg(self):
234	test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)	225		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
235	test_runner.working_data[ref_messagename] = ref_message	226		test_runner.working_data[ref_messagename] = ref_message
236		227
237	test_runner.do_single_action(ref_action)	228		test_runner.do_single_action(ref_action)
238		229
239	assert test_runner.working_data[ref_exportname] == ref_message	230		assert test_runner.working_data[ref_exportname] == ref_message
240		231
241	def test_runner_do_single_dummy_action_exitcode_None(self):	232		def test_runner_do_single_dummy_action_exitcode_None(self):
242	ref_action = {'name': 'Return exitcode', 'exitcode': {'tap_last': self.ref_tap_passed}}	233		ref_action = {'name': 'Return exitcode', 'exitcode': {'result_last': self.ref_yaml_passed}}
243		234
244	test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)	235		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
245		236
246	test_runner.do_single_action(ref_action)	237		test_runner.do_single_action(ref_action)
247		238
248	assert test_runner.exitcode == exitcode_directive.SUCCESS	239		assert test_runner.exitcode == exitcode_directive.SUCCESS
249		240
250	def test_runner_do_single_dummy_action_exitcode_success(self):	241		def test_runner_do_single_dummy_action_exitcode_success(self):
251	ref_action = {'name': 'Return exitcode', 'exitcode': {'tap_last': self.ref_tap_passed}}	242		ref_action = {'name': 'Return exitcode', 'exitcode': {'result_last': self.ref_yaml_passed}}
252		243
253	test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)	244		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
254		245
255	test_runner.do_single_action(ref_action)	246		test_runner.do_single_action(ref_action)
256		247
257	assert test_runner.exitcode == exitcode_directive.SUCCESS	248		assert test_runner.exitcode == exitcode_directive.SUCCESS
258		249
259	def test_runner_do_single_dummy_action_exitcode_failure(self):	250		def test_runner_do_single_dummy_action_exitcode_failure(self):
260	ref_action = {'name': 'Return exitcode', 'exitcode': {'tap_last': self.ref_tap_failed}}	251		ref_action = {'name': 'Return exitcode', 'exitcode': {'result_last': self.ref_yaml_failed}}
261		252
262	test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)	253		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
263		254
264	test_runner.do_single_action(ref_action)	255		test_runner.do_single_action(ref_action)
265		256
266	assert test_runner.exitcode == exitcode_directive.FAILURE	257		assert test_runner.exitcode == exitcode_directive.FAILURE
267		258
268	def test_runner_do_single_dummy_action_exitcode_failure_overwrite_success(self):	259		def test_runner_do_single_dummy_action_exitcode_failure_overwrite_success(self):
269	ref_action = {'name': 'Return exitcode', 'exitcode': {'tap_last': self.ref_tap_failed}}	260		ref_action = {'name': 'Return exitcode', 'exitcode': {'result_last': self.ref_yaml_failed}}
270		261
271	test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)	262		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
272	test_runner.exitcode = exitcode_directive.SUCCESS	263		test_runner.exitcode = exitcode_directive.SUCCESS
273	test_runner.do_single_action(ref_action)	264		test_runner.do_single_action(ref_action)
274		265
275	assert test_runner.exitcode == exitcode_directive.FAILURE	266		assert test_runner.exitcode == exitcode_directive.FAILURE
276		267
277	def test_runner_do_single_dummy_action_exitcode_success_not_overwrite_failure(self):	268		def test_runner_do_single_dummy_action_exitcode_success_not_overwrite_failure(self):
278	ref_action = {'name': 'Return exitcode', 'exitcode': {'tap_last': self.ref_tap_passed}}	269		ref_action = {'name': 'Return exitcode', 'exitcode': {'result_last': self.ref_yaml_passed}}
279		270
280	test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)	271		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
281	test_runner.exitcode = exitcode_directive.FAILURE	272		test_runner.exitcode = exitcode_directive.FAILURE
282	test_runner.do_single_action(ref_action)	273		test_runner.do_single_action(ref_action)
283		274
284	assert test_runner.exitcode == exitcode_directive.FAILURE	275		assert test_runner.exitcode == exitcode_directive.FAILURE
285		276
286		277
▲ Show 20 Lines • Show All 235 Lines • Show Last 20 Lines

Switched from TAP to YAMLClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 1534

docs/source/index.rst

docs/source/resultyaml.rst

docs/source/tapformat.rst

docs/source/writingtasks.rst

libtaskotron.spec

libtaskotron/check.py

libtaskotron/directives/bodhi_comment_directive.py

libtaskotron/directives/exitcode_directive.py

libtaskotron/directives/python_directive.py

libtaskotron/directives/resultsdb_directive.py

readme.rst

requirements.txt

testing/test_bodhi_comment_directive.py

testing/test_check.py

testing/test_exitcode_directive.py

testing/test_resultsdb_directive.py

testing/test_runner.py

Switched from TAP to YAML
ClosedPublic
Actions