implement check.export_TAP()
ClosedPublic
Actions

Authored by kparal on Mar 14 2014, 2:29 PM.

Details

Reviewers

tflink
jskladan

Maniphest Tasks

T88: Generate TAP output from CheckDetail
T97: Add CheckDetail.getTAP()

Commits

rLTRN23d387af43f9: implement check.export_TAP()

Summary

Rename some CheckDetail attributes to match TAP and ResultsDB
terminology:
CheckDetail.name -> CheckDetail.item
CheckDetail.result -> CheckDetail.outcome
CheckDetail.result_order -> CheckDetail.outcome_priority
CheckDetail.update_result -> CheckDetail.update_outcome

Add new attribute CheckDetail.report_type and introduce enum ReportType.

Example TAP output from 3 CheckDetail instances:

ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
  ---
  details:
    output: |-
      xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
      xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
      xchat.x86_64: W: no-manual-page-for-binary xchat
  item: xchat-0.5-1.fc20
  outcome: PASSED
  summary: 5 ERRORS, 10 WARNINGS
  type: koji_build
  ...
not ok - $CHECKNAME for Bodhi update foobar-1.2-3.fc20  # FAIL 
  ---
  item: foobar-1.2-3.fc20
  outcome: FAILED
  summary: dependency error
  type: bodhi_update
  ...
ok - $CHECKNAME for Yum repository f20-updates
  ---
  item: f20-updates
  outcome: INFO
  summary: 2 stale updates
  type: yum_repository
  ...

I consider this to be a preliminary review. There are still some unsolved issues which might or might not get included in this patch.

a) the libraries can't access the name of the currently running check, only directives can. From that reason I place $CHECKNAME in the TAP result line. Of course, we don't really need to have the name there, it just makes it a bit more readable. This either needs to be solved in a different ticket or the word needs to be removed.

b) the TAP structure is not yet finalized, but it's something that @jskladan proposed in T84

c) I'm not fully convinced that result -> outcome rename is intuitive. On the other hand, ResultsDB uses "result" differently, it's a whole object which contains 'outcome' as an attribute:
https://fedoraproject.org/wiki/AutoQA_resultsdb_schema
http://resultsdb.qa.fedoraproject.org/resultsdb_frontend/jobs
So I'd like to stay consistent with our ResultsDB UI. Maybe we should even rename whole CheckDetail to CheckResult?

d) Maybe export_TAP() should not be in check module, but we should rather move it to tap module as tap.fromCheckDetail(). Is that more logical or not? But in that case we need to solve T101 first.

e) the TAP output produced by mcepl's bayeux library is not fully matching the spec (for example there should be no # FAIL directive in the second output, also the - separator between ok and rest of the line is redundant. We might want to adjust it in the future.

Test Plan

Lots of new unit tests.

Diff Detail

Lint

Lint Skipped

Unit

Unit Tests Skipped

a) the libraries can't access the name of the currently running check, only directives can. From that reason I place $CHECKNAME in the TAP result line. Of course, we don't really need to have the name there, it just makes it a bit more readable. This either needs to be solved in a different ticket or the word needs to be removed.

It's something that we could add to the data passed into libraries or possibly do a form of search-and-replace on TAP output to insert the $CHECKNAME

c) I'm not fully convinced that result -> outcome rename is intuitive. On the other hand, ResultsDB uses "result" differently, it's a whole object which contains 'outcome' as an attribute:

https://fedoraproject.org/wiki/AutoQA_resultsdb_schema
http://resultsdb.qa.fedoraproject.org/resultsdb_frontend/jobs
So I'd like to stay consistent with our ResultsDB UI. Maybe we should even rename whole CheckDetail to CheckResult?

I'd like to avoid confusion between the result sent to something like resultsdb (status, logs, what build/update/etc is related) and the PASS/FAIL/NEEDS_INSPECTION/etc result that is a component of the other result. Do you have any ideas on how to make the naming more intuitive?

d) Maybe export_TAP() should not be in check module, but we should rather move it to tap module as tap.fromCheckDetail(). Is that more logical or not? But in that case we need to solve T101 first.

That makes sense to me, it doesn't have TAP specific code in the CheckDetail which would hopefully make any transition to subunit less painful in the future.

e) the TAP output produced by mcepl's bayeux library is not fully matching the spec (for example there should be no # FAIL directive in the second output, also the - separator between ok and rest of the line is redundant. We might want to adjust it in the future.

Hrm, I hadn't looked at the output from bayeux that closely. It sounds good enough to get away with for the time being, though?

Looks good to me, though.

It's something that we could add to the data passed into libraries or possibly do a form of search-and-replace on TAP output to insert the $CHECKNAME

My idea was to create a global singleton Context class. Could be used like this:

from Context import context
print context.check_name
# either str or None, depending how you run it (runner or standalone)

We could also implement context.check_name_fallback() that would return os.path.basename(sys.modules['__main__'].__file__) or something like that.

The Context would of course be useful for lots of purposes. It's not always viable to pass all values through method parameters, sometimes you need... well, some context :-)

I'd like to avoid confusion between the result sent to something like resultsdb (status, logs, what build/update/etc is related) and the PASS/FAIL/NEEDS_INSPECTION/etc result that is a component of the other result. Do you have any ideas on how to make the naming more intuitive?

Using 'outcome' word for PASSED/FAILED/etc is one way to have consistency. The other way is to use 'result' for that and rename Result table in ResultsDB-sense to something else, but I don't have any idea what that should be.

That makes sense to me, it doesn't have TAP specific code in the CheckDetail which would hopefully make any transition to subunit less painful in the future.

What exactly do you mean by subunit? I'm a bit lost.

Hrm, I hadn't looked at the output from bayeux that closely. It sounds good enough to get away with for the time being, though?

Yes, it's perfectly fine for the time being, I believe.

@jskladan, are you OK with the sample TAP output as well?

Josef has become sick, I'll push this now.

Closed by commit rLTRN23d387af43f9 (authored by @kparal).

Revision Contents

			Path
M			libtaskotron/check.py (278 lines)
M			libtaskotron/tap.py (4 lines)
M			libtaskotron/yamlish.py (4 lines)
M			testing/test_check.py (245 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	74	d962f51		Mar 14 2014, 2:29 PM	★	★
Diff 2	75	d962f51	Commit rLTRN23d387af43f95b4c750242b96235cdefedb586c8	Mar 20 2014, 9:49 AM	★	★

Commit	Tree	Parents	Author	Summary	Date
5695ba4f9990	c8ef7d8b900d	d962f5135cc2	Kamil Páral	implement check.export_TAP() (Show More…)	Mar 13 2014, 5:16 PM

Diff 75

View Options

libtaskotron/check.py

1	# -- coding: utf-8 --	1	# -- coding: utf-8 --
2	# Copyright 2014, Red Hat, Inc.	2	# Copyright 2014, Red Hat, Inc.
3	# License: GNU General Public License version 2 or later	3	# License: GNU General Public License version 2 or later
4		4
5	'''Helper tools for managing check status, results, and output.'''	5	'''Helper tools for managing check status, outcome, and output.'''
		6
		7	import pprint
6		8
7	from . import python_utils	9	from . import python_utils
8	from . import exceptions as exc	10	from . import exceptions as exc
		11	from . import tap
9		12
10		13
11	class CheckDetail(object):	14	class CheckDetail(object):
12	'''Class encompassing everything related to the outcome of a check run.	15	'''Class encompassing everything related to the outcome of a check run.
13	It contains convenience methods for tracking, evaluating and reporting	16	It contains convenience methods for tracking, evaluating and reporting
14	check results.	17	check results.
15		18
16	For some checks, it's trivial to parse its output at the end of its	19	For some checks, it's trivial to parse its output at the end of its
17	execution and evaluate the results. For some, however, it's much easier	20	execution and evaluate the results. For some, however, it's much easier
18	to do this continuously during check execution. This is especially true	21	to do this continuously during check execution. This is especially true
19	for multi-result checks (a single check providing results for many items -	22	for multi-item checks (a single check providing results for many items -
20	builds, updates, etc). That's when this class comes very handy (it can be	23	builds, updates, etc). That's when this class comes very handy (it can be
21	used even for the simple use cases, of course).	24	used even for the simple use cases, of course).
22		25
23	@classattr result_order a tuple of result keywords sorted by priority from	26	@classattr outcome_priority a tuple of outcome keywords sorted by priority
24	the least important to the most important	27	from the least important to the most important
25	@attr name a string describing the item being tested; for example a build	28	@attr item a string describing the item being tested; for example a build
26	NVR ('vpnc-0.5-1.fc20'), update ID ('FEDORA-2014-3309') or a	29	NVR ('vpnc-0.5-1.fc20'), update ID ('FEDORA-2014-3309') or a
27	repository name ('f20-updates')	30	repository name ('f20-updates')
28	@attr result a keyword specifying the final result of the check. Available	31	@attr report_type a definition of the type of the object being checked; for
29	result keywords:	32	example 'a Koji build', 'a Bodhi update' or 'a yum
30	* PASSED - everything went well	33	repository name'. The allowed values are attributes in
31	* INFO - everything went well, but there is some important	34	@class ReportType. You don't have to fill this in (or you
32	information that needs to be pointed out	35	can provide a custom string value), but the reporting
33	* FAILED - the item in question fails the check	36	directives react only to the known types (you can still
34	* NEEDS_INSPECTION - the result can't be determined and a	37	find it in ResultsDB, though).
35	manual inspection is needed	38	@attr outcome a keyword specifying the final outcome of the check. Available
36	* ABORTED - the check aborted itself because of some unexpected	39	outcome keywords:
37	problems, i.e. a necessary network server is not	40	* PASSED - everything went well
38	reachable. Running this check with the same	41	* INFO - everything went well, but there is some important
39	arguments later can help to mitigate this problem.	42	information that needs to be pointed out
40	* CRASHED - the check crashed and did not provide usable	43	* FAILED - the item in question fails the check
41	results	44	* NEEDS_INSPECTION - the outcome can't be determined and a
42	If no result is set, this attribute returns NEEDS_INSPECTION.	45	manual inspection is needed
43	@attr summary a few words or one-sentence summary about the outcome of the	46	* ABORTED - the check aborted itself because of some
		47	unexpected problems, i.e. a necessary network
		48	server is not reachable. Running this check with
		49	the same arguments later can help to mitigate
		50	this problem.
		51	* CRASHED - the check crashed and did not provide usable
		52	results
		53	If no outcome is set, this attribute returns NEEDS_INSPECTION.
		54	@attr summary a few words or one-sentence summary about the result of the
44	check run (e.g. '5 WARNINGS, 1 ERROR' for an rpmlint result).	55	check run (e.g. '5 WARNINGS, 1 ERROR' for an rpmlint result).
45	Should not unnecessarily duplicate @attr name or @attr result,	56	Should not unnecessarily duplicate self.item or self.outcome,
46	if possible.	57	if possible.
47	@attr output output from the check run (a list of strings). This will form	58	@attr output output from the check run (a list of strings). You can easily
48	the check log. You can easily populate this by using	59	populate this by using @method store(), or you can modify it
49	@method log(), or you can modify it directly (for example	60	directly (for example filter out some messages you don't want
50	filter out some messages you don't want to see in the check	61	to see in the check output).
51	log).
52	@attr keyvals all key-value pairs in this dictionary are stored in ResultsDB	62	@attr keyvals all key-value pairs in this dictionary are stored in ResultsDB
53	as 'extra data'. This can be used to store e.g. a compose id	63	as 'extra data'. This can be used to store e.g. a compose id
54	or a kernel version, if that information is important for	64	or a kernel version, if that information is important for
55	querying the results. Keep it as short as possible.	65	querying the results. Keep it as short as possible.
56	'''	66	'''
57		67
58	result_order = ('PASSED', 'INFO', 'FAILED', 'NEEDS_INSPECTION',	68	outcome_priority = ('PASSED', 'INFO', 'FAILED', 'NEEDS_INSPECTION',
59	'ABORTED', 'CRASHED')	69	'ABORTED', 'CRASHED')
60		70
61		71
62	def __init__(self, name='', result=None, summary='', output=None):	72	def __init__(self, item, report_type=None, outcome=None, summary='',
		73	output=None):
63	'''Create new CheckDetail.	74	'''Create new CheckDetail.
64	For param descriptions see class docstring.'''	75	For param descriptions see class docstring.'''
65		76
66	# validate input	77	# validate input
67	if output is not None and not python_utils.listlike(output):	78	if output is not None and not python_utils.listlike(output):
68	raise exc.TaskotronValueError("'output' parameter must be a list. "	79	raise exc.TaskotronValueError("'output' parameter must be a list. "
69	"Yours was: %s" % type(output))	80	"Yours was: %s" % type(output))
70		81
71	self.name = name	82	self.item = item
72	self._result = None	83	self.report_type = report_type
73	if result:	84	self._outcome = None
74	self.result = result	85	if outcome:
		86	self.outcome = outcome
75	self.summary = summary	87	self.summary = summary
76	self.output = output or []	88	self.output = output or []
77	self.keyvals = {}	89	self.keyvals = {}
78		90
79		91
80	@property	92	@property
81	def result(self):	93	def outcome(self):
82	'''See class documentation.'''	94	'''See class documentation.'''
83	return self._result or 'NEEDS_INSPECTION'	95	return self._outcome or 'NEEDS_INSPECTION'
84		96
85		97
86	@result.setter	98	@outcome.setter
87	def result(self, value):	99	def outcome(self, value):
88	'''See class documentation.'''	100	'''See class documentation.'''
89	if value not in self.result_order:	101	if value not in self.outcome_priority:
90	raise exc.TaskotronValueError('Unknown result keyword: %s' % value)	102	raise exc.TaskotronValueError('Unknown outcome keyword: %s' % value)
91	self._result = value	103	self._outcome = value
92		104
93		105
94	def update_result(self, result):	106	def update_outcome(self, outcome):
95	'''Update self.result with the provided value only and only if it has	107	'''Update self.outcome with the provided value only and only if it has
96	a higher priority than its current value (according to	108	a higher priority than its current value (according to
97	self.result_order). Otherwise this call does nothing.	109	self.outcome_priority). Otherwise this call does nothing.
98		110
99	This is useful if your check performs a number of 'sub-tasks', each	111	This is useful if your check performs a number of 'sub-tasks', each
100	passing or failing, and you want to final result to be the worst/most	112	passing or failing, and you want to final outcome to be the worst/most
101	serious one of those. You can just keep calling self.update_result()	113	serious one of those. You can just keep calling self.update_outcome()
102	and the highest priority result type will be stored at self.result at	114	and the highest priority outcome type will be stored at self.outcome at
103	the end.	115	the end.
104		116
105	@param result the result value to assign to self.result if it has a	117	@param outcome the outcome value to assign to self.outcome if it has a
106	higher priority than its current value. Handles None	118	higher priority than its current value. Handles None
107	values gracefully (no action).	119	values gracefully (no action).
108	@throw TaskotronValueError if any of the result keywords are not	120	@throw TaskotronValueError if any of the outcome keywords are not
109	specified in @classatr result_order'''	121	specified in @classatr outcome_priority'''
110	if self.cmp_result(result, self._result) > 0:	122	if self.cmp_outcome(outcome, self._outcome) > 0:
111	self.result = result	123	self.outcome = outcome
112		124
113		125
114	def broken(self):	126	def broken(self):
115	'''Tells whether the check result is set to one of the broken states	127	'''Tells whether the check outcome is set to one of the broken states
116	(i.e. ABORTED or CRASHED).'''	128	(i.e. ABORTED or CRASHED).'''
117	return self.result in ['ABORTED', 'CRASHED']	129	return self.outcome in ['ABORTED', 'CRASHED']
118		130
119		131
120	def store(self, message, printout=True):	132	def store(self, message, printout=True):
121	'''Add a string to the CheckDetail output, and print it optionally as	133	'''Add a string to the CheckDetail output, and print it optionally as
122	well.	134	well.
123		135
124	This is just a convenience method for most common use case, you can	136	This is just a convenience method for most common use case, you can
125	of course always access and modify self.output directly, and print or	137	of course always access and modify self.output directly, and print or
126	use logging facilities directly as well. This combines both into a	138	use logging facilities directly as well. This combines both into a
127	single call.	139	single call.
128		140
129	@param message a string to store in self.output	141	@param message a string to store in self.output
130	@param printout whether to print to standard output or not	142	@param printout whether to print to standard output or not
131	'''	143	'''
132		144
133	self.output.append(message)	145	self.output.append(message)
134	if printout:	146	if printout:
135	print message	147	print message
136		148
137		149
138	@classmethod	150	@classmethod
139	def cmp_result(cls, result1, result2):	151	def cmp_outcome(cls, outcome1, outcome2):
140	'''Compare two results according to self.result_order and return -1/0/1	152	'''Compare two outcomes according to self.outcome_priority and return
141	if result1 has lower/equal/higher priority than result2.	153	-1/0/1 if outcome1 has lower/equal/higher priority than outcome2.
142		154
143	@param result1 a result keyword to compare	155	@param outcome1 a outcome keyword to compare
144	@param result2 a result keyword to compare	156	@param outcome2 a outcome keyword to compare
145	@throw TaskotronValueError if any of the result keywords are not	157	@throw TaskotronValueError if any of the outcome keywords are not
146	specified in @classatr result_order	158	specified in @classatr outcome_priority
147	'''	159	'''
148		160
149	# validate input	161	# validate input
150	for result in [result1, result2]:	162	for outcome in [outcome1, outcome2]:
151	if (result not in cls.result_order) and (result is not None):	163	if (outcome not in cls.outcome_priority) and (outcome is not None):
152	raise exc.TaskotronValueError('Unknown result keyword: %s' %	164	raise exc.TaskotronValueError('Unknown outcome keyword: %s' %
153	result)	165	outcome)
154		166
155	index1 = cls.result_order.index(result1) if result1 is not None else -1	167	index1 = cls.outcome_priority.index(outcome1) if outcome1 is not None \
156	index2 = cls.result_order.index(result2) if result2 is not None else -1	168	else -1
		169	index2 = cls.outcome_priority.index(outcome2) if outcome2 is not None \
		170	else -1
157	return cmp(index1, index2)	171	return cmp(index1, index2)
158		172
159		173
160	@classmethod	174	@classmethod
161	def create_multi_item_summary(cls, results):	175	def create_multi_item_summary(cls, outcomes):
162	'''Create a string containing a sum of all results, like this:	176	'''Create a string containing a sum of all outcomes, like this:
163	"3 PASSED, 1 INFO, 2 FAILED"	177	"3 PASSED, 1 INFO, 2 FAILED"
164		178
165	@param results either one CheckDetail instance or an iterable of	179	@param outcomes either one CheckDetail instance or an iterable of
166	CheckDetails or an iterable of result strings	180	CheckDetails or an iterable of outcome strings
167	'''	181	'''
168	# validate input	182	# validate input
169	if isinstance(results, CheckDetail):	183	if isinstance(outcomes, CheckDetail):
170	results = (results,)	184	outcomes = (outcomes,)
171		185
172	if len(results) <= 0:	186	if len(outcomes) <= 0:
173	return ''	187	return ''
174		188
175	if isinstance(iter(results).next(), CheckDetail):	189	if isinstance(iter(outcomes).next(), CheckDetail):
176	all_results = [detail.result for detail in results]	190	all_outcomes = [detail.outcome for detail in outcomes]
177	else: # list of strings	191	else: # list of strings
178	if not python_utils.listlike(results):	192	if not python_utils.listlike(outcomes):
179	raise exc.TaskotronValueError("'results' parameter type is "	193	raise exc.TaskotronValueError("'outcomes' parameter type is "
180	'incorrect: %s' % type(results))	194	'incorrect: %s' % type(outcomes))
181	all_results = results	195	all_outcomes = outcomes
182		196
183	# create the summary	197	# create the summary
184	summary = []	198	summary = []
185	for res in cls.result_order:	199	for res in cls.outcome_priority:
186	count = all_results.count(res)	200	count = all_outcomes.count(res)
187	if count > 0:	201	if count > 0:
188	summary.append('%d %s' % (count, res))	202	summary.append('%d %s' % (count, res))
189	summary = ', '.join(summary)	203	summary = ', '.join(summary)
190		204
191	return summary	205	return summary
		206
		207
		208	def __str__(self):
		209	# Make this object more readable when printing
		210	# But don't show self.output, because that can be huuge
		211	attrs = vars(self)
		212	attrs['output'] = '<stripped out>'
		213	return '<%s: %s>' % (self.__class__.__name__, pprint.pformat(attrs))
		214
		215
		216	class ReportType(object):
		217	''' Enum for different types of CheckDetail.report_type'''
		218	# the values are used as identifiers in a TAP export
		219	KOJI_BUILD = 'koji_build'
		220	BODHI_UPDATE = 'bodhi_update'
		221	YUM_REPOSITORY = 'yum_repository'
		222
		223
		224	def export_TAP(*check_details):
		225	'''Generate TAP output used for reporting to ResultsDB.
		226
		227	@param *check_details any number of CheckDetail instances
		228	@return a string containing TAP output with results for every CheckDetail
		229	instance provided
		230	@raise TaskotronValueError if CheckDetail.item is empty for any parameter
		231	provided
		232
		233	Example output:
		234	~~~~~~~~~~~~~~~~~~~~~~~
		235	ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
		236	---
		237	details:
		238	output: \|-
		239	xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
		240	xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps.schemas
		241	xchat.x86_64: W: no-manual-page-for-binary xchat
		242	item: xchat-0.5-1.fc20
		243	outcome: PASSED
		244	summary: 5 ERRORS, 10 WARNINGS
		245	type: koji_build
		246	...
		247	~~~~~~~~~~~~~~~~~~~~~~~
		248	'''
		249
		250	# validate input
		251	for detail in check_details:
		252	if not detail.item:
		253	raise exc.TaskotronValueError('CheckDetail.item is empty for: %s' %
		254	detail)
		255
		256	# TAP test line will be more readable for recognized report types
		257	pretty_type_name = {
		258	ReportType.KOJI_BUILD: 'Koji build',
		259	ReportType.BODHI_UPDATE: 'Bodhi update',
		260	ReportType.YUM_REPOSITORY: 'Yum repository',
		261	}
		262
		263	tapgen = tap.TAPGenerator()
		264	tapout = []
		265
		266	for detail in check_details:
		267	# TAP test line
		268	tap_outcome = tap.FAIL
		269	if detail.outcome in ['PASSED', 'INFO']:
		270	tap_outcome = tap.PASS
		271
		272	name = '$CHECKNAME for '
		273	if detail.report_type:
		274	name += '%s ' % pretty_type_name.get(detail.report_type,
		275	detail.report_type)
		276	name += detail.item
		277
		278	# TAP YAML block
		279	data = {}
		280	data['outcome'] = detail.outcome
		281	data['item'] = detail.item
		282	if detail.report_type:
		283	data['type'] = detail.report_type
		284	if detail.summary:
		285	data['summary'] = detail.summary
		286	data['details'] = {}
		287	if detail.output:
		288	data['details']['output'] = '\n'.join(detail.output)
		289	if not data['details']:
		290	del data['details']
		291
		292	# generate
		293	output = tapgen.format_TAP_msg(result=tap_outcome, name=name, data=data)
		294	tapout.append(output)
		295
		296	return '\n'.join(tapout)
		297

View Options

libtaskotron/tap.py

Show All 12 Lines
13	#	13	#
14	#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR	14	#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15	#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,	15	#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16	#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE	16	#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17	#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER	17	#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18	#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,	18	#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19	#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE	19	#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20	#SOFTWARE.	20	#SOFTWARE.
		21
		22	# This code is taken from http://luther.ceplovi.cz/git/bayeux.git/
		23	# Also available at https://pypi.python.org/pypi/bayeux
		24
21	import yamlish	25	import yamlish
22	import logging	26	import logging
23		27
24	PASS = 0	28	PASS = 0
25	SKIP = 1	29	SKIP = 1
26	TBD = 2 #IGNORE:W0511	30	TBD = 2 #IGNORE:W0511
27	FAIL = 3	31	FAIL = 3
28	ERROR = 4	32	ERROR = 4
▲ Show 20 Lines • Show All 58 Lines • Show Last 20 Lines

View Options

libtaskotron/yamlish.py

Show First 20 Lines • Show All 102 Lines • ▼ Show 20 Line(s)
103	diagnostic information to be passed from test scripts to	103	diagnostic information to be passed from test scripts to
104	the Perl module `TAP::Harness`. That means that if you're writing	104	the Perl module `TAP::Harness`. That means that if you're writing
105	a testing system that needs to output TAP version 13 or later	105	a testing system that needs to output TAP version 13 or later
106	syntax you might find this module useful.	106	syntax you might find this module useful.
107		107
108	Read more about TAP and YAMLish on `<http://testanything.org/wiki>`	108	Read more about TAP and YAMLish on `<http://testanything.org/wiki>`
109		109
110	"""	110	"""
		111
		112	# This code is taken from http://luther.ceplovi.cz/git/yamlish.git/
		113	# Also available at https://pypi.python.org/pypi/yamlish
		114
111	from __future__ import absolute_import, print_function, unicode_literals	115	from __future__ import absolute_import, print_function, unicode_literals
112	import logging	116	import logging
113	import yaml	117	import yaml
114		118
115	__docformat__ = 'reStructuredText'	119	__docformat__ = 'reStructuredText'
116	__version__ = "0.10"	120	__version__ = "0.10"
117	__author__ = u"Matěj Cepl <mcepl_at_redhat_dot_com>"	121	__author__ = u"Matěj Cepl <mcepl_at_redhat_dot_com>"
118		122
▲ Show 20 Lines • Show All 88 Lines • Show Last 20 Lines

View Options

testing/test_check.py

1	# -- coding: utf-8 --	1	# -- coding: utf-8 --
2	# Copyright 2014, Red Hat, Inc.	2	# Copyright 2014, Red Hat, Inc.
3	# License: GNU General Public License version 2 or later	3	# License: GNU General Public License version 2 or later
4		4
5	'''Unit tests for libtaskotron/check.py'''	5	'''Unit tests for libtaskotron/check.py'''
6		6
7	import pytest	7	import pytest
		8	import yaml
8		9
9	from libtaskotron import check	10	from libtaskotron import check
10	from libtaskotron import exceptions as exc	11	from libtaskotron import exceptions as exc
11		12
12	class TestCheckDetail:	13	class TestCheckDetail(object):
13		14
14	def setup_method(self, method):	15	def setup_method(self, method):
15	'''Run this before every test invocation'''	16	'''Run this before every test invocation'''
16	# for most methods, we want a default empty CheckDetail instance	17	# for most methods, we want a default empty CheckDetail instance
17	self.cd = check.CheckDetail()	18	self.cd = check.CheckDetail(None)
18		19
19	def test_init_empty(self):	20	def test_init_empty(self):
20	'''Test instance creation with empty parameters'''	21	'''Test instance creation with empty parameters'''
21	# by default, result should be 'NEEDS_INSPECTION', not empty	22	# by default, outcome should be 'NEEDS_INSPECTION', not empty
22	assert self.cd.result == 'NEEDS_INSPECTION'	23	assert self.cd.outcome == 'NEEDS_INSPECTION'
23	assert not self.cd.name	24	assert not self.cd.item
24	assert not self.cd.summary	25	assert not self.cd.summary
25	assert not self.cd.output	26	assert not self.cd.output
26	assert not self.cd.keyvals	27	assert not self.cd.keyvals
27		28
28	def test_init_params(self):	29	def test_init_params(self):
29	'''Test instance creation with non-empty parameters'''	30	'''Test instance creation with non-empty parameters'''
30	cd = check.CheckDetail(name='foo-1.2-3.fc20', result='FAILED',	31	cd = check.CheckDetail(item='foo-1.2-3.fc20', outcome='FAILED',
31	summary='untested', output=['line1', 'line2'])	32	summary='untested', output=['line1', 'line2'])
32	assert cd.name == 'foo-1.2-3.fc20'	33	assert cd.item == 'foo-1.2-3.fc20'
33	assert cd.result == 'FAILED'	34	assert cd.outcome == 'FAILED'
34	assert cd.summary == 'untested'	35	assert cd.summary == 'untested'
35	assert cd.output == ['line1', 'line2']	36	assert cd.output == ['line1', 'line2']
36		37
37	def test_init_raise(self):	38	def test_init_raise(self):
38	'''Test instance creation that raises an error - invalid parameters'''	39	'''Test instance creation that raises an error - invalid parameters'''
39	with pytest.raises(exc.TaskotronValueError):	40	with pytest.raises(exc.TaskotronValueError):
40	check.CheckDetail(result='INVALID TYPE')	41	check.CheckDetail(None, outcome='INVALID TYPE')
41		42
42	with pytest.raises(exc.TaskotronValueError):	43	with pytest.raises(exc.TaskotronValueError):
43	check.CheckDetail(output='foobar')	44	check.CheckDetail(None, output='foobar')
44		45
45	def test_result(self):	46	def test_outcome(self):
46	'''Test 'result' attribute'''	47	'''Test 'outcome' attribute'''
47	# default value test	48	# default value test
48	# (already tested in constructor test, but it logically belongs here,	49	# (already tested in constructor test, but it logically belongs here,
49	# so let's include it again, in case constructor test changes)	50	# so let's include it again, in case constructor test changes)
50	assert self.cd.result == 'NEEDS_INSPECTION'	51	assert self.cd.outcome == 'NEEDS_INSPECTION'
51		52
52	# test setter method with correct values	53	# test setter method with correct values
53	self.cd.result = 'INFO'	54	self.cd.outcome = 'INFO'
54	assert self.cd.result == 'INFO'	55	assert self.cd.outcome == 'INFO'
55		56
56	self.cd.result = 'PASSED'	57	self.cd.outcome = 'PASSED'
57	assert self.cd.result == 'PASSED'	58	assert self.cd.outcome == 'PASSED'
58		59
59	def test_result_raise(self):	60	def test_outcome_raise(self):
60	'''Test 'result' attribute when it should raise an error'''	61	'''Test 'outcome' attribute when it should raise an error'''
61	with pytest.raises(exc.TaskotronValueError):	62	with pytest.raises(exc.TaskotronValueError):
62	self.cd.result = 'INVALID RESULT TYPE'	63	self.cd.outcome = 'INVALID OUTCOME TYPE'
63		64
64	def test_update_result(self):	65	def test_update_outcome(self):
65	'''Test 'update_result' method'''	66	'''Test 'update_outcome' method'''
66	# basic use case	67	# basic use case
67	self.cd.update_result('FAILED')	68	self.cd.update_outcome('FAILED')
68	assert self.cd.result == 'FAILED'	69	assert self.cd.outcome == 'FAILED'
69		70
70	# update_result respects the result type ordering, it never overrides	71	# update_outcome respects the outcome priority ordering, it never
71	# a more important result with a less important one; therefore it should	72	# overrides a more important outcome with a less important one;
72	# retain FAILED here	73	# therefore it should retain FAILED here
73	self.cd.update_result('PASSED')	74	self.cd.update_outcome('PASSED')
74	assert self.cd.result == 'FAILED'	75	assert self.cd.outcome == 'FAILED'
75		76
76	# None should do nothing	77	# None should do nothing
77	self.cd.update_result(None)	78	self.cd.update_outcome(None)
78	assert self.cd.result == 'FAILED'	79	assert self.cd.outcome == 'FAILED'
79		80
80	def test_update_result_raise(self):	81	def test_update_outcome_raise(self):
81	'''Test 'update_result' method when it should raise an error'''	82	'''Test 'update_outcome' method when it should raise an error'''
82	with pytest.raises(exc.TaskotronValueError):	83	with pytest.raises(exc.TaskotronValueError):
83	self.cd.update_result('INVALID RESULT TYPE')	84	self.cd.update_outcome('INVALID OUTCOME TYPE')
84		85
85	def test_broken(self):	86	def test_broken(self):
86	'''Test 'broken' method'''	87	'''Test 'broken' method'''
87	# by default we're not broken	88	# by default we're not broken
88	assert not self.cd.broken()	89	assert not self.cd.broken()
89		90
90	# two values should make it look broken	91	# two values should make it look broken
91	self.cd.result = 'ABORTED'	92	self.cd.outcome = 'ABORTED'
92	assert self.cd.broken()	93	assert self.cd.broken()
93		94
94	self.cd.result = 'CRASHED'	95	self.cd.outcome = 'CRASHED'
95	assert self.cd.broken()	96	assert self.cd.broken()
96		97
97	# and we can go back to a non-broken state	98	# and we can go back to a non-broken state
98	self.cd.result = 'FAILED'	99	self.cd.outcome = 'FAILED'
99	assert not self.cd.broken()	100	assert not self.cd.broken()
100		101
101	def test_store(self, capsys):	102	def test_store(self, capsys):
102	'''Test 'store' method'''	103	'''Test 'store' method'''
103	# default use case where we fill 'output' and print to stdout as well	104	# default use case where we fill 'output' and print to stdout as well
104	self.cd.store('foobar')	105	self.cd.store('foobar')
105	assert self.cd.output == ['foobar']	106	assert self.cd.output == ['foobar']
106	out, err = capsys.readouterr()	107	out, err = capsys.readouterr()
107	assert out == 'foobar\n'	108	assert out == 'foobar\n'
108	assert not err	109	assert not err
109		110
110	# use case without printing to stdout	111	# use case without printing to stdout
111	self.cd.store('barbaz', printout=False)	112	self.cd.store('barbaz', printout=False)
112	assert self.cd.output == ['foobar', 'barbaz']	113	assert self.cd.output == ['foobar', 'barbaz']
113	out, err = capsys.readouterr()	114	out, err = capsys.readouterr()
114	assert not out	115	assert not out
115	assert not err	116	assert not err
116		117
117	def test_cmp_result(self):	118	def test_cmp_outcome(self):
118	'''Test 'cmp_result' method'''	119	'''Test 'cmp_outcome' method'''
119	# sample comparisons where the first parameter is 'larger' (more	120	# sample comparisons where the first parameter is 'larger' (more
120	# important) than the second one	121	# important) than the second one
121	assert check.CheckDetail.cmp_result('FAILED', 'PASSED') == 1	122	assert check.CheckDetail.cmp_outcome('FAILED', 'PASSED') == 1
122	assert check.CheckDetail.cmp_result('NEEDS_INSPECTION', 'INFO') == 1	123	assert check.CheckDetail.cmp_outcome('NEEDS_INSPECTION', 'INFO') == 1
123	assert check.CheckDetail.cmp_result('CRASHED', 'ABORTED') == 1	124	assert check.CheckDetail.cmp_outcome('CRASHED', 'ABORTED') == 1
124	assert check.CheckDetail.cmp_result('PASSED', None) == 1	125	assert check.CheckDetail.cmp_outcome('PASSED', None) == 1
125		126
126	# sample comparisons where both the parameters are equal (same	127	# sample comparisons where both the parameters are equal (same
127	#importance)	128	#importance)
128	assert check.CheckDetail.cmp_result('PASSED', 'PASSED') == 0	129	assert check.CheckDetail.cmp_outcome('PASSED', 'PASSED') == 0
129	assert check.CheckDetail.cmp_result(None, None) == 0	130	assert check.CheckDetail.cmp_outcome(None, None) == 0
130		131
131	# sample comparisons where the first parameter is 'smaller' (less	132	# sample comparisons where the first parameter is 'smaller' (less
132	# important) than the second one	133	# important) than the second one
133	assert check.CheckDetail.cmp_result('PASSED', 'FAILED') == -1	134	assert check.CheckDetail.cmp_outcome('PASSED', 'FAILED') == -1
134	assert check.CheckDetail.cmp_result(None, 'PASSED') == -1	135	assert check.CheckDetail.cmp_outcome(None, 'PASSED') == -1
135		136
136	def test_cmp_result_raise(self):	137	def test_cmp_outcome_raise(self):
137	'''Test 'cmp_result' method when it should raise an error'''	138	'''Test 'cmp_outcome' method when it should raise an error'''
138	# test invalid result types as an input	139	# test invalid outcome types as an input
139	with pytest.raises(exc.TaskotronValueError):	140	with pytest.raises(exc.TaskotronValueError):
140	assert check.CheckDetail.cmp_result('PASSED', 'INVALID RESULT')	141	assert check.CheckDetail.cmp_outcome('PASSED', 'INVALID OUTCOME')
141		142
142	with pytest.raises(exc.TaskotronValueError):	143	with pytest.raises(exc.TaskotronValueError):
143	assert check.CheckDetail.cmp_result('INVALID RESULT', 'FAILED')	144	assert check.CheckDetail.cmp_outcome('INVALID OUTCOME', 'FAILED')
144		145
145	with pytest.raises(exc.TaskotronValueError):	146	with pytest.raises(exc.TaskotronValueError):
146	assert check.CheckDetail.cmp_result('FOOBAR', 'FOOBAR')	147	assert check.CheckDetail.cmp_outcome('FOOBAR', 'FOOBAR')
147		148
148	def test_create_multi_item_summary(self):	149	def test_create_multi_item_summary(self):
149	'''Test create_multi_item_summary method'''	150	'''Test create_multi_item_summary method'''
150	# a basic use case with a list of strings	151	# a basic use case with a list of strings
151	summary = check.CheckDetail.create_multi_item_summary(['PASSED',	152	summary = check.CheckDetail.create_multi_item_summary(['PASSED',
152	'PASSED', 'FAILED', 'ABORTED', 'INFO', 'INFO'])	153	'PASSED', 'FAILED', 'ABORTED', 'INFO', 'INFO'])
153	assert summary == '2 PASSED, 2 INFO, 1 FAILED, 1 ABORTED'	154	assert summary == '2 PASSED, 2 INFO, 1 FAILED, 1 ABORTED'
154		155
155	# an empty list should return an empty string	156	# an empty list should return an empty string
156	summary = check.CheckDetail.create_multi_item_summary([])	157	summary = check.CheckDetail.create_multi_item_summary([])
157	assert summary == ''	158	assert summary == ''
158		159
159	# a use case with one CheckDetail instance input	160	# a use case with one CheckDetail instance input
160	summary = check.CheckDetail.create_multi_item_summary(self.cd)	161	summary = check.CheckDetail.create_multi_item_summary(self.cd)
161	assert summary == '1 NEEDS_INSPECTION'	162	assert summary == '1 NEEDS_INSPECTION'
162		163
163	# a use case with multiple CheckDetail instances input	164	# a use case with multiple CheckDetail instances input
164	cd1 = check.CheckDetail(result='PASSED')	165	cd1 = check.CheckDetail(None, outcome='PASSED')
165	cd2 = check.CheckDetail(result='PASSED')	166	cd2 = check.CheckDetail(None, outcome='PASSED')
166	cd3 = check.CheckDetail(result='INFO')	167	cd3 = check.CheckDetail(None, outcome='INFO')
167	summary = check.CheckDetail.create_multi_item_summary([self.cd,	168	summary = check.CheckDetail.create_multi_item_summary([self.cd,
168	cd1, cd2, cd3])	169	cd1, cd2, cd3])
169	assert summary == '2 PASSED, 1 INFO, 1 NEEDS_INSPECTION'	170	assert summary == '2 PASSED, 1 INFO, 1 NEEDS_INSPECTION'
170		171
171	def test_create_multi_item_summary_raise(self):	172	def test_create_multi_item_summary_raise(self):
172	'''Test create_multi_item_summary method when it should raise an	173	'''Test create_multi_item_summary method when it should raise an
173	error'''	174	error'''
174	# a single string input is not allowed, only a list of them	175	# a single string input is not allowed, only a list of them
175	with pytest.raises(exc.TaskotronValueError):	176	with pytest.raises(exc.TaskotronValueError):
176	assert check.CheckDetail.create_multi_item_summary('PASSED')	177	assert check.CheckDetail.create_multi_item_summary('PASSED')
		178
		179
		180	class TestExportTAP(object):
		181	'''Test @method export_TAP'''
		182	item = 'xchat-0.5-1.fc20'
		183	outcome = 'PASSED'
		184	summary='5 ERRORS, 10 WARNINGS'
		185	report_type=check.ReportType.KOJI_BUILD
		186	output = '''\
		187	xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
		188	xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
		189	xchat.x86_64: W: no-manual-page-for-binary xchat'''
		190
		191	def setup_method(self, method):
		192	self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,
		193	summary=self.summary,
		194	report_type=self.report_type)
		195	self.cd.output.append(self.output)
		196
		197	def _tap_to_yaml(self, tap):
		198	'''Remove all TAP-related contents and return only its YAML block(s)'''
		199	lines = tap.splitlines()
		200
		201	# split the output to TAP result lines and YAML blocks
		202	tap_lines = [line for line in lines if line.startswith('ok') or
		203	line.startswith('not ok')]
		204	# remove left padding - 2 spaces
		205	yaml_lines = [line[2:] for line in lines if line not in tap_lines]
		206
		207	return '\n'.join(yaml_lines)
		208
		209	def test_single_tap(self):
		210	'''Test export with a single item section. Check generic TAP lines.'''
		211	tap = check.export_TAP(self.cd)
		212	lines = tap.splitlines()
		213
		214	# generic lines must exist
		215	assert lines[0].startswith('ok ')
		216	assert lines[1].strip() == '---'
		217	assert lines[-1].strip() == '...'
		218
		219	def test_single_yaml(self):
		220	'''Test export with a single item section. Check specific YAML lines.'''
		221	tap = check.export_TAP(self.cd)
		222
		223	# specific lines must exist
		224	yaml_block = self._tap_to_yaml(tap)
		225	yaml_obj = yaml.load(yaml_block)
		226
		227	assert yaml_obj['item'] == self.item
		228	assert yaml_obj['outcome'] == self.outcome
		229	assert yaml_obj['summary'] == self.summary
		230	assert yaml_obj['type'] == self.report_type
		231	assert yaml_obj['details']
		232	assert yaml_obj['details']['output'] == self.output
		233
		234	def test_multi(self):
		235	'''Test export with multiple item sections'''
		236	cd2 = check.CheckDetail(item='foobar-1.2-3.fc20', outcome='FAILED',
		237	summary='dependency error',
		238	report_type=check.ReportType.BODHI_UPDATE)
		239	cd3 = check.CheckDetail(item='f20-updates', outcome='INFO',
		240	summary='2 stale updates',
		241	report_type=check.ReportType.YUM_REPOSITORY)
		242	tap = check.export_TAP(self.cd, cd2, cd3)
		243	lines = tap.splitlines()
		244	yaml_blocks = self._tap_to_yaml(tap)
		245
		246	# test that we have a correct number of TAP lines
		247	tap_lines = [line for line in lines if line.startswith('ok') or
		248	line.startswith('not ok')]
		249	assert len(tap_lines) == 3
		250
		251	# test that we have a correct number (and rough contents) of YAML blocks
		252	yaml_gen = yaml.load_all(yaml_blocks)
		253	for index, yaml_obj in enumerate(yaml_gen):
		254	assert 'item' in yaml_obj
		255	assert index == 2
		256
		257	def test_invalid(self):
		258	'''Test invalid input parameters'''
		259	with pytest.raises(exc.TaskotronValueError):
		260	self.cd.item = None
		261	check.export_TAP(self.cd)
		262
		263	def test_minimal(self):
		264	'''Test that empty CheckDetail values don't produce empty TAP lines,
		265	for example 'summary:' should not be present if there's no
		266	CheckDetail.summary'''
		267	cd = check.CheckDetail('foo')
		268	tap = check.export_TAP(cd)
		269	yaml_block = self._tap_to_yaml(tap)
		270	yaml_obj = yaml.load(yaml_block)
		271
		272	# the output should look like this:
		273	#
		274	# ok - $CHECKNAME for XXX
		275	# ---
		276	# item: XXX
		277	# outcome: XXX
		278	# ...
		279	#
		280	# ('outcome:' is technically not a mandatory line, but with CheckDetail
		281	# export we produce it every time)
		282	assert len(yaml_obj) == 2
		283
		284	def test_tap_ok(self):
		285	'''Test that TAP header contains 'ok' for appropriate outcomes'''
		286	# by default the outcome is PASSED
		287	assert check.export_TAP(self.cd).startswith('ok')
		288	self.cd.outcome = 'INFO'
		289	assert check.export_TAP(self.cd).startswith('ok')
		290
		291	def test_tap_not_ok(self):
		292	'''Test that TAP header contains 'not ok' for appropriate outcomes'''
		293	self.cd.outcome = 'FAILED'
		294	assert check.export_TAP(self.cd).startswith('not ok')
		295	self.cd.outcome = 'NEEDS_INSPECTION'
		296	assert check.export_TAP(self.cd).startswith('not ok')
		297	self.cd.outcome = 'ABORTED'
		298	assert check.export_TAP(self.cd).startswith('not ok')
		299	self.cd.outcome = 'CRASHED'
		300	assert check.export_TAP(self.cd).startswith('not ok')
		301
		302	def test_custom_report_type(self):
		303	'''Test that user can specify a custom CheckDetail.report_type'''
		304	self.cd.report_type = 'My Custom Type'
		305	tap = check.export_TAP(self.cd)
		306	yaml_block = self._tap_to_yaml(tap)
		307	yaml_obj = yaml.load(yaml_block)
		308
		309	assert yaml_obj['type'] == self.cd.report_type

implement check.export_TAP()ClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 75

libtaskotron/check.py

libtaskotron/tap.py

libtaskotron/yamlish.py

testing/test_check.py

implement check.export_TAP()
ClosedPublic
Actions