Differential D329

Add optional checkname field to the TAP's YAML section
ClosedPublic
Actions

Authored by jskladan on Apr 7 2015, 2:19 PM.

Details

Reviewers
kparal
tflink
mkrizek
Maniphest Tasks
T446: Add optional `checkname` field to the TAP's YAML section
Commits
rLTRNf355f23e56e7: Add optional checkname field to the TAP's YAML section
Summary

As discussed in T446, we want to be able to have a single test report results for multiple checks
(e.g. rpmgrill). Adding checkname field to the CheckDetail, and then handling it properly in
import/exportTAP and the resultsdb reporting will allow us to do so.

Test Plan

Added unittests

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
jskladan retitled this revision from to Add optional checkname field to the TAP's YAML section.Apr 7 2015, 2:19 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added reviewers: kparal, tflink, mkrizek.
kparal accepted this revision.Apr 7 2015, 3:18 PM

LGTM

libtaskotron/check.py
84–88

In order to stay consistent in terminology, maybe this could be adjusted?
result -> either outcome or more probably CheckDetail (?)
non needed -> not needed
tasks metadata -> task metadata or task's metadata
single test -> single task

This revision is now accepted and ready to land.Apr 7 2015, 3:18 PM
jskladan updated this revision to Diff 897.Apr 7 2015, 3:26 PM
  • added tests for check.py
jskladan updated this revision to Diff 898.Apr 7 2015, 3:29 PM
  • fixed terminology according to review
tflink accepted this revision.Apr 7 2015, 4:05 PM
mkrizek accepted this revision.Apr 8 2015, 6:11 AM
Closed by commit rLTRNf355f23e56e7: Add optional checkname field to the TAP's YAML section (authored by Josef Skladanka <jskladan@redhat.com>). · Explain WhyApr 8 2015, 1:08 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 908

View Options

libtaskotron/check.py

Show All 13 Lines
14from . import python_utils 14​from . import python_utils
15from . import exceptions as exc 15​from . import exceptions as exc
16from .logger import log 16​from .logger import log
17 17​
18from pytap13 import TAP13 18​from pytap13 import TAP13
19 19​
20# a list of reserved keywords for TAP output, which can't be overridden in 20​# a list of reserved keywords for TAP output, which can't be overridden in
21# keyvals 21​# keyvals
22RESERVED_KEYS = ('item', 'type', 'outcome', 'summary', 'details') 22​RESERVED_KEYS = ('item', 'type', 'outcome', 'summary', 'details', 'checkname')
23 23​
24class CheckDetail(object): 24​class CheckDetail(object):
25 '''Class encompassing everything related to the outcome of a check run. 25​ '''Class encompassing everything related to the outcome of a check run.
26 It contains convenience methods for tracking, evaluating and reporting 26​ It contains convenience methods for tracking, evaluating and reporting
27 check results. 27​ check results.
28 28​
29 For some checks, it's trivial to parse its output at the end of its 29​ For some checks, it's trivial to parse its output at the end of its
30 execution and evaluate the results. For some, however, it's much easier 30​ execution and evaluate the results. For some, however, it's much easier
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Line(s)75 :ivar output: output from the check run (a list of strings). You can easily
76 populate this by using @method store(), or you can modify it 76​ populate this by using @method store(), or you can modify it
77 directly (for example filter out some messages you don't want 77​ directly (for example filter out some messages you don't want
78 to see in the check output). 78​ to see in the check output).
79 :ivar dict keyvals: all key-value pairs in this dictionary are stored in 79​ :ivar dict keyvals: all key-value pairs in this dictionary are stored in
80 ResultsDB as 'extra data'. This can be used to store 80​ ResultsDB as 'extra data'. This can be used to store
81 e.g. a compose id or a kernel version, if that 81​ e.g. a compose id or a kernel version, if that
82 information is important for querying the results. Keep 82​ information is important for querying the results. Keep
83 it as short as possible. 83​ it as short as possible.
84​ :ivar str checkname: name of the check, which the CheckDetail belongs to. This
85​ is usually not needed, as the check name is devised from
86​ the task metadata, but if a single task produces
87​ results for multiple checks, this is the way to override
88​ the default behavior.
kparalUnsubmitted
Not Done

In order to stay consistent in terminology, maybe this could be adjusted?
result -> either outcome or more probably CheckDetail (?)
non needed -> not needed
tasks metadata -> task metadata or task's metadata
single test -> single task

84 ''' 89​ '''
85 90​
86 outcome_priority = ('PASSED', 'INFO', 'FAILED', 'NEEDS_INSPECTION', 91​ outcome_priority = ('PASSED', 'INFO', 'FAILED', 'NEEDS_INSPECTION',
87 'ABORTED', 'CRASHED') 92​ 'ABORTED', 'CRASHED')
88 93​
89 94​
90 def __init__(self, item, report_type=None, outcome=None, summary='', 95​ def __init__(self, item, report_type=None, outcome=None, summary='',
91 output=None, keyvals=None): 96​ output=None, keyvals=None, checkname=None):
92 # validate input 97​ # validate input
93 if (output is not None and 98​ if (output is not None and
94 not python_utils.sequence(output, basestring, mutable=True)): 99​ not python_utils.sequence(output, basestring, mutable=True)):
95 raise exc.TaskotronValueError("'output' parameter must be a " 100​ raise exc.TaskotronValueError("'output' parameter must be a "
96 "mutable sequence of strings. Yours was: %s" % type(output)) 101​ "mutable sequence of strings. Yours was: %s" % type(output))
97 if keyvals is not None and not isinstance(keyvals, collections.Mapping): 102​ if keyvals is not None and not isinstance(keyvals, collections.Mapping):
98 raise exc.TaskotronValueError("'keyvals' parameter must be a " 103​ raise exc.TaskotronValueError("'keyvals' parameter must be a "
99 "mapping. Yours was: %s" % type(keyvals)) 104​ "mapping. Yours was: %s" % type(keyvals))
100 105​
101 self.item = item 106​ self.item = item
102 self.report_type = report_type 107​ self.report_type = report_type
103 self._outcome = None 108​ self._outcome = None
104 if outcome: 109​ if outcome:
105 self.outcome = outcome 110​ self.outcome = outcome
111​ self.checkname = checkname
106 self.summary = summary 112​ self.summary = summary
107 self.output = output or [] 113​ self.output = output or []
108 self.keyvals = keyvals or {} 114​ self.keyvals = keyvals or {}
109 115​
110 116​
111 @property 117​ @property
112 def outcome(self): 118​ def outcome(self):
113 return self._outcome or 'NEEDS_INSPECTION' 119​ return self._outcome or 'NEEDS_INSPECTION'
▲ Show 20 Lines • Show All 195 Lines • ▼ Show 20 Line(s)303 for detail in check_details:
309 # TAP YAML block 315​ # TAP YAML block
310 data = {} 316​ data = {}
311 data['outcome'] = detail.outcome 317​ data['outcome'] = detail.outcome
312 data['item'] = detail.item 318​ data['item'] = detail.item
313 if detail.report_type: 319​ if detail.report_type:
314 data['type'] = detail.report_type 320​ data['type'] = detail.report_type
315 if detail.summary: 321​ if detail.summary:
316 data['summary'] = detail.summary 322​ data['summary'] = detail.summary
323​ if detail.checkname:
324​ data['checkname'] = detail.checkname
317 data['details'] = {} 325​ data['details'] = {}
318 if detail.output: 326​ if detail.output:
319 data['details']['output'] = '\n'.join(detail.output) 327​ data['details']['output'] = '\n'.join(detail.output)
320 if not data['details']: 328​ if not data['details']:
321 del data['details'] 329​ del data['details']
322 330​
323 for key, value in detail.keyvals.iteritems(): 331​ for key, value in detail.keyvals.iteritems():
324 if key in RESERVED_KEYS: 332​ if key in RESERVED_KEYS:
Show All 25 Lines345def import_TAP(source):
350 check_details = [] 358​ check_details = []
351 for test in tap.tests: 359​ for test in tap.tests:
352 y = test.yaml or {} 360​ y = test.yaml or {}
353 item = y.get('item', None) 361​ item = y.get('item', None)
354 report_type = y.get('type', None) 362​ report_type = y.get('type', None)
355 outcome = 'PASSED' if test.result == 'ok' else 'FAILED' 363​ outcome = 'PASSED' if test.result == 'ok' else 'FAILED'
356 outcome = y.get('outcome', outcome) 364​ outcome = y.get('outcome', outcome)
357 summary = y.get('summary', '') 365​ summary = y.get('summary', '')
366​ checkname = y.get('checkname', None)
358 details = y.get('details', {}) 367​ details = y.get('details', {})
359 output = details.get('output', None) 368​ output = details.get('output', None)
360 if output is not None: 369​ if output is not None:
361 output = [output] 370​ output = [output]
362 371​
363 # TODO: is there a better way to do this? 372​ # TODO: is there a better way to do this?
364 other_keys = set(y.keys()) - set(RESERVED_KEYS) 373​ other_keys = set(y.keys()) - set(RESERVED_KEYS)
365 keyvals = dict([(k,y[k]) for k in other_keys]) 374​ keyvals = dict([(k,y[k]) for k in other_keys])
366 375​
367 cd = CheckDetail(item, report_type, outcome, summary, output) 376​ cd = CheckDetail(item, report_type, outcome, summary, output, checkname=checkname)
368 377​
369 cd.keyvals = keyvals 378​ cd.keyvals = keyvals
370 check_details.append(cd) 379​ check_details.append(cd)
371 380​
372 return check_details 381​ return check_details
373 382​
View Options

libtaskotron/directives/resultsdb_directive.py

Show First 20 Lines • Show All 94 Lines • ▼ Show 20 Line(s)92 def __init__(self, resultsdb = None):
95 conf = config.get_config() 95​ conf = config.get_config()
96 self.masterurl = conf.taskotron_master 96​ self.masterurl = conf.taskotron_master
97 self.task_stepname = conf.buildbot_task_step 97​ self.task_stepname = conf.buildbot_task_step
98 self.execdb_server = "%s/jobs" % conf.execdb_server 98​ self.execdb_server = "%s/jobs" % conf.execdb_server
99 99​
100 if self.resultsdb is None: 100​ if self.resultsdb is None:
101 self.resultsdb = resultsdb_api.ResultsDBapi(conf.resultsdb_server) 101​ self.resultsdb = resultsdb_api.ResultsDBapi(conf.resultsdb_server)
102 102​
103​ self._ensured_testcases = []
104​
103 def ensure_testcase_exists(self, name): 105​ def ensure_testcase_exists(self, name):
104 """ Make sure that the testcase exists in resultsdb, otherwise create 106​ """ Make sure that the testcase exists in resultsdb, otherwise create
105 the testcase using a dummy url as a reference 107​ the testcase using a dummy url as a reference
106 108​
107 :param str name: name of the testcase to check for 109​ :param str name: name of the testcase to check for
108 """ 110​ """
109 111​
112​ if name in self._ensured_testcases:
113​ return
114​
110 try: 115​ try:
111 self.resultsdb.get_testcase(name) 116​ self.resultsdb.get_testcase(name)
117​ self._ensured_testcases.append(name)
112 return 118​ return
113 except resultsdb_api.ResultsDBapiException, e: 119​ except resultsdb_api.ResultsDBapiException, e:
114 if e.message != 'Testcase not found': 120​ if e.message != 'Testcase not found':
115 raise e 121​ raise e
116 122​
117 # since the testcase doesn't exist, create it with a dummy value for url 123​ # since the testcase doesn't exist, create it with a dummy value for url
118 # it can be updated later when it's not blocking results reporting 124​ # it can be updated later when it's not blocking results reporting
119 dummyurl = 'http://faketestcasesRus.com/%s' % name 125​ dummyurl = 'http://faketestcasesRus.com/%s' % name
120 self.resultsdb.create_testcase(name, dummyurl) 126​ self.resultsdb.create_testcase(name, dummyurl)
127​ self._ensured_testcases.append(name)
121 128​
122 def create_resultsdb_job(self, name, uuid=None, refurl=None): 129​ def create_resultsdb_job(self, name, uuid=None, refurl=None):
123 """ Create a job in resultsdb for reporting results against 130​ """ Create a job in resultsdb for reporting results against
124 131​
125 :param str name: name of job to report against 132​ :param str name: name of job to report against
126 :param str uuid: UUID of the job (most probably provided by ExecDB) 133​ :param str uuid: UUID of the job (most probably provided by ExecDB)
127 :param str refurl: url pointing to the execution overview. 134​ :param str refurl: url pointing to the execution overview.
128 If set to None, ExecDB url is created from UUID 135​ If set to None, ExecDB url is created from UUID
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 Line(s)158 def process(self, input_data, env_data):
182 # the job needs to be 'RUNNING' in order to append any results 189​ # the job needs to be 'RUNNING' in order to append any results
183 job_url, log_url = buildbot_utils.get_urls(env_data['jobid'], 190​ job_url, log_url = buildbot_utils.get_urls(env_data['jobid'],
184 self.masterurl, 191​ self.masterurl,
185 self.task_stepname) 192​ self.task_stepname)
186 job_data = self.create_resultsdb_job(env_data['checkname'], 193​ job_data = self.create_resultsdb_job(env_data['checkname'],
187 uuid=env_data['uuid'], 194​ uuid=env_data['uuid'],
188 ) 195​ )
189 196​
190 self.ensure_testcase_exists(env_data['checkname']) ​
191 197​
192 output = [] 198​ output = []
193 199​
194 log.info('Posting %s results to ResultsDB...' % len(check_details)) 200​ log.info('Posting %s results to ResultsDB...' % len(check_details))
195 for detail in check_details: 201​ for detail in check_details:
202​ checkname = detail.checkname or env_data['checkname']
203​ self.ensure_testcase_exists(checkname)
204​
196 try: 205​ try:
197 result = self.resultsdb.create_result( 206​ result = self.resultsdb.create_result(
198 job_id = job_data['id'], 207​ job_id = job_data['id'],
199 testcase_name = env_data['checkname'], 208​ testcase_name = checkname,
200 outcome = detail.outcome, 209​ outcome = detail.outcome,
201 summary = detail.summary or None, 210​ summary = detail.summary or None,
202 log_url = log_url, 211​ log_url = log_url,
203 item = detail.item, 212​ item = detail.item,
204 type = detail.report_type, 213​ type = detail.report_type,
205 **detail.keyvals) 214​ **detail.keyvals)
206 output.append(result) 215​ output.append(result)
207 216​
208 except resultsdb_api.ResultsDBapiException, e: 217​ except resultsdb_api.ResultsDBapiException, e:
209 log.error(e) 218​ log.error(e)
210 log.error("Failed to store ResulsDB: `%s` `%s` `%s`", 219​ log.error("Failed to store ResulsDB: `%s` `%s` `%s`",
211 detail.item, env_data['checkname'], detail.outcome) 220​ detail.item, checkname, detail.outcome)
212 221​
213 # move job to completed 222​ # move job to completed
214 self.complete_resultsdb_job(job_data['id']) 223​ self.complete_resultsdb_job(job_data['id'])
215 224​
216 return output 225​ return output
217 226​
218 227​
219def report_summary(tap, checkname): 228​def report_summary(tap, checkname):
Show All 27 Lines
View Options

testing/test_check.py

Show All 21 Lines15class TestCheckDetail(object):
22 def test_init_empty(self): 22​ def test_init_empty(self):
23 '''Test instance creation with empty parameters''' 23​ '''Test instance creation with empty parameters'''
24 # by default, outcome should be 'NEEDS_INSPECTION', not empty 24​ # by default, outcome should be 'NEEDS_INSPECTION', not empty
25 assert self.cd.outcome == 'NEEDS_INSPECTION' 25​ assert self.cd.outcome == 'NEEDS_INSPECTION'
26 assert not self.cd.item 26​ assert not self.cd.item
27 assert not self.cd.summary 27​ assert not self.cd.summary
28 assert not self.cd.output 28​ assert not self.cd.output
29 assert not self.cd.keyvals 29​ assert not self.cd.keyvals
30​ assert not self.cd.checkname
30 31​
31 def test_init_params(self): 32​ def test_init_params(self):
32 '''Test instance creation with non-empty parameters''' 33​ '''Test instance creation with non-empty parameters'''
33 cd = check.CheckDetail(item='foo-1.2-3.fc20', outcome='FAILED', 34​ cd = check.CheckDetail(item='foo-1.2-3.fc20', outcome='FAILED',
34 summary='untested', output=['line1', 'line2']) 35​ summary='untested', output=['line1', 'line2'],
36​ checkname='testcheck')
35 assert cd.item == 'foo-1.2-3.fc20' 37​ assert cd.item == 'foo-1.2-3.fc20'
36 assert cd.outcome == 'FAILED' 38​ assert cd.outcome == 'FAILED'
37 assert cd.summary == 'untested' 39​ assert cd.summary == 'untested'
38 assert cd.output == ['line1', 'line2'] 40​ assert cd.output == ['line1', 'line2']
41​ assert cd.checkname == 'testcheck'
39 42​
40 def test_init_raise(self): 43​ def test_init_raise(self):
41 '''Test instance creation that raises an error - invalid parameters''' 44​ '''Test instance creation that raises an error - invalid parameters'''
42 with pytest.raises(exc.TaskotronValueError): 45​ with pytest.raises(exc.TaskotronValueError):
43 check.CheckDetail(None, outcome='INVALID TYPE') 46​ check.CheckDetail(None, outcome='INVALID TYPE')
44 47​
45 with pytest.raises(exc.TaskotronValueError): 48​ with pytest.raises(exc.TaskotronValueError):
46 check.CheckDetail(None, output='foobar') 49​ check.CheckDetail(None, output='foobar')
▲ Show 20 Lines • Show All 138 Lines • ▼ Show 20 Line(s)185class TestExportTAP(object):
185 outcome = 'PASSED' 188​ outcome = 'PASSED'
186 summary = '5 ERRORS, 10 WARNINGS' 189​ summary = '5 ERRORS, 10 WARNINGS'
187 report_type = check.ReportType.KOJI_BUILD 190​ report_type = check.ReportType.KOJI_BUILD
188 output = '''\ 191​ output = '''\
189xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog 192​xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
190xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas 193​xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
191xchat.x86_64: W: no-manual-page-for-binary xchat''' 194​xchat.x86_64: W: no-manual-page-for-binary xchat'''
192 keyvals = {"foo": "bar", "moo": 11} 195​ keyvals = {"foo": "bar", "moo": 11}
196​ checkname = 'testcheck'
193 197​
194 def setup_method(self, method): 198​ def setup_method(self, method):
195 self.cd = check.CheckDetail(item=self.item, outcome=self.outcome, 199​ self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,
196 summary=self.summary, 200​ summary=self.summary,
197 report_type=self.report_type, 201​ report_type=self.report_type,
198 keyvals = self.keyvals) 202​ keyvals=self.keyvals,
203​ checkname=self.checkname)
199 self.cd.output.append(self.output) 204​ self.cd.output.append(self.output)
200 205​
201 def _tap_to_yaml(self, rendered_tap): 206​ def _tap_to_yaml(self, rendered_tap):
202 '''Remove all TAP-related contents and return only its YAML block(s)''' 207​ '''Remove all TAP-related contents and return only its YAML block(s)'''
203 lines = rendered_tap.splitlines()[2:] 208​ lines = rendered_tap.splitlines()[2:]
204 209​
205 # split the output to TAP result lines and YAML blocks 210​ # split the output to TAP result lines and YAML blocks
206 tap_lines = [line for line in lines if line.startswith('ok') or 211​ tap_lines = [line for line in lines if line.startswith('ok') or
Show All 26 Lines230 def test_single_yaml(self):
233 assert yaml_obj['item'] == self.item 238​ assert yaml_obj['item'] == self.item
234 assert yaml_obj['outcome'] == self.outcome 239​ assert yaml_obj['outcome'] == self.outcome
235 assert yaml_obj['summary'] == self.summary 240​ assert yaml_obj['summary'] == self.summary
236 assert yaml_obj['type'] == self.report_type 241​ assert yaml_obj['type'] == self.report_type
237 assert yaml_obj['details'] 242​ assert yaml_obj['details']
238 assert yaml_obj['details']['output'] == self.output 243​ assert yaml_obj['details']['output'] == self.output
239 assert yaml_obj['foo'] == self.keyvals['foo'] 244​ assert yaml_obj['foo'] == self.keyvals['foo']
240 assert yaml_obj['moo'] == self.keyvals['moo'] 245​ assert yaml_obj['moo'] == self.keyvals['moo']
246​ assert yaml_obj['checkname'] == self.checkname
241 247​
242 def test_invalid_keyvals(self): 248​ def test_invalid_keyvals(self):
243 '''Test export with keyvals containing reserved keys.''' 249​ '''Test export with keyvals containing reserved keys.'''
244 cd = deepcopy(self.cd) 250​ cd = deepcopy(self.cd)
245 251​
246 for key in check.RESERVED_KEYS: 252​ for key in check.RESERVED_KEYS:
247 cd.keyvals[key] = 'foo' 253​ cd.keyvals[key] = 'foo'
248 254​
249 tap_output = check.export_TAP(cd) 255​ tap_output = check.export_TAP(cd)
250 256​
251 # specific lines must exist 257​ # specific lines must exist
252 yaml_block = self._tap_to_yaml(tap_output) 258​ yaml_block = self._tap_to_yaml(tap_output)
253 yaml_obj = yaml.load(yaml_block) 259​ yaml_obj = yaml.load(yaml_block)
254 260​
255 assert yaml_obj['item'] == self.item 261​ assert yaml_obj['item'] == self.item
256 assert yaml_obj['outcome'] == self.outcome 262​ assert yaml_obj['outcome'] == self.outcome
257 assert yaml_obj['summary'] == self.summary 263​ assert yaml_obj['summary'] == self.summary
258 assert yaml_obj['type'] == self.report_type 264​ assert yaml_obj['type'] == self.report_type
259 assert yaml_obj['details'] 265​ assert yaml_obj['details']
260 assert yaml_obj['details']['output'] == self.output 266​ assert yaml_obj['details']['output'] == self.output
261 assert yaml_obj['foo'] == self.keyvals['foo'] 267​ assert yaml_obj['foo'] == self.keyvals['foo']
262 assert yaml_obj['moo'] == self.keyvals['moo'] 268​ assert yaml_obj['moo'] == self.keyvals['moo']
269​ assert yaml_obj['checkname'] == self.checkname
263 270​
264 def test_multi(self): 271​ def test_multi(self):
265 '''Test export with multiple item sections''' 272​ '''Test export with multiple item sections'''
266 cd2 = check.CheckDetail(item='foobar-1.2-3.fc20', outcome='FAILED', 273​ cd2 = check.CheckDetail(item='foobar-1.2-3.fc20', outcome='FAILED',
267 summary='dependency error', 274​ summary='dependency error',
268 report_type=check.ReportType.BODHI_UPDATE) 275​ report_type=check.ReportType.BODHI_UPDATE)
269 cd3 = check.CheckDetail(item='f20-updates', outcome='INFO', 276​ cd3 = check.CheckDetail(item='f20-updates', outcome='INFO',
270 summary='2 stale updates', 277​ summary='2 stale updates',
▲ Show 20 Lines • Show All 89 Lines • ▼ Show 20 Line(s)364class TestImportTAP(object):
360 outcome = 'PASSED' 367​ outcome = 'PASSED'
361 summary = '5 ERRORS, 10 WARNINGS' 368​ summary = '5 ERRORS, 10 WARNINGS'
362 report_type = check.ReportType.KOJI_BUILD 369​ report_type = check.ReportType.KOJI_BUILD
363 output = '''\ 370​ output = '''\
364xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog 371​xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
365xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas 372​xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
366xchat.x86_64: W: no-manual-page-for-binary xchat''' 373​xchat.x86_64: W: no-manual-page-for-binary xchat'''
367 keyvals = {"foo": "bar", "moo": 11} 374​ keyvals = {"foo": "bar", "moo": 11}
375​ checkname = 'testcheck'
368 376​
369 tap_output = """ 377​ tap_output = """
370TAP version 13 378​TAP version 13
3711..1 379​1..1
372ok - $CHECKNAME for Koji build xchat-0.5-1.fc20 380​ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
373 --- 381​ ---
374 details: 382​ details:
375 output: |- 383​ output: |-
376 xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog 384​ xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
377 xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas 385​ xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
378 xchat.x86_64: W: no-manual-page-for-binary xchat 386​ xchat.x86_64: W: no-manual-page-for-binary xchat
379 item: xchat-0.5-1.fc20 387​ item: xchat-0.5-1.fc20
380 outcome: PASSED 388​ outcome: PASSED
381 summary: 5 ERRORS, 10 WARNINGS 389​ summary: 5 ERRORS, 10 WARNINGS
382 type: koji_build 390​ type: koji_build
383 foo: bar 391​ foo: bar
384 moo: 11 392​ moo: 11
393​ checkname: testcheck
385 ...""" 394​ ..."""
386 395​
387 def setup_method(self, method): 396​ def setup_method(self, method):
388 self.cd = check.CheckDetail(item=self.item, outcome=self.outcome, 397​ self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,
389 summary=self.summary, 398​ summary=self.summary,
390 report_type=self.report_type, 399​ report_type=self.report_type,
391 keyvals=self.keyvals) 400​ keyvals=self.keyvals,
401​ checkname=self.checkname)
392 self.cd.output.append(self.output) 402​ self.cd.output.append(self.output)
393 403​
394 def test_import(self): 404​ def test_import(self):
395 cds = check.import_TAP(self.tap_output) 405​ cds = check.import_TAP(self.tap_output)
396 406​
397 assert len(cds) == 1 407​ assert len(cds) == 1
398 assert isinstance(cds[0], check.CheckDetail) 408​ assert isinstance(cds[0], check.CheckDetail)
399 409​
400 cd = cds[0] 410​ cd = cds[0]
401 assert cd.item == self.cd.item 411​ assert cd.item == self.cd.item
402 assert cd.outcome == self.cd.outcome 412​ assert cd.outcome == self.cd.outcome
403 assert cd.summary == self.cd.summary 413​ assert cd.summary == self.cd.summary
404 assert cd.report_type == self.cd.report_type 414​ assert cd.report_type == self.cd.report_type
405 assert cd.output == self.cd.output 415​ assert cd.output == self.cd.output
406 assert cd.keyvals == self.keyvals 416​ assert cd.keyvals == self.keyvals
417​ assert cd.checkname == self.checkname
407 418​
408 def test_import_exported(self): 419​ def test_import_exported(self):
409 tap_output = check.export_TAP(self.cd) 420​ tap_output = check.export_TAP(self.cd)
410 421​
411 cds = check.import_TAP(tap_output) 422​ cds = check.import_TAP(tap_output)
412 423​
413 assert len(cds) == 1 424​ assert len(cds) == 1
414 assert isinstance(cds[0], check.CheckDetail) 425​ assert isinstance(cds[0], check.CheckDetail)
415 426​
416 cd = cds[0] 427​ cd = cds[0]
417 assert cd.item == self.cd.item 428​ assert cd.item == self.cd.item
418 assert cd.outcome == self.cd.outcome 429​ assert cd.outcome == self.cd.outcome
419 assert cd.summary == self.cd.summary 430​ assert cd.summary == self.cd.summary
420 assert cd.report_type == self.cd.report_type 431​ assert cd.report_type == self.cd.report_type
421 assert cd.output == self.cd.output 432​ assert cd.output == self.cd.output
422 assert cd.keyvals == self.keyvals 433​ assert cd.keyvals == self.keyvals
434​ assert cd.checkname == self.checkname
423 435​
424 436​
425 def test_invalid_tap(self): 437​ def test_invalid_tap(self):
426 tap_output = """ 438​ tap_output = """
427TAP version 13 439​TAP version 13
428ok 2 foo 440​ok 2 foo
429ok 1 bar""" 441​ok 1 bar"""
430 with pytest.raises(exc.TaskotronValueError): 442​ with pytest.raises(exc.TaskotronValueError):
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 Line(s)487...'''
477 assert len(cds) == 1 489​ assert len(cds) == 1
478 assert isinstance(cds[0], check.CheckDetail) 490​ assert isinstance(cds[0], check.CheckDetail)
479 491​
480 cd = cds[0] 492​ cd = cds[0]
481 assert not cd.item 493​ assert not cd.item
482 assert cd.outcome == 'FAILED' 494​ assert cd.outcome == 'FAILED'
483 assert not cd.summary 495​ assert not cd.summary
484 assert not cd.report_type 496​ assert not cd.report_type
485 assert not cd.output ​
486 assert not cd.keyvals 497​ assert not cd.keyvals
498​ assert not cd.output
499​ assert not cd.checkname
View Options

testing/test_resultsdb_directive.py

Show First 20 Lines • Show All 200 Lines • ▼ Show 20 Line(s)22class TestResultsdbReport():
201 def test_ensure_testcase_creation_exists(self): 201​ def test_ensure_testcase_creation_exists(self):
202 # check to make sure that a testcase is _not_ created in the case that 202​ # check to make sure that a testcase is _not_ created in the case that
203 # it already exists 203​ # it already exists
204 204​
205 self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname']) 205​ self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
206 206​
207 rdb_calls = self.stub_rdb.calls() 207​ rdb_calls = self.stub_rdb.calls()
208 208​
209 # dingus doesn't record calls to exception_raiser 209​ assert len(rdb_calls) == 1
210​ assert rdb_calls[0][0] == 'get_testcase'
211​
212​ def test_ensure_testcase_cached(self):
213​ # check to make sure that ensure_testcase_exists does not query resultsdb unnecessarily
214​
215​ self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
216​ self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
217​
218​ rdb_calls = self.stub_rdb.calls()
219​
210 assert len(rdb_calls) == 1 220​ assert len(rdb_calls) == 1
211 assert rdb_calls[0][0] == 'get_testcase' 221​ assert rdb_calls[0][0] == 'get_testcase'