Differential D395

bodhi_comment_directive's log_url can point to ResultsDB
ClosedPublic
Actions

Authored by jskladan on Jun 17 2015, 12:21 PM.

Details

Reviewers
kparal
tflink
mkrizek
Maniphest Tasks
T502: Change bodhi_comment_directive so it links to the specific result in ResultsDB
Commits
rLTRN39a62f7a1c5c: bodhi_comment_directive: log_url can point to ResultsDB status page
Summary

Now that we have artifacts that only contain the 'sane' subset of tests' output,
lets change the URL in Bodhi-comment, so it points to the respective ResultsDB's
result page, from which it is then possible to either go to the log, or view the
other results.

In order to do that, it is necessary for the resultsdb_directive to produce
information about item->result_id pairing. After a discussion with @kparal
I've added another reserved key to the TAP's YAML section - __internal.
The value is meant to be a dictionary, that will contain the implementation
specific bits of information, when we need to transfer data from one directive
to another.

Test Plan

added unittests

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
jskladan retitled this revision from to bodhi_comment_directive's log_url can point to ResultsDB.Jun 17 2015, 12:21 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added reviewers: kparal, mkrizek, tflink.
jskladan added a subscriber: kparal.
kparal added a comment.Jun 17 2015, 1:06 PM

A few nitpicks and documentation requests.

libtaskotron/check.py
24

The attribute has a single underscore, but this has two. It's not very important, but if we synchronized it (I think one underscore is enough), it would look prettier.

122

We probably don't want to document this in a docstring, but at least a comment explaining what it is for would be great.

libtaskotron/directives/bodhi_comment_directive.py
61–98

I think we will need to include another example here to show how to send Bodhi comments which include links to ResultDB and not directly to BuildBot. And it would be great to also shortly mention it in the directive description.

183–184

Why exactly do we check for AttributeError here? Just a few lines below we use detail.keyvals and we don't check it either. If someone fills the attribute with invalid value type, let's just crash? Also, even if we keep it, at least log the error?

libtaskotron/directives/resultsdb_directive.py
257

The return value has changed, but the documentation hasn't been adjusted. It would be great not only to mention what it returns, but also why (in case you need to use it later, e.g. in bodhi comments directive).

jskladan updated this revision to Diff 1051.Jun 17 2015, 1:27 PM
kparal accepted this revision.Jun 17 2015, 1:32 PM
kparal added inline comments.
libtaskotron/directives/bodhi_comment_directive.py
77

typo :P

This revision is now accepted and ready to land.Jun 17 2015, 1:32 PM
jskladan updated this revision to Diff 1052.Jun 17 2015, 1:35 PM
  • fixed typo and bodhi_comment_directive docs
mkrizek accepted this revision.Jun 23 2015, 8:53 AM
Closed by commit rLTRN39a62f7a1c5c: bodhi_comment_directive: log_url can point to ResultsDB status page (authored by Josef Skladanka <jskladan@redhat.com>, committed by kparal). · Explain WhyJun 23 2015, 8:57 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 1071

View Options

libtaskotron/check.py

Show All 13 Lines
14from . import python_utils 14​from . import python_utils
15from . import exceptions as exc 15​from . import exceptions as exc
16from .logger import log 16​from .logger import log
17 17​
18from pytap13 import TAP13 18​from pytap13 import TAP13
19 19​
20# a list of reserved keywords for TAP output, which can't be overridden in 20​# a list of reserved keywords for TAP output, which can't be overridden in
21# keyvals 21​# keyvals
22RESERVED_KEYS = ('item', 'type', 'outcome', 'summary', 'details', 'checkname', 'artifact') 22​RESERVED_KEYS = ('item', 'type', 'outcome',
23​ 'summary', 'details', 'checkname',
24​ 'artifact', '_internal')
kparalUnsubmitted
Not Done

The attribute has a single underscore, but this has two. It's not very important, but if we synchronized it (I think one underscore is enough), it would look prettier.

23 25​
24class CheckDetail(object): 26​class CheckDetail(object):
25 '''Class encompassing everything related to the outcome of a check run. 27​ '''Class encompassing everything related to the outcome of a check run.
26 It contains convenience methods for tracking, evaluating and reporting 28​ It contains convenience methods for tracking, evaluating and reporting
27 check results. 29​ check results.
28 30​
29 For some checks, it's trivial to parse its output at the end of its 31​ For some checks, it's trivial to parse its output at the end of its
30 execution and evaluate the results. For some, however, it's much easier 32​ execution and evaluate the results. For some, however, it's much easier
▲ Show 20 Lines • Show All 81 Lines • ▼ Show 20 Line(s)101 def __init__(self, item, report_type=None, outcome=None, summary='',
112 self._outcome = None 114​ self._outcome = None
113 if outcome: 115​ if outcome:
114 self.outcome = outcome 116​ self.outcome = outcome
115 self.checkname = checkname 117​ self.checkname = checkname
116 self.summary = summary 118​ self.summary = summary
117 self.output = output or [] 119​ self.output = output or []
118 self.keyvals = keyvals or {} 120​ self.keyvals = keyvals or {}
119 self.artifact = artifact 121​ self.artifact = artifact
122​ # this dictionary will hold implementation-specific data, that needs
kparalUnsubmitted
Not Done

We probably don't want to document this in a docstring, but at least a comment explaining what it is for would be great.

123​ # to be transfered in between directives (like resultsdb_result_id)
124​ self._internal = {}
120 125​
121 126​
122 @property 127​ @property
123 def outcome(self): 128​ def outcome(self):
124 return self._outcome or 'NEEDS_INSPECTION' 129​ return self._outcome or 'NEEDS_INSPECTION'
125 130​
126 131​
127 @outcome.setter 132​ @outcome.setter
▲ Show 20 Lines • Show All 198 Lines • ▼ Show 20 Line(s)330 if detail.artifact:
326 data['artifact'] = detail.artifact 331​ data['artifact'] = detail.artifact
327 if detail.report_type: 332​ if detail.report_type:
328 data['type'] = detail.report_type 333​ data['type'] = detail.report_type
329 if detail.summary: 334​ if detail.summary:
330 data['summary'] = detail.summary 335​ data['summary'] = detail.summary
331 if detail.checkname: 336​ if detail.checkname:
332 data['checkname'] = detail.checkname 337​ data['checkname'] = detail.checkname
333 data['details'] = {} 338​ data['details'] = {}
339​ if detail._internal:
340​ data['_internal'] = detail._internal
334 if detail.output: 341​ if detail.output:
335 data['details']['output'] = '\n'.join(detail.output) 342​ data['details']['output'] = '\n'.join(detail.output)
336 if not data['details']: 343​ if not data['details']:
337 del data['details'] 344​ del data['details']
338 345​
339 for key, value in detail.keyvals.iteritems(): 346​ for key, value in detail.keyvals.iteritems():
340 if key in RESERVED_KEYS: 347​ if key in RESERVED_KEYS:
341 log.warn("Reserved key '%s' found in keyvals. Ignoring for " 348​ log.warn("Reserved key '%s' found in keyvals. Ignoring for "
Show All 28 Lines374 for test in tap.tests:
370 report_type = y.get('type', None) 377​ report_type = y.get('type', None)
371 artifact = y.get('artifact', None) 378​ artifact = y.get('artifact', None)
372 outcome = 'PASSED' if test.result == 'ok' else 'FAILED' 379​ outcome = 'PASSED' if test.result == 'ok' else 'FAILED'
373 outcome = y.get('outcome', outcome) 380​ outcome = y.get('outcome', outcome)
374 summary = y.get('summary', '') 381​ summary = y.get('summary', '')
375 checkname = y.get('checkname', None) 382​ checkname = y.get('checkname', None)
376 details = y.get('details', {}) 383​ details = y.get('details', {})
377 output = details.get('output', None) 384​ output = details.get('output', None)
385​ _internal = y.get('_internal', {})
378 if output is not None: 386​ if output is not None:
379 output = [output] 387​ output = [output]
380 388​
381 # TODO: is there a better way to do this? 389​ # TODO: is there a better way to do this?
382 other_keys = set(y.keys()) - set(RESERVED_KEYS) 390​ other_keys = set(y.keys()) - set(RESERVED_KEYS)
383 keyvals = dict([(k,y[k]) for k in other_keys]) 391​ keyvals = dict([(k,y[k]) for k in other_keys])
384 392​
385 cd = CheckDetail(item, report_type, outcome, summary, output, checkname=checkname, artifact=artifact) 393​ cd = CheckDetail(item, report_type, outcome, summary, output, checkname=checkname, artifact=artifact)
386 394​
387 cd.keyvals = keyvals 395​ cd.keyvals = keyvals
396​ cd._internal = _internal
388 check_details.append(cd) 397​ check_details.append(cd)
389 398​
390 return check_details 399​ return check_details
391 400​
View Options

libtaskotron/config_defaults.py

Show First 20 Lines • Show All 49 Lines • ▼ Show 20 Line(s)23class Config(object):
50 report_to_bodhi = True #: 50​ report_to_bodhi = True #:
51 report_to_resultsdb = True #: 51​ report_to_resultsdb = True #:
52 52​
53 koji_url = 'http://koji.fedoraproject.org/kojihub' #: 53​ koji_url = 'http://koji.fedoraproject.org/kojihub' #:
54 pkg_url = 'http://kojipkgs.fedoraproject.org/packages' #: 54​ pkg_url = 'http://kojipkgs.fedoraproject.org/packages' #:
55 bodhi_server = 'https://admin.fedoraproject.org/updates/' #: 55​ bodhi_server = 'https://admin.fedoraproject.org/updates/' #:
56 execdb_server = 'http://127.0.0.1/execdb/' #: 56​ execdb_server = 'http://127.0.0.1/execdb/' #:
57 resultsdb_server = 'http://127.0.0.1/resultsdb/api/v1.0/' #: 57​ resultsdb_server = 'http://127.0.0.1/resultsdb/api/v1.0/' #:
58​ resultsdb_frontend = 'http://127.0.0.1/resultsdb_frontend/' #:
58 taskotron_master = 'http://127.0.0.1/taskmaster/' #: 59​ taskotron_master = 'http://127.0.0.1/taskmaster/' #:
59 artifacts_baseurl = 'http://127.0.0.1/artifacts/' #: 60​ artifacts_baseurl = 'http://127.0.0.1/artifacts/' #:
60 buildbot_task_step = 'runtask' #: 61​ buildbot_task_step = 'runtask' #:
61 download_cache_enabled = True #: 62​ download_cache_enabled = True #:
62 63​
63 bodhi_posting_comments_span = 4320 #: 64​ bodhi_posting_comments_span = 4320 #:
64 # 3 days (3*24*60 = 4320) 65​ # 3 days (3*24*60 = 4320)
65 bodhi_request_max_retries = 3 #: 66​ bodhi_request_max_retries = 3 #:
▲ Show 20 Lines • Show All 51 Lines • Show Last 20 Lines
View Options

libtaskotron/directives/bodhi_comment_directive.py

Show All 14 Lines10description: |
15 .. _Bodhi: https://admin.fedoraproject.org/updates/ 15​ .. _Bodhi: https://admin.fedoraproject.org/updates/
16 16​
17 The comment will roughly look like this:: 17​ The comment will roughly look like this::
18 18​
19 Taskotron: rpmlint test PASSED on x86_64. Result log: <url> 19​ Taskotron: rpmlint test PASSED on x86_64. Result log: <url>
20 20​
21 .. note:: You have to have ``fas_username`` and ``fas_password`` configured 21​ .. note:: You have to have ``fas_username`` and ``fas_password`` configured
22 in your config file, otherwise the comments won't be posted. 22​ in your config file, otherwise the comments won't be posted.
23​
24​ .. note:: The <url> will point to the Buildbot's full stdio log.
25​ It is possible to make the <url> point to the respective ResultsDB's
26​ result page, if you use the ``resultsdb_directive``'s exported output
27​ as an input for this directive (see Examples section).
23parameters: 28​parameters:
24 results: 29​ results:
25 required: true 30​ required: true
26 description: single or multiple check results in TAP13 format 31​ description: single or multiple check results in TAP13 format
27 type: str 32​ type: str
28 doreport: 33​ doreport:
29 required: true 34​ required: true
30 description: | 35​ description: |
Show All 17 Lines
48raises: | 53​raises: |
49 * :class:`.TaskotronDirectiveError`: if ``results`` is not a string in valid 54​ * :class:`.TaskotronDirectiveError`: if ``results`` is not a string in valid
50 TAP13 format 55​ TAP13 format
51 * :class:`.TaskotronValueError`: if ``results`` refer to a Bodhi update which 56​ * :class:`.TaskotronValueError`: if ``results`` refer to a Bodhi update which
52 can't be found in Bodhi 57​ can't be found in Bodhi
53version_added: 0.4 58​version_added: 0.4
54""" 59​"""
55 60​
56EXAMPLES = """ 61​EXAMPLES = """
57Run ``upgradepath`` check and then report all results to Bodhi (using 62​Run ``upgradepath`` check and then report all results to Bodhi (using
58``onchange`` approach):: 63​``onchange`` approach)::
59 64​
60 - name: run upgradepath 65​ - name: run upgradepath
61 python: 66​ python:
62 file: upgradepath.py 67​ file: upgradepath.py
63 callable: main 68​ callable: main
64 custom_args: [--debug, "${koji_tag}"] 69​ custom_args: [--debug, "${koji_tag}"]
65 export: upgradepath_output 70​ export: upgradepath_output
66 71​
67 - name: report results to Bodhi 72​ - name: report results to Bodhi
68 bodhi_comment: 73​ bodhi_comment:
69 results: ${upgradepath_output} 74​ results: ${upgradepath_output}
70 doreport: 'onchange' 75​ doreport: 'onchange'
76​
77​Run ``upgradepath`` check, report results to ResultsDB and then to
kparalUnsubmitted
Not Done

typo :P

78​Bodhi (using ``onchange`` approach). This also causes the url of the
79​logs in the Bodhi comment to point to the ResultsDB's respective
80​result page, instead of the full stdio log.::
81​
82​ - name: run upgradepath
83​ python:
84​ file: upgradepath.py
85​ callable: main
86​ custom_args: [--debug, "${koji_tag}"]
87​ export: upgradepath_output
88​
89​ - name: report results to ResultsDB
90​ resultsdb:
91​ results: ${upgradepath_output}
92​ export: ${resultsdb_output}
93​
94​ - name: report results to Bodhi
95​ bodhi_comment:
96​ results: ${resultsdb_output}
97​ doreport: 'onchange'
71""" 98​"""
kparalUnsubmitted
Not Done

I think we will need to include another example here to show how to send Bodhi comments which include links to ResultDB and not directly to BuildBot. And it would be great to also shortly mention it in the directive description.

72 99​
73from libtaskotron.directives import BaseDirective 100​from libtaskotron.directives import BaseDirective
74 101​
75from libtaskotron.logger import log 102​from libtaskotron.logger import log
76from libtaskotron import check 103​from libtaskotron import check
77from libtaskotron import config 104​from libtaskotron import config
78from libtaskotron import bodhi_utils 105​from libtaskotron import bodhi_utils
79from libtaskotron import buildbot_utils 106​from libtaskotron import buildbot_utils
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Line(s)136 def process(self, input_data, env_data):
144 171​
145 _, log_url = buildbot_utils.get_urls(env_data['jobid'], 172​ _, log_url = buildbot_utils.get_urls(env_data['jobid'],
146 Config.taskotron_master, 173​ Config.taskotron_master,
147 Config.buildbot_task_step) 174​ Config.buildbot_task_step)
148 175​
149 log.info('Posting %s results to Bodhi...', len(check_details)) 176​ log.info('Posting %s results to Bodhi...', len(check_details))
150 177​
151 for index, detail in enumerate(check_details, start=1): 178​ for index, detail in enumerate(check_details, start=1):
179​ # Check for the presence of resultsdb's result id, and create
180​ # alternate log_url, than points to the respective frontend page.
181​ resultsdb_id = detail._internal.get('resultsdb_result_id', None)
182​ if resultsdb_id:
183​ log_url = "%s/results/%s" % (Config.resultsdb_frontend, resultsdb_id)
184​
kparalUnsubmitted
Not Done

Why exactly do we check for AttributeError here? Just a few lines below we use detail.keyvals and we don't check it either. If someone fills the attribute with invalid value type, let's just crash? Also, even if we keep it, at least log the error?

152 outcome = _post_testresult(self.bodhi_api, detail.item, 185​ outcome = _post_testresult(self.bodhi_api, detail.item,
153 env_data['checkname'], 186​ env_data['checkname'],
154 detail.outcome, 187​ detail.outcome,
155 url = log_url, 188​ url = log_url,
156 doreport = input_data['doreport'], 189​ doreport = input_data['doreport'],
157 arch = detail.keyvals.get('arch', 'noarch'), 190​ arch = detail.keyvals.get('arch', 'noarch'),
158 ) 191​ )
159 192​
▲ Show 20 Lines • Show All 332 Lines • Show Last 20 Lines
View Options

libtaskotron/directives/resultsdb_directive.py

Show All 18 Lines10description: |
19 19​
20 .. _TAP13 format: http://testanything.org/tap-version-13-specification.html 20​ .. _TAP13 format: http://testanything.org/tap-version-13-specification.html
21parameters: 21​parameters:
22 results: 22​ results:
23 required: true 23​ required: true
24 description: The output of the check in TAP13 format 24​ description: The output of the check in TAP13 format
25 type: str 25​ type: str
26returns: | 26​returns: |
27 A list of HTTP response objects, one for every HTTP request sent to create a 27​ TAP13 formatted output as taken from input, enhanced with result-id's for each
28 result in ResultsDB (i.e. there should be the same number of responses as 28​ successfully stored result. The result-id information then can be used by
29 there are results in ``results`` parameter). The response objects are created 29​ other directives - e.g. ``bodhi_comment_directive``.
30 by :meth:`requests.Response.json`, i.e. they are python dictionaries created ​
31 from ResultsDB's response JSON object. ​
32raises: | 30​raises: |
33 * :class:`.TaskotronDirectiveError`: if ``results`` is not in a valid TAP13 31​ * :class:`.TaskotronDirectiveError`: if ``results`` is not in a valid TAP13
34 format, or when ``item`` or ``type`` are missing in the TAP's result data 32​ format, or when ``item`` or ``type`` are missing in the TAP's result data
35 * :class:`resultsdb_api.ResultsDBapiException`: for any errors coming from 33​ * :class:`resultsdb_api.ResultsDBapiException`: for any errors coming from
36 the ResultsDB server 34​ the ResultsDB server
37version_added: 0.4 35​version_added: 0.4
38""" 36​"""
39 37​
40EXAMPLES = """ 38​EXAMPLES = """
41These two actions first run ``run_rpmlint.py`` and export its TAP13 output to 39​These two actions first run ``run_rpmlint.py`` and export its TAP13 output to
42``${rpmlint_output}``, and then feed this TAP13 output to ``resultsdb`` 40​``${rpmlint_output}``, and then feed this TAP13 output to ``resultsdb``
43directive:: 41​directive::
44 42​
45 - name: run rpmlint on downloaded rpms 43​ - name: run rpmlint on downloaded rpms
46 python: 44​ python:
47 file: run_rpmlint.py 45​ file: run_rpmlint.py
48 callable: run 46​ callable: run
49 workdir: ${workdir} 47​ workdir: ${workdir}
50 export: rpmlint_output 48​ export: rpmlint_output
51 49​
52 - name: report results to resultsdb 50​ - name: report results to resultsdb
53 resultsdb: 51​ resultsdb:
54 results: ${rpmlint_output} 52​ results: ${rpmlint_output}
53​ export: resultsdb_output
55 54​
56If ResultsDB reporting is configured in the config file, it will get saved on 55​If ResultsDB reporting is configured in the config file, it will get saved on
57the ResultsDB server, otherwise only TAP13 compliance will get checked and some 56​the ResultsDB server, otherwise only TAP13 compliance will get checked and some
58summary will be printed out into the log, like this:: 57​summary will be printed out into the log, like this::
59 58​
60 [libtaskotron:resultsdb_directive.py:143] 2014-06-24 13:55:27 INFO TAP is OK. 59​ [libtaskotron:resultsdb_directive.py:143] 2014-06-24 13:55:27 INFO TAP is OK.
61 [libtaskotron:resultsdb_directive.py:144] 2014-06-24 13:55:27 INFO Reporting to ResultsDB is disabled. 60​ [libtaskotron:resultsdb_directive.py:144] 2014-06-24 13:55:27 INFO Reporting to ResultsDB is disabled.
62 [libtaskotron:resultsdb_directive.py:145] 2014-06-24 13:55:27 INFO Once enabled, the following would be reported: 61​ [libtaskotron:resultsdb_directive.py:145] 2014-06-24 13:55:27 INFO Once enabled, the following would be reported:
▲ Show 20 Lines • Show All 151 Lines • ▼ Show 20 Line(s)182 def process(self, input_data, env_data):
214 # the job needs to be 'RUNNING' in order to append any results 213​ # the job needs to be 'RUNNING' in order to append any results
215 job_url, log_url = buildbot_utils.get_urls(env_data['jobid'], 214​ job_url, log_url = buildbot_utils.get_urls(env_data['jobid'],
216 self.masterurl, 215​ self.masterurl,
217 self.task_stepname) 216​ self.task_stepname)
218 job_data = self.create_resultsdb_job(env_data['checkname'], 217​ job_data = self.create_resultsdb_job(env_data['checkname'],
219 uuid=env_data['uuid'], 218​ uuid=env_data['uuid'],
220 ) 219​ )
221 220​
222 ​
223 output = [] ​
224 ​
225 log.info('Posting %s results to ResultsDB...' % len(check_details)) 221​ log.info('Posting %s results to ResultsDB...' % len(check_details))
226 for detail in check_details: 222​ for detail in check_details:
227 checkname = detail.checkname or env_data['checkname'] 223​ checkname = detail.checkname or env_data['checkname']
228 self.ensure_testcase_exists(checkname) 224​ self.ensure_testcase_exists(checkname)
229 result_log_url = log_url 225​ result_log_url = log_url
230 if detail.artifact: 226​ if detail.artifact:
231 artifact_path = self.get_artifact_path( 227​ artifact_path = self.get_artifact_path(
232 env_data['artifactsdir'], 228​ env_data['artifactsdir'],
Show All 10 Lines238 result = self.resultsdb.create_result(
243 job_id = job_data['id'], 239​ job_id = job_data['id'],
244 testcase_name = checkname, 240​ testcase_name = checkname,
245 outcome = detail.outcome, 241​ outcome = detail.outcome,
246 summary = detail.summary or None, 242​ summary = detail.summary or None,
247 log_url=result_log_url, 243​ log_url=result_log_url,
248 item = detail.item, 244​ item = detail.item,
249 type = detail.report_type, 245​ type = detail.report_type,
250 **detail.keyvals) 246​ **detail.keyvals)
251 output.append(result) 247​ detail._internal['resultsdb_result_id'] = result['id']
252 248​
253 except resultsdb_api.ResultsDBapiException, e: 249​ except resultsdb_api.ResultsDBapiException, e:
254 log.error(e) 250​ log.error(e)
255 log.error("Failed to store ResulsDB: `%s` `%s` `%s`", 251​ log.error("Failed to store ResulsDB: `%s` `%s` `%s`",
256 detail.item, checkname, detail.outcome) 252​ detail.item, checkname, detail.outcome)
257 253​
258 # move job to completed 254​ # move job to completed
259 self.complete_resultsdb_job(job_data['id']) 255​ self.complete_resultsdb_job(job_data['id'])
260 256​
261 return output 257​ return check.export_TAP(check_details)
kparalUnsubmitted
Not Done

The return value has changed, but the documentation hasn't been adjusted. It would be great not only to mention what it returns, but also why (in case you need to use it later, e.g. in bodhi comments directive).

262 258​
263 259​
264def report_summary(tap, checkname): 260​def report_summary(tap, checkname):
265 '''Create a pretty summary of what will/would be reported into ResultsDB. 261​ '''Create a pretty summary of what will/would be reported into ResultsDB.
266 The summary is based on TAP output, with the TAP header stripped out and 262​ The summary is based on TAP output, with the TAP header stripped out and
267 most of the check output stripped out. 263​ most of the check output stripped out.
268 264​
269 :param str tap: TAP-formatted results 265​ :param str tap: TAP-formatted results
Show All 22 Lines
View Options

testing/test_bodhi_comment_directive.py

Show First 20 Lines • Show All 293 Lines • ▼ Show 20 Line(s)283 def test_tap_mapping(self, prepare_for_bodhi_comment):
294 # Select the positional arguments of that call 294​ # Select the positional arguments of that call
295 call_data = call[1] 295​ call_data = call[1]
296 296​
297 assert call_data[0] == "update_title" 297​ assert call_data[0] == "update_title"
298 assert call_data[1] == ref_comment 298​ assert call_data[1] == ref_comment
299 assert call_data[2] == 0 299​ assert call_data[2] == 0
300 assert call_data[3] is False 300​ assert call_data[3] is False
301 301​
302​ def test_tap_mapping_resultsdb(self, prepare_for_bodhi_comment):
303​ """Checks whether TAP output is mapped correctly to
304​ the bodhi comment call when resultsdb_result_id is provided"""
305​
306​ Config = config.get_config()
307​
308​ cd = check.CheckDetail(
309​ item='update_title',
310​ report_type=check.ReportType.BODHI_UPDATE,
311​ outcome='PASSED',
312​ )
313​ cd._internal = {'resultsdb_result_id': 1234}
314​ log_url = "%s/results/1234" % Config.resultsdb_frontend
315​
316​ tap = check.export_TAP(cd)
317​
318​ ref_input = {
319​ 'results': tap,
320​ 'doreport': 'onchange',
321​ }
322​
323​ ref_comment = "Taskotron: depcheck test PASSED on noarch. "\
324​ "Result log: %s (results are informative only)" % log_url
325​ self.stub_bodhi.client = Dingus('bodhi_api.bodhi')
326​
327​ self.directive.process(ref_input, self.ref_envdata)
328​
329​ # Select the first call of "comment" method.
330​ call = [call for call in self.stub_bodhi.client.calls() if call[0] == 'comment'][0]
331​ # Select the positional arguments of that call
332​ call_data = call[1]
333​
334​ assert call_data[0] == "update_title"
335​ assert call_data[1] == ref_comment
336​ assert call_data[2] == 0
337​ assert call_data[3] is False
338​
302 def test_is_comment_email_needed(self, monkeypatch): 339​ def test_is_comment_email_needed(self, monkeypatch):
303 def mock_BodhiUpdateState_factory(current_state, new_state): 340​ def mock_BodhiUpdateState_factory(current_state, new_state):
304 341​
305 class MockBodhiUpdateState(Dingus): 342​ class MockBodhiUpdateState(Dingus):
306 def __init__(self, *args, **kwargs): 343​ def __init__(self, *args, **kwargs):
307 Dingus.__init__(self, *args, **kwargs) 344​ Dingus.__init__(self, *args, **kwargs)
308 self.state = current_state 345​ self.state = current_state
309 346​
▲ Show 20 Lines • Show All 42 Lines • Show Last 20 Lines
View Options

testing/test_check.py

Show First 20 Lines • Show All 190 Lines • ▼ Show 20 Line(s)186class TestExportTAP(object):
191 report_type = check.ReportType.KOJI_BUILD 191​ report_type = check.ReportType.KOJI_BUILD
192 output = '''\ 192​ output = '''\
193xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog 193​xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
194xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas 194​xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
195xchat.x86_64: W: no-manual-page-for-binary xchat''' 195​xchat.x86_64: W: no-manual-page-for-binary xchat'''
196 keyvals = {"foo": "bar", "moo": 11} 196​ keyvals = {"foo": "bar", "moo": 11}
197 checkname = 'testcheck' 197​ checkname = 'testcheck'
198 artifact = 'logfile.log' 198​ artifact = 'logfile.log'
199​ _internal = {'resultsdb_result_id': 1234}
199 200​
200 def setup_method(self, method): 201​ def setup_method(self, method):
201 self.cd = check.CheckDetail(item=self.item, outcome=self.outcome, 202​ self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,
202 summary=self.summary, 203​ summary=self.summary,
203 report_type=self.report_type, 204​ report_type=self.report_type,
204 keyvals=self.keyvals, 205​ keyvals=self.keyvals,
205 checkname=self.checkname, 206​ checkname=self.checkname,
206 artifact=self.artifact) 207​ artifact=self.artifact)
208​ self.cd._internal = self._internal
207 self.cd.output.append(self.output) 209​ self.cd.output.append(self.output)
208 210​
209 def _tap_to_yaml(self, rendered_tap): 211​ def _tap_to_yaml(self, rendered_tap):
210 '''Remove all TAP-related contents and return only its YAML block(s)''' 212​ '''Remove all TAP-related contents and return only its YAML block(s)'''
211 lines = rendered_tap.splitlines()[2:] 213​ lines = rendered_tap.splitlines()[2:]
212 214​
213 # split the output to TAP result lines and YAML blocks 215​ # split the output to TAP result lines and YAML blocks
214 tap_lines = [line for line in lines if line.startswith('ok') or 216​ tap_lines = [line for line in lines if line.startswith('ok') or
Show All 27 Lines235 def test_single_yaml(self):
242 assert yaml_obj['outcome'] == self.outcome 244​ assert yaml_obj['outcome'] == self.outcome
243 assert yaml_obj['summary'] == self.summary 245​ assert yaml_obj['summary'] == self.summary
244 assert yaml_obj['type'] == self.report_type 246​ assert yaml_obj['type'] == self.report_type
245 assert yaml_obj['details'] 247​ assert yaml_obj['details']
246 assert yaml_obj['details']['output'] == self.output 248​ assert yaml_obj['details']['output'] == self.output
247 assert yaml_obj['foo'] == self.keyvals['foo'] 249​ assert yaml_obj['foo'] == self.keyvals['foo']
248 assert yaml_obj['moo'] == self.keyvals['moo'] 250​ assert yaml_obj['moo'] == self.keyvals['moo']
249 assert yaml_obj['checkname'] == self.checkname 251​ assert yaml_obj['checkname'] == self.checkname
252​ assert yaml_obj['_internal'] == self._internal
250 253​
251 def test_invalid_keyvals(self): 254​ def test_invalid_keyvals(self):
252 '''Test export with keyvals containing reserved keys.''' 255​ '''Test export with keyvals containing reserved keys.'''
253 cd = deepcopy(self.cd) 256​ cd = deepcopy(self.cd)
254 257​
255 for key in check.RESERVED_KEYS: 258​ for key in check.RESERVED_KEYS:
256 cd.keyvals[key] = 'foo' 259​ cd.keyvals[key] = 'foo'
257 260​
258 tap_output = check.export_TAP(cd) 261​ tap_output = check.export_TAP(cd)
259 262​
260 # specific lines must exist 263​ # specific lines must exist
261 yaml_block = self._tap_to_yaml(tap_output) 264​ yaml_block = self._tap_to_yaml(tap_output)
262 yaml_obj = yaml.load(yaml_block) 265​ yaml_obj = yaml.load(yaml_block)
263 266​
264 assert yaml_obj['item'] == self.item 267​ assert yaml_obj['item'] == self.item
265 assert yaml_obj['outcome'] == self.outcome 268​ assert yaml_obj['outcome'] == self.outcome
266 assert yaml_obj['summary'] == self.summary 269​ assert yaml_obj['summary'] == self.summary
267 assert yaml_obj['type'] == self.report_type 270​ assert yaml_obj['type'] == self.report_type
268 assert yaml_obj['details'] 271​ assert yaml_obj['details']
269 assert yaml_obj['details']['output'] == self.output 272​ assert yaml_obj['details']['output'] == self.output
270 assert yaml_obj['foo'] == self.keyvals['foo'] 273​ assert yaml_obj['foo'] == self.keyvals['foo']
271 assert yaml_obj['moo'] == self.keyvals['moo'] 274​ assert yaml_obj['moo'] == self.keyvals['moo']
272 assert yaml_obj['checkname'] == self.checkname 275​ assert yaml_obj['checkname'] == self.checkname
276​ assert yaml_obj['_internal'] == self._internal
273 277​
274 def test_multi(self): 278​ def test_multi(self):
275 '''Test export with multiple item sections''' 279​ '''Test export with multiple item sections'''
276 cd2 = check.CheckDetail(item='foobar-1.2-3.fc20', outcome='FAILED', 280​ cd2 = check.CheckDetail(item='foobar-1.2-3.fc20', outcome='FAILED',
277 summary='dependency error', 281​ summary='dependency error',
278 report_type=check.ReportType.BODHI_UPDATE) 282​ report_type=check.ReportType.BODHI_UPDATE)
279 cd3 = check.CheckDetail(item='f20-updates', outcome='INFO', 283​ cd3 = check.CheckDetail(item='f20-updates', outcome='INFO',
280 summary='2 stale updates', 284​ summary='2 stale updates',
▲ Show 20 Lines • Show All 91 Lines • ▼ Show 20 Line(s)371class TestImportTAP(object):
372 report_type = check.ReportType.KOJI_BUILD 376​ report_type = check.ReportType.KOJI_BUILD
373 output = '''\ 377​ output = '''\
374xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog 378​xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
375xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas 379​xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
376xchat.x86_64: W: no-manual-page-for-binary xchat''' 380​xchat.x86_64: W: no-manual-page-for-binary xchat'''
377 keyvals = {"foo": "bar", "moo": 11} 381​ keyvals = {"foo": "bar", "moo": 11}
378 checkname = 'testcheck' 382​ checkname = 'testcheck'
379 artifact = 'logfile.log' 383​ artifact = 'logfile.log'
384​ _internal = {'resultsdb_result_id': 1234}
380 385​
381 tap_output = """ 386​ tap_output = """
382TAP version 13 387​TAP version 13
3831..1 388​1..1
384ok - $CHECKNAME for Koji build xchat-0.5-1.fc20 389​ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
385 --- 390​ ---
386 details: 391​ details:
387 output: |- 392​ output: |-
388 xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog 393​ xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
389 xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas 394​ xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps_xchat_url_handler.schemas
390 xchat.x86_64: W: no-manual-page-for-binary xchat 395​ xchat.x86_64: W: no-manual-page-for-binary xchat
391 item: xchat-0.5-1.fc20 396​ item: xchat-0.5-1.fc20
392 outcome: PASSED 397​ outcome: PASSED
393 artifact: logfile.log 398​ artifact: logfile.log
394 summary: 5 ERRORS, 10 WARNINGS 399​ summary: 5 ERRORS, 10 WARNINGS
395 type: koji_build 400​ type: koji_build
396 foo: bar 401​ foo: bar
397 moo: 11 402​ moo: 11
398 checkname: testcheck 403​ checkname: testcheck
404​ _internal:
405​ resultsdb_result_id: 1234
399 ...""" 406​ ..."""
400 407​
401 def setup_method(self, method): 408​ def setup_method(self, method):
402 self.cd = check.CheckDetail(item=self.item, outcome=self.outcome, 409​ self.cd = check.CheckDetail(item=self.item, outcome=self.outcome,
403 summary=self.summary, 410​ summary=self.summary,
404 report_type=self.report_type, 411​ report_type=self.report_type,
405 keyvals=self.keyvals, 412​ keyvals=self.keyvals,
406 checkname=self.checkname, 413​ checkname=self.checkname,
407 artifact=self.artifact) 414​ artifact=self.artifact)
415​ self.cd._internal = self._internal
408 self.cd.output.append(self.output) 416​ self.cd.output.append(self.output)
409 417​
410 def test_import(self): 418​ def test_import(self):
411 cds = check.import_TAP(self.tap_output) 419​ cds = check.import_TAP(self.tap_output)
412 420​
413 assert len(cds) == 1 421​ assert len(cds) == 1
414 assert isinstance(cds[0], check.CheckDetail) 422​ assert isinstance(cds[0], check.CheckDetail)
415 423​
416 cd = cds[0] 424​ cd = cds[0]
417 assert cd.item == self.cd.item 425​ assert cd.item == self.cd.item
418 assert cd.outcome == self.cd.outcome 426​ assert cd.outcome == self.cd.outcome
419 assert cd.summary == self.cd.summary 427​ assert cd.summary == self.cd.summary
420 assert cd.report_type == self.cd.report_type 428​ assert cd.report_type == self.cd.report_type
421 assert cd.output == self.cd.output 429​ assert cd.output == self.cd.output
422 assert cd.keyvals == self.keyvals 430​ assert cd.keyvals == self.keyvals
423 assert cd.checkname == self.checkname 431​ assert cd.checkname == self.checkname
424 assert cd.artifact == self.artifact 432​ assert cd.artifact == self.artifact
433​ assert cd._internal == self._internal
425 434​
426 def test_import_exported(self): 435​ def test_import_exported(self):
427 tap_output = check.export_TAP(self.cd) 436​ tap_output = check.export_TAP(self.cd)
428 437​
429 cds = check.import_TAP(tap_output) 438​ cds = check.import_TAP(tap_output)
430 439​
431 assert len(cds) == 1 440​ assert len(cds) == 1
432 assert isinstance(cds[0], check.CheckDetail) 441​ assert isinstance(cds[0], check.CheckDetail)
433 442​
434 cd = cds[0] 443​ cd = cds[0]
435 assert cd.item == self.cd.item 444​ assert cd.item == self.cd.item
436 assert cd.outcome == self.cd.outcome 445​ assert cd.outcome == self.cd.outcome
437 assert cd.summary == self.cd.summary 446​ assert cd.summary == self.cd.summary
438 assert cd.report_type == self.cd.report_type 447​ assert cd.report_type == self.cd.report_type
439 assert cd.output == self.cd.output 448​ assert cd.output == self.cd.output
440 assert cd.keyvals == self.keyvals 449​ assert cd.keyvals == self.keyvals
441 assert cd.checkname == self.checkname 450​ assert cd.checkname == self.checkname
442 assert cd.artifact == self.artifact 451​ assert cd.artifact == self.artifact
452​ assert cd._internal == self._internal
443 453​
444 454​
445 def test_invalid_tap(self): 455​ def test_invalid_tap(self):
446 tap_output = """ 456​ tap_output = """
447TAP version 13 457​TAP version 13
448ok 2 foo 458​ok 2 foo
449ok 1 bar""" 459​ok 1 bar"""
450 with pytest.raises(exc.TaskotronValueError): 460​ with pytest.raises(exc.TaskotronValueError):
Show All 30 Lines486ok'''
481 cd = cds[0] 491​ cd = cds[0]
482 assert not cd.item 492​ assert not cd.item
483 assert cd.outcome == 'PASSED' 493​ assert cd.outcome == 'PASSED'
484 assert not cd.summary 494​ assert not cd.summary
485 assert not cd.report_type 495​ assert not cd.report_type
486 assert not cd.output 496​ assert not cd.output
487 assert not cd.keyvals 497​ assert not cd.keyvals
488 assert not cd.artifact 498​ assert not cd.artifact
499​ assert not cd._internal
489 500​
490 501​
491 def test_empty_yaml(self): 502​ def test_empty_yaml(self):
492 tap_output = ''' 503​ tap_output = '''
493TAP version 13 504​TAP version 13
4941..1 505​1..1
495not ok 506​not ok
496--- 507​---
497...''' 508​...'''
498 cds = check.import_TAP(tap_output) 509​ cds = check.import_TAP(tap_output)
499 assert len(cds) == 1 510​ assert len(cds) == 1
500 assert isinstance(cds[0], check.CheckDetail) 511​ assert isinstance(cds[0], check.CheckDetail)
501 512​
502 cd = cds[0] 513​ cd = cds[0]
503 assert not cd.item 514​ assert not cd.item
504 assert cd.outcome == 'FAILED' 515​ assert cd.outcome == 'FAILED'
505 assert not cd.summary 516​ assert not cd.summary
506 assert not cd.report_type 517​ assert not cd.report_type
507 assert not cd.keyvals 518​ assert not cd.keyvals
508 assert not cd.output 519​ assert not cd.output
509 assert not cd.checkname 520​ assert not cd.checkname
510 assert not cd.artifact 521​ assert not cd.artifact
522​ assert not cd._internal
View Options

testing/test_resultsdb_directive.py

Show First 20 Lines • Show All 41 Lines • ▼ Show 20 Line(s)26 def setup(self, monkeypatch):
42 self.ref_envdata = { 42​ self.ref_envdata = {
43 'resultsdb_job_id': 1, 43​ 'resultsdb_job_id': 1,
44 'checkname': 'test_resultsdb_report', 44​ 'checkname': 'test_resultsdb_report',
45 'jobid': 'all/123', 45​ 'jobid': 'all/123',
46 'uuid': 'c25237a4-b6b3-11e4-b98a-3c970e018701', 46​ 'uuid': 'c25237a4-b6b3-11e4-b98a-3c970e018701',
47 'artifactsdir': '/some/directory/', 47​ 'artifactsdir': '/some/directory/',
48 } 48​ }
49 49​
50​ self.ref_resultdata = {u'id': 1234}
51​
50 self.ref_jobid = 1234 52​ self.ref_jobid = 1234
51 self.ref_refurl = u'http://example.com/%s' % self.ref_envdata['checkname'] 53​ self.ref_refurl = u'http://example.com/%s' % self.ref_envdata['checkname']
52 self.ref_jobdata = { u'end_time': None, 54​ self.ref_jobdata = { u'end_time': None,
53 u'href': u'http://127.0.0.1/api/v1.0/jobs/%d' % self.ref_jobid, 55​ u'href': u'http://127.0.0.1/api/v1.0/jobs/%d' % self.ref_jobid,
54 u'id': self.ref_jobid, 56​ u'id': self.ref_jobid,
55 u'name': self.ref_envdata['checkname'], 57​ u'name': self.ref_envdata['checkname'],
56 u'ref_url': self.ref_refurl, 58​ u'ref_url': self.ref_refurl,
57 u'results': [], 59​ u'results': [],
58 u'results_count': 0, 60​ u'results_count': 0,
59 u'start_time': None, 61​ u'start_time': None,
60 u'status': u'SCHEDULED'} 62​ u'status': u'SCHEDULED'}
61 63​
62 self.stub_rdb = Dingus('resultsdb', get_testcase__returns={}, 64​ self.stub_rdb = Dingus('resultsdb', get_testcase__returns={},
63 create_job__returns=self.ref_jobdata) 65​ create_job__returns=self.ref_jobdata,
66​ create_result__returns=self.ref_resultdata,
67​ )
64 self.test_rdb = resultsdb_directive.ResultsdbDirective(self.stub_rdb) 68​ self.test_rdb = resultsdb_directive.ResultsdbDirective(self.stub_rdb)
65 69​
66 # while it appears useless, this actually sets config in several tests 70​ # while it appears useless, this actually sets config in several tests
67 monkeypatch.setattr(config, '_config', None) 71​ monkeypatch.setattr(config, '_config', None)
68 self.conf = config.get_config() 72​ self.conf = config.get_config()
69 self.conf.reporting_enabled = True 73​ self.conf.reporting_enabled = True
70 self.conf.report_to_resultsdb = True 74​ self.conf.report_to_resultsdb = True
71 75​
▲ Show 20 Lines • Show All 207 Lines • ▼ Show 20 Line(s)281 def test_ensure_testcase_cached(self):
279 283​
280 self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname']) 284​ self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
281 self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname']) 285​ self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
282 286​
283 rdb_calls = self.stub_rdb.calls() 287​ rdb_calls = self.stub_rdb.calls()
284 288​
285 assert len(rdb_calls) == 1 289​ assert len(rdb_calls) == 1
286 assert rdb_calls[0][0] == 'get_testcase' 290​ assert rdb_calls[0][0] == 'get_testcase'
291​
292​ def test_tap_output(self):
293​ """Checks whether TAP is correctly mapped to the reporting method's
294​ arguments."""
295​
296​ output = self.test_rdb.process(self.ref_input, self.ref_envdata)
297​ data = check.import_TAP(output)
298​
299​ assert data[0]._internal['resultsdb_result_id'] == 1234