Differential D1155

Report test 'scenario' to ResultsDB
ClosedPublic
Actions

Authored by adamwill on Feb 28 2017, 11:05 PM.

Details

Reviewers
jsedlak
jskladan
Commits
rOPENQA4c7613cd8b70: Report test 'scenario' to ResultsDB
Summary

See https://lists.fedoraproject.org/archives/list/resultsdb-users@lists.fedoraproject.org/message/DJSIOLA3YEH6HNSJEXVPLIZT5TW4CDED/
for some discussion of this. To make life easier for consumers
of our results from ResultsDB, we should report the 'scenario'
of the test. This allows consumers to conveniently deduplicate
results: results for the same 'item' and 'scenario' are dupes,
so usually, consumers will only want to consider the most recent
one.

Test Plan

Check the tests pass, report some results to a test
ResultsDB and check the scenario item in the extradata looks
correct.

Diff Detail

Repository
rOPENQA fedora_openqa
Branch
scenario (branched from master)
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1086
Build 1086: arc lint + arc unit
adamwill retitled this revision from to Report test 'scenario' to ResultsDB.Feb 28 2017, 11:05 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jsedlak, jskladan.
jsedlak accepted this revision.Mar 3 2017, 12:00 PM

Code looks OK and if we need it right now, feel free to merge it, but I wonder how will the discussion evolve.

This revision is now accepted and ready to land.Mar 3 2017, 12:00 PM
adamwill added a subscriber: kparal.Mar 3 2017, 11:58 PM

I think we're OK to go with this for now, too, at least if I'm reading @kparal correctly - if anything, his remaining question is whether the test name should be in the scenario. But even if we decided to take it out in future, I don't think that's a problem, intended usages wouldn't be affected by the change I don't think. So I'm gonna go ahead and merge this.

adamwill closed this revision.Mar 3 2017, 11:59 PM

Revision Contents

PathPackages
Mfedora_openqa/report.py (5 lines)
Mtests/test_report.py (2 lines)

Diff 2941

View Options

fedora_openqa/report.py

Show All 26 Lines
27import logging 27​import logging
28import re 28​import re
29from functools import partial 29​from functools import partial
30from operator import attrgetter 30​from operator import attrgetter
31 31​
32# External dependencies 32​# External dependencies
33import mwclient.errors 33​import mwclient.errors
34from openqa_client.client import OpenQA_Client 34​from openqa_client.client import OpenQA_Client
35​from openqa_client.const import JOB_SCENARIO_WITH_MACHINE_KEYS
35from resultsdb_api import ResultsDBapi, ResultsDBapiException 36​from resultsdb_api import ResultsDBapi, ResultsDBapiException
36from resultsdb_conventions.fedora import FedoraImageResult, FedoraComposeResult, FedoraBodhiResult 37​from resultsdb_conventions.fedora import FedoraImageResult, FedoraComposeResult, FedoraBodhiResult
37from wikitcms.wiki import Wiki, ResTuple 38​from wikitcms.wiki import Wiki, ResTuple
38 39​
39# Internal dependencies 40​# Internal dependencies
40from . import conf_test_suites 41​from . import conf_test_suites
41from .config import CONFIG 42​from .config import CONFIG
42 43​
▲ Show 20 Lines • Show All 359 Lines • ▼ Show 20 Line(s)402 if module["result"] == "failed" and job["result"] == "softfailed":
402 kwargs["note"] = "non-important module {0} failed".format(module["name"]) 403​ kwargs["note"] = "non-important module {0} failed".format(module["name"])
403 404​
404 # create the Result instance 405​ # create the Result instance
405 rdb_object = rdbpartial(**kwargs) 406​ rdb_object = rdbpartial(**kwargs)
406 407​
407 # Add some more extradata items 408​ # Add some more extradata items
408 rdb_object.extradata.update({ 409​ rdb_object.extradata.update({
409 'firmware': 'uefi' if 'UEFI' in job['settings'] else 'bios', 410​ 'firmware': 'uefi' if 'UEFI' in job['settings'] else 'bios',
410 'arch': job['settings']['ARCH'] 411​ 'arch': job['settings']['ARCH'],
412​ 'scenario': '.'.join(job['settings'][key] for key in JOB_SCENARIO_WITH_MACHINE_KEYS)
411 }) 413​ })
414​
412 # FIXME: use overall_url as a group ref_url 415​ # FIXME: use overall_url as a group ref_url
413 416​
414 rdb_object.report(rdb_instance) 417​ rdb_object.report(rdb_instance)
415 418​
416# vim: set textwidth=120 ts=8 et sw=4: 419​# vim: set textwidth=120 ts=8 et sw=4:
View Options

tests/test_report.py

Show First 20 Lines • Show All 354 Lines • ▼ Show 20 Line(s)352class TestResultsDBReport:
355 def test_simple(self, fakeres, jobdict01): 355​ def test_simple(self, fakeres, jobdict01):
356 """Check resultsdb_report behaviour with our stock fake data.""" 356​ """Check resultsdb_report behaviour with our stock fake data."""
357 fosreport.resultsdb_report(jobs=[1]) 357​ fosreport.resultsdb_report(jobs=[1])
358 assert fakeres.call_args[1]['item'] == 'Fedora-Server-dvd-x86_64-Rawhide-20170207.n.0.iso' 358​ assert fakeres.call_args[1]['item'] == 'Fedora-Server-dvd-x86_64-Rawhide-20170207.n.0.iso'
359 assert fakeres.call_args[1]['ref_url'] == 'https://some.url/tests/70581' 359​ assert fakeres.call_args[1]['ref_url'] == 'https://some.url/tests/70581'
360 assert fakeres.call_args[1]['testcase']['name'] == 'compose.server_realmd_join_kickstart' 360​ assert fakeres.call_args[1]['testcase']['name'] == 'compose.server_realmd_join_kickstart'
361 assert fakeres.call_args[1]['firmware'] == 'bios' 361​ assert fakeres.call_args[1]['firmware'] == 'bios'
362 assert fakeres.call_args[1]['outcome'] == 'PASSED' 362​ assert fakeres.call_args[1]['outcome'] == 'PASSED'
363​ scenario = 'fedora.Rawhide.Server-dvd-iso.x86_64.server_realmd_join_kickstart.64bit'
364​ assert fakeres.call_args[1]['scenario'] == scenario
363 365​
364 def test_update(self, fakeres, oqaclientmock, jobdict02): 366​ def test_update(self, fakeres, oqaclientmock, jobdict02):
365 """Check report behaviour with an update test job (rather than 367​ """Check report behaviour with an update test job (rather than
366 a compose test job). 368​ a compose test job).
367 """ 369​ """
368 # adjust the OpenQA_Client instance mock to return jobdict02 370​ # adjust the OpenQA_Client instance mock to return jobdict02
369 instmock = oqaclientmock[1] 371​ instmock = oqaclientmock[1]
370 instmock.get_jobs.return_value = [jobdict02] 372​ instmock.get_jobs.return_value = [jobdict02]
▲ Show 20 Lines • Show All 147 Lines • Show Last 20 Lines