Differential D932

allow wiki reporting per test module, report FreeIPA_web_ui
ClosedPublic
Actions

Authored by adamwill on Jul 12 2016, 9:22 PM.

Details

Reviewers
garretraziel
jskladan
Commits
rOPENQA6475a2d6c780: allow wiki reporting per test module, report FreeIPA_web_ui
Summary

This tweaks the wiki result reporting bits to be a bit more
flexible and allow for reporting the QA:Testcase_FreeIPA_web_ui
results. We added that test case as a single, non-fatal test
module within the realmd_join_cockpit test suite. That means
that when openQA runs that test suite, it will run the FreeIPA
web UI test module and record the result of that module, but
if that module fails, the openQA job will not stop nor will it
be considered to have failed; if all the other modules pass,
the job is considered to have passed. So for wiki reporting
purposes we need to be able to say 'report a pass for this
Wikitcms test case only if this specific test module within the
job passed'.

We allow this by letting entries in the TESTSUITES dict be
dicts instead of lists, with a special key for the normal
'these test cases passed if the job passed' list and other keys
representing individual modules, with lists of test cases that
passed only if those specific test modules passed. There's a
bit more detail in the comments.

Test Plan

Generate ResTups for a run of realmd_join_cockpit
where both the job and the freeipa_webui_postinstall module
passed - e.g. https://openqa.stg.fedoraproject.org/tests/24944

  • and make sure a ResTup for QA:Testcase_FreeIPA_web_ui is

generated. If possible, also generate ResTups for a run where
the job passed but the module failed, and make sure there are
ResTups for the *other* test cases but *not* that one, and
also generate ResTups for a run where the job failed and make
sure we get none.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to allow wiki reporting per test module, report FreeIPA_web_ui.Jul 12 2016, 9:22 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: garretraziel, jskladan.
adamwill updated this revision to Diff 2373.Jul 12 2016, 10:33 PM

be a bit more robust against expected things not being there

garretraziel added a comment.Jul 18 2016, 9:47 AM

Looks good, but I was unable to try this code; arcanist says that this diff is against commit f88c8944, which is nowhere to be found in my tree (branch freeipa-webui-report is not in origin).

scheduler/fedora_openqa_schedule/report.py
130

Please, add comment documenting what is happening here.

adamwill updated this revision to Diff 2443.Aug 3 2016, 8:25 PM

rebase on current develop

adamwill updated this revision to Diff 2444.Aug 3 2016, 8:31 PM

add requested comment

adamwill updated this revision to Diff 2452.Aug 5 2016, 10:54 PM

fix wiki environment, add password_change reporting also

as we merged the freeipa_password_change test, we should also
report those results. And the environment (the wiki column name)
was wrong in the TESTCASES dict (should be 'Result' not 'x86'),
so fix that.

garretraziel accepted this revision.Aug 8 2016, 10:46 AM

Looks OK.

This revision is now accepted and ready to land.Aug 8 2016, 10:46 AM
Closed by commit rOPENQA6475a2d6c780: allow wiki reporting per test module, report FreeIPA_web_ui (authored by adamwill). · Explain WhyAug 8 2016, 6:26 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2455

View Options

scheduler/fedora_openqa_schedule/conf_test_suites.py

Show First 20 Lines • Show All 355 Lines • ▼ Show 20 Line(s)26TESTCASES = {
356 "QA:Testcase_Server_cockpit_default": { 356​ "QA:Testcase_Server_cockpit_default": {
357 "env": "x86", 357​ "env": "x86",
358 "type": "Server", 358​ "type": "Server",
359 }, 359​ },
360 "QA:Testcase_Server_cockpit_basic": { 360​ "QA:Testcase_Server_cockpit_basic": {
361 "env": "x86", 361​ "env": "x86",
362 "type": "Server", 362​ "type": "Server",
363 }, 363​ },
364​ "QA:Testcase_FreeIPA_web_ui": {
365​ "env": "Result",
366​ "type": "Server",
367​ },
368​ "QA:Testcase_FreeIPA_password_change": {
369​ "env": "Result",
370​ "type": "Server",
371​ },
364 # "": { 372​ # "": {
365 # "name": "", # optional, use when same testcase occurs on multiple rows with different link text 373​ # "name": "", # optional, use when same testcase occurs on multiple rows with different link text
366 # "section": "", # optional, some result pages have no sections 374​ # "section": "", # optional, some result pages have no sections
367 # "env": "x86", 375​ # "env": "x86",
368 # "type": "Installation", 376​ # "type": "Installation",
369 # }, 377​ # },
370} 378​}
371 379​
372# This is used to update TESTCASES dict for ResultsDB reporting 380​# This is used to update TESTCASES dict for ResultsDB reporting
373TESTCASES_RESULTSDB_EXTRADATA = { 381​TESTCASES_RESULTSDB_EXTRADATA = {
374 "QA:Testcase_partitioning_guided_shrink": { 382​ "QA:Testcase_partitioning_guided_shrink": {
375 "filesystem": "$FS$", # needed to convert test_fs to only one testcase 383​ "filesystem": "$FS$", # needed to convert test_fs to only one testcase
376 }, 384​ },
377} 385​}
378 386​
379TESTSUITES = { 387​TESTSUITES = {
388​ # each entry in this dict is named for an openQA test suite, and
389​ # represents the Wikitcms test cases that passed if that openQA
390​ # test suite passed. The entries can be simple lists of test case
391​ # names, or they can be dicts. In the dict form, the special dict
392​ # entry named 'PASS' is a list of test cases that passed if the
393​ # overall openQA job passed, as in the simple list form. The other
394​ # dict entries are named for individual openQA test modules, and
395​ # are lists of test cases that passed if the overall openQA job
396​ # passed *and* that specific test module passed. This handles
397​ # test suites like realmd_join_cockpit, which has an optional
398​ # module that covers QA:Testcase_FreeIPA_web_ui ; if that module
399​ # passes we want to report a pass for that test case, if the job
400​ # passes but that module fails we do *not* want to report a pass.
380 "install_default": [ 401​ "install_default": [
381 "QA:Testcase_Boot_default_install", 402​ "QA:Testcase_Boot_default_install",
382 "QA:Testcase_install_to_VirtIO", 403​ "QA:Testcase_install_to_VirtIO",
383 "QA:Testcase_partitioning_guided_empty", 404​ "QA:Testcase_partitioning_guided_empty",
384 "QA:Testcase_Anaconda_User_Interface_Graphical", 405​ "QA:Testcase_Anaconda_User_Interface_Graphical",
385 "QA:Testcase_Anaconda_user_creation", 406​ "QA:Testcase_Anaconda_user_creation",
386 ], 407​ ],
387 "install_default_upload": [ 408​ "install_default_upload": [
▲ Show 20 Lines • Show All 297 Lines • ▼ Show 20 Line(s)705 "server_firewall_default": [
685 "QA:Testcase_Server_firewall_default", 706​ "QA:Testcase_Server_firewall_default",
686 ], 707​ ],
687 "server_cockpit_default": [ 708​ "server_cockpit_default": [
688 "QA:Testcase_Server_cockpit_default", 709​ "QA:Testcase_Server_cockpit_default",
689 ], 710​ ],
690 "server_cockpit_basic": [ 711​ "server_cockpit_basic": [
691 "QA:Testcase_Server_cockpit_basic", 712​ "QA:Testcase_Server_cockpit_basic",
692 ], 713​ ],
693 "realmd_join_cockpit": [ 714​ "realmd_join_cockpit": {
694 "QA:Testcase_realmd_join_cockpit", 715​ "PASS": [
695 "QA:Testcase_FreeIPA_realmd_login", 716​ "QA:Testcase_realmd_join_cockpit",
696 "QA:Testcase_domain_client_authenticate", 717​ "QA:Testcase_FreeIPA_realmd_login",
697 ], 718​ "QA:Testcase_domain_client_authenticate",
719​ ],
720​ "freeipa_webui_postinstall": [
721​ "QA:Testcase_FreeIPA_web_ui",
722​ ],
723​ "freeipa_password_change_postinstall": [
724​ "QA:Testcase_FreeIPA_password_change",
725​ ],
726​ },
698 "realmd_join_sssd": [ 727​ "realmd_join_sssd": [
699 "QA:Testcase_realmd_join_sssd", 728​ "QA:Testcase_realmd_join_sssd",
700 "QA:Testcase_FreeIPA_realmd_login", 729​ "QA:Testcase_FreeIPA_realmd_login",
701 "QA:Testcase_domain_client_authenticate", 730​ "QA:Testcase_domain_client_authenticate",
702 ], 731​ ],
703 "install_iscsi": [ 732​ "install_iscsi": [
704 "QA:Testcase_install_to_iSCSI_no_authentication", 733​ "QA:Testcase_install_to_iSCSI_no_authentication",
705 "QA:Testcase_Anaconda_User_Interface_Graphical", 734​ "QA:Testcase_Anaconda_User_Interface_Graphical",
706 "QA:Testcase_Anaconda_user_creation", 735​ "QA:Testcase_Anaconda_user_creation",
707 "QA:Testcase_Package_Sets_Minimal_Package_Install", 736​ "QA:Testcase_Package_Sets_Minimal_Package_Install",
708 ], 737​ ],
709} 738​}
View Options

scheduler/fedora_openqa_schedule/report.py

Show First 20 Lines • Show All 100 Lines • ▼ Show 20 Line(s)99 if job['result'] == 'passed':
101 # what the validation event for it is. 101​ # what the validation event for it is.
102 composeid = job['settings']['BUILD'] 102​ composeid = job['settings']['BUILD']
103 if composeid.endswith('EXTRA'): 103​ if composeid.endswith('EXTRA'):
104 # this is a 'dirty' test run with extra parameters 104​ # this is a 'dirty' test run with extra parameters
105 # (usually a test with an updates.img), we never want 105​ # (usually a test with an updates.img), we never want
106 # to report results for these 106​ # to report results for these
107 logger.debug("Job was run with extra params! Will not report") 107​ logger.debug("Job was run with extra params! Will not report")
108 continue 108​ continue
109 testsuite = job['settings']['TEST'] 109​ tsname = job['settings']['TEST']
110 # There usually ought to be an entry in TESTSUITES for all 110​ # There usually ought to be an entry in TESTSUITES for all
111 # tests, but just in case someone messed up, let's be safe 111​ # tests, but just in case someone messed up, let's be safe
112 if testsuite not in conf_test_suites.TESTSUITES: 112​ if tsname not in conf_test_suites.TESTSUITES:
113 logger.warning("No TESTSUITES entry found for test {0}!".format(testsuite)) 113​ logger.warning("No TESTSUITES entry found for test {0}!".format(tsname))
114 continue 114​ continue
115 for testcase in conf_test_suites.TESTSUITES[testsuite]: 115​ passed = []
116​ testsuite = conf_test_suites.TESTSUITES[tsname]
117​ # testsuite can be simply a list of test case names - in which case all those test
118​ # cases are considered 'passed' if the openQA job overall 'passed' - or a dict of
119​ # lists of test case names. In the dict form, there is one special key, 'PASS'; the
120​ # 'PASS' list works just like the simple list form, all test cases in that list
121​ # 'passed' if the openQA job overall passed. Any other keys are the names of
122​ # individual openQA test modules, which should be included in the job as non-fatal
123​ # modules; if that module passed within the job, the test cases in the list 'passed'.
124​ try:
125​ # this is the dict case. first take all test cases from the 'PASS' list...
126​ passed.extend(testsuite.pop('PASS', []))
127​ # now check each individual test module status
128​ for (modname, cases) in testsuite.items():
129​ try:
130​ # find the matching test module in the job's list (note this
garretrazielUnsubmitted
Not Done

Please, add comment documenting what is happening here.

131​ # assumes we don't run the same module multiple times, which
132​ # is something upstream has recently started allowing...)
133​ module = [module for module in job['modules'] if module['name'] == modname][0]
134​ except IndexError:
135​ logger.warning("Did not find expected module %s in job data!", modname)
136​ continue
137​ if module.get('result', '') == 'passed':
138​ passed.extend(cases)
139​ except TypeError:
140​ # this is the simple list case.
141​ passed = testsuite
142​ for testcase in passed:
116 # each 'testsuite' is a list using testcase names to indicate which Wikitcms tests 143​ # each 'testsuite' is a list using testcase names to indicate which Wikitcms tests
117 # have passed if this job passes. Each testcase name is the name of a dict in the 144​ # have passed if this job passes. Each testcase name is the name of a dict in the
118 # TESTCASES dict-of-dicts which more precisely identifies the 'test instance' (when 145​ # TESTCASES dict-of-dicts which more precisely identifies the 'test instance' (when
119 # there is more than one for a testcase) and environment for which the result 146​ # there is more than one for a testcase) and environment for which the result
120 # should be filed. We do a deepcopy here because otherwise the dict in TESTCASES 147​ # should be filed. We do a deepcopy here because otherwise the dict in TESTCASES
121 # itself is modified in-place and other jobs for the same testcase (but a different 148​ # itself is modified in-place and other jobs for the same testcase (but a different
122 # environment, section or testname) will read the modified values and be messed up. 149​ # environment, section or testname) will read the modified values and be messed up.
123 150​
▲ Show 20 Lines • Show All 119 Lines • Show Last 20 Lines