add a script for sanity checking conf_test_suites
ClosedPublic

Authored by adamwill on Sep 23 2016, 2:39 AM.

Details

Summary

It bugs me that it's not very visible if we've messed up wiki
reporting: we really have to be paying attention to spot that
we don't have reporting hooked up for a test, or if we have it
hooked up but we didn't quite get the settings right and
wikitcms won't manage to report the result. The latter case is
partly logged, but that still means we have to go poke through
the server logs to spot it, and there are in fact cases where
wikitcms won't report insufficient / incorrect restups (I
should probably fix that), and it means we don't catch the
issue till after we've committed the changes; we can't check
before commit.

So I fixed it! Here's a script that checks the TESTSUITES and
TESTCASES dicts correspond, that we have an entry in TESTSUITES
for every test suite in the openQA templates, and that wikitcms
can actually find a resultrow and environment for a set of
faked-up 'passed jobs' derived from the openQA templates. Now
we can use this to sanity check commits that add wiki reports.

We also fix an error it discovered, for KDE_Package_Install.
The result would actually have gotten reported anyway in this
case (as wikitcms doesn't bother checking the restup's 'env'
value if the resultrow only has *one* env), but it's good to
be correct.

I guess this is *one* way to spend an 8 hour train ride...

Test Plan

Run the script, see what it tells you. On current
develop it should spot that there is no reporting of the
graphical update tests yet. You can introduce intentional
errors to conf_test_suites to see if it catches them.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to add a script for sanity checking conf_test_suites.Sep 23 2016, 2:39 AM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jskladan, garretraziel.

It also catches that we don't have anywhere in the wiki for the Atomic DVD installer results to go.

Otherwise than inline comment, LGTM. Thanks, that's very useful tool!

scheduler/check-wiki.py
155 ↗(On Diff #2587)

I think that it would be better to create fakejobs only from items that exist in TESTCASES (don't include items that are listed in test suite but not in TESTCASES), otherwise this fails before it can check anything:

Test case QA:Testcase_base_selinux listed in test suite base_selinux is not in TESTCASES!
Test case QA:Testcase_base_salinux from TESTCASES is never used in TESTSUITES!
...
Traceback (most recent call last):
  File "check-wiki.py", line 155, in <module>
    restups = report.get_passed_testcases(fakejobs)
  File "/home/jsedlak/Projects/openqa_fedora_tools/scheduler/fedora_openqa_schedule/report.py", line 156, in get_passed_testcases
    uniqueres = _uniqueres_replacements(job, conf_test_suites.TESTCASES[testcase])
KeyError: 'QA:Testcase_base_selinux'
garretraziel accepted this revision.Sep 23 2016, 1:28 PM
This revision is now accepted and ready to land.Sep 23 2016, 1:28 PM

Yeah, I noticed that too, and was kinda on the fence about whether to 'fix' it. But on balance probably a good idea to, so you don't have to fix the missing TESTCASE entry before you can find out if there are any other reporting problems...will do.

oh, we could also get rid of the terrible perl bit if we convert the templates to json, but I dunno how much you care about that. It doesn't bother me too much since we'll only be using this ourselves probably. just don't run this on an untrusted templates file of course, since it happily runs anything it finds in that file. do is not a magic function that only evaluates safe expressions or anything, sadly. :P

This revision was automatically updated to reflect the committed changes.

In fact it turned out the best way to fix the issue @garretraziel identified was to fix it in get_passed_testcases, since the same error could actually occur in production use if a testcase listed in TESTSUITES was missing from TESTCASES. It's much like the 'test suite missing from TESTSUITES' case I made log and continue (instead of crashing) a few months back. So I did that, moved all the main logic into a main() function to make pylint happy, and merged.