Differential D594

allow result reporting by build, use it in CLI (for now?) T619
ClosedPublic
Actions

Authored by adamwill on Sep 17 2015, 3:43 AM.

Details

Reviewers
garretraziel
jskladan
Commits
rOPENQAd0ef10a74827: allow result reporting by build, use it in CLI (for now?) T619
Summary

This rejigs things so that result reporting can be done by
BUILD as well as by list of job IDs. This is specifically
intended to help with T619: it will handle cases where jobs
get restarted by openQA. It has another significant benefit,
though. As things stand, for a list of job IDs, the client has
to run one API query per job, which takes longer and increases
the risk of hitting a timeout or something. For a build, the
client only has to run a single query. So this should be more
robust as well.

I have a PR upstream to allow for retrieving a bunch of jobs by
ID in a single query:
https://github.com/os-autoinst/openQA/pull/434
so if that gets merged, we could re-consider this. We'd still
need some code for catching when jobs get cloned, though.

You can imagine cases where this reports results from jobs
that weren't actually created by this specific submission,
but I can't think of any cases where it would do anything
*wrong*, really.

Test Plan

Check that result submission still works as before,
except that cloned jobs are handled and there are fewer API
calls.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to allow result reporting by build, use it in CLI (for now?) T619.Sep 17 2015, 3:43 AM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jskladan, garretraziel.
garretraziel accepted this revision.Sep 21 2015, 10:58 AM

Looks OK.

fedora_openqa_schedule/report.py
47–91

This is much better now!

This revision is now accepted and ready to land.Sep 21 2015, 10:58 AM
Closed by commit rOPENQAd0ef10a74827: allow result reporting by build, use it in CLI (for now?) T619 (authored by adamwill). · Explain WhySep 21 2015, 3:57 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 1552

View Options

fedora_openqa_schedule/cli.py

Show First 20 Lines • Show All 44 Lines • ▼ Show 20 Line(s)
45### SUB-COMMAND METHODS 45​### SUB-COMMAND METHODS
46 46​
47def command_compose(args, wiki): 47​def command_compose(args, wiki):
48 """run OpenQA on a specified compose, optionally reporting 48​ """run OpenQA on a specified compose, optionally reporting
49 results if a matching wikitcms ValidationEvent is found by 49​ results if a matching wikitcms ValidationEvent is found by
50 relval/wikitcms. 50​ relval/wikitcms.
51 """ 51​ """
52 try: 52​ try:
53 jobs = schedule.jobs_from_compose( 53​ (ff_rel, jobs) = schedule.jobs_from_compose(
54 args.release, args.milestone, args.compose, arches=args.arch, force=args.force, 54​ args.release, args.milestone, args.compose, arches=args.arch, force=args.force,
55 wait=args.wait) 55​ wait=args.wait)
56 56​
57 except ValueError as err: 57​ except ValueError as err:
58 logger.critical("Could not find compose %s %s %s! Error: %s", 58​ logger.critical("Could not find compose %s %s %s! Error: %s",
59 args.release, args.milestone, args.compose, err) 59​ args.release, args.milestone, args.compose, err)
60 sys.exit(1) 60​ sys.exit(1)
61 61​
62 except schedule.TriggerException as err: 62​ except schedule.TriggerException as err:
63 logger.warning("No jobs run! Compose may not exist? %s", err) 63​ logger.warning("No jobs run! Compose may not exist? %s", err)
64 sys.exit(1) 64​ sys.exit(1)
65 65​
66 except schedule.WaitError: 66​ except schedule.WaitError:
67 logger.critical("Waited too long for compose to appear!") 67​ logger.critical("Waited too long for compose to appear!")
68 sys.exit(1) 68​ sys.exit(1)
69 69​
70 if jobs: 70​ if jobs:
71 report.report_results(jobs, wiki=wiki, report=args.submit) 71​ build = '_'.join((ff_rel.release, ff_rel.milestone, ff_rel.compose))
72​ report.wait_and_report(build=build, wiki=wiki, report=args.submit)
72 else: 73​ else:
73 msg = "No jobs run!" 74​ msg = "No jobs run!"
74 if not args.force: 75​ if not args.force:
75 msg = "{0} {1}".format( 76​ msg = "{0} {1}".format(
76 msg, "You did not pass --force: maybe jobs exist for all ISOs/flavors") 77​ msg, "You did not pass --force: maybe jobs exist for all ISOs/flavors")
77 logger.warning(msg) 78​ logger.warning(msg)
78 sys.exit(1) 79​ sys.exit(1)
79 80​
80 logger.debug("finished") 81​ logger.debug("finished")
81 sys.exit() 82​ sys.exit()
82 83​
83def command_current(args, wiki): 84​def command_current(args, wiki):
84 """run OpenQA for current release validation event, if we have 85​ """run OpenQA for current release validation event, if we have
85 not already done it. 86​ not already done it.
86 """ 87​ """
87 logger.info("running on current release") 88​ logger.info("running on current release")
88 try: 89​ try:
89 jobs = schedule.jobs_from_current(wiki) 90​ (ff_rel, jobs) = schedule.jobs_from_current(wiki)
90 except schedule.TriggerException as e: 91​ except schedule.TriggerException as e:
91 logger.debug("No jobs run: %s", e) 92​ logger.debug("No jobs run: %s", e)
92 sys.exit(1) 93​ sys.exit(1)
93 # wait for jobs to finish and display results 94​ # wait for jobs to finish and display results
94 if jobs: 95​ if jobs:
95 logger.info("waiting for jobs: %s", ' '.join(str(j) for j in jobs)) 96​ logger.info("waiting for jobs")
96 report.report_results(jobs, wiki=wiki, report=args.submit) 97​ build = '_'.join((ff_rel.release, ff_rel.milestone, ff_rel.compose))
98​ report.wait_and_report(build=build, wiki=wiki, report=args.submit)
97 logger.debug("finished") 99​ logger.debug("finished")
98 sys.exit() 100​ sys.exit()
99 101​
100def command_report(args, wiki): 102​def command_report(args, wiki):
101 """Map a list of openQA job IDs to Wikitcms test results, and 103​ """Map a list of openQA job IDs to Wikitcms test results, and
102 either display the ResTups for inspection or report the 104​ either display the ResTups for inspection or report the
103 results to the wiki. 105​ results to the wiki.
104 """ 106​ """
105 try: 107​ try:
106 report.report_results( 108​ report.wait_and_report(
107 args.jobs, wiki=wiki, report=args.submit) 109​ job_ids=args.jobs, wiki=wiki, report=args.submit)
108 except report.LoginError: 110​ except report.LoginError:
109 # Module logs for us, no need to dupe it. 111​ # Module logs for us, no need to dupe it.
110 sys.exit(1) 112​ sys.exit(1)
111 113​
112### ARGUMENT PARSING AND SUB-COMMAND INIT 114​### ARGUMENT PARSING AND SUB-COMMAND INIT
113 115​
114def run(): 116​def run():
115 """Read in args, set up logging and wikitcms, and run sub-command 117​ """Read in args, set up logging and wikitcms, and run sub-command
▲ Show 20 Lines • Show All 101 Lines • Show Last 20 Lines
View Options

fedora_openqa_schedule/report.py

Show All 38 Lines
39logger = logging.getLogger(__name__) 39​logger = logging.getLogger(__name__)
40 40​
41 41​
42class LoginError(Exception): 42​class LoginError(Exception):
43 """Raised when cannot log in to wiki to submit results.""" 43​ """Raised when cannot log in to wiki to submit results."""
44 pass 44​ pass
45 45​
46 46​
47def get_passed_testcases(job_ids, waittime=None): 47​def get_passed_testcases(jobs):
48​ """Given an iterable of job dicts - any waiting, filtering and so
49​ on is assumed to have already happened - returns a list of
50​ wikitcms ResTuples derived from any passed tests.
48 """ 51​ """
49 job_ids ~ list of int (job ids) ​
50 waittime ~ int or None - how long to wait for jobs, None == read from config ​
51 Returns ~ list of str - names of passed testcases ​
52 """ ​
53 if waittime is None: ​
54 waittime = CONFIG.getint('report', 'jobs-wait') ​
55 # Use the openQA client lib to wait for jobs to be done. Will ​
56 # raise an openQA client error on wait expiry. ​
57 client = OpenQA_Client() ​
58 jobs = client.wait_jobs(job_ids, waittime=waittime) ​
59 passed_testcases = set() 52​ passed_testcases = set()
60 for job in jobs: 53​ for job in jobs:
61 if job['result'] == 'passed': 54​ if job['result'] == 'passed':
62 (release, milestone, compose) = job['settings']['BUILD'].split('_') 55​ (release, milestone, compose) = job['settings']['BUILD'].split('_')
63 testsuite = job['settings']['TEST'] 56​ testsuite = job['settings']['TEST']
64 # We need to differentiate tests with UEFI from BIOS 57​ # We need to differentiate tests with UEFI from BIOS
65 uefi = job['settings'].has_key('UEFI') 58​ uefi = job['settings'].has_key('UEFI')
66 arch = job['settings']['ARCH'] 59​ arch = job['settings']['ARCH']
67 flavor = job['settings']['FLAVOR'] 60​ flavor = job['settings']['FLAVOR']
68 61​
69 for testcase in conf_test_suites.TESTSUITES[testsuite]: 62​ for testcase in conf_test_suites.TESTSUITES[testsuite]:
70 # each 'testsuite' is a list using testcase names to indicate which Wikitcms tests 63​ # each 'testsuite' is a list using testcase names to indicate which Wikitcms tests
71 # have passed if this job passes. Each testcase name is the name of a dict in the 64​ # have passed if this job passes. Each testcase name is the name of a dict in the
72 # TESTCASES dict-of-dicts which more precisely identifies the 'test instance' (when 65​ # TESTCASES dict-of-dicts which more precisely identifies the 'test instance' (when
73 # there is more than one for a testcase) and environment for which the result 66​ # there is more than one for a testcase) and environment for which the result
74 # should be filed. 67​ # should be filed.
75 uniqueres = conf_test_suites.TESTCASES[testcase] 68​ uniqueres = conf_test_suites.TESTCASES[testcase]
76 testname = '' 69​ testname = ''
77 if 'name_cb' in uniqueres: 70​ if 'name_cb' in uniqueres:
78 testname = uniqueres['name_cb'](flavor) 71​ testname = uniqueres['name_cb'](flavor)
79 # Decide what column should be used for report. 72​ # Decide what column should be used for report.
80 # Testcases with more columns now have to use $variable$ so it 73​ # Testcases with more columns now have to use $variable$ so it
81 # could be decided where to report the result 74​ # could be decided where to report the result
82 if uniqueres['env'] == '$RUNARCH_OR_UEFI$': 75​ if uniqueres['env'] == '$RUNARCH_OR_UEFI$':
83 env = 'UEFI' if uefi else arch 76​ env = 'UEFI' if uefi else arch
84 elif uniqueres['env'] == '$FS$': 77​ elif uniqueres['env'] == '$FS$':
85 env = testsuite.split('_')[-1] 78​ env = testsuite.split('_')[-1]
86 elif uniqueres['env'] == '$RUNARCH$': 79​ elif uniqueres['env'] == '$RUNARCH$':
87 env = arch 80​ env = arch
88 elif uniqueres['env'] == '$BOOTMETHOD$': 81​ elif uniqueres['env'] == '$BOOTMETHOD$':
89 env = 'x86 UEFI' if uefi else 'x86 BIOS' 82​ env = 'x86 UEFI' if uefi else 'x86 BIOS'
90 else: 83​ else:
91 env = uniqueres['env'] 84​ env = uniqueres['env']
92 result = ResTuple( 85​ result = ResTuple(
93 testtype=uniqueres['type'], release=release, milestone=milestone, 86​ testtype=uniqueres['type'], release=release, milestone=milestone,
94 compose=compose, testcase=testcase, section=uniqueres['section'], 87​ compose=compose, testcase=testcase, section=uniqueres['section'],
95 testname=testname, env=env, status='pass', bot=True) 88​ testname=testname, env=env, status='pass', bot=True)
96 passed_testcases.add(result) 89​ passed_testcases.add(result)
97 90​
98 return sorted(list(passed_testcases), key=attrgetter('testcase')) 91​ return sorted(list(passed_testcases), key=attrgetter('testcase'))
garretrazielUnsubmitted
Not Done

This is much better now!

99 92​
100 93​
101def report_results(job_ids, wiki=None, report=None): 94​def wait_and_report(job_ids=None, build=None, wiki=None, report=None, waittime=None):
102 """Report results from a list of openQA job IDs to Wikitcms. If 95​ """Find some openQA jobs and report the results to Wikitcms (via
103 report is False, will just print out the python-wikitcms ResTups 96​ report_results). Either job_ids (an iterable of IDs) or build (an
104 for inspection. If report is None, will read the setting from the 97​ openQA BUILD string) is required. wiki and report are passed to
105 config file; if no config file is present, default is True. 98​ report_results and documented there. waittime (int) is how long to
99​ wait for jobs to complete before giving up; if 'build' is used,
100​ will also wait for jobs to appear. If jobs are not present and
101​ complete when waittime expires,
102​ openqa_client.exceptions.WaitException will be raised. If waittime
103​ is 0, no waiting is done, and if jobs are not complete, the
104​ exception will raise immediately.
105​ """
106​ if waittime is None:
107​ waittime = CONFIG.getint('report', 'jobs-wait')
108​ # Use the openQA client lib to wait for jobs to be done. Will
109​ # raise an openQA client error on wait expiry.
110​ client = OpenQA_Client()
111​ if job_ids:
112​ jobs = client.wait_jobs(job_ids, waittime=waittime)
113​ elif build:
114​ jobs = client.wait_build_jobs(build, waittime=waittime)
115​ else:
116​ raise ValueError("wait_and_report requires either job_ids or build.")
117​ return report_results(jobs, wiki=wiki, report=report)
118​
119​
120​def report_results(jobs, wiki=None, report=None):
121​ """Report results from openQA jobs to Wikitcms. jobs is an
122​ iterable of openQA job dicts. If report is False, will just print
123​ out the python-wikitcms ResTups for inspection. If report is None,
124​ will read the setting from the config file; if no config file is
125​ present, default is True.
106 """ 126​ """
107 passed_testcases = get_passed_testcases(job_ids) 127​ passed_testcases = get_passed_testcases(jobs)
108 logger.info("passed testcases: %s", passed_testcases) 128​ logger.info("passed testcases: %s", passed_testcases)
109 if report is None: 129​ if report is None:
110 report = CONFIG.getboolean('report', 'submit') 130​ report = CONFIG.getboolean('report', 'submit')
111 131​
112 if report: 132​ if report:
113 logger.info("reporting test passes") 133​ logger.info("reporting test passes")
114 if not wiki: 134​ if not wiki:
115 wiki = Wiki() 135​ wiki = Wiki()
Show All 14 Lines
View Options

fedora_openqa_schedule/schedule.py

Show First 20 Lines • Show All 177 Lines • ▼ Show 20 Line(s)146def jobs_from_current(wiki, force=False, arches=None):
178 logger.info("jobs_from_current: planned jobs: %s", ' '.join(str(j) for j in jobs)) 178​ logger.info("jobs_from_current: planned jobs: %s", ' '.join(str(j) for j in jobs))
179 179​
180 # write info about latest versions 180​ # write info about latest versions
181 f = open(CONFIG.get('schedule', 'persistent'), "w") 181​ f = open(CONFIG.get('schedule', 'persistent'), "w")
182 f.write(json.dumps(json_parsed)) 182​ f.write(json.dumps(json_parsed))
183 f.close() 183​ f.close()
184 logger.debug("written info about newest version") 184​ logger.debug("written info about newest version")
185 185​
186 return jobs 186​ return (currev.ff_release, jobs)
187 187​
188 188​
189def jobs_from_compose(release='', milestone='', compose='', arches=None, force=False, wait=None): 189​def jobs_from_compose(release='', milestone='', compose='', arches=None, force=False, wait=None):
190 """Schedule jobs against a specific release/compose. Returns the 190​ """Schedule jobs against a specific release/compose. Returns the
191 list of job IDs. 191​ list of job IDs.
192 192​
193 'release', 'milestone' and 'compose' identify a release according 193​ 'release', 'milestone' and 'compose' identify a release according
194 to fedfind conventions (e.g. '23', 'Beta', 'TC2' or '24', 194​ to fedfind conventions (e.g. '23', 'Beta', 'TC2' or '24',
Show All 27 Lines221 if wait:
222 logger.info("jobs_from_compose: Waiting up to %s mins for compose", str(wait)) 222​ logger.info("jobs_from_compose: Waiting up to %s mins for compose", str(wait))
223 try: 223​ try:
224 ff_release.wait(waittime=wait) 224​ ff_release.wait(waittime=wait)
225 except fedfind.exceptions.WaitError as err: 225​ except fedfind.exceptions.WaitError as err:
226 raise WaitError(err) 226​ raise WaitError(err)
227 227​
228 jobs = jobs_from_fedfind(ff_release, arches=arches, force=force) 228​ jobs = jobs_from_fedfind(ff_release, arches=arches, force=force)
229 logger.info("jobs_from_compose: planned jobs: %s", ' '.join(str(j) for j in jobs)) 229​ logger.info("jobs_from_compose: planned jobs: %s", ' '.join(str(j) for j in jobs))
230 return jobs 230​ return (ff_release, jobs)
231 231​
232 232​
233def jobs_from_fedfind(ff_release, force=False, arches=None, imagetypes=None, payloads=None): 233​def jobs_from_fedfind(ff_release, force=False, arches=None, imagetypes=None, payloads=None):
234 """Given a fedfind.Release object, find the ISOs we want and run 234​ """Given a fedfind.Release object, find the ISOs we want and run
235 jobs on them. arches is an iterable of arches to run on, if not 235​ jobs on them. arches is an iterable of arches to run on, if not
236 specified, we'll use value from config file. If force is False, 236​ specified, we'll use value from config file. If force is False,
237 for each ISO, jobs will only be scheduled if there are no 237​ for each ISO, jobs will only be scheduled if there are no
238 existing, non-cancelled jobs for the same ISO and flavor. 238​ existing, non-cancelled jobs for the same ISO and flavor.
▲ Show 20 Lines • Show All 72 Lines • Show Last 20 Lines