Differential D925

add reporting to ResultsDB
ClosedPublic
Actions

Authored by garretraziel on Jul 7 2016, 10:34 AM.

Details

Reviewers
adamwill
Commits
rOPENQA3cdc4a7b8a6a: add reporting to ResultsDB
Summary

This adds possibility to report results to ResultsDB. It
changes cli args a little bit (now you specify either --wiki or
--resultsdb or both and --submit if you want to submit your
results). It sets more variables in openQA during test scheduling,
namely subvariant and imagetype (we COULD retrieve those things
from BUILD, but it's safer this way) as well as job ID of ResultsDB
job (that had to be created during test scheduling).

In contrast to Wiki reporting, we want to report both passes and
fails to ResultsDB (we probably aren't going to use ResultsDB directly,
we are planning to write some service on top of it).

We are using shiny new convention of naming tests with dot separator,
so QA:Testcase_Boot_default_install becomes
openqa.installation.boot_default_install. We are also trying to put
links where appropriate, so job overview links to openQA's overview,
testcase links to testcase page on wiki and result links to openQA
job run.

Right now, ResultsDB consumer is missing, I'll add it in some future
DR.

Test Plan

Run ResultsDB and ResultsDB-frontend locally, schedule
some tests, let it report to ResultsDB, observe results.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
garretraziel retitled this revision from to add reporting to ResultsDB.Jul 7 2016, 10:34 AM
garretraziel updated this object.
garretraziel edited the test plan for this revision. (Show Details)
garretraziel added a reviewer: adamwill.
garretraziel added a subscriber: jskladan.
adamwill requested changes to this revision.Jul 8 2016, 9:32 PM

A few comments in-line.

scheduler/fedora_openqa_schedule/cli.py
93

typo.

169

I don't like this, and it will break our production deployments (I'd have to change things so prod explicitly passes --submit). Why did you change it? The existing setup allows for it to be specified by config but overridden on the CLI.

169

The help text here is a bit confusing, as really what the wiki and resultsdb args control is whether results of that type are *produced*, not whether they are *submitted*. submit controls whether whatever results are produced get *submitted* (or just printed).

scheduler/fedora_openqa_schedule/config.py
37

I don't think either of these will actually affect reporting, because the way you changed args.submit, the do_report arg for report_results() and resultsdb_report() will never be None, it will always be True (if --submit was specified) or False (if not). submit_resultsdb does affect whether a resultsdb 'job' gets created at all, though.

scheduler/fedora_openqa_schedule/report.py
224

'additional' (two d's)

scheduler/fedora_openqa_schedule/schedule.py
315

this probably needs to account for the case where the job has already been created, and take the job_id of the existing job?

we also have nothing that will ever change the status *away from* RUNNING, right, since there's no consumer yet? that seems like a problem.

The submit config settings / CLI args are really feeling pretty mushy right around now, what with this config setting being read here. I'm having trouble keeping track of them, they could probably do with a group-up rethink?

scheduler/schedule.conf.sample
18

I think this is some kinda rebase mistake? Don't think this exists any more.

20

so this - and wiki_url - seems to only be used for producing links that go in the test results; it's not actually the URL we connect to in order to retrieve the test results, the openQA-python-client configuration is used there (as we initiate OpenQA_Client with no args). I can see a use for this - in case we want our OpenQA_Client to connect to http://localhost but we want to use https://openqa.fedoraproject.org in the links that are produced in the results data, for instance. But it needs to be more clearly explained at least, and generally, the mismatches in how all these different URL settings are actually used is a bit confusing I think.

26

AFAICT this is not used at all.

This revision now requires changes to proceed.Jul 8 2016, 9:32 PM
garretraziel added a comment.Jul 11 2016, 9:55 AM

I'll correct mistakes and resend this.

scheduler/fedora_openqa_schedule/cli.py
169

I thought that we are using moksha consumers for reporting in prod, so cli is used only by us, during development. We thought that we are rarely (if ever) using cli for reporting to wiki. But I don't have strong opinion either way, I can change it back/make whole cli usage more backward compatible.

scheduler/fedora_openqa_schedule/config.py
37

I'll look at it.

scheduler/fedora_openqa_schedule/schedule.py
315

We can get ID of existing job, but question is whether we want. This way, if we (for whatever reason) run openQA for one compose twice, it will show as two jobs (but with the same ref_url and name). Or is there any chance this code will run twice for one openQA run for one compose?

I'll ask jskladan how ResultsDB works, but from what I heard, "job status" is more of a relic they want to get rid of and I am not sure whether it's problem if we leave job "RUNNING".

I agree that I should unify and clear "submit" behavior and usage.

scheduler/schedule.conf.sample
18

I put it there intentionally because I saw that we are setting this value in config.py. But yeah, I forgot that we aren't using this value anymore, so correct change is to remove CONFIG.set('report', 'jobs-wait', '360') from config.py.

20

I can change comment (or variable name) to make this more clear. OpenQA_Client is taking this URL from its configuration, but I didn't want to read configuration of some foreign library (and besides, as you said, there might be use case for it - connecting to localhost vs. creating links that are visitable from outside).

26

You are right, I put it there before I realized that we want to produce links to prod wiki every time.

adamwill added inline comments.Jul 11 2016, 10:47 PM
scheduler/fedora_openqa_schedule/cli.py
169

yeah, that's true actually. and the consumers do their own 'report/don't report' configuration. So it's not that significant, I just got a bit lost trying to trace it all out in review.

scheduler/fedora_openqa_schedule/schedule.py
315

OK, I see. It's a bit of a squishy problem, then.

The 'normal' case is pretty simple: this will just get hit once, automatically, per compose. In which case there's really no problem. But there *are* awkward cases occasionally which require re-submitting the jobs manually for some reason or other, say something goes wrong and the tests all fail when they shouldn't, I might have to log in and re-submit them manually, or something like that.

I guess I don't really know whether it'd be better to try and re-use the existing RDB 'job' (and hence effectively group all openQA submissions for the same compose into the same RDB 'job') or create a new one per submission, I'm really not sure (and I don't know how significant a difference it is). One thing to note, I suppose, would be that if we keep it this way, you'll get a different result for re-submitting openQA jobs via this tool (new RDB job) vs. restarting an openQA job through the web UI (same RDB job as the original openQA job).

garretraziel updated this revision to Diff 2405.Jul 22 2016, 8:37 AM
  • few changes
garretraziel updated this revision to Diff 2408.Jul 22 2016, 10:57 AM
  • resultsdb job creation finalization
jskladan added inline comments.Jul 22 2016, 11:30 AM
scheduler/fedora_openqa_schedule/schedule.py
315

We wouldn't really care about the job state - the Job state is kind of a leftover from the times when ResultsDB also stood in as execution state reporting tool. So having the jobs on RUNNING is not a problem. In the next ReslutsDB patch, the constraint on the job status will be removed, so one less problem to think about (it could all be in some random state then).

On the job-id front - I'd rather create new job - we'll be consuming the results anyway, the (number of) jobs is irrelevant - the most recent results for the specified item(s) are returned by resultsdb anyway. The job acts more as a group - in the next resultsdb version (sometimes in the future) I'll make the semantics clearer, and might even make the job/group not required.

garretraziel added a comment.Jul 25 2016, 2:27 PM

So, about changes in newest DR:

  1. I haven't changed CLI behaviour back. We should discuss it (and if you have different opinion, please say so!), but AFAIK CLI is used mostly during development, where you probably don't want to submit results to our wiki/ResultsDB. In production, we are using fedmsg consumers and I would also argue that this is preferred way of deployment. So I vote for CLI not submitting by default and fedmsg consumer submitting by default - that's the reason I put -s to CLI. But again, it's possible that I am wrong and I don't have strong opinion either way - I can change it back and make it somewhat backward-compatible (thoughts on how CLI should be used to submit/print results to Wiki/ResultsDB?).
  1. I have added --create_resultsdb_job argument to CLI. The thing is, you want to create job in ResultsDB during scheduling, but when we have schedule and report as different subcommands, one cannot find out whether user wants to really submit to ResultsDB before user runs report subcommand. We could do something like "search for existing job with this compose and if it doesn't exist, create one", but jskladan described it as "not ideal". Job from ResultsDB perspective describes "one run" (as in "one CLI schedule command" or "one fedmsg that caused openQA scheduling on compose") and is somewhat akin to buildbot job (and according to jskladan, is relic and was replaced by ExecDB). That's the reason we had to set it to "RUNNING" state, even though we couldn't care less. It would be great (and jskladan said that it's in the works) if "job" in ResultsDB wasn't mandatory and we could use some different result grouping. I don't think that not being able to submit jobs without having job ID will create any problems, but I can add something like possibility to specify job ID in CLI, if you think that it is necessary. So now it works like this:
    • If user will want to submit results to ResultDB, he needs to run CLI with --create_resultsdb_job. It creates new job in ResultsDB and then adds its ID to settings during job scheduling.
    • After job is finished, user can use report with --resultsdb. If job was run without --create_resultsdb_job, job doesn't have job ID in its settings and nothing will be reported to ResultsDB.
  1. Config files and, specifically, submit behaviour. Code above doesn't use config file for CLI for submit. Reasoning is that CLI is used mainly for development, so it either has default behaviour (not submit) or you can specify --submit. I would vote for reading submit value from config file only for fedmsg consumer-part of scheduler. But again, this is more of a proposal and opportunity for discussion.

All in all, I wanted to have often used variant as default solution for things. There is unfortunate dichotomy in that we have fedmsg consumer that is run automatically and is deployed in production and CLI tool that is run by hand and more often used for development and I think that we should decide how to behave in what conditions.

garretraziel updated this revision to Diff 2430.Jul 26 2016, 12:05 PM
  • another bunch of changes
garretraziel added a comment.Jul 26 2016, 12:09 PM

So I have actually changed behaviour little bit.

  • running report without any additional argument will just print out the results, adding --wiki submits results to wiki, adding --resultsdb submits results to ResultsDB (and you can, of course, use both arguments)
  • user can specify ResultsDB job ID from CLI during reporting (so user can report openQA jobs to different ResultsDB job or report job for which ResultsDB job ID wasn't created during scheduling)
adamwill accepted this revision.Aug 3 2016, 9:03 PM

OK, so aside from the inline comments I guess this is fine...the whole thing with the config files and the CLI parameters really comes from when we just had cron jobs running the CLI to submit results, so there weren't the different paths. What you have now is fine, I guess. My main use case for reporting results with the CLI is when something went wrong with the fedmsg stuff and it needs fixing manually, or something like that.

scheduler/fedora_openqa_schedule/schedule.py
315

compose.split('-')[1] is not entirely safe, as we have some composes now where the distribution name is 'Fedora-Atomic' - e.g. 'Fedora-Atomic-24-20160803.0'. I had this problem in fedfind too, which now uses compose.split('-')[-2] instead.

scheduler/schedule.conf.sample
24

So now you made this be used, but do we actually need it? What's the use case for changing it, ever?

This revision is now accepted and ready to land.Aug 3 2016, 9:03 PM
garretraziel added inline comments.Aug 8 2016, 10:40 AM
scheduler/schedule.conf.sample
24

No, I've removed (unused) wiki_stg_url. wiki_url was actually used before. I didn't want to hardcode wiki URL and figured out that you "might" want testcase detail URL to lead somewhere else (but TBH I don't believe someone is going to change this ever). Do you think I should hardcode this?

Closed by commit rOPENQA3cdc4a7b8a6a: add reporting to ResultsDB (authored by garretraziel). · Explain WhyAug 8 2016, 10:47 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2453

View Options

scheduler/fedora_openqa_schedule/cli.py

Show All 22 Lines
23 23​
24"""CLI module for fedora-openqa-schedule.""" 24​"""CLI module for fedora-openqa-schedule."""
25 25​
26# Standard libraries 26​# Standard libraries
27import argparse 27​import argparse
28import logging 28​import logging
29import sys 29​import sys
30 30​
31​# External dependencies
32​from resultsdb_api import ResultsDBapiException
33​
31# Internal dependencies 34​# Internal dependencies
32import fedora_openqa_schedule.schedule as schedule 35​import fedora_openqa_schedule.schedule as schedule
33import fedora_openqa_schedule.report as report 36​import fedora_openqa_schedule.report as report
34from fedora_openqa_schedule.config import CONFIG 37​from fedora_openqa_schedule.config import CONFIG
35 38​
36logger = logging.getLogger(__name__) 39​logger = logging.getLogger(__name__)
37 40​
41​
38### SUB-COMMAND METHODS 42​### SUB-COMMAND METHODS
39 43​
40def command_compose(args, wiki_url): 44​def command_compose(args, *_, **__):
41 """run OpenQA on a specified compose, optionally reporting 45​ """run OpenQA on a specified compose, optionally reporting
42 results if a matching wikitcms ValidationEvent is found by 46​ results if a matching wikitcms ValidationEvent is found by
43 relval/wikitcms. 47​ relval/wikitcms.
44 """ 48​ """
45 extraparams = None 49​ extraparams = None
46 if args.updates: 50​ if args.updates:
47 extraparams = {'GRUBADD': "inst.updates={0}".format(args.updates)} 51​ extraparams = {'GRUBADD': "inst.updates={0}".format(args.updates)}
48 try: 52​ try:
49 (compose, jobs) = schedule.jobs_from_compose( 53​ (compose, jobs) = schedule.jobs_from_compose(
50 args.location, force=args.force, extraparams=extraparams) 54​ args.location, force=args.force, extraparams=extraparams, create_resultsdb_job=args.create_resultsdb_job)
51 except schedule.TriggerException as err: 55​ except schedule.TriggerException as err:
52 logger.warning("No jobs run! %s", err) 56​ logger.warning("No jobs run! %s", err)
53 sys.exit(1) 57​ sys.exit(1)
54 58​
55 if jobs: 59​ if jobs:
56 print("Scheduled jobs: {0}".format(jobs)) 60​ print("Scheduled jobs: {0}".format(jobs))
57 sys.exit() 61​ sys.exit()
58 else: 62​ else:
59 msg = "No jobs run!" 63​ msg = "No jobs run!"
60 if not args.force: 64​ if not args.force:
61 msg = "{0} {1}".format( 65​ msg = "{0} {1}".format(
62 msg, "You did not pass --force: maybe jobs exist for all ISOs/flavors") 66​ msg, "You did not pass --force: maybe jobs exist for all ISOs/flavors")
63 logger.warning(msg) 67​ logger.warning(msg)
64 sys.exit(1) 68​ sys.exit(1)
65 69​
66 logger.debug("finished") 70​ logger.debug("finished")
67 sys.exit() 71​ sys.exit()
68 72​
73​
69def command_report(args, wiki_url): 74​def command_report(args, wiki_url):
70 """Map a list of openQA job IDs and/or builds to Wikitcms test 75​ """Map a list of openQA job IDs and/or builds to Wikitcms test
71 results, and either display the ResTups for inspection or report 76​ results, and either display the ResTups for inspection or report
72 the results to the wiki. 77​ the results to the wiki and/or ResultsDB.
73 """ 78​ """
74 jobs = [int(job) for job in args.jobs if job.isdigit()] 79​ jobs = [int(job) for job in args.jobs if job.isdigit()]
75 builds = [build for build in args.jobs if not build.isdigit()] 80​ builds = [build for build in args.jobs if not build.isdigit()]
76 try: 81​ resultsdb_url = CONFIG.get('report', 'resultsdb_url')
77 if jobs: 82​
78 report.report_results( 83​ if jobs:
79 wiki_url, jobs=jobs, do_report=args.submit) 84​ try:
80 if builds: 85​ if args.wiki:
81 for build in builds: 86​ report.wiki_report(wiki_url, jobs=jobs, do_report=True)
82 report.report_results( 87​ if args.resultsdb:
83 wiki_url, build=build, do_report=args.submit) 88​ report.resultsdb_report(
84 except report.LoginError: 89​ resultsdb_url, jobs=jobs, resultsdb_job_id=args.resultsdb_job_id, do_report=True)
85 # Module logs for us, no need to dupe it. 90​ if not args.wiki and not args.resultsdb:
86 sys.exit(1) 91​ # use wiki_report to only print results
92​ report.wiki_report(wiki_url, jobs=jobs, do_report=False)
93​ except (report.LoginError, ResultsDBapiException) as e:
adamwillUnsubmitted
Not Done

typo.

94​ logger.error("Reporting failed: %s", e)
95​ if builds:
96​ for build in builds:
97​ try:
98​ if args.wiki:
99​ report.wiki_report(wiki_url, build=build, do_report=True)
100​ if args.resultsdb:
101​ report.resultsdb_report(
102​ resultsdb_url, build=build, resultsdb_job_id=args.resultsdb_job_id, do_report=True)
103​ if not args.wiki and not args.resultsdb:
104​ # use wiki_report to only print results
105​ report.wiki_report(wiki_url, build=build, do_report=False)
106​ except (report.LoginError, ResultsDBapiException) as e:
107​ logger.error("Reporting failed: %s", e)
108​
87 109​
88### ARGUMENT PARSING AND SUB-COMMAND INIT 110​### ARGUMENT PARSING AND SUB-COMMAND INIT
89 111​
90def run(): 112​def run():
91 """Read in args, set up logging and wikitcms, and run sub-command 113​ """Read in args, set up logging and wikitcms, and run sub-command
92 function. 114​ function.
93 """ 115​ """
94 args = parse_args() 116​ args = parse_args()
Show All 9 Lines
104 126​
105 wiki_url = "fedoraproject.org" 127​ wiki_url = "fedoraproject.org"
106 if args.test: 128​ if args.test:
107 logger.debug("using test wiki") 129​ logger.debug("using test wiki")
108 wiki_url = "stg.fedoraproject.org" 130​ wiki_url = "stg.fedoraproject.org"
109 131​
110 args.func(args, wiki_url) 132​ args.func(args, wiki_url)
111 133​
134​
112def parse_args(): 135​def parse_args():
113 """Parse arguments with argparse.""" 136​ """Parse arguments with argparse."""
114 test_help = "Operate on the staging wiki (for testing)" 137​ test_help = "Operate on the staging wiki (for testing)"
115 parser = argparse.ArgumentParser(description=( 138​ parser = argparse.ArgumentParser(description=(
116 "Run OpenQA tests for a release validation test event.")) 139​ "Run OpenQA tests for a release validation test event."))
117 subparsers = parser.add_subparsers() 140​ subparsers = parser.add_subparsers()
118 141​
119 parser_compose = subparsers.add_parser( 142​ parser_compose = subparsers.add_parser(
120 'compose', description="Run for a specific compose (TC/RC or nightly). If a matching " 143​ 'compose', description="Run for a specific compose (TC/RC or nightly). If a matching "
121 "release validation test event can be found and --submit-results is passed, results " 144​ "release validation test event can be found and --submit-results is passed, results "
122 "will be reported.") 145​ "will be reported.")
123 parser_compose.add_argument( 146​ parser_compose.add_argument(
124 'location', help="The top-level URL of the compose", 147​ 'location', help="The top-level URL of the compose",
125 metavar="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-24-20160113.n.1/compose") 148​ metavar="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-24-20160113.n.1/compose")
126 parser_compose.add_argument( 149​ parser_compose.add_argument(
127 '--force', '-f', help="For each ISO/flavor combination, schedule jobs even if there " 150​ '--force', '-f', help="For each ISO/flavor combination, schedule jobs even if there "
128 "are existing, non-cancelled jobs for that combination", action='store_true') 151​ "are existing, non-cancelled jobs for that combination", action='store_true')
129 parser_compose.add_argument( 152​ parser_compose.add_argument(
130 '--updates', '-u', help="URL to an updates image to load for all tests. The tests that " 153​ '--updates', '-u', help="URL to an updates image to load for all tests. The tests that "
131 "test updates image loading will fail when you use this") 154​ "test updates image loading will fail when you use this")
155​ parser_compose.add_argument(
156​ '--create_resultsdb_job', '-r', help="Create job for this run in ResultsDB and add its ID "
157​ "to openQA job settings. This is needed if you want to submit results of scheduled tests into ResultsDB",
158​ action="store_true", default=False)
132 parser_compose.set_defaults(func=command_compose) 159​ parser_compose.set_defaults(func=command_compose)
133 160​
134 parser_report = subparsers.add_parser( 161​ parser_report = subparsers.add_parser(
135 'report', description="Map openQA job results to Wikitcms test results and optionally " 162​ 'report', description="Map openQA job results to Wikitcms test results and either log them to output or "
136 "submit them to the wiki.") 163​ "submit them to the wiki and/or ResultsDB.")
137 parser_report.add_argument('jobs', nargs='+', help="openQA job IDs or builds (e.g. " 164​ parser_report.add_argument(
138 "'Fedora-24-20160113.n.1'). For each build included, the latest jobs will be reported.") 165​ 'jobs', nargs='+', help="openQA job IDs or builds (e.g. "
166​ "'Fedora-24-20160113.n.1'). For each build included, the latest jobs will be reported.")
139 parser_report.set_defaults(func=command_report) 167​ parser_report.set_defaults(func=command_report)
140 168​ parser_report.add_argument(
141 submitgroup = parser.add_mutually_exclusive_group() 169​ "--wiki", action="store_true", default=False, help="Submit results to wiki")
adamwillUnsubmitted
Not Done

I don't like this, and it will break our production deployments (I'd have to change things so prod explicitly passes --submit). Why did you change it? The existing setup allows for it to be specified by config but overridden on the CLI.

garretrazielAuthorUnsubmitted
Not Done

I thought that we are using moksha consumers for reporting in prod, so cli is used only by us, during development. We thought that we are rarely (if ever) using cli for reporting to wiki. But I don't have strong opinion either way, I can change it back/make whole cli usage more backward compatible.

adamwillUnsubmitted
Not Done

yeah, that's true actually. and the consumers do their own 'report/don't report' configuration. So it's not that significant, I just got a bit lost trying to trace it all out in review.

adamwillUnsubmitted
Not Done

The help text here is a bit confusing, as really what the wiki and resultsdb args control is whether results of that type are *produced*, not whether they are *submitted*. submit controls whether whatever results are produced get *submitted* (or just printed).

142 submitgroup.add_argument('--submit', '-s', '--submit-results', action='store_true', 170​ parser_report.add_argument(
143 dest="submit", help="Where appropriate, submit results to the wiki") 171​ "--resultsdb", action="store_true", default=False, help="Submit results to ResultsDB")
144 submitgroup.add_argument('--no-submit', '-ns', '--no-submit-results', action='store_false', 172​ parser_report.add_argument(
145 dest="submit", help="Do not submit results to the wiki") 173​ "--resultsdb_job_id", "-r", help="ResultsDB job ID (if it wasn't set automatically by scheduling with "
146 # This will result in getting the 'submit' value from config file 174​ "--create_resultsdb_job argument)")
147 # if not explicitly specified, overriding the implied defaults of ​
148 # store_true and store_false ​
149 parser.set_defaults(submit=None) ​
150 175​
151 parser.add_argument( 176​ parser.add_argument(
152 '--test', '-t', help=test_help, action='store_true') 177​ '--test', '-t', help=test_help, action='store_true')
153 parser.add_argument( 178​ parser.add_argument(
154 '--log-file', '-f', help="If given, log into specified file. When not provided, stdout" 179​ '--log-file', '-f', help="If given, log into specified file. When not provided, stdout"
155 " is used", default=CONFIG.get('cli', 'log-file')) 180​ " is used", default=CONFIG.get('cli', 'log-file'))
156 parser.add_argument( 181​ parser.add_argument(
157 '--log-level', '-l', help="Specify log level to be outputted", 182​ '--log-level', '-l', help="Specify log level to be outputted",
158 choices=('debug', 'info', 'warning', 'error', 'critical'), 183​ choices=('debug', 'info', 'warning', 'error', 'critical'),
159 default=CONFIG.get('cli', 'log-level')) 184​ default=CONFIG.get('cli', 'log-level'))
160 185​
161 return parser.parse_args() 186​ return parser.parse_args()
162 187​
188​
163def main(): 189​def main():
164 """Main loop.""" 190​ """Main loop."""
165 try: 191​ try:
166 run() 192​ run()
167 except KeyboardInterrupt: 193​ except KeyboardInterrupt:
168 sys.stderr.write("Interrupted, exiting...\n") 194​ sys.stderr.write("Interrupted, exiting...\n")
169 sys.exit(1) 195​ sys.exit(1)
170 196​
171if __name__ == '__main__': 197​if __name__ == '__main__':
172 main() 198​ main()
View Options

scheduler/fedora_openqa_schedule/conf_test_suites.py

Show First 20 Lines • Show All 363 Lines • ▼ Show 20 Line(s)26TESTCASES = {
364 # "": { 364​ # "": {
365 # "name": "", # optional, use when same testcase occurs on multiple rows with different link text 365​ # "name": "", # optional, use when same testcase occurs on multiple rows with different link text
366 # "section": "", # optional, some result pages have no sections 366​ # "section": "", # optional, some result pages have no sections
367 # "env": "x86", 367​ # "env": "x86",
368 # "type": "Installation", 368​ # "type": "Installation",
369 # }, 369​ # },
370} 370​}
371 371​
372​# This is used to update TESTCASES dict for ResultsDB reporting
373​TESTCASES_RESULTSDB_EXTRADATA = {
374​ "QA:Testcase_partitioning_guided_shrink": {
375​ "filesystem": "$FS$", # needed to convert test_fs to only one testcase
376​ },
377​}
372 378​
373TESTSUITES = { 379​TESTSUITES = {
374 "install_default": [ 380​ "install_default": [
375 "QA:Testcase_Boot_default_install", 381​ "QA:Testcase_Boot_default_install",
376 "QA:Testcase_install_to_VirtIO", 382​ "QA:Testcase_install_to_VirtIO",
377 "QA:Testcase_partitioning_guided_empty", 383​ "QA:Testcase_partitioning_guided_empty",
378 "QA:Testcase_Anaconda_User_Interface_Graphical", 384​ "QA:Testcase_Anaconda_User_Interface_Graphical",
379 "QA:Testcase_Anaconda_user_creation", 385​ "QA:Testcase_Anaconda_user_creation",
▲ Show 20 Lines • Show All 324 Lines • Show Last 20 Lines
View Options

scheduler/fedora_openqa_schedule/config.py

Show All 28 Lines
29# Read in config from /etc/fedora-openqa/schedule.conf or 29​# Read in config from /etc/fedora-openqa/schedule.conf or
30# ~/.config/fedora-openqa/schedule.conf after setting some default 30​# ~/.config/fedora-openqa/schedule.conf after setting some default
31# values. 31​# values.
32CONFIG = configparser.ConfigParser() 32​CONFIG = configparser.ConfigParser()
33CONFIG.add_section('cli') 33​CONFIG.add_section('cli')
34CONFIG.add_section('report') 34​CONFIG.add_section('report')
35CONFIG.set('cli', 'log-file', '') 35​CONFIG.set('cli', 'log-file', '')
36CONFIG.set('cli', 'log-level', 'info') 36​CONFIG.set('cli', 'log-level', 'info')
37CONFIG.set('report', 'submit', 'true') 37​CONFIG.set('report', 'submit_wiki', 'true')
adamwillUnsubmitted
Not Done

I don't think either of these will actually affect reporting, because the way you changed args.submit, the do_report arg for report_results() and resultsdb_report() will never be None, it will always be True (if --submit was specified) or False (if not). submit_resultsdb does affect whether a resultsdb 'job' gets created at all, though.

garretrazielAuthorUnsubmitted
Not Done

I'll look at it.

38CONFIG.set('report', 'jobs-wait', '360') 38​CONFIG.set('report', 'submit_resultsdb', 'true')
39​CONFIG.set('report', 'openqa_url', 'http://localhost')
40​CONFIG.set('report', 'resultsdb_url', 'http://localhost:5001/api/v1.0/')
41​CONFIG.set('report', 'wiki_url', 'https://fedoraproject.org/wiki')
42​CONFIG.set('report', 'wiki_stg_url', 'https://stg.fedoraproject.org/wiki')
39CONFIG.read('/etc/fedora-openqa/schedule.conf') 43​CONFIG.read('/etc/fedora-openqa/schedule.conf')
40CONFIG.read('{0}/.config/fedora-openqa/schedule.conf'.format(os.path.expanduser('~'))) 44​CONFIG.read('{0}/.config/fedora-openqa/schedule.conf'.format(os.path.expanduser('~')))
41 45​
42# The default set of tested images. The format intentionally follows 46​# The default set of tested images. The format intentionally follows
43# the format of images.json closely, and this set can be overridden by 47​# the format of images.json closely, and this set can be overridden by
44# an 'images.json' file in /etc/fedora-openqa or 48​# an 'images.json' file in /etc/fedora-openqa or
45# ~/.config/fedora-openqa. 49​# ~/.config/fedora-openqa.
46 50​
▲ Show 20 Lines • Show All 157 Lines • Show Last 20 Lines
View Options

scheduler/fedora_openqa_schedule/consumer.py

Show First 20 Lines • Show All 87 Lines • ▼ Show 20 Line(s)81class OpenQAWikiConsumer(fedmsg.consumers.FedmsgConsumer):
88 openQA but retrieving the jobs with those same IDs from the other 88​ openQA but retrieving the jobs with those same IDs from the other
89 openQA). 89​ openQA).
90 """ 90​ """
91 91​
92 def consume(self, message): 92​ def consume(self, message):
93 """Consume incoming message.""" 93​ """Consume incoming message."""
94 job = message['body']['msg']['id'] 94​ job = message['body']['msg']['id']
95 self.log.info("%s: reporting results for %s", self.__class__.__name__, job) 95​ self.log.info("%s: reporting results for %s", self.__class__.__name__, job)
96 results = report.report_results(self.url, jobs=[job], do_report=self.report) 96​ results = report.wiki_report(self.url, jobs=[job], do_report=self.report)
97 if not self.report: 97​ if not self.report:
98 for res in results: 98​ for res in results:
99 self.log.info("%s: would report %s", self.__class__.__name__, res) 99​ self.log.info("%s: would report %s", self.__class__.__name__, res)
100 100​
101 101​
102class OpenQAProductionWikiConsumer(OpenQAWikiConsumer): 102​class OpenQAProductionWikiConsumer(OpenQAWikiConsumer):
103 """A result reporting consumer that listens for production fedmsgs 103​ """A result reporting consumer that listens for production fedmsgs
104 and reports to the production wiki. Only one of these should ever 104​ and reports to the production wiki. Only one of these should ever
Show All 36 Lines
View Options

scheduler/fedora_openqa_schedule/report.py

Show All 18 Lines
19# Josef Skladanka <jskladan@redhat.com> 19​# Josef Skladanka <jskladan@redhat.com>
20# Adam Williamson <awilliam@redhat.com> 20​# Adam Williamson <awilliam@redhat.com>
21 21​
22"""report module for fedora-openqa-schedule. Functions related to job 22​"""report module for fedora-openqa-schedule. Functions related to job
23result reporting go here. 23​result reporting go here.
24""" 24​"""
25 25​
26# standard libraries 26​# standard libraries
27​import re
27import copy 28​import copy
28import time 29​import time
29import logging 30​import logging
30from operator import attrgetter 31​from operator import attrgetter
31 32​
32# External dependencies 33​# External dependencies
33from openqa_client.client import OpenQA_Client 34​from openqa_client.client import OpenQA_Client
34from wikitcms.wiki import Wiki, ResTuple 35​from wikitcms.wiki import Wiki, ResTuple
36​from resultsdb_api import ResultsDBapi, ResultsDBapiException
35 37​
36# Internal dependencies 38​# Internal dependencies
37import fedora_openqa_schedule.conf_test_suites as conf_test_suites 39​import fedora_openqa_schedule.conf_test_suites as conf_test_suites
38from fedora_openqa_schedule.config import CONFIG 40​from fedora_openqa_schedule.config import CONFIG
39 41​
40logger = logging.getLogger(__name__) 42​logger = logging.getLogger(__name__)
41 43​
42 44​
▲ Show 20 Lines • Show All 82 Lines • ▼ Show 20 Line(s)126 result = ResTuple(
125 testtype=uniqueres['type'], testcase=testcase, 127​ testtype=uniqueres['type'], testcase=testcase,
126 section=uniqueres.get('section'), testname=uniqueres.get('name', ''), 128​ section=uniqueres.get('section'), testname=uniqueres.get('name', ''),
127 env=uniqueres.get('env', ''), status='pass', bot=True, cid=composeid) 129​ env=uniqueres.get('env', ''), status='pass', bot=True, cid=composeid)
128 passed_testcases.add(result) 130​ passed_testcases.add(result)
129 131​
130 return sorted(list(passed_testcases), key=attrgetter('testcase')) 132​ return sorted(list(passed_testcases), key=attrgetter('testcase'))
131 133​
132 134​
133def report_results(wiki_url, jobs=None, build=None, do_report=None): 135​def wiki_report(wiki_url, jobs=None, build=None, do_report=None):
134 """Report results from openQA jobs to Wikitcms. Either jobs (an 136​ """Report results from openQA jobs to Wikitcms. Either jobs (an
135 iterable of job IDs) or build (an openQA BUILD string, usually a 137​ iterable of job IDs) or build (an openQA BUILD string, usually a
136 Fedora compose ID) is required (if neither is specified, the 138​ Fedora compose ID) is required (if neither is specified, the
137 openQA client will raise TypeError). If do_report is False, will 139​ openQA client will raise TypeError). If do_report is False, will
138 just print out the python-wikitcms ResTups for inspection. If 140​ just print out the python-wikitcms ResTups for inspection. If
139 do_report is None, will read the setting from the config file; if 141​ do_report is None, will read the setting from the config file; if
140 no config file is present, default is True. 142​ no config file is present, default is True.
141 """ 143​ """
142 client = OpenQA_Client() 144​ client = OpenQA_Client()
143 jobs = client.get_jobs(jobs=jobs, build=build) 145​ jobs = client.get_jobs(jobs=jobs, build=build)
144 passed_testcases = get_passed_testcases(jobs) 146​ passed_testcases = get_passed_testcases(jobs)
145 logger.info("passed testcases: %s", passed_testcases) 147​ logger.info("passed testcases: %s", passed_testcases)
146 if do_report is None: 148​ if do_report is None:
147 do_report = CONFIG.getboolean('report', 'submit') 149​ do_report = CONFIG.getboolean('report', 'submit_wiki')
148 150​
149 if do_report: 151​ if do_report:
150 logger.info("reporting test passes") 152​ logger.info("reporting test passes")
151 wiki = Wiki(('https', wiki_url), '/w/') 153​ wiki = Wiki(('https', wiki_url), '/w/')
152 if not wiki.logged_in: 154​ if not wiki.logged_in:
153 wiki.login() 155​ wiki.login()
154 if not wiki.logged_in: 156​ if not wiki.logged_in:
155 logger.error("could not log in to wiki") 157​ logger.error("could not log in to wiki")
156 raise LoginError 158​ raise LoginError
157 159​
158 # Submit the results 160​ # Submit the results
159 (insuffs, dupes) = wiki.report_validation_results(passed_testcases) 161​ (insuffs, dupes) = wiki.report_validation_results(passed_testcases)
160 for dupe in dupes: 162​ for dupe in dupes:
161 tmpl = "already reported result for test %s, env %s! Will not report dupe." 163​ tmpl = "already reported result for test %s, env %s! Will not report dupe."
162 logger.info(tmpl, dupe.testcase, dupe.env) 164​ logger.info(tmpl, dupe.testcase, dupe.env)
163 for insuff in insuffs: 165​ for insuff in insuffs:
164 tmpl = "insufficient data for test %s, env %s! Will not report." 166​ tmpl = "insufficient data for test %s, env %s! Will not report."
165 logger.info(tmpl, insuff.testcase, insuff.env) 167​ logger.info(tmpl, insuff.testcase, insuff.env)
166 return [] 168​ return []
167 169​
168 else: 170​ else:
169 logger.warning("no reporting is done") 171​ logger.warning("no reporting is done")
170 return passed_testcases 172​ return passed_testcases
173​
174​
175​def resultsdb_report(resultsdb_url, jobs=None, build=None, resultsdb_job_id=None, do_report=None):
176​ """Report results from openQA jobs to ResultsDB. Either jobs (an
177​ iterable of job IDs) or build (an openQA BUILD string, usually a
178​ Fedora compose ID) is required (if neither is specified, the
179​ openQA client will raise TypeError). If set, report jobs to ResultsDB
180​ job, specified by resultsdb_job_id. If not set, try to obtain ID from
181​ openQA job settings. If do_report is False, will
182​ just print out list of jobs to report for inspection. If
183​ do_report is None, will read the setting from the config file; if
184​ no config file is present, default is True.
185​ """
186​ if do_report is None:
187​ do_report = CONFIG.getboolean('report', 'submit_resultsdb')
188​
189​ if do_report:
190​ try:
191​ rdb_instance = ResultsDBapi(resultsdb_url)
192​ except ResultsDBapiException as e:
193​ logger.error(e)
194​ return
195​
196​ client = OpenQA_Client()
197​ jobs = client.get_jobs(jobs=jobs, build=build)
198​
199​ tcname_safeify = re.compile(r"\W+") # allow only words to be used in testcase name
200​
201​ for job in jobs:
202​ # ResultsDB's job_id must be set in argument or during test scheduling so we know what job to report to
203​ rdb_job_id = resultsdb_job_id if resultsdb_job_id else job['settings'].get('RESULTSDB_JOB_ID', None)
204​ if not rdb_job_id:
205​ logger.warning("Job %s doesn't have ResultsDB job ID in settings nor was it set by CLI argument", job['id'])
206​ continue
207​ testsuite = job['settings']['TEST']
208​ if testsuite not in conf_test_suites.TESTSUITES:
209​ logger.warning("No TESTSUITES entry found for test {0}!".format(testsuite))
210​ continue
211​ # map openQA's result to ResultsDB's result
212​ rdb_result = {'passed': "PASSED", 'failed': "FAILED"}.get(job['result'], 'NEEDS_INSPECTION')
213​ for testcase in conf_test_suites.TESTSUITES[testsuite]:
214​ # create dot-separated and safe name from testcase name from wiki
215​ tc_name = tcname_safeify.sub("_", testcase).lower()
216​ tc_name = tc_name[len("qa_testcase_"):] if tc_name.startswith("qa_testcase_") else tc_name
217​ tc_name = "openqa.%s.%s" % (conf_test_suites.TESTCASES[testcase]['type'].lower(), tc_name)
218​
219​ # append all extra data that could be useful
220​ extradata = {}
221​ extradata['arch'] = job['settings'].get('ARCH', None)
222​ extradata['firmware'] = 'uefi' if 'UEFI' in job['settings'] else 'bios'
223​ extradata['item'] = job['settings'].get('BUILD', None)
224​ extradata['type'] = "compose"
adamwillUnsubmitted
Not Done

'additional' (two d's)

225​ extradata['subvariant'] = job['settings'].get('SUBVARIANT', None)
226​ extradata['imagetype'] = job['settings'].get('IMAGETYPE', None)
227​ # append all additional data
228​ if testcase in conf_test_suites.TESTCASES_RESULTSDB_EXTRADATA:
229​ extradata.update(_uniqueres_replacements(job, conf_test_suites.TESTCASES_RESULTSDB_EXTRADATA[testcase]))
230​
231​ # create link back to openQA testrun
232​ job_url = "%s/tests/%s" % (CONFIG.get('report', 'openqa_url'), job['id'])
233​ # create link to testcase page on Fedora wiki
234​ testcase_url = "%s/%s" % (CONFIG.get('report', 'wiki_url'), testcase)
235​
236​ logger.info("ResultsDB: %s, %s, %s, %s", rdb_job_id, tc_name, rdb_result, extradata)
237​ if do_report:
238​ rdb_instance.create_result(rdb_job_id, tc_name, rdb_result, log_url=job_url, **extradata)
239​ # this updates testcase link in ResultsDB with every POST. otherwise we would check whether
240​ # this testcase exists in ResultsDB and update link if needed, but it would actually take more
241​ # requests than this solution
242​ rdb_instance.update_testcase(tc_name, testcase_url)
View Options

scheduler/fedora_openqa_schedule/schedule.py

Show All 28 Lines
29import os.path 29​import os.path
30import re 30​import re
31 31​
32# External dependencies 32​# External dependencies
33import fedfind.helpers 33​import fedfind.helpers
34from openqa_client.client import OpenQA_Client 34​from openqa_client.client import OpenQA_Client
35from six.moves.urllib.request import urlopen 35​from six.moves.urllib.request import urlopen
36from six.moves.urllib.error import URLError, HTTPError 36​from six.moves.urllib.error import URLError, HTTPError
37​from resultsdb_api import ResultsDBapi, ResultsDBapiException
37 38​
38# Internal dependencies 39​# Internal dependencies
39from fedora_openqa_schedule.config import CONFIG, WANTED 40​from fedora_openqa_schedule.config import CONFIG, WANTED
40 41​
41logger = logging.getLogger(__name__) 42​logger = logging.getLogger(__name__)
42 43​
43FORMAT_TO_PARAM = { 44​FORMAT_TO_PARAM = {
44 "iso": "ISO_URL", 45​ "iso": "ISO_URL",
Show All 22 Lines65def _get_atomic_installer(location, cid):
67 are not done with Pungi 4 at present so do not have the metadata 68​ are not done with Pungi 4 at present so do not have the metadata
68 we want to use. This should die as soon as all composes we want to 69​ we want to use. This should die as soon as all composes we want to
69 test are done in Pungi 4 style. 70​ test are done in Pungi 4 style.
70 """ 71​ """
71 url = '{0}/Cloud_Atomic/x86_64/iso/Fedora-Cloud_Atomic-x86_64-{1}.iso'.format( 72​ url = '{0}/Cloud_Atomic/x86_64/iso/Fedora-Cloud_Atomic-x86_64-{1}.iso'.format(
72 location, cid) 73​ location, cid)
73 try: 74​ try:
74 resp = urlopen(url) 75​ resp = urlopen(url)
75 return [('Atomic-boot-iso', 'x86_64', 0, {'ISO_URL': url})] 76​ return [('Atomic-boot-iso', 'x86_64', 0, {'ISO_URL': url}, 'Atomic', 'boot')]
76 except (ValueError, URLError, HTTPError): 77​ except (ValueError, URLError, HTTPError):
77 raise TriggerException("Compose not found, or failed!") 78​ raise TriggerException("Compose not found, or failed!")
78 79​
79 80​
80def _get_compose_id(location): 81​def _get_compose_id(location):
81 """Given a compose location, find the compose ID. Really we'd like 82​ """Given a compose location, find the compose ID. Really we'd like
82 taskotron to give us this, as fedmsg provides it, but taskotron is 83​ taskotron to give us this, as fedmsg provides it, but taskotron is
83 kinda tied to giving us just *one* variable property of the fedmsg 84​ kinda tied to giving us just *one* variable property of the fedmsg
Show All 12 Lines95def _get_dkboot_urls(location, arch='armhfp'):
96 files. 97​ files.
97 """ 98​ """
98 pxeboot_url = '{0}/Everything/{1}/os/images/pxeboot/{2}' 99​ pxeboot_url = '{0}/Everything/{1}/os/images/pxeboot/{2}'
99 return pxeboot_url.format(location, arch, 'vmlinuz'), pxeboot_url.format(location, arch, 'initrd.img') 100​ return pxeboot_url.format(location, arch, 'vmlinuz'), pxeboot_url.format(location, arch, 'initrd.img')
100 101​
101 102​
102def _get_images(location, wanted=WANTED): 103​def _get_images(location, wanted=WANTED):
103 """Given a Pungi compose top-level location, this returns a list 104​ """Given a Pungi compose top-level location, this returns a list
104 of (flavor, arch, score, {param: url}) tuples for images to be tested. 105​ of (flavor, arch, score, {param: url}, subvariant, imagetype) tuples for images to be tested.
105 """ 106​ """
106 try: 107​ try:
107 resp = urlopen('{0}/metadata/images.json'.format(location)) 108​ resp = urlopen('{0}/metadata/images.json'.format(location))
108 metadata = json.load(resp) 109​ metadata = json.load(resp)
109 except (ValueError, URLError, HTTPError): 110​ except (ValueError, URLError, HTTPError):
110 raise TriggerException("Compose not found, or failed!") 111​ raise TriggerException("Compose not found, or failed!")
111 images = [] 112​ images = []
112 for variant in wanted.keys(): 113​ for variant in wanted.keys():
Show All 10 Lines122 for wantimg in wantimgs:
123 for foundimg in foundimgs: 124​ for foundimg in foundimgs:
124 # see if the foundimg matches the wantimg 125​ # see if the foundimg matches the wantimg
125 if not all(item in foundimg.items() for item in matchdict.items()): 126​ if not all(item in foundimg.items() for item in matchdict.items()):
126 continue 127​ continue
127 score = wantimg.get('score', 0) 128​ score = wantimg.get('score', 0)
128 # assign a 'flavor' by combining a few productmd 129​ # assign a 'flavor' by combining a few productmd
129 # values, with dashes replaced by underscores so 130​ # values, with dashes replaced by underscores so
130 # we can split this back up again later for report 131​ # we can split this back up again later for report
131 flavor = [item.replace('-', '_') 132​ subvariant = foundimg['subvariant']
132 for item in foundimg['subvariant'], foundimg['type'], 133​ imagetype = foundimg['type']
133 foundimg['format']] 134​ imageformat = foundimg['format']
135​ flavor = [item.replace('-', '_') for item in (subvariant, imagetype, imageformat)]
134 flavor = '-'.join(flavor) 136​ flavor = '-'.join(flavor)
135 url = "{0}/{1}".format(location, foundimg['path']) 137​ url = "{0}/{1}".format(location, foundimg['path'])
136 logger.debug("Found image %s for arch %s at %s", flavor, arch, url) 138​ logger.debug("Found image %s for arch %s at %s", flavor, arch, url)
137 139​
138 # some tests need more than one file, so let's collect them now 140​ # some tests need more than one file, so let's collect them now
139 param_urls = { 141​ param_urls = {
140 FORMAT_TO_PARAM[foundimg['format']]: url 142​ FORMAT_TO_PARAM[foundimg['format']]: url
141 } 143​ }
142 # TODO: WARNING! - workaround for ARM disk images that need kernel for direct kernel boot 144​ # TODO: WARNING! - workaround for ARM disk images that need kernel for direct kernel boot
143 if arch == "armhfp" and foundimg['format'] == "raw.xz": 145​ if arch == "armhfp" and foundimg['format'] == "raw.xz":
144 kernel_url, initrd_url = _get_dkboot_urls(location, arch) 146​ kernel_url, initrd_url = _get_dkboot_urls(location, arch)
145 param_urls["KERNEL_URL"] = kernel_url 147​ param_urls["KERNEL_URL"] = kernel_url
146 param_urls["INITRD_URL"] = initrd_url 148​ param_urls["INITRD_URL"] = initrd_url
147 149​
148 images.append((flavor, arch, score, param_urls)) 150​ images.append((flavor, arch, score, param_urls, subvariant, imagetype))
149 return images 151​ return images
150 152​
151 153​
152def _find_duplicate_jobs(client, param_urls, flavor): 154​def _find_duplicate_jobs(client, param_urls, flavor):
153 """Check if we have any existing non-cancelled jobs for this 155​ """Check if we have any existing non-cancelled jobs for this
154 ISO/HDD and flavor (checking flavor is important otherwise we'd 156​ ISO/HDD and flavor (checking flavor is important otherwise we'd
155 bail on doing the per-ISO jobs for the ISO we use for the 157​ bail on doing the per-ISO jobs for the ISO we use for the
156 'universal' tests). ISO/HDD are taken from param_urls dict. 158​ 'universal' tests). ISO/HDD are taken from param_urls dict.
Show All 16 Lines174 jobs = [job for job in jobs if
173 job.get('state') != 'cancelled' and job.get('result') != 'user_cancelled'] 175​ job.get('state') != 'cancelled' and job.get('result') != 'user_cancelled']
174 if jobs: 176​ if jobs:
175 logger.info("run_openqa_jobs: Existing jobs found for asset %s flavor %s, and force " 177​ logger.info("run_openqa_jobs: Existing jobs found for asset %s flavor %s, and force "
176 "not set! No jobs scheduled.", assetname, flavor) 178​ "not set! No jobs scheduled.", assetname, flavor)
177 return jobs 179​ return jobs
178 return [] 180​ return []
179 181​
180 182​
181def run_openqa_jobs(param_urls, flavor, arch, build, force=False, extraparams=None): 183​def run_openqa_jobs(param_urls, flavor, arch, subvariant, imagetype, build, force=False, extraparams=None, resultsdb_job_id=None):
182 """# run OpenQA 'isos' job on ISO at urls from 'param_urls', with given arch 184​ """# run OpenQA 'isos' job on ISO at urls from 'param_urls', with given arch
183 and a build identifier. **NOTE**: 'build' is passed to openQA as 185​ and a build identifier. **NOTE**: 'build' is passed to openQA as
184 BUILD and is later retrieved and parsed by report.py for wiki 186​ BUILD and is later retrieved and parsed by report.py for wiki
185 report generation. Returns list of job IDs. If force is False, 187​ report generation. Returns list of job IDs. If force is False,
186 jobs will only be scheduled if there are no existing, 188​ jobs will only be scheduled if there are no existing,
187 non-cancelled jobs for the same ISO and flavor. If extraparams 189​ non-cancelled jobs for the same ISO and flavor. If extraparams
188 is specified, it must be a dict or something else that can be 190​ is specified, it must be a dict or something else that can be
189 combined with a dict using `update`; it adds additional parameters 191​ combined with a dict using `update`; it adds additional parameters
Show All 17 Lines
207 params = { 209​ params = {
208 'DISTRI': 'fedora', 210​ 'DISTRI': 'fedora',
209 'VERSION': build.split('-')[1], 211​ 'VERSION': build.split('-')[1],
210 'FLAVOR': flavor, 212​ 'FLAVOR': flavor,
211 'ARCH': arch, 213​ 'ARCH': arch,
212 'BUILD': build, 214​ 'BUILD': build,
213 'CURRREL': currrel, 215​ 'CURRREL': currrel,
214 'PREVREL': prevrel, 216​ 'PREVREL': prevrel,
217​ 'SUBVARIANT': subvariant,
218​ 'IMAGETYPE': imagetype
215 } 219​ }
216 if extraparams: 220​ if extraparams:
217 params.update(extraparams) 221​ params.update(extraparams)
218 # mung the BUILD so this is not considered a 'real' test run 222​ # mung the BUILD so this is not considered a 'real' test run
219 params['BUILD'] = "{0}-EXTRA".format(params['BUILD']) 223​ params['BUILD'] = "{0}-EXTRA".format(params['BUILD'])
224​ if resultsdb_job_id:
225​ params['RESULTSDB_JOB_ID'] = resultsdb_job_id
220 226​
221 if arch == 'armhfp': 227​ if arch == 'armhfp':
222 params.update({ 228​ params.update({
223 'ARCH': 'arm', 229​ 'ARCH': 'arm',
224 'FAMILY': 'arm', 230​ 'FAMILY': 'arm',
225 # ARM kernel arguments since we are using direct kernel boot 231​ # ARM kernel arguments since we are using direct kernel boot
226 'APPEND': 'rw root=LABEL=_/ rootwait console=ttyAMA0 console=tty0', 232​ 'APPEND': 'rw root=LABEL=_/ rootwait console=ttyAMA0 console=tty0',
227 }) 233​ })
Show All 18 Lines
246 252​
247 output = client.openqa_request('POST', 'isos', params) 253​ output = client.openqa_request('POST', 'isos', params)
248 logger.debug("run_openqa_jobs: executed") 254​ logger.debug("run_openqa_jobs: executed")
249 logger.debug("run_openqa_jobs: planned jobs: %s", output["ids"]) 255​ logger.debug("run_openqa_jobs: planned jobs: %s", output["ids"])
250 256​
251 return output["ids"] 257​ return output["ids"]
252 258​
253 259​
254def jobs_from_compose(location, wanted=WANTED, force=False, extraparams=None): 260​def jobs_from_compose(location, wanted=WANTED, force=False, extraparams=None, create_resultsdb_job=None):
255 """Schedule jobs against a specific compose. Returns a 2-tuple 261​ """Schedule jobs against a specific compose. Returns a 2-tuple
256 of the compose ID and the list of job IDs. 262​ of the compose ID and the list of job IDs.
257 263​
258 location is the top level of the compose. Note this value is 264​ location is the top level of the compose. Note this value is
259 provided by fedmsg 'pungi.compose.status.change' messages as 265​ provided by fedmsg 'pungi.compose.status.change' messages as
260 'location'. 266​ 'location'.
261 267​
262 wanted is a dict defining which images from the compose we should 268​ wanted is a dict defining which images from the compose we should
263 schedule tests for. It is passed direct to _get_images(). The 269​ schedule tests for. It is passed direct to _get_images(). The
264 default set of tested images is specified in config.py. It can be 270​ default set of tested images is specified in config.py. It can be
265 overridden by a system-wide or per-user file as well as with this 271​ overridden by a system-wide or per-user file as well as with this
266 argument. The layout is, intentionally, a subset of the pungi 272​ argument. The layout is, intentionally, a subset of the pungi
267 images.json metadata file (except for the 'score' item). Check 273​ images.json metadata file (except for the 'score' item). Check
268 config.py to see the layout and for more information. 274​ config.py to see the layout and for more information.
269 275​
270 If force is False, for each ISO, jobs will only be scheduled if 276​ If force is False, for each ISO, jobs will only be scheduled if
271 there are no existing, non-cancelled jobs for the same ISO and 277​ there are no existing, non-cancelled jobs for the same ISO and
272 flavor. 278​ flavor.
273 279​
274 extraparams is passed through to `run_openqa_jobs()`. It can be 280​ extraparams is passed through to `run_openqa_jobs()`. It can be
275 a dict (or anything else that can be passed to `dict.update()`) 281​ a dict (or anything else that can be passed to `dict.update()`)
276 containing arbitrary extra parameters to be included in the ISO 282​ containing arbitrary extra parameters to be included in the ISO
277 post request (usually this will be to specify extra openQA vars). 283​ post request (usually this will be to specify extra openQA vars).
284​
285​ create_resultsdb_job controls whether to create job in ResultsDB for
286​ this compose.
278 """ 287​ """
279 location = location.strip('/') 288​ location = location.strip('/')
280 # trigger ugly special-casing for non-Pungi4-ified Two Week Atomic 289​ # trigger ugly special-casing for non-Pungi4-ified Two Week Atomic
281 # nightlies 290​ # nightlies
282 if 'alt/atomic/testing' in location: 291​ if 'alt/atomic/testing' in location:
283 compose = _get_atomic_id(location) 292​ compose = _get_atomic_id(location)
284 logger.debug("Finding images for Atomic compose %s in location %s", compose, location) 293​ logger.debug("Finding images for Atomic compose %s in location %s", compose, location)
285 images = _get_atomic_installer(location, compose) 294​ images = _get_atomic_installer(location, compose)
286 else: 295​ else:
287 compose = _get_compose_id(location) 296​ compose = _get_compose_id(location)
288 logger.debug("Finding images for compose %s in location %s", compose, location) 297​ logger.debug("Finding images for compose %s in location %s", compose, location)
289 images = _get_images(location, wanted=wanted) 298​ images = _get_images(location, wanted=wanted)
290 if len(images) == 0: 299​ if len(images) == 0:
291 raise TriggerException("Compose found, but no available images") 300​ raise TriggerException("Compose found, but no available images")
292 jobs = [] 301​ jobs = []
293 univs = {} 302​ univs = {}
294 303​
304​ # create jobs instance in resultsdb if necessary
305​ rdb_job_id = None
306​
307​ if create_resultsdb_job is None:
308​ create_resultsdb_job = CONFIG.get('report', 'submit_resultsdb')
309​
310​ if create_resultsdb_job:
311​ try:
312​ rdb_instance = ResultsDBapi(CONFIG.get('report', 'resultsdb_url'))
313​ # add link to page with overall results
314​ ref_url = "%s/tests/overview?distri=fedora&version=%s&build=%s" % (CONFIG.get('report', 'openqa_url'),
315​ compose.split('-')[-2], compose)
adamwillUnsubmitted
Not Done

this probably needs to account for the case where the job has already been created, and take the job_id of the existing job?

we also have nothing that will ever change the status *away from* RUNNING, right, since there's no consumer yet? that seems like a problem.

The submit config settings / CLI args are really feeling pretty mushy right around now, what with this config setting being read here. I'm having trouble keeping track of them, they could probably do with a group-up rethink?

garretrazielAuthorUnsubmitted
Not Done

We can get ID of existing job, but question is whether we want. This way, if we (for whatever reason) run openQA for one compose twice, it will show as two jobs (but with the same ref_url and name). Or is there any chance this code will run twice for one openQA run for one compose?

I'll ask jskladan how ResultsDB works, but from what I heard, "job status" is more of a relic they want to get rid of and I am not sure whether it's problem if we leave job "RUNNING".

I agree that I should unify and clear "submit" behavior and usage.

adamwillUnsubmitted
Not Done

OK, I see. It's a bit of a squishy problem, then.

The 'normal' case is pretty simple: this will just get hit once, automatically, per compose. In which case there's really no problem. But there *are* awkward cases occasionally which require re-submitting the jobs manually for some reason or other, say something goes wrong and the tests all fail when they shouldn't, I might have to log in and re-submit them manually, or something like that.

I guess I don't really know whether it'd be better to try and re-use the existing RDB 'job' (and hence effectively group all openQA submissions for the same compose into the same RDB 'job') or create a new one per submission, I'm really not sure (and I don't know how significant a difference it is). One thing to note, I suppose, would be that if we keep it this way, you'll get a different result for re-submitting openQA jobs via this tool (new RDB job) vs. restarting an openQA job through the web UI (same RDB job as the original openQA job).

jskladanUnsubmitted
Not Done

We wouldn't really care about the job state - the Job state is kind of a leftover from the times when ResultsDB also stood in as execution state reporting tool. So having the jobs on RUNNING is not a problem. In the next ReslutsDB patch, the constraint on the job status will be removed, so one less problem to think about (it could all be in some random state then).

On the job-id front - I'd rather create new job - we'll be consuming the results anyway, the (number of) jobs is irrelevant - the most recent results for the specified item(s) are returned by resultsdb anyway. The job acts more as a group - in the next resultsdb version (sometimes in the future) I'll make the semantics clearer, and might even make the job/group not required.

adamwillUnsubmitted
Not Done

compose.split('-')[1] is not entirely safe, as we have some composes now where the distribution name is 'Fedora-Atomic' - e.g. 'Fedora-Atomic-24-20160803.0'. I had this problem in fedfind too, which now uses compose.split('-')[-2] instead.

316​ job = rdb_instance.create_job(ref_url=ref_url, name=compose)
317​ rdb_job_id = job["id"]
318​ except ResultsDBapiException as e:
319​ logger.error(e)
320​
295 # schedule per-image jobs, keeping track of the highest score 321​ # schedule per-image jobs, keeping track of the highest score
296 # per arch along the way 322​ # per arch along the way
297 for (flavor, arch, score, param_urls) in images: 323​ for (flavor, arch, score, param_urls, subvariant, imagetype) in images:
298 jobs.extend(run_openqa_jobs(param_urls, flavor, arch, build=compose, force=force, 324​ jobs.extend(run_openqa_jobs(param_urls, flavor, arch, subvariant, imagetype, compose, force=force,
299 extraparams=extraparams)) 325​ extraparams=extraparams, resultsdb_job_id=rdb_job_id))
300 if score > univs.get(arch, ['', '', 0])[2]: 326​ if score > univs.get(arch, [None, 0])[1]:
301 univs[arch] = (param_urls, flavor, score) 327​ univs[arch] = (param_urls, score, subvariant, imagetype)
302 328​
303 # now schedule universal jobs 329​ # now schedule universal jobs
304 if univs: 330​ if univs:
305 for (arch, (param_urls, _, _)) in univs.items(): 331​ for (arch, (param_urls, _, subvariant, imagetype)) in univs.items():
306 # We are assuming that ISO_URL is present in param_urls. This could create problem when 332​ # We are assuming that ISO_URL is present in param_urls. This could create problem when
307 # unversal tests are run on product that doesn't have ISO. OTOH, only product without ISO 333​ # unversal tests are run on product that doesn't have ISO. OTOH, only product without ISO
308 # is ARM and there would be whole lot of other problems if universal tests are run on ARM. 334​ # is ARM and there would be whole lot of other problems if universal tests are run on ARM.
309 logger.info("running universal tests for %s with %s", arch, param_urls['ISO_URL']) 335​ logger.info("running universal tests for %s with %s", arch, param_urls['ISO_URL'])
310 jobs.extend(run_openqa_jobs(param_urls, 'universal', arch, build=compose, force=force, 336​ jobs.extend(run_openqa_jobs(param_urls, 'universal', arch, subvariant, imagetype, compose,
311 extraparams=extraparams)) 337​ force=force, extraparams=extraparams, resultsdb_job_id=rdb_job_id))
312 338​
313 return (compose, jobs) 339​ return (compose, jobs)
View Options

scheduler/schedule.conf.sample

1# fedora-openqa-schedule (both CLI and module) will read configuration from 1​# fedora-openqa-schedule (both CLI and module) will read configuration from
2# /etc/fedora-qa/schedule.conf or ~/.config/fedora-qa/schedule.conf (with the 2​# /etc/fedora-qa/schedule.conf or ~/.config/fedora-qa/schedule.conf (with the
3# latter taking priority if both exist). Most of these settings may be 3​# latter taking priority if both exist). Most of these settings may be
4# overridden with CLI arguments, when using the CLI. Values in this sample 4​# overridden with CLI arguments, when using the CLI. Values in this sample
5# config indicate the defaults. 5​# config indicate the defaults.
6 6​
7[cli] 7​[cli]
8# File to log to (if not set, will log to stdout) 8​# File to log to (if not set, will log to stdout)
9log-file: 9​log-file:
10 10​
11# Logging level (debug, info, warning, error, critical) 11​# Logging level (debug, info, warning, error, critical)
12log-level: info 12​log-level: info
13 13​
14[report] 14​[report]
15# Whether to submit results to the Wiki (if false, just log them) 15​# Whether to submit results to the Wiki from fedmsg consumer (if false, just log them)
16submit: true 16​submit_wiki: true
17​# Whether to submit results to the ResultsDB from fedmsg consumer
18​submit_resultsdb: true
adamwillUnsubmitted
Not Done

I think this is some kinda rebase mistake? Don't think this exists any more.

garretrazielAuthorUnsubmitted
Not Done

I put it there intentionally because I saw that we are setting this value in config.py. But yeah, I forgot that we aren't using this value anymore, so correct change is to remove CONFIG.set('report', 'jobs-wait', '360') from config.py.

19​# URL of openQA instance for creating links in ResultsDB
20​openqa_url: http://localhost
adamwillUnsubmitted
Not Done

so this - and wiki_url - seems to only be used for producing links that go in the test results; it's not actually the URL we connect to in order to retrieve the test results, the openQA-python-client configuration is used there (as we initiate OpenQA_Client with no args). I can see a use for this - in case we want our OpenQA_Client to connect to http://localhost but we want to use https://openqa.fedoraproject.org in the links that are produced in the results data, for instance. But it needs to be more clearly explained at least, and generally, the mismatches in how all these different URL settings are actually used is a bit confusing I think.

garretrazielAuthorUnsubmitted
Not Done

I can change comment (or variable name) to make this more clear. OpenQA_Client is taking this URL from its configuration, but I didn't want to read configuration of some foreign library (and besides, as you said, there might be use case for it - connecting to localhost vs. creating links that are visitable from outside).

21​# URL of ResultsDB instance
22​resultsdb_url: http://localhost/resultsdb_api/api/v1.0/
23​# URL of Fedora wiki for creating links in ResultsDB
24​wiki_url: https://fedoraproject.org/wiki
adamwillUnsubmitted
Not Done

AFAICT this is not used at all.

garretrazielAuthorUnsubmitted
Not Done

You are right, I put it there before I realized that we want to produce links to prod wiki every time.

adamwillUnsubmitted
Not Done

So now you made this be used, but do we actually need it? What's the use case for changing it, ever?

garretrazielAuthorUnsubmitted
Not Done

No, I've removed (unused) wiki_stg_url. wiki_url was actually used before. I didn't want to hardcode wiki URL and figured out that you "might" want testcase detail URL to lead somewhere else (but TBH I don't believe someone is going to change this ever). Do you think I should hardcode this?

View Options

scheduler/setup.py

Show All 24 Lines12setup(
25 }, 25​ },
26 author = "Fedora QA devel team", 26​ author = "Fedora QA devel team",
27 author_email = "qa-devel@lists.fedoraproject.org", 27​ author_email = "qa-devel@lists.fedoraproject.org",
28 description = "Fedora openQA scheduler", 28​ description = "Fedora openQA scheduler",
29 license = "GPLv3+", 29​ license = "GPLv3+",
30 keywords = "fedora openqa test qa", 30​ keywords = "fedora openqa test qa",
31 url = "https://bitbucket.org/rajcze/openqa_fedora_tools", 31​ url = "https://bitbucket.org/rajcze/openqa_fedora_tools",
32 packages = ["fedora_openqa_schedule"], 32​ packages = ["fedora_openqa_schedule"],
33 install_requires = ['fedfind>=1.5', 'fedmsg', 'openqa-client>=1.1', 'setuptools', 'six'], 33​ install_requires = ['fedfind>=1.5', 'fedmsg', 'openqa-client>=1.1', 'setuptools', 'six', 'resultsdb_api'],
34 long_description=read('README.md'), 34​ long_description=read('README.md'),
35 classifiers=[ 35​ classifiers=[
36 "Development Status :: 3 - Alpha", 36​ "Development Status :: 3 - Alpha",
37 "Topic :: Utilities", 37​ "Topic :: Utilities",
38 "License :: OSI Approved :: GNU General Public License v3 or later " 38​ "License :: OSI Approved :: GNU General Public License v3 or later "
39 "(GPLv3+)", 39​ "(GPLv3+)",
40 ], 40​ ],
41) 41​)