Differential D802

allow passing extra params to ISO post, updates URL to CLI
ClosedPublic
Actions

Authored by adamwill on Apr 1 2016, 12:31 AM.

Details

Reviewers
garretraziel
jskladan
Commits
rOPENQAa8c83b6929bb: allow passing extra params to ISO post, updates URL to CLI
Summary

this commit adds a fairly generic mechanism for passing extra
params into the ISO post command when scheduling jobs, and an
arg to the compose sub-command which takes advantage of this
to let you specify an updates image URL which will cause the
new GRUBADD var to be specified when POSTing the jobs, thus
loading that updates image for all the tests. This should give
us a handy way to run a full openQA test run on any updates
image we like.

When extra parameters are specified, the BUILD value is changed
so this test run can be identified as not being a 'pure' one.
get_passed_testcases is tweaked to make sure we never report
results to the wiki for such tests.

This goes with D801 for the tests.

Test Plan

With D801 applied to the tests, try something like
fedora-openqa-schedule.py compose -f --updates http://dlehman.fedorapeople.org/updates/1322943.0.img https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20160331.n.0/compose/
and see what happens; it should add a GRUBADD var to all
tests which should cause that updates image to be loaded. The
updates image loading tests will fail, but all other tests
should now get further than they would without the updates
image (which fixes an early crasher bug). The build should be
the compose ID with -EXTRA appended, and if you try reporting
results to the wiki it should find no passed testcases, with
a debug log message. I will need to adjust the check-compose
consumer to ignore such builds (though we should never run
tests like this on the production openQA, I don't think, at
most we might do it on staging) and I'll also adjust the wiki
report consumer not to even bother trying to report results
for these composes, just to double the protection.

Diff Detail

Repository
rOPENQA fedora_openqa
Branch
test-updates
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 524
Build 524: arc lint + arc unit
adamwill retitled this revision from to allow passing extra params to ISO post, updates URL to CLI.Apr 1 2016, 12:31 AM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jskladan, garretraziel.
garretraziel accepted this revision.Apr 5 2016, 11:58 AM

Looks OK.

This revision is now accepted and ready to land.Apr 5 2016, 11:58 AM
adamwill closed this revision.Apr 8 2016, 8:24 PM

I landed this manually as dealing with diffs-on-diffs is a bit of a pain with the way the commit messages get rewritten and IDs changed.

Revision Contents

Diff 2046

View Options

scheduler/fedora_openqa_schedule/cli.py

Show All 36 Lines
37 37​
38### SUB-COMMAND METHODS 38​### SUB-COMMAND METHODS
39 39​
40def command_compose(args, wiki_url): 40​def command_compose(args, wiki_url):
41 """run OpenQA on a specified compose, optionally reporting 41​ """run OpenQA on a specified compose, optionally reporting
42 results if a matching wikitcms ValidationEvent is found by 42​ results if a matching wikitcms ValidationEvent is found by
43 relval/wikitcms. 43​ relval/wikitcms.
44 """ 44​ """
45​ extraparams = None
46​ if args.updates:
47​ extraparams = {'GRUBADD': "inst.updates={0}".format(args.updates)}
45 try: 48​ try:
46 (compose, jobs) = schedule.jobs_from_compose(args.location, force=args.force) 49​ (compose, jobs) = schedule.jobs_from_compose(
50​ args.location, force=args.force, extraparams=extraparams)
47 except schedule.TriggerException as err: 51​ except schedule.TriggerException as err:
48 logger.warning("No jobs run! %s", err) 52​ logger.warning("No jobs run! %s", err)
49 sys.exit(1) 53​ sys.exit(1)
50 54​
51 if jobs: 55​ if jobs:
52 report.wait_and_report(wiki_url, build=compose, do_report=args.submit) 56​ report.wait_and_report(wiki_url, build=compose, do_report=args.submit)
53 else: 57​ else:
54 msg = "No jobs run!" 58​ msg = "No jobs run!"
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Line(s)118 parser_compose = subparsers.add_parser(
116 "release validation test event can be found and --submit-results is passed, results " 120​ "release validation test event can be found and --submit-results is passed, results "
117 "will be reported.") 121​ "will be reported.")
118 parser_compose.add_argument( 122​ parser_compose.add_argument(
119 'location', help="The top-level URL of the compose", 123​ 'location', help="The top-level URL of the compose",
120 metavar="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-24-20160113.n.1/compose") 124​ metavar="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-24-20160113.n.1/compose")
121 parser_compose.add_argument( 125​ parser_compose.add_argument(
122 '--force', '-f', help="For each ISO/flavor combination, schedule jobs even if there " 126​ '--force', '-f', help="For each ISO/flavor combination, schedule jobs even if there "
123 "are existing, non-cancelled jobs for that combination", action='store_true') 127​ "are existing, non-cancelled jobs for that combination", action='store_true')
128​ parser_compose.add_argument(
129​ '--updates', '-u', help="URL to an updates image to load for all tests. The tests that "
130​ "test updates image loading will fail when you use this")
124 parser_compose.set_defaults(func=command_compose) 131​ parser_compose.set_defaults(func=command_compose)
125 132​
126 parser_report = subparsers.add_parser( 133​ parser_report = subparsers.add_parser(
127 'report', description="Map openQA job results to Wikitcms test results and optionally " 134​ 'report', description="Map openQA job results to Wikitcms test results and optionally "
128 "submit them to the wiki.") 135​ "submit them to the wiki.")
129 parser_report.add_argument('jobs', nargs='+', help="openQA job IDs or builds (e.g. " 136​ parser_report.add_argument('jobs', nargs='+', help="openQA job IDs or builds (e.g. "
130 "'Fedora-24-20160113.n.1'). For each build included, the latest jobs will be reported.") 137​ "'Fedora-24-20160113.n.1'). For each build included, the latest jobs will be reported.")
131 parser_report.set_defaults(func=command_report) 138​ parser_report.set_defaults(func=command_report)
Show All 33 Lines
View Options

scheduler/fedora_openqa_schedule/report.py

Show First 20 Lines • Show All 88 Lines • ▼ Show 20 Line(s)86def get_passed_testcases(jobs):
89 wikitcms ResTuples derived from any passed tests. 89​ wikitcms ResTuples derived from any passed tests.
90 """ 90​ """
91 passed_testcases = set() 91​ passed_testcases = set()
92 for job in jobs: 92​ for job in jobs:
93 if job['result'] == 'passed': 93​ if job['result'] == 'passed':
94 # it's wikitcms' job to take a compose ID and figure out 94​ # it's wikitcms' job to take a compose ID and figure out
95 # what the validation event for it is. 95​ # what the validation event for it is.
96 composeid = job['settings']['BUILD'] 96​ composeid = job['settings']['BUILD']
97​ if composeid.endswith('EXTRA'):
98​ # this is a 'dirty' test run with extra parameters
99​ # (usually a test with an updates.img), we never want
100​ # to report results for these
101​ logger.debug("Job was run with extra params! Will not report")
102​ continue
97 testsuite = job['settings']['TEST'] 103​ testsuite = job['settings']['TEST']
98 # There usually ought to be an entry in TESTSUITES for all 104​ # There usually ought to be an entry in TESTSUITES for all
99 # tests, but just in case someone messed up, let's be safe 105​ # tests, but just in case someone messed up, let's be safe
100 if testsuite not in conf_test_suites.TESTSUITES: 106​ if testsuite not in conf_test_suites.TESTSUITES:
101 logger.warning("No TESTSUITES entry found for test {0}!".format(testsuite)) 107​ logger.warning("No TESTSUITES entry found for test {0}!".format(testsuite))
102 continue 108​ continue
103 for testcase in conf_test_suites.TESTSUITES[testsuite]: 109​ for testcase in conf_test_suites.TESTSUITES[testsuite]:
104 # each 'testsuite' is a list using testcase names to indicate which Wikitcms tests 110​ # each 'testsuite' is a list using testcase names to indicate which Wikitcms tests
▲ Show 20 Lines • Show All 79 Lines • Show Last 20 Lines
View Options

scheduler/fedora_openqa_schedule/schedule.py

Show First 20 Lines • Show All 118 Lines • ▼ Show 20 Line(s)117 flavor = [item.replace('-', '_')
119 foundimg['format']] 119​ foundimg['format']]
120 flavor = '-'.join(flavor) 120​ flavor = '-'.join(flavor)
121 url = "{0}/{1}".format(location, foundimg['path']) 121​ url = "{0}/{1}".format(location, foundimg['path'])
122 logger.debug("Found image %s for arch %s at %s", flavor, arch, url) 122​ logger.debug("Found image %s for arch %s at %s", flavor, arch, url)
123 images.add((url, flavor, arch, score)) 123​ images.add((url, flavor, arch, score))
124 return images 124​ return images
125 125​
126 126​
127def run_openqa_jobs(url, flavor, arch, build, force=False): 127​def run_openqa_jobs(url, flavor, arch, build, force=False, extraparams=None):
128 """# run OpenQA 'isos' job on ISO at 'url', with given arch 128​ """# run OpenQA 'isos' job on ISO at 'url', with given arch
129 and a build identifier. **NOTE**: 'build' is passed to openQA as 129​ and a build identifier. **NOTE**: 'build' is passed to openQA as
130 BUILD and is later retrieved and parsed by report.py for wiki 130​ BUILD and is later retrieved and parsed by report.py for wiki
131 report generation. Returns list of job IDs. If force is False, 131​ report generation. Returns list of job IDs. If force is False,
132 jobs will only be scheduled if there are no existing, 132​ jobs will only be scheduled if there are no existing,
133 non-cancelled jobs for the same ISO and flavor. 133​ non-cancelled jobs for the same ISO and flavor. If extraparams
134​ is specified, it must be a dict or something else that can be
135​ combined with a dict using `update`; it adds additional parameters
136​ (usually openQA variables) to the POST request. When extraparams
137​ is used, the BUILD value has '-EXTRA' appended to signify that
138​ this should not be considered a clean test run for the build.
134 """ 139​ """
135 logger.info("sending jobs to openQA") 140​ logger.info("sending jobs to openQA")
136 # find current and previous releases; these are used to determine 141​ # find current and previous releases; these are used to determine
137 # the hard disk image file names for the upgrade tests 142​ # the hard disk image file names for the upgrade tests
138 try: 143​ try:
139 currrel = str(fedfind.helpers.get_current_release()) 144​ currrel = str(fedfind.helpers.get_current_release())
140 prevrel = str(int(currrel) - 1) 145​ prevrel = str(int(currrel) - 1)
141 except ValueError: 146​ except ValueError:
142 # we don't really want to bail entirely if fedfind failed for 147​ # we don't really want to bail entirely if fedfind failed for
143 # some reason, let's just run the other tests and set a value 148​ # some reason, let's just run the other tests and set a value
144 # that shows what went wrong for the upgrade tests 149​ # that shows what went wrong for the upgrade tests
145 currrel = prevrel = "FEDFINDERROR" 150​ currrel = prevrel = "FEDFINDERROR"
146 151​
147 # starts OpenQA jobs 152​ # starts OpenQA jobs
148 params = { 153​ params = {
149 'ISO_URL': url, 154​ 'ISO_URL': url,
150 'DISTRI': 'fedora', 155​ 'DISTRI': 'fedora',
151 'VERSION': build.split('-')[1], 156​ 'VERSION': build.split('-')[1],
152 'FLAVOR': flavor, 157​ 'FLAVOR': flavor,
153 'ARCH': arch, 158​ 'ARCH': arch,
154 'BUILD': build, 159​ 'BUILD': build,
155 'CURRREL': currrel, 160​ 'CURRREL': currrel,
156 'PREVREL': prevrel, 161​ 'PREVREL': prevrel,
157 } 162​ }
163​ if extraparams:
164​ params.update(extraparams)
165​ # mung the BUILD so this is not considered a 'real' test run
166​ params['BUILD'] = "{0}-EXTRA".format(params['BUILD'])
158 client = OpenQA_Client() 167​ client = OpenQA_Client()
159 if not force: 168​ if not force:
160 isoname = url.split('/')[-1] 169​ isoname = url.split('/')[-1]
161 # Check if we have any existing non-cancelled jobs for this 170​ # Check if we have any existing non-cancelled jobs for this
162 # ISO and flavor (checking flavor is important otherwise we'd 171​ # ISO and flavor (checking flavor is important otherwise we'd
163 # bail on doing the per-ISO jobs for the ISO we use for the 172​ # bail on doing the per-ISO jobs for the ISO we use for the
164 # 'universal' tests). 173​ # 'universal' tests).
165 jobs = client.openqa_request('GET', 'jobs', params={'iso': isoname})['jobs'] 174​ jobs = client.openqa_request('GET', 'jobs', params={'iso': isoname})['jobs']
166 jobs = [job for job in jobs if job['settings']['FLAVOR'] == flavor] 175​ jobs = [job for job in jobs if job['settings']['FLAVOR'] == flavor]
167 jobs = [job for job in jobs if 176​ jobs = [job for job in jobs if
168 job.get('state') != 'cancelled' and job.get('result') != 'user_cancelled'] 177​ job.get('state') != 'cancelled' and job.get('result') != 'user_cancelled']
169 if jobs: 178​ if jobs:
170 logger.info("run_openqa_jobs: Existing jobs found for ISO %s flavor %s, and force " 179​ logger.info("run_openqa_jobs: Existing jobs found for ISO %s flavor %s, and force "
171 "not set! No jobs scheduled.", isoname, flavor) 180​ "not set! No jobs scheduled.", isoname, flavor)
172 logger.debug("Existing jobs found: %s", ' '.join(str(job['id']) for job in jobs)) 181​ logger.debug("Existing jobs found: %s", ' '.join(str(job['id']) for job in jobs))
173 return [] 182​ return []
174 183​
175 output = client.openqa_request('POST', 'isos', params) 184​ output = client.openqa_request('POST', 'isos', params)
176 logger.debug("run_openqa_jobs: executed") 185​ logger.debug("run_openqa_jobs: executed")
177 logger.debug("run_openqa_jobs: planned jobs: %s", output["ids"]) 186​ logger.debug("run_openqa_jobs: planned jobs: %s", output["ids"])
178 187​
179 return output["ids"] 188​ return output["ids"]
180 189​
181 190​
182def jobs_from_compose(location, wanted=WANTED, force=False): 191​def jobs_from_compose(location, wanted=WANTED, force=False, extraparams=None):
183 """Schedule jobs against a specific compose. Returns a 2-tuple 192​ """Schedule jobs against a specific compose. Returns a 2-tuple
184 of the compose ID and the list of job IDs. 193​ of the compose ID and the list of job IDs.
185 194​
186 location is the top level of the compose. Note this value is 195​ location is the top level of the compose. Note this value is
187 provided by fedmsg 'pungi.compose.status.change' messages as 196​ provided by fedmsg 'pungi.compose.status.change' messages as
188 'location'. 197​ 'location'.
189 198​
190 wanted is a dict defining which images from the compose we should 199​ wanted is a dict defining which images from the compose we should
191 schedule tests for. It is passed direct to _get_images(). The 200​ schedule tests for. It is passed direct to _get_images(). The
192 default set of tested images is specified in config.py. It can be 201​ default set of tested images is specified in config.py. It can be
193 overridden by a system-wide or per-user file as well as with this 202​ overridden by a system-wide or per-user file as well as with this
194 argument. The layout is, intentionally, a subset of the pungi 203​ argument. The layout is, intentionally, a subset of the pungi
195 images.json metadata file (except for the 'score' item). Check 204​ images.json metadata file (except for the 'score' item). Check
196 config.py to see the layout and for more information. 205​ config.py to see the layout and for more information.
197 206​
198 If force is False, for each ISO, jobs will only be scheduled if 207​ If force is False, for each ISO, jobs will only be scheduled if
199 there are no existing, non-cancelled jobs for the same ISO and 208​ there are no existing, non-cancelled jobs for the same ISO and
200 flavor. 209​ flavor.
210​
211​ extraparams is passed through to `run_openqa_jobs()`. It can be
212​ a dict (or anything else that can be passed to `dict.update()`)
213​ containing arbitrary extra parameters to be included in the ISO
214​ post request (usually this will be to specify extra openQA vars).
201 """ 215​ """
202 location = location.strip('/') 216​ location = location.strip('/')
203 # trigger ugly special-casing for non-Pungi4-ified Two Week Atomic 217​ # trigger ugly special-casing for non-Pungi4-ified Two Week Atomic
204 # nightlies 218​ # nightlies
205 if 'alt/atomic/testing' in location: 219​ if 'alt/atomic/testing' in location:
206 compose = _get_atomic_id(location) 220​ compose = _get_atomic_id(location)
207 logger.debug("Finding images for Atomic compose %s in location %s", compose, location) 221​ logger.debug("Finding images for Atomic compose %s in location %s", compose, location)
208 images = _get_atomic_installer(location, compose) 222​ images = _get_atomic_installer(location, compose)
209 else: 223​ else:
210 compose = _get_compose_id(location) 224​ compose = _get_compose_id(location)
211 logger.debug("Finding images for compose %s in location %s", compose, location) 225​ logger.debug("Finding images for compose %s in location %s", compose, location)
212 images = _get_images(location, wanted=wanted) 226​ images = _get_images(location, wanted=wanted)
213 if len(images) == 0: 227​ if len(images) == 0:
214 raise TriggerException("Compose found, but no available images") 228​ raise TriggerException("Compose found, but no available images")
215 jobs = [] 229​ jobs = []
216 univs = {} 230​ univs = {}
217 231​
218 # schedule per-image jobs, keeping track of the highest score 232​ # schedule per-image jobs, keeping track of the highest score
219 # per arch along the way 233​ # per arch along the way
220 for (url, flavor, arch, score) in images: 234​ for (url, flavor, arch, score) in images:
221 jobs.extend(run_openqa_jobs(url, flavor, arch, build=compose, force=force)) 235​ jobs.extend(run_openqa_jobs(url, flavor, arch, build=compose, force=force,
236​ extraparams=extraparams))
222 if score > univs.get(arch, ['', 0])[1]: 237​ if score > univs.get(arch, ['', 0])[1]:
223 univs[arch] = (url, score) 238​ univs[arch] = (url, score)
224 239​
225 # now schedule universal jobs 240​ # now schedule universal jobs
226 if univs: 241​ if univs:
227 for (arch, (url, _)) in univs.items(): 242​ for (arch, (url, _)) in univs.items():
228 logger.info("running universal tests for %s with %s", arch, url) 243​ logger.info("running universal tests for %s with %s", arch, url)
229 jobs.extend(run_openqa_jobs(url, 'universal', arch, build=compose, force=force)) 244​ jobs.extend(run_openqa_jobs(url, 'universal', arch, build=compose, force=force,
245​ extraparams=extraparams))
230 246​
231 return (compose, jobs) 247​ return (compose, jobs)