Differential D1020

put compose string into vmlinuz and initrd names
ClosedPublic
Actions

Authored by garretraziel on Oct 6 2016, 11:24 AM.

Details

Reviewers
adamwill
jskladan
Commits
rOPENQAc90db2ea242d: make direct kernel boot more general, put compose string into vmlinuz and…
Summary

This should be merged only after https://github.com/os-autoinst/openQA/pull/920
gets into production.

Test Plan

Schedule ARM tests, observe that they downloaded into {composeid}.vmlinuz and
{composeid}.initrd.img.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
garretraziel retitled this revision from to put compose string into vmlinuz and initrd names.Oct 6 2016, 11:24 AM
garretraziel updated this object.
garretraziel edited the test plan for this revision. (Show Details)
garretraziel added reviewers: adamwill, jskladan.
adamwill added a comment.Oct 6 2016, 7:14 PM

Um - what about the existing code to do this, that's there but commented out?

# add KERNEL and INITRD arguments when needed
# TODO: this will not work until https://github.com/os-autoinst/openQA/pull/673 gets merged
# if 'KERNEL_URL' in params:
#     params['KERNEL'] = "vmlinuz.{0}".format(build)
# if 'INITRD_URL' in params:
#     params['INITRD'] = "initrd.img.{0}".format(build)

it's around lines 214-219.

adamwill added a comment.Oct 6 2016, 7:17 PM

oh, and yeah, I noticed that the upstream fix had been merged, I'll get it built downstream and running on the official hosts ASAP.

adamwill added a comment.Oct 6 2016, 11:41 PM

http://koji.fedoraproject.org/koji/taskinfo?taskID=15977110 is a scratch openQA build with the fix included, it's live on staging ATM, I'll send it to prod in a couple of days if it seems OK on staging.

garretraziel added a comment.Oct 11 2016, 11:47 AM

I've tested it and scratchbuild works for me. I'll wait until it'll be in production with merge.

garretraziel added a comment.Oct 11 2016, 12:11 PM
In D1020#19210, @adamwill wrote:

Um - what about the existing code to do this, that's there but commented out?

# add KERNEL and INITRD arguments when needed
# TODO: this will not work until https://github.com/os-autoinst/openQA/pull/673 gets merged
# if 'KERNEL_URL' in params:
#     params['KERNEL'] = "vmlinuz.{0}".format(build)
# if 'INITRD_URL' in params:
#     params['INITRD'] = "initrd.img.{0}".format(build)

it's around lines 214-219.

Hmm, you are right. I can either uncomment this (and remove changes in this diff), or remove this comment. What do you think is better to do?

Change in this diff is better in that it has all special-case ARM-handling in one place. OTOH, existing commented code would work even when we would schedule direct kernel boot test for non-ARM system (I think that in that case, we should also put architecture into the filename). It looks like it boils down into whether we are counting on using DKB with other architectures.

garretraziel updated this revision to Diff 2616.Oct 11 2016, 12:12 PM
  • remove comment
garretraziel updated this revision to Diff 2618.Oct 13 2016, 10:47 AM
  • make direct kernel boot more general
garretraziel added a comment.Oct 13 2016, 10:53 AM

So current code makes direct kernel boot more general, we are now able to use direct kernel boot even with other architectures. It also means that we can remove almost all ARM-specific hardcoding.

adamwill accepted this revision.Oct 14 2016, 1:50 AM

This looks pretty good to me aside from the in-line notes, didn't try it yet but on an eyeball I'm OK with it. I'll ack it for now, I'm also doing an official build of openqa package with the necessary upstream fix and will roll it out to prod (staging has been working fine). thanks.

scheduler/fedora_openqa_schedule/schedule.py
78

this feels a bit messy to me now, we might be able to redesign it to be a bit nicer somehow? but I don't think it's worth blocking on. I might look into it if I get the motivation.

211

is losing this going to be a problem?

This revision is now accepted and ready to land.Oct 14 2016, 1:50 AM
garretraziel added a comment.Oct 14 2016, 9:49 AM

I'll merge it once openQA with bugfix will be in stable.

scheduler/fedora_openqa_schedule/schedule.py
211

I have checked and FAMILY isn't really used anywhere. I was planning to use it for START_AFTER with ARM (have START_AFTER=default_install_%FAMILY%, move current default_install to default_install_x86 and have default_install_arm that does only initial setup), but then I have simply duplicated all jobs we want to run on ARM.

adamwill added a comment.Oct 14 2016, 3:17 PM

Cool. You can merge now if you like, the updated openQA is already live on prod.

Closed by commit rOPENQAc90db2ea242d: make direct kernel boot more general, put compose string into vmlinuz and… (authored by garretraziel). · Explain WhyOct 17 2016, 10:47 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2630

View Options

scheduler/fedora_openqa_schedule/config.py

Show First 20 Lines • Show All 65 Lines • ▼ Show 20 Line(s)
66# but instead influence the behaviour of the scheduler. 66​# but instead influence the behaviour of the scheduler.
67 67​
68# The "score" item is used to decide which images to run the universal 68​# The "score" item is used to decide which images to run the universal
69# tests with. For each arch that has universal tests, the found image 69​# tests with. For each arch that has universal tests, the found image
70# with the highest score is used. If no image with a score > 0 is 70​# with the highest score is used. If no image with a score > 0 is
71# found, the universal tests are skipped. If score is not specified 71​# found, the universal tests are skipped. If score is not specified
72# the image is considered to have a score of 0. 72​# the image is considered to have a score of 0.
73 73​
74​# If "dkboot" is set to True, scheduler also downloads kernel and initrd
75​# files for specified architecture and then schedules job to run in Direct
76​# Kernel Boot mode. Default value (used when dkboot item is not specified)
77​# is False.
78​
74WANTED = { 79​WANTED = {
75 "Server": { 80​ "Server": {
76 "x86_64": [ 81​ "x86_64": [
77 { 82​ {
78 "match": { 83​ "match": {
79 "type": "boot", 84​ "type": "boot",
80 "format": "iso", 85​ "format": "iso",
81 }, 86​ },
▲ Show 20 Lines • Show All 95 Lines • ▼ Show 20 Line(s)164 "Spins": {
177 ], 182​ ],
178 "armhfp": [ 183​ "armhfp": [
179 { 184​ {
180 "match": { 185​ "match": {
181 "subvariant": "Minimal", 186​ "subvariant": "Minimal",
182 "type": "raw-xz", 187​ "type": "raw-xz",
183 "format": "raw.xz", 188​ "format": "raw.xz",
184 }, 189​ },
190​ "dkboot": True,
185 }, 191​ },
186 ], 192​ ],
187 }, 193​ },
188 "Atomic": { 194​ "Atomic": {
189 "x86_64": [ 195​ "x86_64": [
190 { 196​ {
191 "match": { 197​ "match": {
192 "type": "dvd-ostree", 198​ "type": "dvd-ostree",
Show All 15 Lines
View Options

scheduler/fedora_openqa_schedule/schedule.py

Show First 20 Lines • Show All 68 Lines • ▼ Show 20 Line(s)
69def _get_dkboot_urls(location, arch='armhfp'): 69​def _get_dkboot_urls(location, arch='armhfp'):
70 """Given a compose location (and arch if provided), return URLs for kernel and initrd 70​ """Given a compose location (and arch if provided), return URLs for kernel and initrd
71 files. 71​ files.
72 """ 72​ """
73 pxeboot_url = '{0}/Everything/{1}/os/images/pxeboot/{2}' 73​ pxeboot_url = '{0}/Everything/{1}/os/images/pxeboot/{2}'
74 return pxeboot_url.format(location, arch, 'vmlinuz'), pxeboot_url.format(location, arch, 'initrd.img') 74​ return pxeboot_url.format(location, arch, 'vmlinuz'), pxeboot_url.format(location, arch, 'initrd.img')
75 75​
76 76​
77def _get_images(location, wanted=WANTED): 77​def _get_images(location, compose, wanted=WANTED):
78 """Given a Pungi compose top-level location, this returns a list 78​ """Given a Pungi compose top-level location and string containing compose ID, this returns a list
adamwillUnsubmitted
Not Done

this feels a bit messy to me now, we might be able to redesign it to be a bit nicer somehow? but I don't think it's worth blocking on. I might look into it if I get the motivation.

79 of (flavor, arch, score, {param: url}, subvariant, imagetype) tuples for images to be tested. 79​ of (flavor, arch, score, {param: url}, subvariant, imagetype) tuples for images to be tested.
80 """ 80​ """
81 try: 81​ try:
82 resp = urlopen('{0}/metadata/images.json'.format(location)) 82​ resp = urlopen('{0}/metadata/images.json'.format(location))
83 metadata = json.load(resp) 83​ metadata = json.load(resp)
84 except (ValueError, URLError, HTTPError): 84​ except (ValueError, URLError, HTTPError):
85 raise TriggerException("Compose not found, or failed!") 85​ raise TriggerException("Compose not found, or failed!")
86 images = [] 86​ images = []
Show All 23 Lines98 for foundimg in foundimgs:
110 imagetype = foundimg['type'] 110​ imagetype = foundimg['type']
111 url = "{0}/{1}".format(location, foundimg['path']) 111​ url = "{0}/{1}".format(location, foundimg['path'])
112 logger.debug("Found image %s for arch %s at %s", flavor, arch, url) 112​ logger.debug("Found image %s for arch %s at %s", flavor, arch, url)
113 113​
114 # some tests need more than one file, so let's collect them now 114​ # some tests need more than one file, so let's collect them now
115 param_urls = { 115​ param_urls = {
116 FORMAT_TO_PARAM[foundimg['format']]: url 116​ FORMAT_TO_PARAM[foundimg['format']]: url
117 } 117​ }
118 # TODO: WARNING! - workaround for ARM disk images that need kernel for direct kernel boot 118​ # if direct kernel boot is specified, we need to download kernel and initrd
119 if arch == "armhfp" and foundimg['format'] == "raw.xz": 119​ if wantimg.get('dkboot', False):
120 kernel_url, initrd_url = _get_dkboot_urls(location, arch) 120​ kernel_url, initrd_url = _get_dkboot_urls(location, arch)
121 param_urls["KERNEL_URL"] = kernel_url 121​ param_urls["KERNEL_URL"] = kernel_url
122​ param_urls["KERNEL"] = "{0}.{1}.vmlinuz".format(compose, arch)
122 param_urls["INITRD_URL"] = initrd_url 123​ param_urls["INITRD_URL"] = initrd_url
124​ param_urls["INITRD"] = "{0}.{1}.initrd.img".format(compose, arch)
123 125​
124 images.append((flavor, arch, score, param_urls, subvariant, imagetype)) 126​ images.append((flavor, arch, score, param_urls, subvariant, imagetype))
125 return images 127​ return images
126 128​
127 129​
128def _find_duplicate_jobs(client, param_urls, flavor): 130​def _find_duplicate_jobs(client, param_urls, flavor):
129 """Check if we have any existing non-cancelled jobs for this 131​ """Check if we have any existing non-cancelled jobs for this
130 ISO/HDD and flavor (checking flavor is important otherwise we'd 132​ ISO/HDD and flavor (checking flavor is important otherwise we'd
▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Line(s)179 except ValueError:
180 # that shows what went wrong for the upgrade tests 182​ # that shows what went wrong for the upgrade tests
181 currrel = prevrel = "FEDFINDERROR" 183​ currrel = prevrel = "FEDFINDERROR"
182 184​
183 # starts OpenQA jobs 185​ # starts OpenQA jobs
184 params = { 186​ params = {
185 'DISTRI': 'fedora', 187​ 'DISTRI': 'fedora',
186 'VERSION': build.split('-')[1], 188​ 'VERSION': build.split('-')[1],
187 'FLAVOR': flavor, 189​ 'FLAVOR': flavor,
188 'ARCH': arch, 190​ 'ARCH': arch if arch != 'armhfp' else 'arm', # openQA does something special when `ARCH = arm`
189 'BUILD': build, 191​ 'BUILD': build,
190 'LOCATION': location, 192​ 'LOCATION': location,
191 'CURRREL': currrel, 193​ 'CURRREL': currrel,
192 'PREVREL': prevrel, 194​ 'PREVREL': prevrel,
193 'SUBVARIANT': subvariant, 195​ 'SUBVARIANT': subvariant,
194 'IMAGETYPE': imagetype 196​ 'IMAGETYPE': imagetype
195 } 197​ }
196 if extraparams: 198​ if extraparams:
197 params.update(extraparams) 199​ params.update(extraparams)
198 # mung the BUILD so this is not considered a 'real' test run 200​ # mung the BUILD so this is not considered a 'real' test run
199 params['BUILD'] = "{0}-EXTRA".format(params['BUILD']) 201​ params['BUILD'] = "{0}-EXTRA".format(params['BUILD'])
200 if resultsdb_job_id: 202​ if resultsdb_job_id:
201 params['RESULTSDB_JOB_ID'] = resultsdb_job_id 203​ params['RESULTSDB_JOB_ID'] = resultsdb_job_id
202 204​
203 if arch == 'armhfp': ​
204 params.update({ ​
205 'ARCH': 'arm', ​
206 'FAMILY': 'arm', ​
207 # ARM kernel arguments since we are using direct kernel boot ​
208 'APPEND': 'rw root=LABEL=_/ rootwait console=ttyAMA0 console=tty0 consoleblank=0', ​
209 }) ​
210 else: ​
211 params['FAMILY'] = 'x86' ​
adamwillUnsubmitted
Not Done

is losing this going to be a problem?

garretrazielAuthorUnsubmitted
Not Done

I have checked and FAMILY isn't really used anywhere. I was planning to use it for START_AFTER with ARM (have START_AFTER=default_install_%FAMILY%, move current default_install to default_install_x86 and have default_install_arm that does only initial setup), but then I have simply duplicated all jobs we want to run on ARM.

212 params.update(param_urls) 205​ params.update(param_urls)
213 206​
214 # add KERNEL and INITRD arguments when needed ​
215 # TODO: this will not work until https://github.com/os-autoinst/openQA/pull/673 gets merged ​
216 # if 'KERNEL_URL' in params: ​
217 # params['KERNEL'] = "vmlinuz.{0}".format(build) ​
218 # if 'INITRD_URL' in params: ​
219 # params['INITRD'] = "initrd.img.{0}".format(build) ​
220 ​
221 client = OpenQA_Client() 207​ client = OpenQA_Client()
222 208​
223 if not force: 209​ if not force:
224 duplicates = _find_duplicate_jobs(client, param_urls, flavor) 210​ duplicates = _find_duplicate_jobs(client, param_urls, flavor)
225 if duplicates: 211​ if duplicates:
226 logger.debug("Existing jobs found: %s", ' '.join(str(dupe['id']) for dupe in duplicates)) 212​ logger.debug("Existing jobs found: %s", ' '.join(str(dupe['id']) for dupe in duplicates))
227 return [] 213​ return []
228 214​
Show All 30 Lines222def jobs_from_compose(location, wanted=WANTED, force=False, extraparams=None, create_resultsdb_job=None):
259 post request (usually this will be to specify extra openQA vars). 245​ post request (usually this will be to specify extra openQA vars).
260 246​
261 create_resultsdb_job controls whether to create job in ResultsDB for 247​ create_resultsdb_job controls whether to create job in ResultsDB for
262 this compose. 248​ this compose.
263 """ 249​ """
264 location = location.strip('/') 250​ location = location.strip('/')
265 compose = _get_compose_id(location) 251​ compose = _get_compose_id(location)
266 logger.debug("Finding images for compose %s in location %s", compose, location) 252​ logger.debug("Finding images for compose %s in location %s", compose, location)
267 images = _get_images(location, wanted=wanted) 253​ images = _get_images(location, compose, wanted=wanted)
268 if len(images) == 0: 254​ if len(images) == 0:
269 raise TriggerException("Compose found, but no available images") 255​ raise TriggerException("Compose found, but no available images")
270 jobs = [] 256​ jobs = []
271 univs = {} 257​ univs = {}
272 258​
273 # create jobs instance in resultsdb if necessary 259​ # create jobs instance in resultsdb if necessary
274 rdb_job_id = None 260​ rdb_job_id = None
275 261​
Show All 35 Lines