Differential D917

switch from virt-builder to virt-install (T813)
ClosedPublic
Actions

Authored by adamwill on Jul 1 2016, 12:33 AM.

Details

Reviewers
garretraziel
Commits
rOPENQA635bc639d6ad: switch from virt-builder to virt-install (T813)
rOPENQAb2422a65abe1: switch from virt-builder to virt-install (T813)
Summary

We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.

The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.

We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.

The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).

This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).

Test Plan

Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to switch from virt-builder to virt-install (T813).Jul 1 2016, 12:33 AM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added a reviewer: garretraziel.
adamwill updated this revision to Diff 2332.Jul 1 2016, 6:33 AM

keep the imsettings-qt workaround for now, still needed

garretraziel accepted this revision.Jul 4 2016, 10:37 AM

I had some problems during image building - fair amount of F24 installations failed during boot and dropped to dracut console (although it worked fine on second try). Also mirrorlist returns "bad" URL for initrd download fairly often:

ERROR    Couldn't acquire file https://download.fedoraproject.org/pub/fedora/linux/releases/24/Everything/x86_64/os/images/pxeboot/initrd.img: 403 Client Error: Forbidden for url: http://mirror.onet.pl/pub/mirrors/fedora/linux/releases/24/Everything/x86_64/os/images/pxeboot/initrd.img

Finally, during one install, mirror returned 404 on one package, so it timed out.

But I guess that these are not problems with createhdds code and code looks OK otherwise.

Just mention in readme that libvirt-python3 package is required.

This revision is now accepted and ready to land.Jul 4 2016, 10:37 AM
adamwill added a comment.Jul 4 2016, 2:39 PM

Huh, not sure why you'd get failure during boot - I didn't see that once. As for mirror issues, that won't affect the official deployments as download.fedoraproject.org will always redirect to dl.phx2 for them. (it also doesn't affect me as it always redirects to my private mirror, for me. :>) if we think it's a problem for others we could change it to dl. , but I think it's probably OK.

and yeah, I'll update the readme, thanks.

Closed by commit rOPENQA635bc639d6ad: switch from virt-builder to virt-install (T813) (authored by adamwill). · Explain WhyJul 4 2016, 4:29 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
MInstallGuide.md (2 lines)
Mtools/createhdds.py (154 lines)
DMtools/desktop.commands (11 lines)
AMtools/desktop.ks (17 lines)
Mtools/hdds.json (32 lines)
DMtools/kde.commands (14 lines)
AMtools/kde.ks (20 lines)
DMtools/minimal.commands (3 lines)
AMtools/minimal.ks (16 lines)
DMtools/server.commands (6 lines)
AMtools/server.ks (17 lines)
DMtools/support.commands (4 lines)
AMtools/support.ks (19 lines)

Diff 2340

View Options

InstallGuide.md

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Line(s)44Grab the tests, and initialize the database:
47 git clone https://bitbucket.org/rajcze/openqa_fedora fedora 47​ git clone https://bitbucket.org/rajcze/openqa_fedora fedora
48 chown -R geekotest:geekotest fedora 48​ chown -R geekotest:geekotest fedora
49 cd fedora 49​ cd fedora
50 ./templates 50​ ./templates
51 51​
52To run the tests, you will also need to provide some disk images: 52​To run the tests, you will also need to provide some disk images:
53 53​
54 git clone http://bitbucket.org/rajcze/openqa_fedora_tools.git ~/openqa_fedora_tools 54​ git clone http://bitbucket.org/rajcze/openqa_fedora_tools.git ~/openqa_fedora_tools
55 dnf install libguestfs-tools libguestfs-xfs python3-pexpect python3-fedfind python3-libguestfs 55​ dnf install libguestfs-tools libguestfs-xfs python3-fedfind python3-libguestfs libvirt-daemon-config-network libvirt-python3 virt-install
56 mkdir -p /var/lib/openqa/factory/hdd 56​ mkdir -p /var/lib/openqa/factory/hdd
57 cd /var/lib/openqa/factory/hdd/ 57​ cd /var/lib/openqa/factory/hdd/
58 ~/openqa_fedora_tools/tools/createhdds.py all 58​ ~/openqa_fedora_tools/tools/createhdds.py all
59 chown geekotest * 59​ chown geekotest *
60 60​
61[optional] go to OpenQA web interface, go to "admin" -> "Test suites". 61​[optional] go to OpenQA web interface, go to "admin" -> "Test suites".
62Set `REPOSITORY_GRAPHICAL` and `REPOSITORY_VARIATION` to your nearest mirror 62​Set `REPOSITORY_GRAPHICAL` and `REPOSITORY_VARIATION` to your nearest mirror
63according to this list: https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-23&arch=x86_64, 63​according to this list: https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-23&arch=x86_64,
64excluding `/23/x86_64/os/` at the end of url. For example if your nearest mirror 64​excluding `/23/x86_64/os/` at the end of url. For example if your nearest mirror
65is http://ftp.linux.cz/pub/linux/fedora/linux/development/23/x86_64/os/, set 65​is http://ftp.linux.cz/pub/linux/fedora/linux/development/23/x86_64/os/, set
66`REPOSITORY_GRAPHICAL` and `REPOSITORY_VARIATION` to http://ftp.linux.cz/pub/linux/fedora/linux/development. 66​`REPOSITORY_GRAPHICAL` and `REPOSITORY_VARIATION` to http://ftp.linux.cz/pub/linux/fedora/linux/development.
67 67​
68At this point most of the tests should work. However, a few of the tests require two or more workers to be able to communicate with each other (e.g. the FreeIPA tests, where one job sets up a server, and other jobs set up clients and try to connect to the server). For these tests to work, you must do some quite complex software-defined networking configuration on the worker host(s). Instructions for this are in [AdvancedNetwork.md](AdvancedNetwork.md). 68​At this point most of the tests should work. However, a few of the tests require two or more workers to be able to communicate with each other (e.g. the FreeIPA tests, where one job sets up a server, and other jobs set up clients and try to connect to the server). For these tests to work, you must do some quite complex software-defined networking configuration on the worker host(s). Instructions for this are in [AdvancedNetwork.md](AdvancedNetwork.md).
View Options

tools/createhdds.py

Show All 17 Lines
18# Author: Adam Williamson <awilliam@redhat.com> 18​# Author: Adam Williamson <awilliam@redhat.com>
19 19​
20"""Tool for creating hard disk images for Fedora openQA.""" 20​"""Tool for creating hard disk images for Fedora openQA."""
21 21​
22import argparse 22​import argparse
23import logging 23​import logging
24import json 24​import json
25import os 25​import os
26try: 26​import subprocess
27 import subprocess32 as subprocess ​
28except ImportError: ​
29 import subprocess ​
30import sys 27​import sys
31import time ​
32import tempfile 28​import tempfile
29​import time
33 30​
34import fedfind.helpers 31​import fedfind.helpers
35import guestfs 32​import guestfs
36import pexpect 33​import libvirt
37 34​
38from six.moves.urllib.request import urlopen 35​from six.moves.urllib.request import urlopen
39 36​
40# this is a bit icky, but it means you can run the script from 37​# this is a bit icky, but it means you can run the script from
41# anywhere - we use this to locate hdds.json and the virtbuilder 38​# anywhere - we use this to locate hdds.json and the virtinstall
42# image command files, so they must always be in the same place 39​# image kickstarts, so they must always be in the same place
43# as the script itself. images are checked/created in the working 40​# as the script itself. images are checked/created in the working
44# directory. 41​# directory.
45SCRIPTDIR = os.path.abspath(os.path.dirname(sys.argv[0])) 42​SCRIPTDIR = os.path.abspath(os.path.dirname(sys.argv[0]))
46logger = logging.getLogger('createhdds') 43​logger = logging.getLogger('createhdds')
47 44​
48def handle_size(size): 45​def handle_size(size):
49 """Simple function to handle sizes like '10G' or '100MB', returns 46​ """Simple function to handle sizes like '10G' or '100MB', returns
50 the size in bytes as an int. Used by both image classes. 47​ the size in bytes as an int. Used by both image classes.
▲ Show 20 Lines • Show All 125 Lines • ▼ Show 20 Line(s)169 except:
176 raise 173​ raise
177 finally: 174​ finally:
178 # whether things go right or wrong, we want to close the 175​ # whether things go right or wrong, we want to close the
179 # gfs instance, and rwmj recommends 'shutdown()' too 176​ # gfs instance, and rwmj recommends 'shutdown()' too
180 gfs.shutdown() 177​ gfs.shutdown()
181 gfs.close() 178​ gfs.close()
182 179​
183 180​
184class VirtBuilderImage(object): 181​class VirtInstallImage(object):
185 """Class representing an image created by virt-builder. 'release' 182​ """Class representing an image created by virt-install. 'release'
186 is the release the image will be built for. 'arch' is the arch. 183​ is the release the image will be built for. 'variant' is the
187 'size' is the desired image size, valid formats are a digit string 184​ variant whose install tree should be used. 'arch' is the arch.
188 (size in bytes, digit string plus 'M', 'MB' or 'MiB' (size in 185​ 'size' is the desired image size, in gigabytes. 'imgver' is
189 power of two megabytes), or digit string plus 'G', 'GB' or 'GiB' ​
190 (size in power of two gigabytes). 0 (or any false-y value) means ​
191 the image will be the size of the upstream base image. 'imgver' is ​
192 the image 'version' - in practice it's simply a string that gets 186​ the image 'version' - in practice it's simply a string that gets
193 included in the image file name if specified. 'maxage' is the 187​ included in the image file name if specified. 'maxage' is the
194 maximum age of the image file (in days) - if the image is older 188​ maximum age of the image file (in days) - if the image is older
195 than this, 'check' will report it as 'outdated' and 'all' will 189​ than this, 'check' will report it as 'outdated' and 'all' will
196 rebuild it. 190​ rebuild it.
197 """ 191​ """
198 def __init__(self, name, release, arch, size=0, imgver='', maxage=14): 192​ def __init__(self, name, release, arch, size, variant=None, imgver='', maxage=14):
199 self.name = name 193​ self.name = name
200 self.size = handle_size(size) 194​ self.size = size
201 self.filename = "disk_f{0}_{1}".format(str(release), name) 195​ self.filename = "disk_f{0}_{1}".format(str(release), name)
202 if imgver: 196​ if imgver:
203 self.filename = "{0}_{1}".format(self.filename, imgver) 197​ self.filename = "{0}_{1}".format(self.filename, imgver)
204 self.filename = "{0}_{1}.img".format(self.filename, arch) 198​ self.filename = "{0}_{1}.img".format(self.filename, arch)
205 self.release = release 199​ self.release = release
200​ self.variant = variant
206 self.arch = arch 201​ self.arch = arch
207 self.maxage = maxage 202​ self.maxage = maxage
203​ if variant:
204​ self.variant = variant
205​ else:
206​ if release < 24:
207​ self.variant = "Server"
208​ else:
209​ self.variant = "Everything"
208 210​
209 def create(self, retries=3): 211​ def create(self, retries=3):
210 """Create the image.""" 212​ """Create the image."""
213​ # figure out the best os-variant. this is harder than it ought
214​ # to be thanks to RHBZ #1351718...
215​ found = False
216​ rel = self.release + 1
217​ shortid = ""
218​ while not found:
219​ rel -= 1
220​ shortid = "fedora{0}".format(str(rel))
221​ args = ["osinfo-query", "os", "short-id={0}".format(shortid)]
222​ process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
223​ out = process.communicate()[0].decode()
224​ if shortid in out:
225​ found = True
226​
227​ # destroy and delete the domain we use for all virt-installs
228​ conn = libvirt.open()
229​ try:
230​ dom = conn.lookupByName('createhdds')
231​ try:
232​ dom.destroy()
233​ except libvirt.libvirtError:
234​ # domain may not be running, so this is fine
235​ pass
236​ dom.undefine()
237​ except libvirt.libvirtError:
238​ # domain may not exist, so this is fine
239​ pass
240​ conn.close()
241​
211 tmpfile = "{0}.tmp".format(self.filename) 242​ tmpfile = "{0}.tmp".format(self.filename)
212 try: 243​ try:
213 # Basic creation command with standard params 244​ loctmp = "https://download.fedoraproject.org/pub/fedora/linux/releases/{0}/{1}/{2}/os"
214 args = ["virt-builder", "fedora-{0}".format(str(self.release)), "-o", tmpfile, 245​ arch = self.arch
215 "--arch", self.arch] 246​ if arch == 'i686':
216 if self.size: 247​ arch = 'i386'
217 args.extend(["--size", "{0}b".format(str(int(self.size)))]) 248​ args = ["virt-install", "--disk", "size={0},path={1}".format(self.size, tmpfile),
218 # We use guestfs's ability to read customization commands from 249​ "--os-variant", shortid, "-x",
219 # a file. The convention is to have a file 'name.commands' in 250​ "inst.ks=file:/{0}.ks".format(self.name), "--initrd-inject",
220 # SCRIPTDIR; if this file exists, we pass it to virt-builder. 251​ "{0}/{1}.ks".format(SCRIPTDIR, self.name), "--location",
221 if os.path.isfile("{0}/{1}.commands".format(SCRIPTDIR, self.name)): 252​ loctmp.format(str(self.release), self.variant, arch), "--graphics", "vnc",
222 args.extend(["--commands-from-file", "{0}/{1}.commands".format(SCRIPTDIR, self.name)]) 253​ "--name", "createhdds", "--memory", "2048", "--noreboot", "--noautoconsole",
254​ "--wait", "-1"]
223 # run the command, timing out after 1 hour; sometimes creation 255​ # run the command, timing out after 1 hour; sometimes creation
224 # seems to just get mysteriously stuck, we need to bail and 256​ # seems to just get mysteriously stuck, we need to bail and
225 # retry in this case 257​ # retry in this case
226 try: 258​ try:
259​ logger.info("Install running, connect via VNC to monitor")
227 ret = subprocess.call(args, timeout=3600) 260​ ret = subprocess.call(args, timeout=3600)
228 except subprocess.TimeoutExpired: 261​ except subprocess.TimeoutExpired:
229 logger.warning("Image creation timed out!") 262​ logger.warning("Image creation timed out!")
230 if os.path.isfile(tmpfile): 263​ if os.path.isfile(tmpfile):
231 os.remove(tmpfile) 264​ os.remove(tmpfile)
232 if retries: 265​ if retries:
233 logger.info("Retrying: %s retries remain after this", str(retries)) 266​ logger.info("Retrying: %s retries remain after this", str(retries))
234 return self.create(retries=retries-1) 267​ return self.create(retries=retries-1)
235 else: 268​ else:
236 sys.exit("Image creation timed out too many times!") 269​ sys.exit("Image creation timed out too many times!")
237 if ret > 0: 270​ if ret > 0:
238 # wipe the temp file 271​ # wipe the temp file
239 if os.path.isfile(tmpfile): 272​ if os.path.isfile(tmpfile):
240 os.remove(tmpfile) 273​ os.remove(tmpfile)
241 sys.exit("virt-builder command {0} failed!".format(' '.join(args))) 274​ sys.exit("virt-install command {0} failed!".format(' '.join(args)))
242 # We have to boot the disk to make SELinux relabelling happen; 275​ # at this point the domain should be shut off; if it's
243 # virt-builder can't do it unless the policy version on the host 276​ # anything else, something went wrong: clean up and exit
244 # is the same as the guest(?) There are lots of bad ways to do 277​ conn = libvirt.open()
245 # this, expect is the one we're using. This can also get stuck 278​ dom = conn.lookupByName('createhdds')
246 # apparently, so let's set a 300 sec timeout for each expect 279​ if dom.state()[0] != libvirt.VIR_DOMAIN_SHUTOFF:
247 # and bail if it's hit. 280​ conn.close()
248 try: ​
249 logger.info("Booting image to trigger SELinux relabel...") ​
250 child = pexpect.spawnu("qemu-kvm -m 2G -nographic {0}".format(tmpfile), timeout=600) ​
251 child.expect(u"localhost login:") ​
252 child.sendline(u"root") ​
253 child.expect(u"Password:") ​
254 child.sendline(u"weakpassword") ​
255 child.expect(u"~]#") ​
256 child.sendline(u"poweroff") ​
257 child.expect(u"reboot: Power down") ​
258 child.close() ​
259 except pexpect.TIMEOUT: ​
260 if os.path.isfile(tmpfile): 281​ if os.path.isfile(tmpfile):
261 os.remove(tmpfile) 282​ os.remove(tmpfile)
262 sys.exit("Timed out booting image!") 283​ sys.exit("libvirt domain ('createhdds') is not shutdown! "
263 # we're all done! rename to the correct name 284​ "this is an unexpected condition, aborting.")
285​ # we're all done! rename to the correct name and clean up
286​ # the domain
264 os.rename(tmpfile, self.filename) 287​ os.rename(tmpfile, self.filename)
288​ os.chmod(self.filename, 0o644)
289​ dom.undefine()
290​ conn.close()
265 except: 291​ except:
266 # if anything went wrong, we want to wipe the temp file 292​ # if anything went wrong, we want to wipe the temp file
267 # then raise 293​ # then raise. we leave the domain intact in case we want
294​ # to debug it
268 if os.path.isfile(tmpfile): 295​ if os.path.isfile(tmpfile):
269 os.remove(tmpfile) 296​ os.remove(tmpfile)
270 raise 297​ raise
271 298​
272 @property 299​ @property
273 def outdated(self): 300​ def outdated(self):
274 """Whether the image is outdated - if self.maxage is set and 301​ """Whether the image is outdated - if self.maxage is set and
275 it's older than that. 302​ it's older than that.
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Line(s)356 if len(imggrp.get('filesystems', [])) > 1 or len(filesystems) > 1:
330 name_extras.append(filesystem) 357​ name_extras.append(filesystem)
331 if len(imggrp.get('labels', [])) > 1 or len(labels) > 1: 358​ if len(imggrp.get('labels', [])) > 1 or len(labels) > 1:
332 name_extras.append(label) 359​ name_extras.append(label)
333 img = GuestfsImage( 360​ img = GuestfsImage(
334 name, size, imgver, filesystem, label, parts, writes, uploads, name_extras) 361​ name, size, imgver, filesystem, label, parts, writes, uploads, name_extras)
335 imgs.append(img) 362​ imgs.append(img)
336 return imgs 363​ return imgs
337 364​
338def get_virtbuilder_images(imggrp, nextrel=None, releases=None): 365​def get_virtinstall_images(imggrp, nextrel=None, releases=None):
339 """Passed a single 'image group' dict (usually read out of hdds. 366​ """Passed a single 'image group' dict (usually read out of hdds.
340 json), returns a list of VirtBuilderImage instances. 'nextrel' 367​ json), returns a list of VirtInstallImage instances. 'nextrel'
341 indicates the 'next' release of Fedora: sometimes we determine the 368​ indicates the 'next' release of Fedora: sometimes we determine the
342 release to build image(s) for in relation to this, so we need to 369​ release to build image(s) for in relation to this, so we need to
343 know it. If it's not specified, we ask fedfind to figure it out 370​ know it. If it's not specified, we ask fedfind to figure it out
344 for us (this is the usual case, we just allow specifying it 371​ for us (this is the usual case, we just allow specifying it
345 in case there's an issue with fedfind or you want to test image 372​ in case there's an issue with fedfind or you want to test image
346 creation for the next-next release ahead of time or something). 373​ creation for the next-next release ahead of time or something).
347 The image group dict must include a dict named 'releases' which 374​ The image group dict must include a dict named 'releases' which
348 indicates what releases to build images for and what arches to 375​ indicates what releases to build images for and what arches to
349 build for each release; 'releases' can be used to override that. If 376​ build for each release; 'releases' can be used to override that. If
350 set, the image group 'releases' dict is ignored, and this dict is 377​ set, the image group 'releases' dict is ignored, and this dict is
351 used instead. The dict's keys must be release numbers or negative 378​ used instead. The dict's keys must be release numbers or negative
352 integers: -1 means 'one release lower than the "next" release', 379​ integers: -1 means 'one release lower than the "next" release',
353 -2 means 'two releases lower than the "next" release', and so on. 380​ -2 means 'two releases lower than the "next" release', and so on.
354 The values are the arches to build for that release. 381​ The values are the arches to build for that release.
355 """ 382​ """
356 imgs = [] 383​ imgs = []
357 # Set this here so if we need to calculate it, we only do it once 384​ # Set this here so if we need to calculate it, we only do it once
358 if not nextrel: 385​ if not nextrel:
359 nextrel = 0 386​ nextrel = 0
360 name = imggrp['name'] 387​ name = imggrp['name']
361 # this is the second place we set a default for maxage - bit ugly 388​ # this is the second place we set a default for maxage - bit ugly
362 maxage = int(imggrp.get('maxage', 14)) 389​ maxage = int(imggrp.get('maxage', 14))
390​ # ditto variant
391​ variant = imggrp.get('variant')
363 if not releases: 392​ if not releases:
364 releases = imggrp['releases'] 393​ releases = imggrp['releases']
365 size = imggrp.get('size', 0) 394​ size = imggrp.get('size', 0)
366 imgver = imggrp.get('imgver') 395​ imgver = imggrp.get('imgver')
367 # add an image for each release/arch combination 396​ # add an image for each release/arch combination
368 for (relnum, arches) in releases.items(): 397​ for (relnum, arches) in releases.items():
369 if int(relnum) < 0: 398​ if int(relnum) < 0:
370 # negative relnum indicates 'relative to next release' 399​ # negative relnum indicates 'relative to next release'
371 if not nextrel: 400​ if not nextrel:
372 nextrel = fedfind.helpers.get_current_release() + 1 401​ nextrel = fedfind.helpers.get_current_release() + 1
373 relnum = int(nextrel) + int(relnum) 402​ relnum = int(nextrel) + int(relnum)
374 for arch in arches: 403​ for arch in arches:
375 imgs.append( 404​ imgs.append(
376 VirtBuilderImage(name, relnum, arch, size=size, imgver=imgver, maxage=maxage)) 405​ VirtInstallImage(name, relnum, arch, variant=variant, size=size, imgver=imgver,
406​ maxage=maxage))
377 return imgs 407​ return imgs
378 408​
379def get_all_images(hdds, nextrel=None): 409​def get_all_images(hdds, nextrel=None):
380 """Simply iterates over the 'image group' dicts in hdds.json and 410​ """Simply iterates over the 'image group' dicts in hdds.json and
381 calls the appropriate get_foo_images() function for each, then 411​ calls the appropriate get_foo_images() function for each, then
382 returns the list of all images. No overrides for labels, 412​ returns the list of all images. No overrides for labels,
383 filesystems, releases or arches are provided here, this function 413​ filesystems, releases or arches are provided here, this function
384 is just for painting inside the lines - it's used to determine 414​ is just for painting inside the lines - it's used to determine
385 what images are 'expected' to exist according to hdds.json. We 415​ what images are 'expected' to exist according to hdds.json. We
386 do allow passing of 'nextrel' just in case there's some issue 416​ do allow passing of 'nextrel' just in case there's some issue
387 with the auto-discovery. 417​ with the auto-discovery.
388 """ 418​ """
389 imgs = [] 419​ imgs = []
390 for imggrp in hdds['guestfs']: 420​ for imggrp in hdds['guestfs']:
391 imgs.extend(get_guestfs_images(imggrp)) 421​ imgs.extend(get_guestfs_images(imggrp))
392 422​
393 for imggrp in hdds['virtbuilder']: 423​ for imggrp in hdds['virtinstall']:
394 imgs.extend(get_virtbuilder_images(imggrp, nextrel=nextrel)) 424​ imgs.extend(get_virtinstall_images(imggrp, nextrel=nextrel))
395 return imgs 425​ return imgs
396 426​
397def do_renames(hdds): 427​def do_renames(hdds):
398 """Rename files according to the 'renames' list in hdds.json, 428​ """Rename files according to the 'renames' list in hdds.json,
399 which is just a list of 'old name, new name' pairs. Say there's 429​ which is just a list of 'old name, new name' pairs. Say there's
400 an image we used to only create an mbr version of, but now we add 430​ an image we used to only create an mbr version of, but now we add
401 a gpt version; that means the expected name of the mbr version 431​ a gpt version; that means the expected name of the mbr version
402 changes. This function allows us to specify in hdds.json that the 432​ changes. This function allows us to specify in hdds.json that the
▲ Show 20 Lines • Show All 149 Lines • ▼ Show 20 Line(s)577 if imgtype == 'guestfs':
552 labels = None 582​ labels = None
553 filesystems = None 583​ filesystems = None
554 if args.label: 584​ if args.label:
555 labels = [args.label] 585​ labels = [args.label]
556 if args.filesystem: 586​ if args.filesystem:
557 filesystems = [args.filesystem] 587​ filesystems = [args.filesystem]
558 imgs = get_guestfs_images(imggrp, labels=labels, filesystems=filesystems) 588​ imgs = get_guestfs_images(imggrp, labels=labels, filesystems=filesystems)
559 589​
560 elif imgtype == 'virtbuilder': 590​ elif imgtype == 'virtinstall':
561 # If the user passed args.release, we construct a releases 591​ # If the user passed args.release, we construct a releases
562 # dict to pass to get_virtbuilder_images to override the dict 592​ # dict to pass to get_virtinstall_images to override the dict
563 # from imggrp. If they passed args.arch, we use that arch, 593​ # from imggrp. If they passed args.arch, we use that arch,
564 # otherwise we default to x86_64. If args.release isn't set, 594​ # otherwise we default to x86_64. If args.release isn't set,
565 # we just pass None as release, and the releases dict from 595​ # we just pass None as release, and the releases dict from
566 # imggrp will be used, and images created for all release/ 596​ # imggrp will be used, and images created for all release/
567 # arch combinations listed there. FIXME: if the user passes 597​ # arch combinations listed there. FIXME: if the user passes
568 # args.arch but not args.release, we just ignore it... 598​ # args.arch but not args.release, we just ignore it...
569 releases = None 599​ releases = None
570 if args.release: 600​ if args.release:
571 if args.arch: 601​ if args.arch:
572 arches = [args.arch] 602​ arches = [args.arch]
573 else: 603​ else:
574 arches = ['x86_64'] 604​ arches = ['x86_64']
575 releases = {args.release: arches} 605​ releases = {args.release: arches}
576 imgs = get_virtbuilder_images(imggrp, releases=releases) 606​ imgs = get_virtinstall_images(imggrp, releases=releases)
577 607​
578 for (num, img) in enumerate(imgs, 1): 608​ for (num, img) in enumerate(imgs, 1):
579 logger.info("Creating image %s...[%s/%s]", img.filename, str(num), str(len(imgs))) 609​ logger.info("Creating image %s...[%s/%s]", img.filename, str(num), str(len(imgs)))
580 img.create() 610​ img.create()
581 611​
582def parse_args(hdds): 612​def parse_args(hdds):
583 """Parse arguments with argparse.""" 613​ """Parse arguments with argparse."""
584 parser = argparse.ArgumentParser(description=( 614​ parser = argparse.ArgumentParser(description=(
▲ Show 20 Lines • Show All 59 Lines • ▼ Show 20 Line(s)673 if len(filesystems) > 1:
644 imgparser.add_argument( 674​ imgparser.add_argument(
645 '-f', '--filesystem', help="Create only images with this filesystem", 675​ '-f', '--filesystem', help="Create only images with this filesystem",
646 choices=filesystems) 676​ choices=filesystems)
647 imgparser.set_defaults(func=cli_image, label=None, filesystem=None) 677​ imgparser.set_defaults(func=cli_image, label=None, filesystem=None)
648 # Here we're stuffing the type of the image and the dict from 678​ # Here we're stuffing the type of the image and the dict from
649 # hdds into args for cli_image() to use. 679​ # hdds into args for cli_image() to use.
650 imgparser.set_defaults(imggrp=('guestfs', imggrp)) 680​ imgparser.set_defaults(imggrp=('guestfs', imggrp))
651 681​
652 # For libvirt images, we provide args to override the release/arch 682​ # For virtinstall images, we provide args to override the release/
653 # combination; using args.release will always result in just a 683​ # arch combination; using args.release will always result in just a
654 # single image being built, for x86_64 unless args.arch is set to 684​ # single image being built, for x86_64 unless args.arch is set to
655 # i686. 685​ # i686.
656 for imggrp in hdds['virtbuilder']: 686​ for imggrp in hdds['virtinstall']:
657 imgparser = subparsers.add_parser( 687​ imgparser = subparsers.add_parser(
658 imggrp['name'], description="Create {0} image(s)".format(imggrp['name'])) 688​ imggrp['name'], description="Create {0} image(s)".format(imggrp['name']))
659 imgparser.add_argument( 689​ imgparser.add_argument(
660 '-r', '--release', help="The release to build the image(s) for. If not " 690​ '-r', '--release', help="The release to build the image(s) for. If not "
661 "set or set to 0, createhdds will attempt to determine the current " 691​ "set or set to 0, createhdds will attempt to determine the current "
662 "release and build for appropriate releases relative to that", 692​ "release and build for appropriate releases relative to that",
663 type=int, default=0) 693​ type=int, default=0)
664 imgparser.add_argument( 694​ imgparser.add_argument(
665 '-a', '--arch', help="The arch to build the image(s) for. If neither " 695​ '-a', '--arch', help="The arch to build the image(s) for. If neither "
666 "this nor --release is set, createhdds will decide the appropriate " 696​ "this nor --release is set, createhdds will decide the appropriate "
667 "arch(es) to build for each release. If this is not set but --release " 697​ "arch(es) to build for each release. If this is not set but --release "
668 "is set, only x86_64 image(s) will be built.", 698​ "is set, only x86_64 image(s) will be built.",
669 choices=('x86_64', 'i686')) 699​ choices=('x86_64', 'i686'))
670 imgparser.set_defaults(func=cli_image) 700​ imgparser.set_defaults(func=cli_image)
671 # Here we're stuffing the type of the image and the dict from 701​ # Here we're stuffing the type of the image and the dict from
672 # hdds into args for cli_image() to use. 702​ # hdds into args for cli_image() to use.
673 imgparser.set_defaults(imggrp=('virtbuilder', imggrp)) 703​ imgparser.set_defaults(imggrp=('virtinstall', imggrp))
674 return parser.parse_args() 704​ return parser.parse_args()
675 705​
676def main(): 706​def main():
677 """Main loop - set up logging, parse args, run subcommand 707​ """Main loop - set up logging, parse args, run subcommand
678 function. 708​ function.
679 """ 709​ """
680 try: 710​ try:
681 with open('{0}/hdds.json'.format(SCRIPTDIR), 'r') as fout: 711​ with open('{0}/hdds.json'.format(SCRIPTDIR), 'r') as fout:
Show All 16 Lines
View Options

tools/desktop.commands

  • This file was deleted.
This file was completely deleted. Show File Contents
View Options

tools/desktop.ks

  • This file was added.
1​install
2​bootloader --location=mbr
3​network --bootproto=dhcp
4​url --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
5​repo --name=updates --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch
6​lang en_US.UTF-8
7​keyboard us
8​timezone --utc America/New_York
9​clearpart --all
10​autopart
11​rootpw --plaintext weakpassword
12​user --name=test --password=weakpassword --plaintext
13​poweroff
14​
15​%packages
16​@^workstation-product-environment
17​%end
View Options

tools/hdds.json

Show First 20 Lines • Show All 117 Lines • ▼ Show 20 Line(s)117 "writes" : [
118 { 118​ {
119 "part" : "1", 119​ "part" : "1",
120 "path" : "/testfile", 120​ "path" : "/testfile",
121 "content" : "Hello, world!" 121​ "content" : "Hello, world!"
122 } 122​ }
123 ] 123​ ]
124 } 124​ }
125 ], 125​ ],
126 "virtbuilder" : [ 126​ "virtinstall" : [
127 { 127​ {
128 "name" : "minimal", 128​ "name" : "minimal",
129 "releases" : { 129​ "releases" : {
130 "-1" : ["x86_64"], 130​ "-1" : ["x86_64"],
131 "-2" : ["x86_64"] 131​ "-2" : ["x86_64"]
132 } 132​ },
133​ "size" : "6",
134​ "imgver": "2"
133 }, 135​ },
134 { 136​ {
135 "name" : "desktop", 137​ "name" : "desktop",
136 "releases" : { 138​ "releases" : {
137 "-1" : ["x86_64", "i686"], 139​ "-1" : ["x86_64", "i686"],
138 "-2" : ["x86_64", "i686"] 140​ "-2" : ["x86_64", "i686"]
139 }, 141​ },
140 "size" : "20G", 142​ "size" : "20",
141 "imgver": "2" 143​ "imgver": "3",
144​ "variant": "Workstation"
142 }, 145​ },
143 { 146​ {
144 "name" : "server", 147​ "name" : "server",
145 "releases" : { 148​ "releases" : {
146 "-1" : ["x86_64", "i686"], 149​ "-1" : ["x86_64"],
147 "-2" : ["x86_64", "i686"] 150​ "-2" : ["x86_64"]
148 }, 151​ },
149 "imgver": "2" 152​ "size" : "6",
153​ "imgver": "3",
154​ "variant": "Server"
150 }, 155​ },
151 { 156​ {
152 "name" : "kde", 157​ "name" : "kde",
153 "releases" : { 158​ "releases" : {
154 "-1" : ["x86_64", "i686"], 159​ "-1" : ["x86_64"],
155 "-2" : ["x86_64", "i686"] 160​ "-2" : ["x86_64"]
156 }, 161​ },
157 "size" : "20G", 162​ "size" : "20",
158 "imgver": "2" 163​ "imgver": "3"
159 }, 164​ },
160 { 165​ {
161 "name" : "support", 166​ "name" : "support",
162 "imgver" : "2", ​
163 "releases" : { 167​ "releases" : {
164 "-1" : ["x86_64"] 168​ "-1" : ["x86_64"]
165 } 169​ },
170​ "size" : "6",
171​ "imgver" : "3"
166 } 172​ }
167 ], 173​ ],
168 "renames" : [] 174​ "renames" : []
169} 175​}
View Options

tools/kde.commands

  • This file was deleted.
This file was completely deleted. Show File Contents
View Options

tools/kde.ks

  • This file was added.
1​install
2​bootloader --location=mbr
3​network --bootproto=dhcp
4​url --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
5​repo --name=updates --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch
6​lang en_US.UTF-8
7​keyboard us
8​timezone --utc America/New_York
9​clearpart --all
10​autopart
11​rootpw --plaintext weakpassword
12​user --name=test --password=weakpassword --plaintext
13​poweroff
14​
15​%packages
16​@^kde-desktop-environment
17​imsettings-qt
18​-initial-setup
19​-initial-setup-gui
20​%end
View Options

tools/minimal.commands

  • This file was deleted.
This file was completely deleted. Show File Contents
View Options

tools/minimal.ks

  • This file was added.
1​install
2​bootloader --location=mbr
3​network --bootproto=dhcp
4​url --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
5​repo --name=updates --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch
6​lang en_US.UTF-8
7​keyboard us
8​timezone --utc America/New_York
9​clearpart --all
10​autopart
11​rootpw weakpassword
12​poweroff
13​
14​%packages
15​@core
16​%end
View Options

tools/server.commands

  • This file was deleted.
This file was completely deleted. Show File Contents
View Options

tools/server.ks

  • This file was added.
1​install
2​bootloader --location=mbr
3​network --bootproto=dhcp
4​url --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
5​repo --name=updates --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch
6​lang en_US.UTF-8
7​keyboard us
8​timezone --utc America/New_York
9​clearpart --all
10​autopart
11​rootpw weakpassword
12​user --name=test --password=weakpassword --plaintext
13​poweroff
14​
15​%packages
16​@^server-product-environment
17​%end
View Options

tools/support.commands

  • This file was deleted.
This file was completely deleted. Show File Contents
View Options

tools/support.ks

  • This file was added.
1​install
2​bootloader --location=mbr
3​network --bootproto=dhcp
4​url --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
5​repo --name=updates --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever&arch=$basearch
6​lang en_US.UTF-8
7​keyboard us
8​timezone --utc America/New_York
9​clearpart --all
10​autopart
11​rootpw weakpassword
12​poweroff
13​
14​%packages
15​@core
16​scsi-target-utils
17​nfs-utils
18​dnsmasq
19​%end