Differential D694

Script for triggering builds in ImageFactory
AbandonedPublic
Actions

Authored by jskladan on Dec 16 2015, 12:57 PM.

Details

Reviewers
lbrabec
kparal
garretraziel
tflink
mkrizek
Maniphest Tasks
T646: Deploy system for building images for disposable clients
Summary

Use the trigger_build.py to trigger new builds of images in ImageFactory.
ImageFactory's base-url needs to be configured in the code for now (defaults to localhost:8075).

Example usage:

trigger_build.py build --version 23
trigger_build.py build --version 23 --flavor minimal --arch x86_64

The command now only supports minimal flavor, support for more will come, when needed.

There is also a basic functionality for pruning the old and failed-to-build images.
To remove failed builds:

trigger_build.py prune-failed

To remove old images - this means images of a common version, flavor and arch, which became obsolete by new successfully built ones:

trigger_build.py prune-old

By default, the newest image plus one older are kept. If you wish to keep more in the history, use --keep parameter:

trigger_build.py prune-old --keep X
Test Plan

Tested against locally running ImageFactory

Diff Detail

Lint
Lint Skipped
Unit
Unit Tests Skipped
jskladan retitled this revision from to Script for triggering builds in ImageFactory.Dec 16 2015, 12:57 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added a reviewer: libtaskotron.
jskladan added a project: libtaskotron.
tflink accepted this revision.Dec 17 2015, 7:19 AM

I think it's going to need some tweaking going forward but overall, it looks good

This revision is now accepted and ready to land.Dec 17 2015, 7:19 AM
kparal added a comment.Dec 17 2015, 11:12 AM

I like how commented the code is :) Some comments below.

trigger_build/kickstarts/minimal.ks
1

I see you're using my kickstart, copycat! Firewall disabled, selinux permissive, root password 'fedora', that has to be mine :-)

I'm not sure all of those settings are perfect for our purposes, food for thought.

10

It might be a good idea to set the default local timezone to be the same as UTC timezone (which means Iceland, which seems to be specified as Atlantic/Reykjavik), so that no matter whether the command in question prints in local time or UTC, it will always be same as UTC.

23

This will probably need to be configurable, so that we can set our internal repo here, when deployed in Fedora infrastructure.

trigger_build/tdl/fedora.tdl
10

I guess this should match the kickstart one.

trigger_build/trigger_build.py
12

I'd rather not use the Fedora master mirror as the default one. You could pick one of the high-bandwidth ones in appropriate location, or probably the best default would be download.fedoraproject.org, which automatically redirects to the closest one. But it shouldn't be used more than once consequently (otherwise you might start pulling files from different mirrors, unsynchronized). I don't know whether oz-install supports that (first resolve the url, check redirection, then pull everything from the new location), but I guess it should be fairly easy to do this with requests beforehand and then give the new location to oz-install. (We should start doing this in libtaskotron as well, I've seen several issues caused by the lack of this).

35–40

If you didn't want to hardcode this, we could possible use yumrepoinfo.conf for this (extend it to know about flavors). Same for RAWHIDE.

jskladan abandoned this revision.Feb 8 2016, 8:43 AM

Revision Contents

Diff 1797

View Options

trigger_build/kickstarts/minimal.ks

  • This file was added.
1​install
kparalUnsubmitted
Not Done

I see you're using my kickstart, copycat! Firewall disabled, selinux permissive, root password 'fedora', that has to be mine :-)

I'm not sure all of those settings are perfect for our purposes, food for thought.

2​text
3​firewall --disabled
4​rootpw --plaintext fedora
5​auth --enableshadow --passalgo=md5
6​firstboot --disable
7​keyboard us
8​lang en_US
9​selinux --permissive
10​timezone --isUtc Europe/Prague
kparalUnsubmitted
Not Done

It might be a good idea to set the default local timezone to be the same as UTC timezone (which means Iceland, which seems to be specified as Atlantic/Reykjavik), so that no matter whether the command in question prints in local time or UTC, it will always be same as UTC.

11​network --device link --activate
12​reboot
13​
14​# Wipe all disk
15​zerombr
16​bootloader --timeout=1
17​clearpart --all --initlabel
18​part biosboot --fstype=biosboot --size=1
19​part swap --fstype=swap --size=512
20​part / --fstype=ext4 --grow --size=1
21​
22​# Package source
23​url --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
kparalUnsubmitted
Not Done

This will probably need to be configurable, so that we can set our internal repo here, when deployed in Fedora infrastructure.

24​
25​%packages
26​%end
View Options

trigger_build/tdl/fedora.tdl

  • This file was added.
1​<template>
2​ <name>%(timestamp)s-%(release)s-%(flavor)s-%(arch)s</name>
3​ <os>
4​ <name>Fedora</name>
5​ <version>%(version_oz)s</version>
6​ <arch>%(arch)s</arch>
7​ <install type='url'>
8​ <url>%(repourl)s</url>
9​ </install>
10​ <rootpw>eeeewwwww</rootpw>
kparalUnsubmitted
Not Done

I guess this should match the kickstart one.

11​ </os>
12​ <disk>
13​ <size>%(disk_size)s</size>
14​ </disk>
15​ <description>%(metadata)s</description>
16​</template>
View Options

trigger_build/trigger_build.py

  • This file was added.
1​import os
2​import sys
3​import argparse
4​import requests
5​import datetime
6​import json
7​from lxml import etree
8​
9​# TODO: Add config file to override the values
10​
11​IMGFAC_BASE_URL = "http://127.0.0.1:8075/imagefactory"
12​REPOURL_BASE = "http://dl.fedoraproject.org/pub/fedora/linux/"
kparalUnsubmitted
Not Done

I'd rather not use the Fedora master mirror as the default one. You could pick one of the high-bandwidth ones in appropriate location, or probably the best default would be download.fedoraproject.org, which automatically redirects to the closest one. But it shouldn't be used more than once consequently (otherwise you might start pulling files from different mirrors, unsynchronized). I don't know whether oz-install supports that (first resolve the url, check redirection, then pull everything from the new location), but I guess it should be fairly easy to do this with requests beforehand and then give the new location to oz-install. (We should start doing this in libtaskotron as well, I've seen several issues caused by the lack of this).

13​# TinMan can not handle "rawhide" and needs an integer-release
14​RAWHIDE = 24
15​# set to the scriptdir by default
16​TEMPLATES_BASEDIR = os.path.abspath(os.path.dirname(sys.argv[0]))
17​
18​
19​def trigger_build(release, arch="x86_64", flavor="minimal", disk_size="10G"):
20​ """
21​ Triggers a build of new base-image in ImageFactory based on the given
22​ parameters.
23​ Returns the requests' Response object.
24​ """
25​
26​ with open(os.path.join(TEMPLATES_BASEDIR, 'tdl', 'fedora.tdl')) as f:
27​ tdl_template = f.read()
28​
29​ with open(os.path.join(TEMPLATES_BASEDIR, 'kickstarts', '%s.ks' % flavor)) as f:
30​ kickstart = f.read()
31​
32​ # TinMan plugin can not handle "rawhide" and needs an integer-release
33​ # we also set the url from which kernel and initrd are to be downloaded
34​ release = str(release).lower()
35​ if release == 'rawhide':
36​ version_oz = RAWHIDE
37​ repourl = "%s/development/rawhide/%s/os/" % (REPOURL_BASE, arch)
38​ else:
39​ version_oz = release
40​ repourl = "%s/releases/%s/Server/%s/os/" % (REPOURL_BASE, release, arch)
kparalUnsubmitted
Not Done

If you didn't want to hardcode this, we could possible use yumrepoinfo.conf for this (extend it to know about flavors). Same for RAWHIDE.

41​
42​ # prepare data to use for in-place replacement in the tdl
43​ data = {
44​ "release": release,
45​ "flavor": flavor,
46​ "timestamp": datetime.datetime.utcnow().strftime("%y%m%d_%H%M"),
47​ "version_oz": version_oz,
48​ "arch": arch,
49​ "repourl": repourl,
50​ "disk_size": disk_size,
51​ "os_name": "fedora"
52​ }
53​ # this will be used in the future, probably with some additional info
54​ data['metadata'] = json.dumps(data)
55​
56​ # prepare the json data for the POST request
57​ tdl_template = tdl_template % data
58​ post_data = {
59​ "base_image": {
60​ "template": tdl_template,
61​ "parameters": {
62​ "install_script": kickstart
63​ }
64​ }
65​ }
66​
67​ # Trigger the build
68​ r = requests.post("%s/base_images" % IMGFAC_BASE_URL, json=post_data)
69​
70​ # pretty-print what's returned from the request
71​ if r.ok:
72​ print json.dumps(
73​ r.json(),
74​ sort_keys=True,
75​ indent=4,
76​ separators=(',', ': '))
77​ else:
78​ print r.text
79​
80​ return r
81​
82​def prune_failed():
83​ r = requests.get("%s/base_images" % IMGFAC_BASE_URL)
84​ if not r.ok:
85​ return None
86​ all = r.json()
87​
88​ # Go through the list of all the base images
89​ pruned_images = []
90​ for img in all['base_images']:
91​ # get the whole info of the one image
92​ r = requests.get(img['base_image']['href'])
93​ if not r.ok:
94​ continue
95​
96​ data = r.json()['base_image']
97​ # skip all the non-failed
98​ if data['status'] != "FAILED":
99​ continue
100​
101​ print "Removing: %s" % img['base_image']['href']
102​ r = requests.delete(img['base_image']['href'])
103​ if not r.ok:
104​ print "Failed to delete: %s" % img['base_image']['href']
105​ continue
106​ pruned_images.append(img['base_image']['href'])
107​
108​
109​def prune_old(keep=1):
110​ r = requests.get("%s/base_images" % IMGFAC_BASE_URL)
111​ if not r.ok:
112​ return None
113​ all = r.json()
114​
115​ # Go through the list of all the base images
116​ images = {}
117​ for img in all['base_images']:
118​ # get the whole info of the one image
119​ r = requests.get(img['base_image']['href'])
120​ if not r.ok:
121​ continue
122​
123​ data = r.json()['base_image']
124​ # skip all the failed and incomplete builds
125​ if data['status'] != "COMPLETE":
126​ continue
127​
128​ # extract the <description> tag from the stored template
129​ # this is the TLD file used to build the image and it contains
130​ # additional metadata in the said tag to make the searching easier
131​ xmltext = data['template']
132​ tree = etree.fromstring(xmltext)
133​ description = tree.find("description")
134​
135​ # skip the image, if the tag is not present, or empty
136​ if description is None:
137​ continue
138​ metadata = description.text
139​ if not metadata:
140​ continue
141​
142​ # check whether the tag contains json data
143​ try:
144​ metadata = json.loads(metadata)
145​ except ValueError:
146​ continue
147​
148​ # add url of the image to the metadata
149​ metadata['href'] = img['base_image']['href']
150​
151​ # this groups all the images of the same release-flavor-arch
152​ # into one place, so we can keep the 'latest' later
153​ key = (metadata['release'], metadata['flavor'], metadata['arch'])
154​ if key not in images:
155​ images[key] = []
156​ images[key].append(metadata)
157​
158​ # Sort the groups of images by timestamp (in ascending order)
159​ # and remove all but the newest+keep images
160​ pruned_images = []
161​ for key, value in images.iteritems():
162​ to_prune = sorted(value, key=lambda x: x['timestamp'])[:-(keep+1)]
163​ for image in to_prune:
164​ print "Removing: %s" % image['href']
165​ r = requests.delete(image['href'])
166​ if not r.ok:
167​ print "Failed to delete: %s" % image['href']
168​ continue
169​ pruned_images.append(image['href'])
170​
171​
172​
173​def parse_args():
174​ parser = argparse.ArgumentParser(
175​ description='Build and prune base images via ImageFactory.',
176​ )
177​ subparsers = parser.add_subparsers(dest="subcommand")
178​ subparsers.required = True
179​
180​ build = subparsers.add_parser(
181​ "build",
182​ description="Trigger new build",
183​ formatter_class=argparse.ArgumentDefaultsHelpFormatter
184​ )
185​ build.add_argument(
186​ "--release",
187​ help="Release of the Fedora to build. Number or 'rawhide'",
188​ required=True,
189​ )
190​ build.add_argument(
191​ "--arch",
192​ help="Desired architecture.",
193​ choices=['i386', 'x86_64'],
194​ default="x86_64",
195​ )
196​ build.add_argument(
197​ "--flavor",
198​ help="What kind of image to build",
199​ choices=['minimal'],
200​ default='minimal',
201​ )
202​ build.add_argument(
203​ "--disksize",
204​ help="Disk size of the built image.",
205​ default="10G",
206​ )
207​
208​ prune_failed = subparsers.add_parser("prune-failed", description="Remove the failed builds")
209​
210​
211​ def zero_plus(x):
212​ x = int(x)
213​ if x < 0:
214​ raise argparse.ArgumentError("Miminal value is 0")
215​ return x
216​
217​ prune_old = subparsers.add_parser(
218​ "prune-old", description="Remove old builds",
219​ formatter_class=argparse.ArgumentDefaultsHelpFormatter
220​ )
221​ prune_old.add_argument(
222​ "--keep",
223​ help="How many images to keep in history on top of the newest.",
224​ type=zero_plus,
225​ default=1,
226​ )
227​
228​
229​ return parser.parse_args()
230​
231​if __name__ == "__main__":
232​ args = parse_args()
233​ if args.subcommand == 'build':
234​ trigger_build(args.release, args.arch, args.flavor, args.disksize)
235​ elif args.subcommand == 'prune-failed':
236​ prune_failed()
237​ elif args.subcommand == 'prune-old':
238​ prune_old(args.keep)
239​