scheduler/fedora_openqa_schedule/schedule.py
71	Leftover debugging print?
133	Looks like `version` isn't used anywhere.

scheduler/fedora_openqa_schedule/schedule.py
71	Ooops, indeed!
133	oh whoops, it's meant to replace the `VERSION` value. I'm kinda uncomfortable with relying on splitting that out of the 'build' (compose ID), it seems safer to pass it explicitly.

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	2711	8b8798c		Dec 8 2016, 1:36 AM	★	★
Diff 2	2712	8b8798c	Drop the consumer stuff	Dec 8 2016, 7:09 AM	★	★
Diff 3	2713	8b8798c	update commit msg	Dec 8 2016, 7:10 AM	★	★
Diff 4	2714	8b8798c	Fix the issues @garretraziel noted in review	Dec 8 2016, 4:55 PM	★	★
Diff 5	2719	8b8798c	Fix WANTED Workstation i386	Dec 9 2016, 6:36 PM	★	★
Diff 6	2726	8b8798c	rOPENQA6d84c0558e8365e15c2d0d9ee771dcdc1b887bee	Dec 14 2016, 6:58 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
9ceabc2b4fd4	a236e0cff0e1	8b8798cfd1e6	Adam Williamson	Support testing the semi-official live-respins, more fedfind (Show More…)	Dec 8 2016, 1:30 AM

Diff 2726

scheduler/fedora_openqa_schedule/config.py

Show All 20 Lines
21		21
22	# Standard libraries	22	# Standard libraries
23	import json	23	import json
24	import os.path	24	import os.path
25		25
26	# External dependencies	26	# External dependencies
27	from six.moves import configparser	27	from six.moves import configparser
28		28
		29
		30	class ConfigError(Exception):
		31	"""Raised when there's an error in a config file."""
		32	pass
		33
29	# Read in config from /etc/fedora-openqa/schedule.conf or	34	# Read in config from /etc/fedora-openqa/schedule.conf or
30	# ~/.config/fedora-openqa/schedule.conf after setting some default	35	# ~/.config/fedora-openqa/schedule.conf after setting some default
31	# values.	36	# values.
32	CONFIG = configparser.ConfigParser()	37	CONFIG = configparser.ConfigParser()
33	CONFIG.add_section('cli')	38	CONFIG.add_section('cli')
34	CONFIG.add_section('report')	39	CONFIG.add_section('report')
35	CONFIG.set('cli', 'log-file', '')	40	CONFIG.set('cli', 'log-file', '')
36	CONFIG.set('cli', 'log-level', 'info')	41	CONFIG.set('cli', 'log-level', 'info')
37	CONFIG.set('report', 'submit_wiki', 'true')	42	CONFIG.set('report', 'submit_wiki', 'true')
38	CONFIG.set('report', 'submit_resultsdb', 'true')	43	CONFIG.set('report', 'submit_resultsdb', 'true')
39	CONFIG.set('report', 'openqa_url', 'http://localhost')	44	CONFIG.set('report', 'openqa_url', 'http://localhost')
40	CONFIG.set('report', 'resultsdb_url', 'http://localhost:5001/api/v1.0/')	45	CONFIG.set('report', 'resultsdb_url', 'http://localhost:5001/api/v1.0/')
41	CONFIG.set('report', 'wiki_url', 'https://fedoraproject.org/wiki')	46	CONFIG.set('report', 'wiki_url', 'https://fedoraproject.org/wiki')
42	CONFIG.set('report', 'wiki_stg_url', 'https://stg.fedoraproject.org/wiki')	47	CONFIG.set('report', 'wiki_stg_url', 'https://stg.fedoraproject.org/wiki')
43	CONFIG.read('/etc/fedora-openqa/schedule.conf')	48	CONFIG.read('/etc/fedora-openqa/schedule.conf')
44	CONFIG.read('{0}/.config/fedora-openqa/schedule.conf'.format(os.path.expanduser('~')))	49	CONFIG.read('{0}/.config/fedora-openqa/schedule.conf'.format(os.path.expanduser('~')))
45		50
46	# The default set of tested images. The format intentionally follows	51	# The default set of tested images. This set can be overridden by an
47	# the format of images.json closely, and this set can be overridden by	52	# 'images.json' file in /etc/fedora-openqa or ~/.config/fedora-openqa.
48	# an 'images.json' file in /etc/fedora-openqa or
49	# ~/.config/fedora-openqa.
50
51	# There is one level added to this dict compared to images.json.
52	# images.json is a dict of dicts representing "variants". Each variant
53	# is a dict of lists representing arches. Each arch list contains
54	# dicts which each contain the properties of a single image as simple
55	# strings.
56		53
57	# In our dict, things are same until we reach the image level. Again,	54	# The format is a list of dicts. Each dict represents a single image
58	# each item in the list for a given arch within a given variant is a	55	# we want to test. The list is compared against the list of image
59	# dict for a single image. However, one of the items in the image dict	56	# dicts fedfind returns for the release being tested. One of the items
60	# is itself a dict, called 'match'. This dict determines whether the	57	# in the image dict is itself a dict called 'match'. This dict
61	# image "matches": if all the items in this "match" dict match the	58	# determines whether the image "matches": if all the items in this
62	# items in the image dict from the upstream metadata, we take that	59	# "match" dict match the items in the fedfind image dict, we take that
63	# image.	60	# image.
64		61
65	# The other items in the image dict here are not used for matching,	62	# The other items in the image dict here are not used for matching,
66	# but instead influence the behaviour of the scheduler.	63	# but instead influence the behaviour of the scheduler.
67		64
68	# The "score" item is used to decide which images to run the universal	65	# The "score" item is used to decide which images to run the universal
69	# tests with. For each arch that has universal tests, the found image	66	# tests with. For each arch that has universal tests, the found image
70	# with the highest score is used. If no image with a score > 0 is	67	# with the highest score is used. If no image with a score > 0 is
71	# found, the universal tests are skipped. If score is not specified	68	# found, the universal tests are skipped. If score is not specified
72	# the image is considered to have a score of 0.	69	# the image is considered to have a score of 0.
73		70
74	# If "dkboot" is set to True, scheduler also downloads kernel and initrd	71	# If "dkboot" is set to True, scheduler also downloads kernel and initrd
75	# files for specified architecture and then schedules job to run in Direct	72	# files for specified architecture and then schedules job to run in Direct
76	# Kernel Boot mode. Default value (used when dkboot item is not specified)	73	# Kernel Boot mode. Default value (used when dkboot item is not specified)
77	# is False.	74	# is False.
78		75
79	WANTED = {	76	WANTED = [
80	"Server": {	77	{
81	"x86_64": [	78	"match": {
82	{	79	"subvariant": "Server",
83	"match": {	80	"type": "boot",
84	"type": "boot",	81	"format": "iso",
85	"format": "iso",	82	"arch": "x86_64",
86	},	83	},
87	"score": 6,	84	"score": 6,
88	},	85	},
89	{	86	{
90	"match": {	87	"match": {
91	"type": "dvd",	88	"subvariant": "Server",
92	"format": "iso",	89	"type": "dvd",
93	},	90	"format": "iso",
94	"score": 10,	91	"arch": "x86_64",
95	},	92	},
96	],	93	"score": 10,
97	"i386": [	94	},
98	{	95	{
99	"match": {	96	"match": {
100	"type": "boot",	97	"subvariant": "Server",
101	"format": "iso",	98	"type": "boot",
102	},	99	"format": "iso",
103	"score": 6,	100	"arch": "i386",
104	},	101	},
105	{	102	"score": 6,
106	"match": {	103	},
107	"type": "dvd",	104	{
108	"format": "iso",	105	"match": {
109	},	106	"subvariant": "Server",
110	"score": 10,	107	"type": "dvd",
111	},	108	"format": "iso",
112	],	109	"arch": "i386",
		110	},
		111	"score": 10,
		112	},
		113	{
		114	"match": {
		115	"subvariant": "Everything",
		116	"type": "boot",
		117	"format": "iso",
		118	"arch": "x86_64",
		119	},
		120	"score": 8,
		121	},
		122	{
		123	"match": {
		124	"subvariant": "Everything",
		125	"type": "boot",
		126	"format": "iso",
		127	"arch": "i386",
		128	},
		129	"score": 8,
		130	},
		131	{
		132	"match": {
		133	"subvariant": "Workstation",
		134	"type": "live",
		135	"format": "iso",
		136	"arch": "x86_64",
		137	},
		138	},
		139	{
		140	"match": {
		141	"subvariant": "Workstation",
		142	"type": "boot",
		143	"format": "iso",
		144	"arch": "x86_64",
		145	},
		146	},
		147	{
		148	"match": {
		149	"subvariant": "Workstation",
		150	"type": "live",
		151	"format": "iso",
		152	"arch": "i386",
		153	},
		154	},
		155	{
		156	"match": {
		157	"subvariant": "Workstation",
		158	"type": "boot",
		159	"format": "iso",
		160	"arch": "i386",
		161	},
113	},	162	},
114	"Everything": {	163	{
115	"x86_64": [	164	"match": {
116	{	165	"subvariant": "KDE",
117	"match": {	166	"type": "live",
118	"type": "boot",	167	"format": "iso",
119	"format": "iso",	168	"arch": "x86_64",
120	},	169	},
121	"score": 8,
122	},
123	],
124	"i386": [
125	{
126	"match": {
127	"type": "boot",
128	"format": "iso",
129	},
130	"score": 8,
131	},
132	],
133	},	170	},
134	"Workstation": {	171	{
135	"x86_64": [	172	"match": {
136	{	173	"subvariant": "KDE",
137	"match": {	174	"type": "live",
138	"type": "live",	175	"format": "iso",
139	"format": "iso",	176	"arch": "i386",
140	},	177	},
141	},
142	{
143	"match": {
144	"type": "boot",
145	"format": "iso",
146	},
147	},
148	],
149	"i386": [
150	{
151	"match": {
152	"type": "live",
153	"format": "iso",
154	},
155	},
156	{
157	"match": {
158	"type": "boot",
159	"format": "iso",
160	},
161	},
162	],
163	},	178	},
164	"Spins": {	179	{
165	"x86_64": [	180	"match": {
166	{	181	"subvariant": "Minimal",
167	"match": {	182	"type": "raw-xz",
168	"subvariant": "KDE",	183	"format": "raw.xz",
169	"type": "live",	184	"arch": "armhfp",
170	"format": "iso",	185	},
171	},	186	"dkboot": True,
172	},
173	],
174	"i386": [
175	{
176	"match": {
177	"subvariant": "KDE",
178	"type": "live",
179	"format": "iso",
180	},
181	},
182	],
183	"armhfp": [
184	{
185	"match": {
186	"subvariant": "Minimal",
187	"type": "raw-xz",
188	"format": "raw.xz",
189	},
190	"dkboot": True,
191	},
192	],
193	},	187	},
194	"Atomic": {	188	{
195	"x86_64": [	189	"match": {
196	{	190	"subvariant": "Atomic",
197	"match": {	191	"type": "dvd-ostree",
198	"type": "dvd-ostree",	192	"format": "iso",
199	"format": "iso",	193	"arch": "x86_64",
200	},	194	},
201	},
202	],
203	},	195	},
204	}	196	]
205		197
206	for path in ('/etc/fedora-openqa',	198	for path in ('/etc/fedora-openqa',
207	'{0}/.config/fedora-openqa'.format(os.path.expanduser('~'))):	199	'{0}/.config/fedora-openqa'.format(os.path.expanduser('~'))):
208	try:	200	try:
209	with open('{0}/images.json'.format(path), 'r') as fout:	201	fname = '{0}/images.json'.format(path)
		202	with open(fname, 'r') as fout:
210	WANTED = json.load(fout)	203	WANTED = json.load(fout)
		204	try:
		205	WANTED.keys()
		206	raise ConfigError("{0} is in old format (dict, not list)!".format(fname))
		207	except AttributeError:
		208	pass
211	except IOError:	209	except IOError:
212	# file not found	210	# file not found
213	pass	211	pass

View Options

scheduler/fedora_openqa_schedule/schedule.py

Show All 25 Lines
26	# Standard libraries	26		# Standard libraries
27	import json	27		import json
28	import logging	28		import logging
29	import os.path	29		import os.path
30	import re	30		import re
31		31
32	# External dependencies	32		# External dependencies
33	import fedfind.helpers	33		import fedfind.helpers
		34		import fedfind.release
34	from openqa_client.client import OpenQA_Client	35		from openqa_client.client import OpenQA_Client
35	from six.moves.urllib.request import urlopen	36		from six.moves.urllib.request import urlopen
36	from six.moves.urllib.error import URLError, HTTPError	37		from six.moves.urllib.error import URLError, HTTPError
37	from resultsdb_api import ResultsDBapi, ResultsDBapiException	38		from resultsdb_api import ResultsDBapi, ResultsDBapiException
38		39
39	# Internal dependencies	40		# Internal dependencies
40	from fedora_openqa_schedule.config import CONFIG, WANTED	41		from fedora_openqa_schedule.config import CONFIG, WANTED
41		42
42	logger = logging.getLogger(__name__)	43		logger = logging.getLogger(__name__)
43		44
44	FORMAT_TO_PARAM = {	45		FORMAT_TO_PARAM = {
45	"iso": "ISO_URL",	46		"iso": "ISO_URL",
46	# let's connect it as second HDD - we can then use NUMDISKS=1 when we don't need it connected	47		# let's connect it as second HDD - we can then use NUMDISKS=1 when we don't need it connected
47	"raw.xz": "HDD_2_DECOMPRESS_URL"	48		"raw.xz": "HDD_2_DECOMPRESS_URL"
48	}	49		}
49		50
50		51
51	class TriggerException(Exception):	52		class TriggerException(Exception):
52	pass	53		pass
53		54
54		55
55	def _get_compose_id(location):
56	"""Given a compose location, find the compose ID. Really we'd like
57	taskotron to give us this, as fedmsg provides it, but taskotron is
58	kinda tied to giving us just one variable property of the fedmsg
59	message. So we read it from the compose metadata's known location.
60	"""
61	try:
62	resp = urlopen('{0}/metadata/composeinfo.json'.format(location))
63	metadata = json.load(resp)
64	except (ValueError, URLError, HTTPError):
65	raise TriggerException("Compose not found, or failed!")
66	return metadata['payload']['compose']['id']
67
68
69	def _get_dkboot_urls(location, arch='armhfp'):	56		def _get_dkboot_urls(location, arch='armhfp'):
70	"""Given a compose location (and arch if provided), return URLs for kernel and initrd	57		"""Given a fedfind release 'generic' location (and arch if provided), return URLs for kernel
71	files.	58		and initrd files.
72	"""	59		"""
73	pxeboot_url = '{0}/Everything/{1}/os/images/pxeboot/{2}'	60		pxeboot_url = '{0}/{1}/os/images/pxeboot/{2}'
74	return pxeboot_url.format(location, arch, 'vmlinuz'), pxeboot_url.format(location, arch, 'initrd.img')	61		return (pxeboot_url.format(location, arch, 'vmlinuz'), pxeboot_url.format(location, arch, 'initrd.img'))
75		62
76		63
77	def _get_images(location, compose, wanted=WANTED):	64		def _get_images(rel, wanted=WANTED):
78	"""Given a Pungi compose top-level location and string containing compose ID, this returns a list	65		"""Given a fedfind Release instance, this returns a list of (flavor, arch, score, {param: url},
79	of (flavor, arch, score, {param: url}, subvariant, imagetype) tuples for images to be tested.	66		subvariant, imagetype) tuples for images to be tested.
80	"""	67		"""
81	try:
82	resp = urlopen('{0}/metadata/images.json'.format(location))
83	metadata = json.load(resp)
84	except (ValueError, URLError, HTTPError):
85	raise TriggerException("Compose not found, or failed!")
86	images = []	68		images = []
87	for variant in wanted.keys():	69		for wantimg in wanted:
88	for arch in wanted[variant].keys():	70		matchdict = wantimg['match'].copy()
89	try:	71		for foundimg in rel.all_images:
			garretrazielUnsubmitted Not Done Leftover debugging print?
			adamwillAuthorUnsubmitted Not Done Ooops, indeed!
90	foundimgs = metadata['payload']['images'][variant][arch]	72		# see if the foundimg matches the wantimg
91	except KeyError:	73		if not all(item in foundimg.items() for item in matchdict.items()):
92	# not found in upstream metadata, move on
93	continue	74		continue
94		75		score = wantimg.get('score', 0)
95	wantimgs = wanted[variant][arch]	76		# assign a 'flavor' using fedfind's 'image identifier'
96	for wantimg in wantimgs:	77		flavor = fedfind.helpers.identify_image(foundimg, undersub=True, out='string')
97	matchdict = wantimg['match'].copy()	78		# get a couple of values we use for other reasons
98	for foundimg in foundimgs:	79		subvariant = foundimg['subvariant']
99	# let fedfind 'correct' the foundimg dict (fixes	80		imagetype = foundimg['type']
100	# problems with upstream metadata values)	81		arch = foundimg['arch']
101	foundimg = fedfind.helpers.correct_image(foundimg)	82		# FIXME: this is technically too simple though it works
102	# see if the foundimg matches the wantimg	83		# for all releases we currently test; see fedfind 'alt'
103	if not all(item in foundimg.items() for item in matchdict.items()):	84		# location handling
104	continue	85		url = "{0}/{1}".format(rel.location, foundimg['path'])
105	score = wantimg.get('score', 0)	86		logger.debug("Found image %s for arch %s at %s", flavor, arch, url)
106	# assign a 'flavor' using fedfind's 'image identifier'	87
107	flavor = fedfind.helpers.identify_image(foundimg, undersub=True, out='string')	88		# some tests need more than one file, so let's collect them now
108	# get a couple of values we use for other reasons	89		param_urls = {
109	subvariant = foundimg['subvariant']	90		FORMAT_TO_PARAM[foundimg['format']]: url
110	imagetype = foundimg['type']	91		}
111	url = "{0}/{1}".format(location, foundimg['path'])	92		# if direct kernel boot is specified, we need to download kernel and initrd
112	logger.debug("Found image %s for arch %s at %s", flavor, arch, url)	93		if wantimg.get('dkboot', False):
113		94		(kernel_url, initrd_url) = _get_dkboot_urls(rel.https_url_generic, arch)
114	# some tests need more than one file, so let's collect them now	95		param_urls["KERNEL_URL"] = kernel_url
115	param_urls = {	96		param_urls["KERNEL"] = "{0}.{1}.vmlinuz".format(rel.cid, arch)
116	FORMAT_TO_PARAM[foundimg['format']]: url	97		param_urls["INITRD_URL"] = initrd_url
117	}	98		param_urls["INITRD"] = "{0}.{1}.initrd.img".format(rel.cid, arch)
118	# if direct kernel boot is specified, we need to download kernel and initrd	99
119	if wantimg.get('dkboot', False):	100		images.append((flavor, arch, score, param_urls, subvariant, imagetype))
120	kernel_url, initrd_url = _get_dkboot_urls(location, arch)
121	param_urls["KERNEL_URL"] = kernel_url
122	param_urls["KERNEL"] = "{0}.{1}.vmlinuz".format(compose, arch)
123	param_urls["INITRD_URL"] = initrd_url
124	param_urls["INITRD"] = "{0}.{1}.initrd.img".format(compose, arch)
125
126	images.append((flavor, arch, score, param_urls, subvariant, imagetype))
127	return images	101		return images
128		102
129		103
130	def _find_duplicate_jobs(client, param_urls, flavor):	104		def _find_duplicate_jobs(client, param_urls, flavor):
131	"""Check if we have any existing non-cancelled jobs for this	105		"""Check if we have any existing non-cancelled jobs for this
132	ISO/HDD and flavor (checking flavor is important otherwise we'd	106		ISO/HDD and flavor (checking flavor is important otherwise we'd
133	bail on doing the per-ISO jobs for the ISO we use for the	107		bail on doing the per-ISO jobs for the ISO we use for the
134	'universal' tests). ISO/HDD are taken from param_urls dict.	108		'universal' tests). ISO/HDD are taken from param_urls dict.
Show All 16 Lines		124	jobs = [job for job in jobs if
151	job.get('state') != 'cancelled' and job.get('result') != 'user_cancelled']	125		job.get('state') != 'cancelled' and job.get('result') != 'user_cancelled']
152	if jobs:	126		if jobs:
153	logger.info("run_openqa_jobs: Existing jobs found for asset %s flavor %s, and force "	127		logger.info("run_openqa_jobs: Existing jobs found for asset %s flavor %s, and force "
154	"not set! No jobs scheduled.", assetname, flavor)	128		"not set! No jobs scheduled.", assetname, flavor)
155	return jobs	129		return jobs
156	return []	130		return []
157		131
158		132
159	def run_openqa_jobs(param_urls, flavor, arch, subvariant, imagetype, build,	133		def run_openqa_jobs(param_urls, flavor, arch, subvariant, imagetype, build, version,
			garretrazielUnsubmitted Not Done Looks like `version` isn't used anywhere.
			adamwillAuthorUnsubmitted Not Done oh whoops, it's meant to replace the `VERSION` value. I'm kinda uncomfortable with relying on splitting that out of the 'build' (compose ID), it seems safer to pass it explicitly.
160	location, force=False, extraparams=None, resultsdb_job_id=None):	134		location, force=False, extraparams=None, resultsdb_job_id=None):
161	"""# run OpenQA 'isos' job on ISO at urls from 'param_urls', with given arch	135		"""# run OpenQA 'isos' job on ISO at urls from 'param_urls', with
162	and a build identifier. NOTE: 'build' is passed to openQA as	136		given URLs, flavor, arch, subvariant, imagetype, build identifier,
163	BUILD and is later retrieved and parsed by report.py for wiki	137		and version. NOTE: 'build' is passed to openQA as BUILD and is
164	report generation. Returns list of job IDs. If force is False,	138		later retrieved and parsed by report.py for wiki reporting; for
165	jobs will only be scheduled if there are no existing,	139		this to work it should be a productmd/Pungi compose ID. Returns
166	non-cancelled jobs for the same ISO and flavor. If extraparams	140		list of job IDs. If force is False, jobs will only be scheduled if
167	is specified, it must be a dict or something else that can be	141		there are no existing, non-cancelled jobs for the same ISO and
168	combined with a dict using `update`; it adds additional parameters	142		flavor. If extraparams is specified, it must be a dict or
169	(usually openQA variables) to the POST request. When extraparams	143		something else that can be combined with a dict using `update`; it
170	is used, the BUILD value has '-EXTRA' appended to signify that	144		adds additional parameters (usually openQA variables) to the POST
171	this should not be considered a clean test run for the build.	145		request. When extraparams is used, the BUILD value has '-EXTRA'
		146		appended to signify that this should not be considered a clean run
		147		for the build.
172	"""	148		"""
173	logger.info("sending jobs to openQA")	149		logger.info("sending jobs to openQA")
174	# find current and previous releases; these are used to determine	150		# find current and previous releases; these are used to determine
175	# the hard disk image file names for the upgrade tests	151		# the hard disk image file names for the upgrade tests
176	try:	152		try:
177	currrel = str(fedfind.helpers.get_current_release())	153		currrel = str(fedfind.helpers.get_current_release())
178	prevrel = str(int(currrel) - 1)	154		prevrel = str(int(currrel) - 1)
179	except ValueError:	155		except ValueError:
180	# we don't really want to bail entirely if fedfind failed for	156		# we don't really want to bail entirely if fedfind failed for
181	# some reason, let's just run the other tests and set a value	157		# some reason, let's just run the other tests and set a value
182	# that shows what went wrong for the upgrade tests	158		# that shows what went wrong for the upgrade tests
183	currrel = prevrel = "FEDFINDERROR"	159		currrel = prevrel = "FEDFINDERROR"
184		160
185	# starts OpenQA jobs	161		# starts OpenQA jobs
186	params = {	162		params = {
187	'DISTRI': 'fedora',	163		'DISTRI': 'fedora',
188	'VERSION': build.split('-')[1],	164		'VERSION': version,
189	'FLAVOR': flavor,	165		'FLAVOR': flavor,
190	'ARCH': arch if arch != 'armhfp' else 'arm', # openQA does something special when `ARCH = arm`	166		'ARCH': arch if arch != 'armhfp' else 'arm', # openQA does something special when `ARCH = arm`
191	'BUILD': build,	167		'BUILD': build,
192	'LOCATION': location,	168		'LOCATION': location,
193	'CURRREL': currrel,	169		'CURRREL': currrel,
194	'PREVREL': prevrel,	170		'PREVREL': prevrel,
195	'SUBVARIANT': subvariant,	171		'SUBVARIANT': subvariant,
196	'IMAGETYPE': imagetype	172		'IMAGETYPE': imagetype
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Line(s)		198	def jobs_from_compose(location, wanted=WANTED, force=False, extraparams=None, create_resultsdb_job=None):
242	extraparams is passed through to `run_openqa_jobs()`. It can be	218		extraparams is passed through to `run_openqa_jobs()`. It can be
243	a dict (or anything else that can be passed to `dict.update()`)	219		a dict (or anything else that can be passed to `dict.update()`)
244	containing arbitrary extra parameters to be included in the ISO	220		containing arbitrary extra parameters to be included in the ISO
245	post request (usually this will be to specify extra openQA vars).	221		post request (usually this will be to specify extra openQA vars).
246		222
247	create_resultsdb_job controls whether to create job in ResultsDB for	223		create_resultsdb_job controls whether to create job in ResultsDB for
248	this compose.	224		this compose.
249	"""	225		"""
250	location = location.strip('/')	226		try:
251	compose = _get_compose_id(location)	227		rel = fedfind.release.get_release(url=location)
252	logger.debug("Finding images for compose %s in location %s", compose, location)	228		except ValueError:
253	images = _get_images(location, compose, wanted=wanted)	229		raise TriggerException("Could not find a release at {0}".format(location))
		230		logger.debug("Finding images for compose %s in location %s", rel.cid, location)
		231		images = _get_images(rel, wanted=wanted)
254	if len(images) == 0:	232		if len(images) == 0:
255	raise TriggerException("Compose found, but no available images")	233		raise TriggerException("Compose found, but no available images")
256	jobs = []	234		jobs = []
257	univs = {}	235		univs = {}
258		236
259	# create jobs instance in resultsdb if necessary	237		# create jobs instance in resultsdb if necessary
260	rdb_job_id = None	238		rdb_job_id = None
261		239
262	if create_resultsdb_job is None:	240		if create_resultsdb_job is None:
263	create_resultsdb_job = CONFIG.getboolean('report', 'submit_resultsdb')	241		create_resultsdb_job = CONFIG.getboolean('report', 'submit_resultsdb')
264		242
265	if create_resultsdb_job:	243		if create_resultsdb_job:
266	try:	244		try:
267	rdb_instance = ResultsDBapi(CONFIG.get('report', 'resultsdb_url'))	245		rdb_instance = ResultsDBapi(CONFIG.get('report', 'resultsdb_url'))
268	# add link to page with overall results	246		# add link to page with overall results
269	ref_url = "%s/tests/overview?distri=fedora&version=%s&build=%s" % (CONFIG.get('report', 'openqa_url'),	247		ref_url = "%s/tests/overview?distri=fedora&version=%s&build=%s" % (CONFIG.get('report', 'openqa_url'),
270	compose.split('-')[-2], compose)	248		rel.release, rel.cid)
271	job = rdb_instance.create_job(ref_url=ref_url, name=compose)	249		job = rdb_instance.create_job(ref_url=ref_url, name=rel.cid)
272	rdb_job_id = job["id"]	250		rdb_job_id = job["id"]
273	except ResultsDBapiException as e:	251		except ResultsDBapiException as e:
274	logger.error(e)	252		logger.error(e)
275		253
276	# schedule per-image jobs, keeping track of the highest score	254		# schedule per-image jobs, keeping track of the highest score
277	# per arch along the way	255		# per arch along the way
278	for (flavor, arch, score, param_urls, subvariant, imagetype) in images:	256		for (flavor, arch, score, param_urls, subvariant, imagetype) in images:
279	jobs.extend(run_openqa_jobs(param_urls, flavor, arch, subvariant, imagetype, compose,	257		jobs.extend(run_openqa_jobs(param_urls, flavor, arch, subvariant, imagetype, rel.cid,
280	location, force=force, extraparams=extraparams,	258		rel.release, location, force=force, extraparams=extraparams,
281	resultsdb_job_id=rdb_job_id))	259		resultsdb_job_id=rdb_job_id))
282	if score > univs.get(arch, [None, 0])[1]:	260		if score > univs.get(arch, [None, 0])[1]:
283	univs[arch] = (param_urls, score, subvariant, imagetype)	261		univs[arch] = (param_urls, score, subvariant, imagetype)
284		262
285	# now schedule universal jobs	263		# now schedule universal jobs
286	if univs:	264		if univs:
287	for (arch, (param_urls, _, subvariant, imagetype)) in univs.items():	265		for (arch, (param_urls, _, subvariant, imagetype)) in univs.items():
288	# We are assuming that ISO_URL is present in param_urls. This could create problem when	266		# We are assuming that ISO_URL is present in param_urls. This could create problem when
289	# unversal tests are run on product that doesn't have ISO. OTOH, only product without ISO	267		# unversal tests are run on product that doesn't have ISO. OTOH, only product without ISO
290	# is ARM and there would be whole lot of other problems if universal tests are run on ARM.	268		# is ARM and there would be whole lot of other problems if universal tests are run on ARM.
291	logger.info("running universal tests for %s with %s", arch, param_urls['ISO_URL'])	269		logger.info("running universal tests for %s with %s", arch, param_urls['ISO_URL'])
292	jobs.extend(run_openqa_jobs(param_urls, 'universal', arch, subvariant, imagetype,	270		jobs.extend(run_openqa_jobs(param_urls, 'universal', arch, subvariant, imagetype,
293	compose, location, force=force, extraparams=extraparams,	271		rel.cid, rel.release, location, force=force,
294	resultsdb_job_id=rdb_job_id))	272		extraparams=extraparams, resultsdb_job_id=rdb_job_id))
295		273
296	return (compose, jobs)	274		return (rel.cid, jobs)

			Path	Packages
M			scheduler/fedora_openqa_schedule/config.py (274 lines)
M			scheduler/fedora_openqa_schedule/schedule.py (156 lines)

Support testing the semi-official live-respins, more fedfind
ClosedPublic
Actions

Details

Diff Detail

Revision Contents

Diff 2726

scheduler/fedora_openqa_schedule/config.py

scheduler/fedora_openqa_schedule/schedule.py

Support testing the semi-official live-respins, more fedfindClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 2726

scheduler/fedora_openqa_schedule/config.py

scheduler/fedora_openqa_schedule/schedule.py

Support testing the semi-official live-respins, more fedfind
ClosedPublic
Actions