rework wiki report 'conditions', for notification tests
ClosedPublic
Actions

Authored by adamwill on Sep 30 2016, 12:19 AM.

Details

Reviewers

garretraziel
jskladan

Commits

rOPENQA3134833bd4e2: rework wiki report 'conditions', for notification tests

Summary

The big change here is a rework of how 'advanced' entries in
conf_test_suites TESTSUITES work. To report the notification
test results we need to allow for this logic: 'report a pass
for wiki test X only if both testsuite A and testsuite B
passed' - neither desktop_notifications_live passing on its
own nor desktop_notifications_postinstall passing on its own
fully covers any of the wiki test cases, but if *both* pass,
then two of the wiki test cases are covered.

I wanted a consistent and vaguely understandable syntax for
expressing this, so I incorporated a rework of the existing
complex case - 'report a pass only if specific openQA test
modules pass' - into this too. Thus the layout of the dicts
used in the 'complex' case is changed so it's less specific
to the 'module' case. The dicts now use test case names as
their keys and dicts of 'conditions' that must be satisfied
for a pass to be reported as the value for each key. An empty
'conditions' dict just means the same as the 'basic' format:
report this test case as passed if the job passed.

Otherwise, the dict can specify various different types of
conditions that must be satisfied *as well as* the job passing
before a wiki pass will be generated for that wiki test case.
Currently there are 'modules' and 'testsuites' conditions, but
with this design we could add more if needed. The 'modules'
value is an iterable of openQA job modules which must have
passed within the job for a wiki pass to be generated; the
'testsuites' value is an iterable of other test suite names,
each of which must have passed for the same openQA BUILD,
MACHINE and FLAVOR for a wiki pass to be generated.

We adjust the FreeIPA tests to the new design, add entries for
the notifications tests using the new design, and adjust
report.get_passed_testcases to do the work. We fix and update
some comments, and split the now-quite-long code for parsing
the TESTSUITES dicts into its own function, just to make things
a bit more readable.

We also adjust checkwiki for the changed design, enhance it
to handle the 'testsuites' condition and run some specific
checks for it, enhance it to check that all 'modules' named in
a condition are present in the openQA distri (as I realized
that's a check that was missing), and refactor it into separate
test functions, as the giant main() was getting unwieldy,
hard to follow, and increasingly likely to suffer some kind of
variable scope issue.

Test Plan

Use checkwiki and mess around with conf_test_suites
a bit, generate results for a test run that has passed FreeIPA
and notifications tests and make sure you get appropriate
ResTups. I tested reporting the 20160927.n.0 results with this
code and you can see it worked:
https://fedoraproject.org/wiki/Test_Results:Fedora_25_Branched_20160927.n.0_Desktop
It'd probably be good to somehow force a case where one of the
notifications tests passed but the other failed, and check we
do *not* report a pass in this case (fake up the jobs or just
find a compose where that happened or whatever).

Diff Detail

Repository

rOPENQA fedora_openqa

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

adamwill retitled this revision from to rework wiki report 'conditions', for notification tests.Sep 30 2016, 12:19 AM

adamwill updated this object.

adamwill edited the test plan for this revision. (Show Details)

adamwill added reviewers: jskladan, garretraziel.

Herald added a subscriber: tflink. · View Herald TranscriptSep 30 2016, 12:19 AM

ACK, but please, see inline comments.

scheduler/fedora_openqa_schedule/report.py
127	This `try: except` is way too long. I think that it would be better to use `if isinstance`, for example: if isinstance(testsuite, list): return testsuite ... # code from try: except block
139	Use `True/False` instead of `1`.
147	Switch `not module.get() in` to `module.get() not in`.

This revision is now accepted and ready to land.Oct 3 2016, 11:47 AM

scheduler/fedora_openqa_schedule/report.py
127	I generally dislike using `isinstance` in that way, as it's really artificially limiting; it prevents the use of any other iterable. I mean, I guess we could use it here since we control the data structure it's reading from pretty tightly, but it still just rubs me the wrong way. I'll see if I can come up with a better solution.
139	hah, too much perl...

Closed by commit rOPENQA3134833bd4e2: rework wiki report 'conditions', for notification tests (authored by adamwill). · Explain WhyOct 3 2016, 7:17 PM

This revision was automatically updated to reflect the committed changes.

Addressed all three issues in landed version, thanks.

Revision Contents

			Path	Packages
M			scheduler/checkwiki (252 lines)
M			scheduler/fedora_openqa_schedule/conf_test_suites.py (113 lines)
M			scheduler/fedora_openqa_schedule/report.py (161 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	2603	3feee6f		Sep 30 2016, 12:19 AM	★	★
Diff 2	2605	3feee6f	rOPENQA3134833bd4e21819a350ed997233951cc8890f57	Oct 3 2016, 7:17 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
8d97cd234795	0e65661f7847	3feee6f07e6f	Adam Williamson	rework wiki report 'conditions', for notification tests (Show More…)	Sep 29 2016, 9:17 PM

Diff 2605

View Options

scheduler/checkwiki

Show All 18 Lines
19		19
20	"""Script to sanity check the openQA -> wiki results mapping in	20		"""Script to sanity check the openQA -> wiki results mapping in
21	conf_test_suites.py.	21		conf_test_suites.py.
22	"""	22		"""
23		23
24	from __future__ import unicode_literals	24		from __future__ import unicode_literals
25	from __future__ import print_function	25		from __future__ import print_function
26		26
		27		import glob
27	import json	28		import json
		29		import os
28	import subprocess32 as subprocess	30		import subprocess32 as subprocess
29	import sys	31		import sys
30		32
31	from wikitcms import wiki	33		from wikitcms import wiki
32	from wikitcms.exceptions import NotFoundError, TooManyError	34		from wikitcms.exceptions import NotFoundError, TooManyError
33		35
34	from fedora_openqa_schedule.conf_test_suites import (TESTCASES, TESTSUITES)	36		from fedora_openqa_schedule.conf_test_suites import (TESTCASES, TESTSUITES)
35	import fedora_openqa_schedule.report as report	37		import fedora_openqa_schedule.report as report
36		38
37	# wikitcms site object	39		# wikitcms site object
38	SITE = wiki.Wiki()	40		SITE = wiki.Wiki()
39		41
40	# cache of ValidationPage objects	42		# cache of ValidationPage objects
41	PAGES = {}	43		PAGES = {}
42		44
		45		## UTILITY CLASSES AND FUNCTIONS
		46
		47		class FakeClient(object):
		48		"""A fake OpenQA_Client which just returns our fake job dicts. We
		49		need this to handle the 'testsuites' condition in TESTSUITES
		50		entries, which allows a test case to be considered passed only if
		51		the parent test suite and a list of other test suites passed;
		52		get_passed_testcases handles that by querying openQA for other
		53		jobs for the same build and checking their result, but obviously
		54		for our fake jobs, we need a fake openQA 'server'..."""
		55		def __init__(self, jobs):
		56		self.jobs = jobs
		57
		58		def openqa_request(self, method, endpoint, params=None):
		59		"""This is nowhere near a full implementation, it's highly
		60		tuned to what get_passed_testcases needs.
		61		"""
		62		gotjobs = self.jobs
		63		if 'machine' in params:
		64		gotjobs = [job for job in gotjobs if job['settings']['MACHINE'] == params['machine']]
		65		if 'flavor' in params:
		66		gotjobs = [job for job in gotjobs if job['settings']['FLAVOR'] == params['flavor']]
		67		return {'jobs': gotjobs}
		68
		69
		70		def read_templates(distripath):
		71		"""Read in the openQA templates from distripath."""
		72		# Locate template file, check it exists
		73		tempfile = os.path.join(distripath, 'templates')
		74		if not os.path.isfile(tempfile):
		75		sys.exit("Could not find templates file at {0}".format(tempfile))
		76
		77		# Use a perl one-liner to convert the templates to JSON. hold onto
		78		# your hats, folks, this be ugly. also note this executes the
		79		# template file as perl, so make sure you trust it
		80		perl = "use JSON; my $data = do '{0}'; print encode_json($data);".format(tempfile)
		81		cmd = ['perl', '-e', perl]
		82		tempjson = subprocess.check_output(cmd, universal_newlines=True)
		83		# Read the JSON and return it
		84		return json.loads(tempjson)
		85
43	def get_page(testtype):	86		def get_page(testtype):
44	"""Get the page if we didn't yet, cache it, and return from cache."""	87		"""Get the page if we didn't yet, cache it, and return from cache."""
45	if not testtype in PAGES:	88		if not testtype in PAGES:
46	PAGES[testtype] = SITE.get_validation_page(testtype=testtype)	89		PAGES[testtype] = SITE.get_validation_page(testtype=testtype)
47	return PAGES[testtype]	90		return PAGES[testtype]
48		91
49	def cases_from_suite(suite):	92		def cases_from_suite(suite):
50	"""Get test case names from a TESTSUITE."""	93		"""Get test case names from a TESTSUITE."""
51	tscases = TESTSUITES[suite]	94		tscases = TESTSUITES[suite]
52	casenames = []	95		casenames = []
53	try:	96		try:
54	# complex (dict) case	97		# complex (dict) case
55	casenames.extend(tscases.pop('PASS', []))	98		casenames.extend(tscases.keys())
56	for (_, cases) in tscases.items():	99		except AttributeError:
57	casenames.extend(cases)
58	except TypeError:
59	# simple list case	100		# simple list case
60	casenames = tscases	101		casenames = tscases
61	return casenames	102		return casenames
62		103
63	def main():	104		def fake_jobs(templates, modules):
64	"""Main function. Could be split into the individual tests if I	105		"""Create a set of fake passed openQA jobs from JobTemplates and
65	get a minute.	106		the module names in TestSuites (as found by conds_modules_match).
66	"""	107		"""
67	if not len(sys.argv) == 2:	108		fakejobs = []
68	sys.exit("Usage: {0} templatefile".format(sys.argv[0]))	109		for jobtemp in templates['JobTemplates']:
69	tempfile = sys.argv[1]	110		job = {
70		111		'settings': {
71	# read in the openQA templates. hold onto your hats, folks, this be ugly	112		'ARCH': jobtemp['product']['arch'],
72	perl = "use JSON; my $data = do '{0}'; print encode_json($data);".format(tempfile)	113		'FLAVOR': jobtemp['product']['flavor'],
73	cmd = ['perl', '-e', perl]	114		'TEST': jobtemp['test_suite']['name'],
74	tempjson = subprocess.check_output(cmd, universal_newlines=True)	115		'MACHINE': jobtemp['machine']['name'],
75	templates = json.loads(tempjson)	116		# we don't need the BUILD value; we're just going to use
		117		# the current validation pages
		118		'BUILD': 'FAKE',
		119		},
		120		'result': 'passed',
		121		# we use both settings['TEST'] and test, we probably shouldn't...
		122		'test': jobtemp['test_suite']['name'],
		123		# we just pretend every single job had all the modules that
		124		# are used for individual module->result mapping; this is
		125		# harmless for the cases where it's a lie, as get_passed_testcases
		126		# will entirely ignore it except for the TESTSUITES that specify
		127		# the modules
		128		'modules': [{'name': module, 'result': 'passed'} for module in modules]
		129		}
		130		# find the desktop by finding the relevant Product and getting it
		131		# from its settings
		132		for setting in [prod for prod in templates['Products'] if
		133		prod['flavor'] == jobtemp['product']['flavor']][0]['settings']:
		134		if setting['key'] == 'DESKTOP':
		135		job['settings']['DESKTOP'] = setting['value']
		136		fakejobs.append(job)
		137		return fakejobs
76		138
77	# get the test case names from TESTCASES and test suite names from TESTSUITES	139		## CHECK FUNCTIONS
78	tcnames = list(TESTCASES.keys())
79	tsnames = list(TESTSUITES.keys())
80		140
		141		def cases_suites_match():
		142		"""Check all case names referred in TESTSUITES are actually in
		143		TESTCASES and vice versa.
		144		"""
81	failed = 0	145		failed = 0
82		146		# Check all case names referred in TESTSUITES are in TESTCASES
83	# Check all case names referred in TESTSUITES are actually in TESTCASES
84	# also store all case names in TESTSUITES for reverse check	147		# also store all case names in TESTSUITES for reverse check
85	alltscases = set()	148		alltscases = set()
86	for tsname in TESTSUITES:	149		for tsname in TESTSUITES.keys():
87	casenames = cases_from_suite(tsname)	150		casenames = cases_from_suite(tsname)
88	alltscases.update(casenames)	151		alltscases.update(casenames)
89	for casename in casenames:	152		for casename in casenames:
90	if casename not in tcnames:	153		if casename not in TESTCASES.keys():
91	print("Test case {0} listed in test suite {1} is not in TESTCASES!".format(	154		print("Test case {0} listed in test suite {1} is not in TESTCASES!".format(
92	casename, tsname))	155		casename, tsname))
93	failed = 1	156		failed = 1
94		157
95	# check all case names in TESTCASES are used in TESTSUITES	158		# check all case names in TESTCASES are used in TESTSUITES
96	for casename in tcnames:	159		for casename in TESTCASES.keys():
97	if casename not in alltscases:	160		if casename not in alltscases:
98	print("Test case {0} from TESTCASES is never used in TESTSUITES!".format(casename))	161		print("Test case {0} from TESTCASES is never used in TESTSUITES!".format(casename))
		162		failed = 1
99		163
100	# check all test suites from templates are in TESTSUITES, and vice versa	164		return failed
101	# except for ones we know are OK (e.g. support_server)	165
		166		def templates_suites_match(templates):
		167		"""Check all test suites from templates are in TESTSUITES, and
		168		vice versa, except for ones we know are OK (e.g. support_server).
		169		"""
		170		failed = 0
102	tempwhite = set(('support_server',))	171		tempwhite = set(('support_server',))
103	tempsuites = [suite['name'] for suite in templates['TestSuites']]	172		tempsuites = [suite['name'] for suite in templates['TestSuites']]
104	onlytemp = set(tempsuites) - set(tsnames) - tempwhite	173		onlytemp = set(tempsuites) - set(TESTSUITES.keys()) - tempwhite
105	for suite in onlytemp:	174		for suite in onlytemp:
106	failed = 1	175		failed = 1
107	print("Test suite {0} is in templates, but not in TESTSUITES!".format(suite))	176		print("Test suite {0} is in templates, but not in TESTSUITES!".format(suite))
108		177
109	onlysuites = set(tsnames) - set(tempsuites)	178		onlysuites = set(TESTSUITES.keys()) - set(tempsuites)
110	for suite in onlysuites:	179		for suite in onlysuites:
111	failed = 1	180		failed = 1
112	print("Test suite {0} is in TESTSUITES, but not in templates!".format(suite))	181		print("Test suite {0} is in TESTSUITES, but not in templates!".format(suite))
113		182
114	# Now try and check that all our wiki mappings are actually correct. This	183		return failed
115	# is kinda involved. First we're going to create a fake job for each	184
116	# JobTemplate from templates, as if we had a pass for every single one.	185		def conds_testsuites_match():
117	# To handle the complex TESTSUITE case where we map individual passed test	186		"""For TESTSUITES entries which use the 'testsuites' condition,
118	# modules to passed test cases, we need to extract all the relevant module	187		check all listed suite names are in TESTSUITES, and warn if there
119	# names from TESTSUITES...	188		is no equivalent reverse mapping (as usually we'd expect one).
120	modules = set()	189		"""
121	for cases in TESTSUITES.values():	190		failed = 0
		191		for (tsname, cases) in TESTSUITES.items():
122	try:	192		try:
123	modules.update(cases.keys())	193		for (tcname, conds) in cases.items():
		194		for othertsname in conds.get('testsuites', []):
		195		othersuite = TESTSUITES.get(othertsname)
		196		if not othersuite:
		197		print("Test suite {0} listed as a cond in case {1} in suite {2} "
		198		"not found in TESTSUITES!".format(othertsname, tcname, tsname))
		199		failed = 1
		200		othercase = othersuite.get(tcname, {})
		201		if tsname not in othercase.get('testsuites', []):
		202		print("Warning: suite {0} case {1} has suite {2} as cond, but suite {2} "
		203		"case {1} does not have suite {0} as cond".format(
		204		tsname, tcname, othertsname))
124	except AttributeError:	205		except AttributeError:
125	pass	206		pass
		207		return failed
126		208
127	fakejobs = []	209		def conds_modules_match(distripath):
128	for jobtemp in templates['JobTemplates']:	210		"""Check all modules listed in TESTSUITES entries which use the
129	job = {	211		'modules' condition actually exist in the distri, and return the
130	'settings': {	212		module names for the wiki check to use.
131	'ARCH': jobtemp['product']['arch'],	213		"""
132	'FLAVOR': jobtemp['product']['flavor'],	214		failed = 0
133	'TEST': jobtemp['test_suite']['name'],	215		distrimods = glob.glob(os.path.join(distripath, 'tests', '*.pm'))
134	# we don't need the BUILD value; we're just going to use	216		distrimods = [os.path.basename(mod) for mod in distrimods]
135	# the current validation pages	217		distrimods = [mod.split('.pm')[0] for mod in distrimods]
136	'BUILD': 'FAKE',	218		modules = set()
137	},	219		for (suite, cases) in TESTSUITES.items():
138	'result': 'passed',	220		try:
139	# we use both settings['TEST'] and test, we probably shouldn't...	221		for (case, conds) in cases.items():
140	'test': jobtemp['test_suite']['name'],	222		for module in conds.get('modules', []):
141	# we just pretend every single job had all the modules that	223		modules.add(module)
142	# are used for individual module->result mapping; this is	224		if module not in distrimods:
143	# harmless for the cases where it's a lie, as get_passed_testcases	225		print("Module {0} listed in case {1} in suite {2} not found in "
144	# will entirely ignore it except for the TESTSUITES that specify	226		"distri!".format(module, case, suite))
145	# the modules	227		failed = 1
146	'modules': [{'name': module, 'result': 'passed'} for module in modules]	228		modules.update(conds.get('modules', []))
147	}	229		except AttributeError:
148	# find the desktop by finding the relevant Product and getting it	230		pass
149	# from its settings	231		return (failed, modules)
150	for setting in [prod for prod in templates['Products'] if
151	prod['flavor'] == jobtemp['product']['flavor']][0]['settings']:
152	if setting['key'] == 'DESKTOP':
153	job['settings']['DESKTOP'] = setting['value']
154	fakejobs.append(job)
155		232
		233		def check_wiki_mapping(templates, modules):
		234		"""Produce wikitcms ResTups for a set of fake passed jobs produced
		235		by fake_jobs, then check we got a ResTup for every test case in
		236		TESTCASES, and we can find a ResultRow and env for every generated
		237		ResTup in the current validation event wiki pages.
		238		"""
		239		failed = 0
		240		fakejobs = fake_jobs(templates, modules)
156	# Now we have a bunch of fake 'passed jobs', we can ask report.py to	241		# Now we have a bunch of fake 'passed jobs', we can ask report.py to
157	# give us ResTups for them	242		# give us ResTups for them
158	restups = report.get_passed_testcases(fakejobs)	243		# Create an instance of our fake 'openQA client' with the fake jobs
		244		fakeclient = FakeClient(fakejobs)
		245		# Call get_passed_testcases with the fake jobs and client
		246		restups = report.get_passed_testcases(fakejobs, fakeclient)
		247
		248		# Catch if there are any test cases we don't produce a single
		249		# result for, even with a full set of 'passed jobs'
		250		restupcases = set([restup.testcase for restup in restups])
		251		for case in set(TESTCASES.keys()) - restupcases:
		252		print("No wiki result generated for test case {0}!".format(case))
		253
159	for restup in restups:	254		for restup in restups:
160	# this will get us the current validation page for the testtype	255		# this will get us the current validation page for the testtype
161	page = get_page(restup.testtype)	256		page = get_page(restup.testtype)
162	# now we use find_resultrow - which is what report_validation_results	257		# now we use find_resultrow - which is what report_validation_results
163	# uses - with the appropriate values from the ResTup	258		# uses - with the appropriate values from the ResTup
164	try:	259		try:
165	row = page.find_resultrow(restup.testcase, restup.section, restup.testname, restup.env)	260		row = page.find_resultrow(restup.testcase, restup.section, restup.testname, restup.env)
166	# if we got a ResultRow, check the environment from the ResTup	261		# if we got a ResultRow, check the environment from the ResTup
Show All 12 Lines		271	if len(cands) > 1:
179	failed = 1	274		failed = 1
180	except NotFoundError:	275		except NotFoundError:
181	print("No ResultRow found for {0}".format(restup))	276		print("No ResultRow found for {0}".format(restup))
182	failed = 1	277		failed = 1
183	except TooManyError:	278		except TooManyError:
184	print("Too many ResultRows found for {0}".format(restup))	279		print("Too many ResultRows found for {0}".format(restup))
185	failed = 1	280		failed = 1
186		281
		282		return failed
		283
		284		## MAIN FUNCTIONS
		285
		286		def main():
		287		"""Main function. Just reads the templates and runs each test."""
		288		if not len(sys.argv) == 2:
		289		sys.exit("Usage: {0} distripath".format(sys.argv[0]))
		290		distripath = sys.argv[1]
		291		templates = read_templates(distripath)
		292		# Ongoing 'did we fail yet?' indicator
		293		failed = 0
		294		# Run the checks
		295		if cases_suites_match():
		296		failed = 1
		297		if templates_suites_match(templates):
		298		failed = 1
		299		if conds_testsuites_match():
		300		failed = 1
		301		(ret, modules) = conds_modules_match(distripath)
		302		if ret:
		303		failed = 1
		304		if check_wiki_mapping(templates, modules):
		305		failed = 1
		306		# Exit appropriately
187	sys.exit(failed)	307		sys.exit(failed)
188		308
189	if __name__ == '__main__':	309		if __name__ == '__main__':
190	try:	310		try:
191	main()	311		main()
192	except KeyboardInterrupt:	312		except KeyboardInterrupt:
193	sys.stderr.write("Interrupted, exiting...\n")	313		sys.stderr.write("Interrupted, exiting...\n")
194	sys.exit(1)	314		sys.exit(1)

View Options

scheduler/fedora_openqa_schedule/conf_test_suites.py

Show First 20 Lines • Show All 374 Lines • ▼ Show 20 Line(s)		373	"QA:Testcase_desktop_browser": {
375	"type": "Desktop",	375		"type": "Desktop",
376	"section": "Release-blocking desktops: <b>x86 / x86_64</b>",	376		"section": "Release-blocking desktops: <b>x86 / x86_64</b>",
377	},	377		},
378	"QA:Testcase_desktop_update_graphical": {	378		"QA:Testcase_desktop_update_graphical": {
379	"env": "$SUBVARIANT$",	379		"env": "$SUBVARIANT$",
380	"type": "Desktop",	380		"type": "Desktop",
381	"section": "Release-blocking desktops: <b>x86 / x86_64</b>",	381		"section": "Release-blocking desktops: <b>x86 / x86_64</b>",
382	},	382		},
		383		"QA:Testcase_desktop_update_notification": {
		384		"env": "$SUBVARIANT$",
		385		"type": "Desktop",
		386		"section": "Release-blocking desktops: <b>x86 / x86_64</b>",
		387		},
		388		"QA:Testcase_desktop_error_checks": {
		389		"env": "$SUBVARIANT$",
		390		"type": "Desktop",
		391		"section": "Release-blocking desktops: <b>x86 / x86_64</b>",
		392		},
383	"QA:Testcase_Server_firewall_default": {	393		"QA:Testcase_Server_firewall_default": {
384	"env": "x86",	394		"env": "x86",
385	"type": "Server",	395		"type": "Server",
386	},	396		},
387	"QA:Testcase_Server_cockpit_default": {	397		"QA:Testcase_Server_cockpit_default": {
388	"env": "x86",	398		"env": "x86",
389	"type": "Server",	399		"type": "Server",
390	},	400		},
Show All 22 Lines		422	TESTCASES_RESULTSDB_EXTRADATA = {
413	"QA:Testcase_partitioning_guided_shrink": {	423		"QA:Testcase_partitioning_guided_shrink": {
414	"filesystem": "$FS$", # needed to convert test_fs to only one testcase	424		"filesystem": "$FS$", # needed to convert test_fs to only one testcase
415	},	425		},
416	}	426		}
417		427
418	TESTSUITES = {	428		TESTSUITES = {
419	# each entry in this dict is named for an openQA test suite, and	429		# each entry in this dict is named for an openQA test suite, and
420	# represents the Wikitcms test cases that passed if that openQA	430		# represents the Wikitcms test cases that passed if that openQA
421	# test suite passed. The entries can be simple lists of test case	431		# test suite passed or soft failed (from here on, read 'passed' to
422	# names, or they can be dicts. In the dict form, the special dict	432		# mean 'passed or soft failed'). The value can either be a simple
423	# entry named 'PASS' is a list of test cases that passed if the	433		# list of test case names (each must be an entry in the TESTCASES
424	# overall openQA job passed, as in the simple list form. The other	434		# dict), or a dict.
425	# dict entries are named for individual openQA test modules, and	435		#
426	# are lists of test cases that passed if the overall openQA job	436		# In the dict form, the dict's keys are the test case names, and
427	# passed and that specific test module passed. This handles	437		# and the values must all be dicts. These dicts indicate extra
428	# test suites like realmd_join_cockpit, which has an optional	438		# conditions that must be met before the test case will be counted
429	# module that covers QA:Testcase_FreeIPA_web_ui ; if that module	439		# as 'passed'. An empty dict works just like the simple list case:
430	# passes we want to report a pass for that test case, if the job	440		# that test case will be considered 'passed' as long as the test
431	# passes but that module fails we do not want to report a pass.	441		# suite passed.
		442		#
		443		# The dict may contain a 'testsuites' key, whose value must be an
		444		# iterable. If so, the test case will only be considered 'passed'
		445		# if both the test suite being considered passed and all the
		446		# other test suites named in the iterable passed, for the same
		447		# build, machine and flavor. e.g.:
		448		#
		449		# "testsuite_a": {
		450		# "testcase_a": {},
		451		# "testcase_b": {"testsuites": ["testsuite_b"],},
		452		# },
		453		#
		454		# In this case, if testsuite_a passed, testcase_a would be counted
		455		# as passed, but testcase_b would only be counted as passed if
		456		# testsuite_b had also already passed. This covers cases like
		457		# desktop notifications, where there are two wiki test cases that
		458		# can be considered passed only if both desktop_notifications_live
		459		# and desktop_notifications_postinstall passed.
		460		#
		461		# NOTE: unless we can be sure one test suite will always run after
		462		# the other, we must include entries for both test suites,
		463		# because we will process the entry for whichever test suite
		464		# passes first and generate no 'pass' because the other one is not
		465		# done yet, then when the other one passes we will process its
		466		# entry and generate a pass so long as the first one also passed;
		467		# if we don't have entries for both test suites and the 'wrong
		468		# one' passes first, we will miss the result.
		469		#
		470		# The dict may contain a 'modules' key, whose value must be an
		471		# iterable. If so, the test case will only be considered 'passed'
		472		# if the overall openQA job passed and each of the individual
		473		# openQA test modules named in the iterable were present in the
		474		# openQA job and passed. e.g.:
		475		#
		476		# "testsuite_c": {
		477		# "testcase_c": {"modules": ["module_1"],},
		478		# },
		479		#
		480		# In this case, testcase_c would only be considered 'passed' if
		481		# both testsuite_c passed and it contained a test module named
		482		# module_1 and that module passed. This handles test suites like
		483		# realmd_join_cockpit, which has an optional module that covers
		484		# QA:Testcase_FreeIPA_web_ui ; if that module passes we want to
		485		# report a pass for that test case, if the job passes but that
		486		# module fails we do not want to report a pass.
432	"install_default": [	487		"install_default": [
433	"QA:Testcase_Boot_default_install",	488		"QA:Testcase_Boot_default_install",
434	"QA:Testcase_install_to_VirtIO",	489		"QA:Testcase_install_to_VirtIO",
435	"QA:Testcase_partitioning_guided_empty",	490		"QA:Testcase_partitioning_guided_empty",
436	"QA:Testcase_Anaconda_User_Interface_Graphical",	491		"QA:Testcase_Anaconda_User_Interface_Graphical",
437	"QA:Testcase_Anaconda_user_creation",	492		"QA:Testcase_Anaconda_user_creation",
438	],	493		],
439	"install_default_upload": [	494		"install_default_upload": [
▲ Show 20 Lines • Show All 306 Lines • ▼ Show 20 Line(s)		800	"desktop_terminal": [
746	"QA:Testcase_desktop_terminal",	801		"QA:Testcase_desktop_terminal",
747	],	802		],
748	"desktop_browser": [	803		"desktop_browser": [
749	"QA:Testcase_desktop_browser",	804		"QA:Testcase_desktop_browser",
750	],	805		],
751	"desktop_update_graphical": [	806		"desktop_update_graphical": [
752	"QA:Testcase_desktop_update_graphical",	807		"QA:Testcase_desktop_update_graphical",
753	],	808		],
		809		"desktop_notifications_live": {
		810		"QA:Testcase_desktop_update_notification": {
		811		"testsuites": ["desktop_notifications_postinstall"],
		812		},
		813		"QA:Testcase_desktop_error_checks": {
		814		"testsuites": ["desktop_notifications_postinstall"],
		815		},
		816		},
		817		"desktop_notifications_postinstall": {
		818		"QA:Testcase_desktop_update_notification": {
		819		"testsuites": ["desktop_notifications_live"],
		820		},
		821		"QA:Testcase_desktop_error_checks": {
		822		"testsuites": ["desktop_notifications_live"],
		823		},
		824		},
754	"server_firewall_default": [	825		"server_firewall_default": [
755	"QA:Testcase_Server_firewall_default",	826		"QA:Testcase_Server_firewall_default",
756	],	827		],
757	"server_cockpit_default": [	828		"server_cockpit_default": [
758	"QA:Testcase_Server_cockpit_default",	829		"QA:Testcase_Server_cockpit_default",
759	],	830		],
760	"server_cockpit_basic": [	831		"server_cockpit_basic": [
761	"QA:Testcase_Server_cockpit_basic",	832		"QA:Testcase_Server_cockpit_basic",
762	],	833		],
763	"server_role_deploy_database_server": [	834		"server_role_deploy_database_server": [
764	"QA:Testcase_Server_role_deploy",	835		"QA:Testcase_Server_role_deploy",
765	],	836		],
766	"server_database_client": [	837		"server_database_client": [
767	"QA:Testcase_database_server_remote_client",	838		"QA:Testcase_database_server_remote_client",
768	],	839		],
769	"realmd_join_cockpit": {	840		"realmd_join_cockpit": {
770	"PASS": [	841		"QA:Testcase_realmd_join_cockpit": {},
771	"QA:Testcase_realmd_join_cockpit",	842		"QA:Testcase_FreeIPA_realmd_login": {},
772	"QA:Testcase_FreeIPA_realmd_login",	843		"QA:Testcase_domain_client_authenticate": {},
773	"QA:Testcase_domain_client_authenticate",	844		"QA:Testcase_FreeIPA_web_ui": {
774	],	845		"modules": ["freeipa_webui"],
775	"freeipa_webui": [	846		},
776	"QA:Testcase_FreeIPA_web_ui",	847		"QA:Testcase_FreeIPA_password_change": {
777	],	848		"modules": ["freeipa_password_change"],
778	"freeipa_password_change": [	849		},
779	"QA:Testcase_FreeIPA_password_change",
780	],
781	},	850		},
782	"realmd_join_sssd": [	851		"realmd_join_sssd": [
783	"QA:Testcase_realmd_join_sssd",	852		"QA:Testcase_realmd_join_sssd",
784	"QA:Testcase_FreeIPA_realmd_login",	853		"QA:Testcase_FreeIPA_realmd_login",
785	"QA:Testcase_domain_client_authenticate",	854		"QA:Testcase_domain_client_authenticate",
786	],	855		],
787	"install_iscsi": [	856		"install_iscsi": [
788	"QA:Testcase_install_to_iSCSI_no_authentication",	857		"QA:Testcase_install_to_iSCSI_no_authentication",
789	"QA:Testcase_Anaconda_User_Interface_Graphical",	858		"QA:Testcase_Anaconda_User_Interface_Graphical",
790	"QA:Testcase_Anaconda_user_creation",	859		"QA:Testcase_Anaconda_user_creation",
791	"QA:Testcase_Package_Sets_Minimal_Package_Install",	860		"QA:Testcase_Package_Sets_Minimal_Package_Install",
792	],	861		],
793	}	862		}

View Options

scheduler/fedora_openqa_schedule/report.py

Show First 20 Lines • Show All 42 Lines • ▼ Show 20 Line(s)
43	logger = logging.getLogger(__name__)	43		logger = logging.getLogger(__name__)
44		44
45		45
46	class LoginError(Exception):	46		class LoginError(Exception):
47	"""Raised when cannot log in to wiki to submit results."""	47		"""Raised when cannot log in to wiki to submit results."""
48	pass	48		pass
49		49
50		50
51	def _uniqueres_replacements(job, uniqueres):	51		def _uniqueres_replacements(job, tcdict):
52	"""Replace some magic values in the 'uniqueres' dict with test job	52		"""Replace some magic values in the 'tcdict' dict with test job
53	properties; this is to distinguish between environments for a test	53		properties; this is to distinguish between environments for a test
54	case, or tests with the same test case but different test names,	54		case, or tests with the same test case but different test names,
55	and so on. Returns nothing because it modifies the uniqueres dict	55		and so on. Returns a new dict with the modifications.
56	in place.
57	"""	56		"""
58	arch = job['settings']['ARCH']	57		arch = job['settings']['ARCH']
59	flavor = job['settings']['FLAVOR']	58		flavor = job['settings']['FLAVOR']
60	fs = job['settings']['TEST'].split('_')[-1]	59		fs = job['settings']['TEST'].split('_')[-1]
61	desktop = job['settings'].get('DESKTOP', '')	60		desktop = job['settings'].get('DESKTOP', '')
62	try:	61		try:
63	(subvariant, imagetype, _) = flavor.split('-')	62		(subvariant, imagetype, _) = flavor.split('-')
64	except ValueError:	63		except ValueError:
65	# the above fails when the flavor is 'universal'. This is just	64		# the above fails when the flavor is 'universal'. This is just
66	# to avoid crashing, these values should never be used	65		# to avoid crashing, these values should never be used
67	subvariant = imagetype = 'universal'	66		subvariant = imagetype = 'universal'
68	imagetype = imagetype.replace('boot', 'netinst')	67		imagetype = imagetype.replace('boot', 'netinst')
69	subvariant = subvariant.replace('_Base', '')	68		subvariant = subvariant.replace('_Base', '')
70	subvariant_or_arm = "ARM" if arch == "arm" else subvariant	69		subvariant_or_arm = "ARM" if arch == "arm" else subvariant
71	if 'UEFI' in job['settings']:	70		if 'UEFI' in job['settings']:
72	uefi = 'UEFI'	71		uefi = 'UEFI'
73	bootmethod = 'x86_64 UEFI'	72		bootmethod = 'x86_64 UEFI'
74	else:	73		else:
75	uefi = arch	74		uefi = arch
76	bootmethod = 'x86_64 BIOS'	75		bootmethod = 'x86_64 BIOS'
77	role = ''	76		role = ''
78	if 'role_deploy_' in job['test']:	77		if 'role_deploy_' in job['test']:
79	role = job['test'].split('role_deploy_')[1]	78		role = job['test'].split('role_deploy_')[1]
80		79
		80		# We effectively deep copy the `tcdict` dict here; if we just modified it directly
		81		# we'd actually be changing it in TESTCASES, so the results for later jobs in this run
		82		# with the same testcase (but a different environment, section or testname) would read
		83		# the modified values and be messed up
81	changed = {}	84		changed = {}
82	for key, value in uniqueres.iteritems():	85		for key, value in tcdict.iteritems():
83	value = value.replace('$RUNARCH_OR_UEFI$', uefi)	86		value = value.replace('$RUNARCH_OR_UEFI$', uefi)
84	value = value.replace('$FS$', fs)	87		value = value.replace('$FS$', fs)
85	value = value.replace('$RUNARCH$', arch)	88		value = value.replace('$RUNARCH$', arch)
86	value = value.replace('$BOOTMETHOD$', bootmethod)	89		value = value.replace('$BOOTMETHOD$', bootmethod)
87	value = value.replace('$SUBVARIANT$', subvariant)	90		value = value.replace('$SUBVARIANT$', subvariant)
88	value = value.replace('$IMAGETYPE$', imagetype)	91		value = value.replace('$IMAGETYPE$', imagetype)
89	value = value.replace('$DESKTOP$', desktop)	92		value = value.replace('$DESKTOP$', desktop)
90	value = value.replace('$SUBVARIANT_OR_ARM$', subvariant_or_arm)	93		value = value.replace('$SUBVARIANT_OR_ARM$', subvariant_or_arm)
91	value = value.replace('$ROLE$', role)	94		value = value.replace('$ROLE$', role)
92	changed[key] = value	95		changed[key] = value
93		96
94	return changed	97		return changed
95		98
96		99
97	def get_passed_testcases(jobs):	100		def _get_passed_tcnames(job, composeid, client=None):
		101		"""Given a job dict, find the corresponding entry from TESTSUITES
		102		and return the test case names that are considered to have passed.
		103		This is splitting out a chunk of logic from the middle of
		104		get_passed_testcases to make it shorter and more readable.
		105		composeid is the compose ID, passed in from get_passed_testcases.
		106		client can be an OpenQA_Client instance (to save us instantiating
		107		a new one, and also so checkwiki can use a fake one); it's used
		108		for entries that use the 'testsuites' condition, we have to go get
		109		the appropriate results from openQA and check them, we can't
		110		assume they're already in the jobs list that get_passed_testcases
		111		got.
		112		"""
		113		tsname = job['settings']['TEST']
		114		# There usually ought to be an entry in TESTSUITES for all
		115		# tests, but just in case someone messed up, let's be safe
		116		if tsname not in conf_test_suites.TESTSUITES:
		117		logger.warning("No TESTSUITES entry found for test {0}!".format(tsname))
		118		return []
		119
		120		passed = []
		121		testsuite = conf_test_suites.TESTSUITES[tsname]
		122		# testsuite can be simply a list of test case names - in which case all those test
		123		# cases are considered 'passed' if the openQA job overall 'passed' - or a dict whose
		124		# keys are test case names and whose values are dicts of conditions indicating when
		125		# that test case should be considered as 'passed'. Here, we handle the conditions for
		126		# this case. See conf_test_suites for more details.
		127		isdict = None
			garretrazielUnsubmitted Not Done This `try: except` is way too long. I think that it would be better to use `if isinstance`, for example: if isinstance(testsuite, list): return testsuite ... # code from try: except block
			adamwillAuthorUnsubmitted Not Done I generally dislike using `isinstance` in that way, as it's really artificially limiting; it prevents the use of any other iterable. I mean, I guess we could use it here since we control the data structure it's reading from pretty tightly, but it still just rubs me the wrong way. I'll see if I can come up with a better solution.
		128		try:
		129		# check if this is dict case
		130		testsuite.items()
		131		isdict = True
		132		except AttributeError:
		133		isdict = False
		134		if isdict:
		135		# dict case.
		136		for (testcase, conds) in testsuite.items():
		137		if not conds:
		138		# life is easy!
		139		passed.append(testcase)
			garretrazielUnsubmitted Not Done Use `True/False` instead of `1`.
			adamwillAuthorUnsubmitted Not Done hah, too much perl...
		140		continue
		141
		142		# otherwise, handle the conditions...
		143		# modules: the test case is only 'passed' if all listed openQA job modules
		144		# are present in the job and passed
		145		if 'modules' in conds:
		146		tcpass = True
		147		for modname in conds['modules']:
			garretrazielUnsubmitted Not Done Switch `not module.get() in` to `module.get() not in`.
		148		try:
		149		# find the matching test module in the job's list (note this
		150		# assumes we don't run the same module multiple times, which
		151		# is something upstream has recently started allowing...)
		152		module = [module for module in job['modules'] if
		153		module['name'] == modname][0]
		154		if module.get('result', '') not in ('passed', 'softfailed'):
		155		tcpass = False
		156		break
		157		except IndexError:
		158		tcpass = False
		159		logger.warning("Did not find module %s in job data!", modname)
		160		break
		161		if not tcpass:
		162		# skip to next testsuite item
		163		continue
		164
		165		# test suites: the test case is only 'passed' if there are jobs for all listed
		166		# test suites for the same build, machine and flavor, and they all passed
		167		if 'testsuites' in conds:
		168		if not client:
		169		client = OpenQA_Client()
		170		# Ideally we could query on multiple test names - I'll send a PR for that.
		171		# As we can't, let's not do multiple single queries, let's just get all
		172		# results for the same build, machine and flavor and filter ourselves...
		173		params = {
		174		'build': composeid,
		175		'machine': job['settings']['MACHINE'],
		176		'flavor': job['settings']['FLAVOR'],
		177		'latest': 'true',
		178		}
		179		candjobs = client.openqa_request('GET', 'jobs', params=params)['jobs']
		180		_jobs = [_job for _job in candjobs if _job['test'] in conds['testsuites']]
		181		if len(_jobs) != len(conds['testsuites']):
		182		continue
		183		if any(_job['result'] not in ('passed', 'softfailed') for _job in _jobs):
		184		continue
		185
		186		# we only get here if all the conditions are satisfied
		187		passed.append(testcase)
		188
		189		else: # isdict - this is the simple list case.
		190		passed = testsuite
		191
		192		return passed
		193
		194		def get_passed_testcases(jobs, client=None):
98	"""Given an iterable of job dicts - any waiting, filtering and so	195		"""Given an iterable of job dicts - any waiting, filtering and so
99	on is assumed to have already happened - returns a list of	196		on is assumed to have already happened - returns a list of
100	wikitcms ResTuples derived from any passed tests.	197		wikitcms ResTuples derived from any passed tests. client can be an
		198		OpenQA_Client instance (to save us instantiating a new one, and
		199		also so checkwiki can use a fake one); it's passed through to
		200		_get_passed_tcnames, which actually uses it.
101	"""	201		"""
102	passed_testcases = set()	202		passed_testcases = set()
103	for job in jobs:	203		for job in jobs:
104	if job['result'] in ('passed', 'softfailed'):	204		if job['result'] in ('passed', 'softfailed'):
105	# it's wikitcms' job to take a compose ID and figure out	205		# it's wikitcms' job to take a compose ID and figure out
106	# what the validation event for it is.	206		# what the validation event for it is.
107	composeid = job['settings']['BUILD']	207		composeid = job['settings']['BUILD']
108	if composeid.endswith('EXTRA'):	208		if composeid.endswith('EXTRA'):
109	# this is a 'dirty' test run with extra parameters	209		# this is a 'dirty' test run with extra parameters
110	# (usually a test with an updates.img), we never want	210		# (usually a test with an updates.img), we never want
111	# to report results for these	211		# to report results for these
112	logger.debug("Job was run with extra params! Will not report")	212		logger.debug("Job was run with extra params! Will not report")
113	continue	213		continue
114	tsname = job['settings']['TEST']	214		# find the TESTSUITES entry for the job and parse it to
115	# There usually ought to be an entry in TESTSUITES for all	215		# get a list of passed test case names (TESTCASES keys)
116	# tests, but just in case someone messed up, let's be safe	216		passed = _get_passed_tcnames(job, composeid, client)
117	if tsname not in conf_test_suites.TESTSUITES:
118	logger.warning("No TESTSUITES entry found for test {0}!".format(tsname))
119	continue
120	passed = []
121	testsuite = conf_test_suites.TESTSUITES[tsname]
122	# testsuite can be simply a list of test case names - in which case all those test
123	# cases are considered 'passed' if the openQA job overall 'passed' - or a dict of
124	# lists of test case names. In the dict form, there is one special key, 'PASS'; the
125	# 'PASS' list works just like the simple list form, all test cases in that list
126	# 'passed' if the openQA job overall passed. Any other keys are the names of
127	# individual openQA test modules, which should be included in the job as non-fatal
128	# modules; if that module passed within the job, the test cases in the list 'passed'.
129	try:
130	# this is the dict case. first take all test cases from the 'PASS' list...
131	passed.extend(testsuite.pop('PASS', []))
132	# now check each individual test module status
133	for (modname, cases) in testsuite.items():
134	try:
135	# find the matching test module in the job's list (note this
136	# assumes we don't run the same module multiple times, which
137	# is something upstream has recently started allowing...)
138	module = [module for module in job['modules'] if module['name'] == modname][0]
139	except IndexError:
140	logger.warning("Did not find expected module %s in job data!", modname)
141	continue
142	if module.get('result', '') in ('passed', 'softfailed'):
143	passed.extend(cases)
144	except TypeError:
145	# this is the simple list case.
146	passed = testsuite
147	for testcase in passed:	217		for testcase in passed:
148	# skip with warning if testcase is not in TESTCASES	218		# skip with warning if testcase is not in TESTCASES
149	if not testcase in conf_test_suites.TESTCASES:	219		if not testcase in conf_test_suites.TESTCASES:
150	logger.warning("No TESTCASES entry found for {0}!".format(testcase))	220		logger.warning("No TESTCASES entry found for {0}!".format(testcase))
151	continue	221		continue
152	# each 'testsuite' is a list using testcase names to indicate which Wikitcms tests	222		# Each testcase name now in the `passed` list is the name of a dict in the
153	# have passed if this job passes. Each testcase name is the name of a dict in the
154	# TESTCASES dict-of-dicts which more precisely identifies the 'test instance' (when	223		# TESTCASES dict-of-dicts which more precisely identifies the 'test instance' (when
155	# there is more than one for a testcase) and environment for which the result	224		# there is more than one for a testcase) and environment for which the result
156	# should be filed. We do a deepcopy here because otherwise the dict in TESTCASES	225		# should be filed.
157	# itself is modified in-place and other jobs for the same testcase (but a different
158	# environment, section or testname) will read the modified values and be messed up.
159		226
160	# replace $FOO$ values in uniqueres	227		# create new dict based on the testcase dict with $FOO$ values replaced
161	uniqueres = _uniqueres_replacements(job, conf_test_suites.TESTCASES[testcase])	228		uniqueres = _uniqueres_replacements(job, conf_test_suites.TESTCASES[testcase])
162	result = ResTuple(	229		result = ResTuple(
163	testtype=uniqueres['type'], testcase=testcase,	230		testtype=uniqueres['type'], testcase=testcase,
164	section=uniqueres.get('section'), testname=uniqueres.get('name', ''),	231		section=uniqueres.get('section'), testname=uniqueres.get('name', ''),
165	env=uniqueres.get('env', ''), status='pass', bot=True, cid=composeid)	232		env=uniqueres.get('env', ''), status='pass', bot=True, cid=composeid)
166	passed_testcases.add(result)	233		passed_testcases.add(result)
167		234
168	return sorted(list(passed_testcases), key=attrgetter('testcase'))	235		return sorted(list(passed_testcases), key=attrgetter('testcase'))
169		236
170		237
171	def wiki_report(wiki_url, jobs=None, build=None, do_report=None):	238		def wiki_report(wiki_url, jobs=None, build=None, do_report=None):
172	"""Report results from openQA jobs to Wikitcms. Either jobs (an	239		"""Report results from openQA jobs to Wikitcms. Either jobs (an
173	iterable of job IDs) or build (an openQA BUILD string, usually a	240		iterable of job IDs) or build (an openQA BUILD string, usually a
174	Fedora compose ID) is required (if neither is specified, the	241		Fedora compose ID) is required (if neither is specified, the
175	openQA client will raise TypeError). If do_report is False, will	242		openQA client will raise TypeError). If do_report is False, will
176	just print out the python-wikitcms ResTups for inspection. If	243		just print out the python-wikitcms ResTups for inspection. If
177	do_report is None, will read the setting from the config file; if	244		do_report is None, will read the setting from the config file; if
178	no config file is present, default is True.	245		no config file is present, default is True.
179	"""	246		"""
180	client = OpenQA_Client()	247		client = OpenQA_Client()
181	jobs = client.get_jobs(jobs=jobs, build=build)	248		jobs = client.get_jobs(jobs=jobs, build=build)
182	passed_testcases = get_passed_testcases(jobs)	249		passed_testcases = get_passed_testcases(jobs, client)
183	logger.info("passed testcases: %s", passed_testcases)	250		logger.info("passed testcases: %s", passed_testcases)
184	if do_report is None:	251		if do_report is None:
185	do_report = CONFIG.getboolean('report', 'submit_wiki')	252		do_report = CONFIG.getboolean('report', 'submit_wiki')
186		253
187	if do_report:	254		if do_report:
188	logger.info("reporting test passes")	255		logger.info("reporting test passes")
189	wiki = Wiki(('https', wiki_url), '/w/')	256		wiki = Wiki(('https', wiki_url), '/w/')
190	if not wiki.logged_in:	257		if not wiki.logged_in:
▲ Show 20 Lines • Show All 92 Lines • Show Last 20 Lines

rework wiki report 'conditions', for notification testsClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 2605

scheduler/checkwiki

scheduler/fedora_openqa_schedule/conf_test_suites.py

scheduler/fedora_openqa_schedule/report.py

rework wiki report 'conditions', for notification tests
ClosedPublic
Actions