transfer modules from AutoQA for upgradepath test to run
ClosedPublic
Actions

Authored by kparal on Feb 20 2014, 1:16 PM.

Details

Reviewers: None
Maniphest Tasks: T79: Port upgradepath from AutoQA to Taskotron
Commits: rLTRN8d1b3de9111b: transfer modules from AutoQA for upgradepath test to run

Summary

This is a set of modules that upgradepath needs to run on Taskotron.

I tried to strip down the available methods in the modules to the
absolute minimum. We can always copy over additional methods if we need
them.

Part of this is also support for config files and repoinfo.conf. We
might want to modify it somehow, but we will most probably need it in
some form anyway.

The test.py module was reduced to a basic skeleton, but still added,
because it is very useful for multi-result tests (and probably not just
for them). It is no longer tied to autotest base class.

Test Plan

Upgradepath runs fine with this patch. You can test it from the corresponding feature branch in upgradepath git.

Diff Detail

Lint

Lint Skipped

Unit

Unit Tests Skipped

Closed by commit rLTRN8d1b3de9111b (authored by @kparal).

Phabricator closed this ticket because I pushed it to git origin, even though it was not master/develop, but just a feature branch. Also, this revision can't be reopened, I'll have to create a new one (I hope that would work). It seems we've discovered a serious bug in Phab.

Revision Contents

		Path
A	M	conf/repoinfo.conf (55 lines)
A	M	conf/secrets.conf (11 lines)
A	M	conf/taskotron.conf (93 lines)
A	M	libtaskotron/bodhi_utils.py (103 lines)
A	M	libtaskotron/config.py (208 lines)
A	M	libtaskotron/exc.py (17 lines)
M		libtaskotron/file_utils.py (25 lines)
A	M	libtaskotron/koji_utils.py (115 lines)
A	M	libtaskotron/python_utils.py (40 lines)
A	M	libtaskotron/repoinfo.py (143 lines)
A	M	libtaskotron/system_utils.py (55 lines)
A	M	libtaskotron/test.py (161 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	23	2a5dc7b		Feb 20 2014, 1:16 PM	★	★
Diff 2	24	2a5dc7b	Commit rLTRN8d1b3de9111b97843ee346fedf8ea9e8a1b34d6d	Feb 20 2014, 1:16 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
10f5f6878893	e7f80ef4c069	2a5dc7b7e7b8	Kamil Páral	transfer modules from AutoQA for upgradepath test to run (Show More…)	Jan 31 2014, 4:50 PM

Diff 24

View Options

conf/repoinfo.conf

This file was added.

		1		[DEFAULT]
		2		# list of parent repos to this repo
		3		parents =
		4		# list of supported architectures
		5		arches = i386, x86_64
		6		# whether installation composes are created for this repo or not
		7		composes = no
		8		# tag defaults to section name
		9		tag = %(__name__)s
		10		baseurl = http://download.fedoraproject.org/pub/fedora/linux
		11		goldurl = %(baseurl)s/releases/%(path)s/Everything/%(arch)s/os
		12		updatesurl = %(baseurl)s/updates/%(path)s/%(arch)s
		13		rawhideurl = %(baseurl)s/%(path)s/%(arch)s/os
		14		# true for "main" repos corresponding to currently supported Fedora releases
		15		isactiverelease =
		16
		17		# Rawhide
		18		[rawhide]
		19		arches = i386, x86_64
		20		path = development/rawhide
		21		url = %(rawhideurl)s
		22		tag = f21
		23
		24		# Fedora 20
		25		[f20]
		26		isactiverelease = yes
		27		path = 20
		28		url = %(goldurl)s
		29
		30		[f20-updates]
		31		path = 20
		32		url = %(updatesurl)s
		33		parents = f20
		34
		35		[f20-updates-testing]
		36		path = testing/20
		37		url = %(updatesurl)s
		38		parents = f20-updates, f20
		39
		40		# Fedora 19
		41		[f19]
		42		isactiverelease = yes
		43		path = 19
		44		url = %(goldurl)s
		45
		46		[f19-updates]
		47		path = 19
		48		url = %(updatesurl)s
		49		parents = f19
		50
		51		[f19-updates-testing]
		52		path = testing/19
		53		url = %(updatesurl)s
		54		parents = f19-updates, f19
		55

View Options

conf/secrets.conf

This file was added.

		1		# All login credentials and other secrets are here. This file should not be
		2		# world readable.
		3
		4		[secrets]
		5
		6		# FAS (Fedora Accounts System) credentials
		7		# These credentials are used when reporting results in the name of Taskotron,
		8		# e.g. posting a comment into Bodhi.
		9		fas_username = taskotron
		10		fas_password =
		11

View Options

conf/taskotron.conf

This file was added.

		1		# Main configuration file for Taskotron
		2		# More information on Taskotron configuration can be found at:
		3		# https://fedoraproject.org/wiki/AutoQA_Configuration
		4		# FIXME: create Taskotron docs and fix the link
		5
		6		[notifications]
		7		# This section controls which notifications you want to send after the test
		8		# execution is completed
		9
		10		# If "true", test results (for selected tests) will be sent as comments to
		11		# Fedora Update System (Bodhi). This requires that you have Bodhi credentials
		12		# filled in secrets.conf.
		13		send_bodhi_comments = false
		14
		15		# How long (in minutes) should we wait before allowing consequent test to
		16		# re-post a 'FAILED' comment into Bodhi once again.
		17		# By default 3 days (32460 = 4320).
		18		bodhi_posting_comments_span = 4320
		19
		20
		21		[resources]
		22		# This section specifies access details to various external services
		23
		24		# URL of Koji instance used for querying about new builds
		25		koji_url = http://koji.fedoraproject.org/kojihub
		26
		27		# URL of repository of all the RPM packages built in Koji
		28		pkg_url = http://kojipkgs.fedoraproject.org/packages
		29
		30		# URL of Bodhi instance used for communication about package updates
		31		# (leave blank for default)
		32		bodhi_server =
		33
		34		# ResultsDB server can store all test results.
		35		# Provide URL or ResultsDB's XMLRPC interface, leave blank to disable.
		36		resultsdb_server =
		37
		38
		39		[bodhi_email]
		40		# These configuration options affect how Taskotron decideds to send emails through
		41		# bodhi in specific situations. This section only applies when
		42		# 'send_bodhi_comments' is enabled.
		43
		44		# Send email even when all tests are passing for an update. Note that this is
		45		# overridden by the state_change configurations if an update's tests transition
		46		# from FAIL -> PASS (as opposed to INCOMPLETE -> PASS which is affected by this
		47		# configuration value).
		48		email_all_passed = false
		49
		50		# Send emails when an update's test state has changed. This would be the
		51		# update as a whole is affected. Example:
		52		# old_status:
		53		# depcheck_i386:PASS, depcheck_x86_64:PASS, upgradepath:FAIL
		54		# new_status:
		55		# depcheck_i386:PASS, depcheck_x86_64:PASS, upgradepath:PASS
		56		#
		57		# Going from old_status -> new_status represents a state change since all
		58		# of the tests are now passing. Assuming that this wasn't overridden by other
		59		# settings, a state change for the update would be detected and an email sent.
		60		#
		61		# Example of no update state change:
		62		# old_status:
		63		# depcheck_i386:PASS, depcheck_x86_64:FAIL, upgradepath:FAIL
		64		# new_status:
		65		# depcheck_i386:PASS, depcheck_x86_64:FAIL, upgradepath:PASS
		66		#
		67		# Since depcheck_x86_64 is still FAIL, the tests as a whole for the update are
		68		# still FAIL and the overall status has not changed.
		69		email_update_state_change = true
		70
		71		# send email on state change for any individual test instead of waiting for an
		72		# update's state to change as a whole (ex: email when upgradepath goes from
		73		# FAIL -> PASS even though depcheck_i386 is still FAILED and the test state as
		74		# a whole for this update haven't changed from FAIL)
		75		email_test_state_change = false
		76
		77		# Always send an email for all bodhi comments
		78		# This WILL override ANY other email settings
		79		email_always_send = false
		80
		81		# send emails when all tests for updates_testing are complete instead of
		82		# waiting for all tests to be done on dist-fXX-updates. This does
		83		# not affect sending emails when all tests are completed for dist-fXX-updates.
		84		email_updates_testing = true
		85
		86
		87		[paths]
		88		# Location of various parts of the project. You don't want to change anything
		89		# in here except for some rare cases.
		90
		91		# Path containing log file for the taskotron runner and watchers
		92		logfile = /var/log/taskotron/taskotron.log
		93

View Options

libtaskotron/bodhi_utils.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2010-2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		'''Utility functions for dealing with Bodhi'''
		6
		7		import fedora.client
		8
		9		from . import system_utils
		10		from .config import Config
		11		from .logger import log
		12
		13
		14		_ops = {'base_url': Config.resources.bodhi_server} if \
		15		Config.resources.bodhi_server else {}
		16		bodhi = fedora.client.bodhi.BodhiClient(username=Config.secrets.fas_username,
		17		password=Config.secrets.fas_password,
		18		**_ops)
		19
		20
		21		def query_update(package):
		22		'''Get the last Bodhi update matching criteria.
		23
		24		@param package package NVR or package name or update title or update ID
		25		Note: Only NVR allowed, not ENVR. See
		26		https://fedorahosted.org/bodhi/ticket/592.
		27
		28		@return most recent Bodhi update object matching criteria, or None if
		29		no such available
		30		'''
		31		log.debug('Searching Bodhi updates for: %s', package)
		32		res = bodhi.query(package=package, limit=1)
		33		if res['updates']:
		34		return res['updates'][0]
		35		else:
		36		return None
		37
		38
		39		def build2update(builds, strict=False):
		40		'''Find matching Bodhi updates for provided builds.
		41
		42		@param builds builds to search for; iterable of strings
		43		@param strict if False, incomplete Bodhi updates are allowed.
		44		if True, every Bodhi update will be compared with the set of
		45		provided builds. If there is an Bodhi update which contains
		46		builds not provided in @param builds, that update is marked
		47		as incomplete and removed from the result - i.e. all builds
		48		from @param builds that were part of this incomplete update
		49		are placed in the second dictionary of the result tuple.
		50		@return tuple of two dictionaries:
		51		* The first dict provides mapping between builds and their updates
		52		where no error occured. {build (string): Bodhi update (Bunch)}
		53		* The second dict provides mapping between builds and their updates
		54		where some error occured.
		55		{build (string): Bodhi update (Bunch) or None}
		56		The value is None if the matching Bodhi update could not be found
		57		(the only possible cause of failure if @param strict=False). Or
		58		the value is a Bodhi update that was incomplete (happens only if
		59		@param strict=True).
		60		'''
		61		build_update = {}
		62		failures = {}
		63		# Bodhi works with NVR only, but we have to ensure we receive and return
		64		# even other formats, like ENVRA. So we need to convert internally.
		65		builds_nvr = set([system_utils.envra(build, 'nvr') for build in builds])
		66
		67		for build in sorted(builds):
		68		# if already found answer for this build, skip it
		69		if build in build_update or build in failures:
		70		continue
		71
		72		# get Bodhi update object
		73		update = query_update(system_utils.envra(build, 'nvr'))
		74		if update is None:
		75		# no such Bodhi update
		76		failures[build] = None
		77		continue
		78
		79		# all builds listed in the update
		80		bodhi_builds = set([build['nvr'] for build in update['builds']])
		81		# builds not provided in @param builds but part of the update (NVRs)
		82		missing_builds = bodhi_builds.difference(builds_nvr)
		83		# builds provided in @param builds and part of the update
		84		matched_builds = [build for build in builds if system_utils.envra(build,
		85		'nvr') in bodhi_builds]
		86
		87		# reject incomplete updates
		88		if missing_builds and strict:
		89		for matched_build in matched_builds:
		90		failures[matched_build] = update
		91		continue
		92
		93		# create item in build_update
		94		for build in matched_builds:
		95		build_update[build] = update
		96
		97		# some build might have been added to both @var failures and
		98		# @var build_update. if that happens, remove it from @var build_update
		99		for build in failures:
		100		build_update.pop(build, None)
		101
		102		return (build_update, failures)
		103

View Options

libtaskotron/config.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2010-2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		'''Global configuration for Taskotron and relevant helper methods.
		6
		7		All configuration is done statically, all options must be defined here in this
		8		file in order to be accessible. They are not created dynamically from
		9		configuration files contents (just their values are).
		10		'''
		11
		12		import os
		13		import ConfigParser
		14
		15		import libtaskotron
		16
		17		from . import python_utils
		18		from . import file_utils
		19		from . import exc
		20		from .logger import log
		21
		22
		23		# directory where all config files are stored
		24		CONF_DIR = '/etc/taskotron'
		25		# a conf dir location in the local checkout, in case you haven't installed the
		26		# config files into a system-wide location
		27		LOCAL_CONF_DIR = os.path.abspath(os.path.dirname(libtaskotron.__file__) +
		28		'/../conf')
		29
		30		# list of all configuration files loaded into Config class
		31		CONF_FILES = ['taskotron.conf',
		32		'secrets.conf',
		33		]
		34
		35
		36		class Config(object):
		37		'''Global configuration for Taskotron.
		38
		39		All configuration options are available as class attributes in the form
		40		of Config.section.option = value. Different config files are not
		41		distinguished, all sections are flattened into a single name space.
		42
		43		The documentation for individual options is available in their .conf
		44		file.
		45
		46		Implementation notes:
		47		* All new options must be provided here, placing them just in a .conf
		48		file is not enough.
		49		* When changing defaults in here, don't forget to also update the
		50		corresponding .conf file.
		51		* Always assign a correct value type to an attribute (e.g. use empty
		52		string instead of None), because values read from .conf files are then
		53		automatically re-typed according to the attribute's current value type.
		54		'''
		55
		56		class _Section(object):
		57		'''An empty class that represents a configuration section (relates to
		58		sections in .conf files). All config options are assigned as instance
		59		attributes.'''
		60		pass
		61
		62		class _SectionPaths(_Section):
		63		'''A special version of @class _Section for configuring file paths. It
		64		makes sure that some selected paths exist after their assignment.'''
		65
		66		def __init__(self):
		67		self._tmpdir = None
		68
		69		@property
		70		def tmpdir(self):
		71		'''Temporary directory for generic Taskotron files. If you set a
		72		value, that directory is automatically created.'''
		73		return self._tmpdir
		74
		75		@tmpdir.setter
		76		def tmpdir(self, value):
		77		self._tmpdir = value
		78		file_utils.makedirs(self._tmpdir)
		79
		80
		81		### config keys definition ###
		82		# == taskotron.conf ===
		83		notifications = _Section()
		84		notifications.send_bodhi_comments = False
		85		notifications.bodhi_posting_comments_span = 4320
		86
		87		resources = _Section()
		88		resources.koji_url = 'http://koji.fedoraproject.org/kojihub'
		89		resources.pkg_url = 'http://kojipkgs.fedoraproject.org/packages'
		90		resources.bodhi_server = ''
		91		resources.resultsdb_server = ''
		92
		93		bodhi_email = _Section()
		94		bodhi_email.email_all_passed = False
		95		bodhi_email.email_update_state_change = True
		96		bodhi_email.email_test_state_change = False
		97		bodhi_email.email_always_send = False
		98		bodhi_email.email_updates_testing = True
		99
		100		paths = _SectionPaths()
		101		paths.cachedir = '/var/cache/taskotron'
		102		paths.tmpdir = '/tmp/taskotron'
		103		paths.lockdir = '/var/run/taskotron'
		104		paths.logfile = '/var/log/taskotron/taskotron.log'
		105		paths.repoinfo_conf = [os.path.join(CONF_DIR, 'repoinfo.conf'),
		106		os.path.join(LOCAL_CONF_DIR, 'repoinfo.conf')]
		107
		108		# == secrets.conf ==
		109		secrets = _Section()
		110		secrets.fas_username = 'taskotron'
		111		secrets.fas_password = ''
		112
		113
		114		@classmethod
		115		def _load(cls, filelist):
		116		'''Load options from a configuration file and then assign their values
		117		to attributes in this class.
		118
		119		Empty values are ignored. Extra options and sections (not existing
		120		here) are also ignored.
		121
		122		@param filelist a list of config file paths (type: list of strings)
		123		'''
		124		hint = ('Maybe you have an outdated configuration file and want to '
		125		'merge it with the latest version? Or if you want to define a '
		126		'new option, you have to edit libtaskotron.config module as '
		127		'well.')
		128
		129		parser = ConfigParser.SafeConfigParser()
		130		log.debug('Reading config files: %s', filelist)
		131		parser.read(filelist)
		132
		133		for section in parser.sections():
		134		if not hasattr(cls, section):
		135		log.warn('Unknown section "%s" in config files "%s"', section,
		136		filelist)
		137		log.warn(hint)
		138		continue
		139		section_conf = getattr(cls, section)
		140
		141		options = parser.items(section)
		142		for (option, value) in options:
		143		if not hasattr(section_conf, option):
		144		log.warn('Unknown option "%s" in section "%s" in config '
		145		'files "%s"', option, section, filelist)
		146		log.warn(hint)
		147		continue
		148
		149		# Only set option if it has a non-empty value.
		150		# That allows users to leave 'option = ' lines in config file
		151		# and still use default values. OTOH user can't empty out a
		152		# default option value this way.
		153		if value:
		154		# re-type to correct type
		155		try:
		156		value = cls._retype(getattr(section_conf, option),
		157		value)
		158		except exc.TaskotronConfigError, e:
		159		log.exception(e)
		160		log.critical(hint)
		161		raise exc.TaskotronConfigError('Incorrect value '
		162		'provided for option "%s"' % option, e)
		163
		164		# assign value
		165		setattr(section_conf, option, value)
		166
		167
		168		@classmethod
		169		def _retype(cls, old_value, new_value):
		170		'''Re-type new_value according to the current type of old_value.
		171
		172		@return re-typed new_value (it may be the same if old_value was of
		173		the same type or None); if exception occurs during
		174		conversion then None is returned
		175		'''
		176
		177		if old_value is not None:
		178		try:
		179		type_ = type(old_value)
		180		# for booleans do advanced checking
		181		if issubclass(type_, bool):
		182		newvalue = python_utils.getbool(new_value)
		183		else:
		184		newvalue = type_(new_value)
		185		return newvalue
		186		except ValueError as e:
		187		log.exception('Error re-typing config value')
		188		raise exc.TaskotronConfigError("Can't re-type '%s' to %s" %
		189		(new_value, type_), e)
		190
		191		return new_value
		192
		193
		194		@classmethod
		195		def init(cls):
		196		'''Initialize the class and load values from all .conf files. This
		197		method must be called before accessing class attributes. The call
		198		should be ideally placed into package initialization module, as soon as
		199		possible.
		200		'''
		201		# load configuration files
		202		for conffile in CONF_FILES:
		203		cls._load([os.path.join(CONF_DIR, conffile),
		204		os.path.join(LOCAL_CONF_DIR, conffile)])
		205
		206
		207		# initialize Config
		208		Config.init()

View Options

libtaskotron/exc.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2010-2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		'''This module contains custom Taskotron exceptions'''
		6
		7		class TaskotronError(Exception):
		8		'''Common ancestor for Taskotron related exceptions'''
		9		pass
		10
		11		class TaskotronValueError(ValueError, TaskotronError):
		12		'''Taskotron-specific ValueError'''
		13		pass
		14
		15		class TaskotronConfigError(TaskotronError):
		16		'''All errors related to Taskotron config files'''
		17		pass

View Options

libtaskotron/file_utils.py

		1		# -- coding: utf-8 --
		2		# Copyright 2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		'''Helper functions for dealing with files and filesystems'''
		6
1	import os	7		import os
2	import sys	8		import sys
3	import urlgrabber.grabber, urlgrabber.progress	9		import urlgrabber.grabber, urlgrabber.progress
4	from libtaskotron.logger import log	10		from .logger import log
5		11
6		12
7	def makedirs(fullpath):	13		def makedirs(fullpath):
8	'''This is the same as @method os.makedirs(), but does not throw an	14		'''This is the same as @method os.makedirs(), but does not throw an
9	exception when the destination directory already exists.'''	15		exception when the destination directory already exists.'''
10	try:	16		try:
11	os.makedirs(fullpath)	17		os.makedirs(fullpath)
12	assert os.path.isdir(fullpath)	18		assert os.path.isdir(fullpath)
13	except OSError, e:	19		except OSError as e:
14	if e.errno == 17: # "[Errno 17] File exists"	20		if e.errno == 17: # "[Errno 17] File exists"
15	# if it is a directory everything is ok	21		# if it is a directory everything is ok
16	if os.path.isdir(fullpath):	22		if os.path.isdir(fullpath):
17	return	23		return
18	# otherwise it is a file/socket/etc and it is an error	24		# otherwise it is a file/socket/etc and it is an error
19	else:	25		else:
		26		log.exception('Error during creating a directory')
20	raise	27		raise
21	else:	28		else:
		29		log.exception('Error during creating a directory')
22	raise	30		raise
23		31
		32
24	def get_urlgrabber(**kwargs):	33		def get_urlgrabber(**kwargs):
25	'''Get a new instance of URLGrabber that is reasonably set (progress	34		'''Get a new instance of URLGrabber that is reasonably set (progress
26	bar when in terminal, automatic retries, connection timeout, etc)	35		bar when in terminal, automatic retries, connection timeout, etc)
27	@kwargs additional kwargs to add to the URLGrabber constructor	36		@kwargs additional kwargs to add to the URLGrabber constructor
28	@return URLGrabber instance	37		@return URLGrabber instance
29	'''	38		'''
30	grabber_args = {'retry': 3, 'timeout': 120}	39		grabber_args = {'retry': 3, 'timeout': 120}
31	grabber_args.update(kwargs)	40		grabber_args.update(kwargs)
Show All 40 Lines		55	def download(url, dest, overwrite=False, grabber=None):
72	# overwrite?	81		# overwrite?
73	if os.path.exists(dest):	82		if os.path.exists(dest):
74	if overwrite:	83		if overwrite:
75	os.remove(dest)	84		os.remove(dest)
76	else:	85		else:
77	log.debug('Already downloaded: %s', os.path.basename(dest))	86		log.debug('Already downloaded: %s', os.path.basename(dest))
78	return dest	87		return dest
79		88
80	log.info('Downloading: %s', url)	89		log.debug('Downloading: %s', url)
81	try:	90		try:
82	grabber.urlgrab(url, dest)	91		grabber.urlgrab(url, dest)
83	except urlgrabber.grabber.URLGrabError, e:	92		except urlgrabber.grabber.URLGrabError as e:
84	log.error('Download error: %s', e)	93		log.exception('Could not download: %s', url)
85	# the file can be incomplete, remove	94		# the file can be incomplete, remove
86	if os.path.exists(dest):	95		if os.path.exists(dest):
87	try:	96		try:
88	os.remove(dest)	97		os.remove(dest)
89	except:	98		except OSError:
90	log.error('Could not delete incomplete file: %s', dest)	99		log.exception('Could not delete incomplete file: %s', dest)
91	raise	100		raise e
92		101
93	return dest	102		return dest
94		103

View Options

libtaskotron/koji_utils.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2009-2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		'''Helper routines for dealing with Koji'''
		6
		7		import xmlrpclib
		8
		9		import koji
		10		import rpmUtils.miscutils
		11
		12		from . import file_utils
		13		from .config import Config
		14		from .logger import log
		15
		16
		17		class TaskotronKojiClient(koji.ClientSession):
		18		'''Convenience wrapper class for interacting with koji'''
		19
		20		def __init__(self, server=Config.resources.koji_url, opts=None):
		21		koji.ClientSession.__init__(self, server, opts)
		22
		23
		24		def check_connection(self):
		25		'''Check the connection to koji.'''
		26		# Idea taken from the koji CLI
		27		try:
		28		ret = self.getAPIVersion()
		29		log.debug('Successfuly connected to Koji server')
		30		except xmlrpclib.ProtocolError:
		31		log.warn("Unable to connect to Koji server")
		32		raise
		33		if ret != koji.API_VERSION:
		34		log.warn("Warning: Koji API mismatch (Server %d, client %d)",
		35		ret, koji.API_VERSION)
		36
		37
		38		def nvr_to_urls(self, nvr, arches=None, debuginfo=False, src=True):
		39		'''Get list of URLs for RPMs corresponding to a build.
		40		@param nvr build NVR
		41		@param arches restrict the arches of builds to provide URLs for. If
		42		basearch i386 is in the list, i686 arch is automatically
		43		added (since that's the arch Koji API uses).
		44		'''
		45		log.debug('Querying Koji for a list of RPMS for: %s', nvr)
		46
		47		# add i686 arch if i386 is present in arches
		48		if arches and 'i386' in arches and 'i686' not in arches:
		49		arches.append('i686')
		50
		51		info = self.getBuild(nvr)
		52		baseurl = '/'.join((Config.resources.pkg_url, info['package_name'],
		53		info['version'], info['release']))
		54
		55		rpms = self.listRPMs(buildID=info['id'], arches=arches)
		56		if not debuginfo:
		57		rpms = filter(lambda r: not r['name'].endswith('-debuginfo'), rpms)
		58		if not src:
		59		rpms = filter(lambda r: not r['arch'] == 'src', rpms)
		60
		61		urls = ['%s/%s' % (baseurl, koji.pathinfo.rpm(r)) for r in rpms]
		62		return sorted(urls)
		63
		64
		65		def get_nvr_rpms(self, nvr, rpm_dir, arches=None, debuginfo=False,
		66		src=False):
		67		'''Retrieve the RPMs associated with a build NVR into the specified
		68		directory.
		69
		70		@param nvr build NVR
		71		@param rpm_dir location where to store the RPMs
		72		@return list of local filenames of the grabbed RPMs
		73		@raise urlgrabber.grabber.URLGrabError if downloading failed
		74		'''
		75
		76		rpm_urls = self.nvr_to_urls(nvr, arches=arches, debuginfo=debuginfo,
		77		src=src)
		78		rpm_files = []
		79		log.debug('Fetching RPMs for: %s', nvr)
		80		for url in rpm_urls:
		81		rpm_file = file_utils.download(url, rpm_dir)
		82		rpm_files.append(rpm_file)
		83
		84		return rpm_files
		85
		86
		87		def compareEVR(nvr1, nvr2):
		88		'''Compare EVR from two NVR objects.
		89		@param nvr1 Koji build object or None
		90		@param nvr2 Koji build object or None
		91		@return The higher of the two builds when compared according to EVR. If one
		92		of the objects is None, then it's always lesser. If both are None,
		93		then they are equal.
		94		'''
		95		# handle empty input
		96		if not nvr1:
		97		return 0 if not nvr2 else -1
		98		if not nvr2:
		99		return 1
		100		# compare
		101		return rpmUtils.miscutils.compareEVR((nvr1['epoch'] or '0',
		102		nvr1['version'], nvr1['release']), (nvr2['epoch'] or '0',
		103		nvr2['version'], nvr2['release']))
		104
		105
		106		def getENVR(build):
		107		''' Get NVR (if epoch is 0) or ENVR (otherwise) from build info
		108		@param build buildinfo dictionary (as returned from koji methods like
		109		getBuild)
		110		@return string NVR or ENVR '''
		111		if build['epoch']:
		112		return '%s:%s' % (build['epoch'], build['nvr'])
		113		else:
		114		return build['nvr']
		115

View Options

libtaskotron/python_utils.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		'''A collection of convenience methods related to Python base libraries.'''
		6
		7		import collections
		8
		9		from . import exc
		10
		11
		12		def getbool(value):
		13		'''Convert a string into a boolean.
		14
		15		Recognizes many usual strings: 1/0, yes/no, true/false, on/off; also
		16		empty strings. Uses default bool conversion for non-string values.
		17		@throw ValueError if @param value is a string but can't be recognized
		18		'''
		19		boolean_states = {'1': True, 'yes': True, 'true': True, 'on': True,
		20		'0': False, 'no': False, 'false': False, 'off': False,
		21		'': False}
		22
		23		if not isinstance(value, str):
		24		return bool(value)
		25
		26		value = value.lower()
		27		if value not in boolean_states:
		28		raise exc.TaskotronValueError('Not a boolean: %r %s' %
		29		(value, type(value)))
		30
		31		return boolean_states[value]
		32
		33
		34		def listlike(obj):
		35		'''Return whether the object is iterable (list, tuple, set, etc), but not
		36		string (which is an iterable, but we don't usually want to consider it as
		37		such).
		38		'''
		39		return isinstance(obj, collections.Iterable) and not isinstance(obj, str)
		40

View Options

libtaskotron/repoinfo.py

This file was added.

		1		#!/usr/bin/env python
		2		# -- coding: utf-8 --
		3		# Copyright 2009-2014, Red Hat, Inc.
		4		# License: GNU General Public License version 2 or later
		5
		6		'''A wrapper for reading repoinfo.conf easily'''
		7
		8		import ConfigParser
		9
		10		from .config import Config
		11		from .logger import log
		12
		13
		14		class RepoinfoConfig(object):
		15		'''This class is a wrapper for easily accessing repoinfo.conf file.'''
		16
		17
		18		def __init__(self, arch="$basearch", filelist=None, defaults=None):
		19		'''Create RepoinfoConfig object.
		20		Attributes:
		21		* arch - architecture for which to adjust repo URLs
		22		* filelist - list of config files to read information from
		23		* defaults - dictionary of default values
		24		'''
		25		self.config = None
		26		if defaults is None:
		27		self.defaults = {}
		28		else:
		29		self.defaults = defaults
		30		self.filelist = filelist or Config.paths.repoinfo_conf[:]
		31		self.setarch(arch) # implies read()
		32
		33
		34		def _getlist(self, name, item):
		35		'''Transform a list of comma separated values into a Python list
		36		Returns empty list for non-existent item'''
		37		itemlist = self.config.get(name, item)
		38		if not itemlist:
		39		return []
		40		else:
		41		return [t.strip() for t in itemlist.split(',')]
		42
		43
		44		def getarches(self, name):
		45		'''Get the list of all supported architectures for a repo'''
		46		return self._getlist(name, 'arches')
		47
		48
		49		def getreleases(self):
		50		'''Return the list of known, non-EOL releases.'''
		51		return [r for r in repoinfo.repos() if self.config.get(r,
		52		'isactiverelease')]
		53
		54
		55		def get_bodhi_releases(self):
		56		'''Return the list of releases known to Bodhi.'''
		57		return [r.upper() for r in self.getreleases()]
		58
		59
		60		def getparents(self, name):
		61		'''Given a repo name, give the names of its parent repos
		62		e.g.: getparents('f20-updates-testing') -> ['f20','f20-updates']'''
		63		return self._getlist(name, 'parents')
		64
		65
		66		def get_top_parents(self, reponame):
		67		'''Get top parents reponames for a specified reponame.
		68		@param reponame repository name
		69		@return list of top parent reponames (list of strings). Usually this
		70		is a single-item list, but in some cases there may be more
		71		than one top parent.
		72		'''
		73		parents = self.getrepo(reponame)['parents']
		74		if not parents:
		75		return set([reponame])
		76		else:
		77		result = set()
		78		for reponame in parents:
		79		result.update(self.get_top_parents(reponame))
		80		return result
		81
		82
		83		def getrepo(self, name):
		84		'''Given a repo name, return the repoinfo dict, with the keys:
		85		'arches', 'parents', 'tag', 'url', 'path', 'name' and 'composes' '''
		86		repo = {'arches': self.getarches(name),
		87		'parents': self.getparents(name),
		88		'tag': self.config.get(name,'tag'),
		89		'url': self.config.get(name,'url'),
		90		'path': self.config.get(name,'path'),
		91		'name': name,
		92		'composes': self.config.getboolean(name,'composes'),
		93		}
		94		return repo
		95
		96
		97		def getrepo_by_tag(self, tag):
		98		'''Given a koji tag, return the corresponding repo
		99		e.g.: getrepo_by_tag('dist-f13')'''
		100		# we don't have repo definition for -candidate tags,
		101		# let's go up in the hierarchy (stripping '-candidate')
		102		if tag.endswith('-candidate'):
		103		tag = tag[:-len('-candidate')]
		104		for r in self.repos():
		105		if self.config.get(r,'tag') == tag:
		106		return self.getrepo(r)
		107
		108
		109		def repos(self):
		110		'''Returns a list of all known repo names'''
		111		return self.config.sections()
		112
		113
		114		def setarch(self, arch):
		115		'''Set architecture for which to adjust repo URLs'''
		116		self.defaults['arch'] = arch
		117		self.read()
		118
		119
		120		def read(self):
		121		'''Apply the defaults and then read first available config file from
		122		the list of provided config files.'''
		123		log.debug('Reading config files: %s', self.filelist)
		124		self.config = ConfigParser.SafeConfigParser(defaults=self.defaults)
		125		self.config.read(self.filelist)
		126
		127
		128		def get(self, repo, key):
		129		'''Get a specific key from a repo'''
		130		return self.config.get(repo, key)
		131
		132
		133		repoinfo = RepoinfoConfig()
		134
		135		if not repoinfo.repos():
		136		log.error("No repos found in (%s)", ', '.join(
		137		Config.paths.repoinfo_conf))
		138
		139
		140		if __name__ == '__main__':
		141		import pprint
		142		rd = dict([(n, repoinfo.getrepo(n)) for n in repoinfo.repos()])
		143		pprint.pprint(rd)

View Options

libtaskotron/system_utils.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		import rpmUtils
		6		import rpmUtils.arch
		7		import rpmUtils.miscutils
		8
		9		from . import exc
		10
		11
		12		def envra(rpmstr, fmt = 'envra'):
		13		"""
		14		@param rpmstr string to be 'manipulated' - can be standard style rpm
		15		filename, ENVRA, ENVR or NVR
		16		@param fmt desired format of the string to be returned. Allowed options are:
		17		ENVRA, ENVR, NVRA, NVR, E, N, V, R, A. If arch is not present in
		18		rpmstr, 'noarch' is returned wherever needed.
		19		@returns string based on the specified format; if epoch is requested but not
		20		present, it is not included in the output string (rather than
		21		prepending '0:')
		22		"""
		23		fmt = fmt.lower()
		24		supported_formats = ['envra', 'envr', 'nvra', 'nvr', 'e', 'n', 'v', 'r',
		25		'a']
		26		if fmt not in supported_formats:
		27		raise exc.TaskotronValueError("Format '%s' not in supported formats "
		28		"(%s)" % (fmt, ', '.join(supported_formats)))
		29
		30		# prepare rpmstr so it can be used with rpmUtils.miscutils.splitFilename()
		31		parts = rpmstr.split('.')
		32		if parts[-1].lower() != 'rpm':
		33		parts.append('rpm')
		34		if parts[-2].lower() not in rpmUtils.arch.arches.keys() + ['noarch', 'src']:
		35		parts.insert(-1, 'noarch')
		36
		37		# split the filename
		38		(n, v, r, e, a) = rpmUtils.miscutils.splitFilename('.'.join(parts))
		39
		40		if len(fmt) == 1:
		41		return {'e':e, 'n':n, 'v':v, 'r':r, 'a':a}[fmt]
		42
		43		# if epoch is not present, we do not want to return it in the
		44		# envr(a) requests as '0:' but rather return nvr(a) (so we take
		45		# advantage of `e` being empty string. This means, that if `e`
		46		# is non-empty, we need to add add ':' to the end, since it's
		47		# not present in the format string (see below)
		48		if e != '':
		49		e += ':'
		50
		51		return {'envra': "%s%s-%s-%s.%s" % (e, n, v, r, a),
		52		'envr' : "%s%s-%s-%s" % (e, n, v ,r),
		53		'nvra' : "%s-%s-%s.%s" % (n, v, r ,a),
		54		'nvr' : "%s-%s-%s" % (n, v, r),
		55		}[fmt]

View Options

libtaskotron/test.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2014, Red Hat, Inc.
		3		# License: GNU General Public License version 2 or later
		4
		5		'''Helper tools for managing test status, results, and output.'''
		6
		7		from . import python_utils
		8		from . import exc
		9
		10
		11		class TestDetail(object):
		12		'''Class encompassing everything related to the outcome of a test run.
		13
		14		@classattr result_order a tuple of result keywords sorted by priority from
		15		the least important to the most important
		16		@attr result a keyword specifying the final result of the test. Available
		17		result keywords:
		18		* PASSED - everything went well
		19		* INFO - everything went well, but there is some important
		20		information that needs to be pointed out
		21		* FAILED - the item in question fails the test
		22		* NEEDS_INSPECTION - the result can't be determined and a
		23		manual inspection is needed
		24		* ABORTED - the test aborted itself because of some unexpected
		25		problems, i.e. a necessary network server is not
		26		reachable. Running this test with the same
		27		arguments later can help to mitigate this problem.
		28		* CRASHED - the test script crashed and did not provide usable
		29		results
		30		If no result is set, this attribute returns NEEDS_INSPECTION.
		31		@attr summary a few words or one-sentence summary about the outcome of the
		32		test run. Should not duplicate @attr result.
		33		@attr output output from the test run (a list of strings). This will form
		34		the test log. You can easily populate this by using
		35		@method log(), or you can modify it directly (for example
		36		filter out some messages you don't want to see in the test
		37		log).
		38		@attr keyvals all key-value pairs in this dictionary are stored in ResultsDB
		39		as 'extra data'. This is most commonly used to store the
		40		information about tested ENVR/compose id/etc.
		41		'''
		42
		43		result_order = ('PASSED', 'INFO', 'FAILED', 'NEEDS_INSPECTION',
		44		'ABORTED', 'CRASHED')
		45
		46
		47		def __init__(self, result=None, summary=None, output=None):
		48		'''Create new TestDetail.
		49		For param descriptions see class docstring.'''
		50
		51		# validate input
		52		if output is not None and not python_utils.listlike(output):
		53		raise exc.TaskotronValueError("'output' parameter must be a list. "
		54		"Yours was: %s" % type(output))
		55
		56		self._result = result
		57		self.summary = summary
		58		self.output = output or []
		59		self.keyvals = {}
		60
		61
		62		@property
		63		def result(self):
		64		'''See class documetation.'''
		65		return self._result or 'NEEDS_INSPECTION'
		66
		67
		68		@result.setter
		69		def result(self, value):
		70		'''See class documetation.'''
		71		if value not in self.result_order:
		72		raise exc.TaskotronValueError('Unknown result keyword: %s' % value)
		73		self._result = value
		74
		75
		76		def update_result(self, result):
		77		'''Update the self.result with provided value if it has higher priority
		78		according to self.result_order than the current value. Handles None
		79		values.'''
		80		if self.cmp_result(result, self._result) > 0:
		81		self.result = result
		82
		83
		84		def broken(self):
		85		'''Tells whether the test result is set to one of the broken states
		86		(i.e. ABORTED or CRASHED).'''
		87		return self.result in ['ABORTED', 'CRASHED']
		88
		89
		90		def log(self, message, printout=True):
		91		'''Add a string to the TestDetail output log, and print it as well.
		92
		93		This is just a convenience method for most common use case, you can
		94		of course always access and modify @attr output directly, and print or
		95		use a logging directly as well. This combines both into a single call.
		96
		97		@param message a string to log
		98		@param printout whether to print to standard output or not
		99		'''
		100
		101		self.output.append(message)
		102		if printout:
		103		print message
		104
		105
		106		@classmethod
		107		def cmp_result(cls, result1, result2):
		108		'''Compare two results according to self.result_order and return -1/0/1
		109		of the result1 has lower/equal/higher priority than result2.
		110
		111		@param result1 a result keyword to compare
		112		@param result2 a result keyword to compare
		113		@throw TaskotronValueError if any of the result keywords are not
		114		specified in @classatr result_order
		115		'''
		116
		117		# validate input
		118		for result in [result1, result2]:
		119		if (result not in cls.result_order) and (result is not None):
		120		raise exc.TaskotronValueError('Unknown result keyword: %s' %
		121		result)
		122
		123		index1 = cls.result_order.index(result1) if result1 is not None else -1
		124		index2 = cls.result_order.index(result2) if result2 is not None else -1
		125		return cmp(index1, index2)
		126
		127
		128		@classmethod
		129		def create_multi_item_summary(cls, results, result_order=None):
		130		'''Create a string containing a sum of all results, like this:
		131		"3 PASSED, 1 INFO, 2 FAILED"
		132
		133		@param results either one TestDetail instance or an iterable of
		134		TestDetails or an iterable of result strings
		135		@param result_order result keywords will be sorted in this order
		136		'''
		137		# validate input
		138		if isinstance(results, TestDetail):
		139		results = (results,)
		140
		141		if result_order is None:
		142		result_order = cls.result_order
		143
		144		if len(results) <= 0:
		145		return ''
		146
		147		if isinstance(iter(results).next(), TestDetail):
		148		all_results = [detail.result for detail in results]
		149		else: # list of strings
		150		all_results = results
		151
		152		# create the summary
		153		summary = []
		154		for res in result_order:
		155		count = all_results.count(res)
		156		if count > 0:
		157		summary.append('%d %s' % (count, res))
		158		summary = ', '.join(summary)
		159
		160		return summary
		161

transfer modules from AutoQA for upgradepath test to runClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 24

conf/repoinfo.conf

conf/secrets.conf

conf/taskotron.conf

libtaskotron/bodhi_utils.py

libtaskotron/config.py

libtaskotron/exc.py

libtaskotron/file_utils.py

libtaskotron/koji_utils.py

libtaskotron/python_utils.py

libtaskotron/repoinfo.py

libtaskotron/system_utils.py

libtaskotron/test.py

transfer modules from AutoQA for upgradepath test to run
ClosedPublic
Actions