libtaskotron/directives/koji_directive.py
34–37	N(E)VR of a Koji build to download (for ``action="download"``) or to search the latest stable build for (for ``action="download_latest_stable"``). Not required for ``action="download_tag"``. Example: ``xchat-2.8.8-21.fc20``
100	target_dir: ${workdir}/stable/
159–164	Just in case you wanted to format this without the long tiny column on the right and didn't know how, this is also PEP8-compliant: output_data['downloaded_rpms'] = self.koji.get_nvr_rpms( nvr, self.target_dir, arches=self.arches, debuginfo=self.debuginfo, src=self.src)
182	`'download_latest_stable'`
190	typo: latest
libtaskotron/koji_utils.py
47–50	I'd like to change this method to `latest_by_tag(self, pkgname, tags)`, and `tags` would be a non-empty list of searched tags. We would then call it like: latest_by_tag('pidgin', ['f22', 'f22-updates']) The reason to do this (instead of calling the method twice) is that we can now use multicall inside that function and make several calls at once, speeding up the query (and decreasing network error possibility).
52	Let's make NVR uppercase, thanks :)
55	Won't this crash if there's no such package?

libtaskotron/koji_utils.py
47	Does it make sense to raise an exception if the package is not tagged in the list of searched koji tags? Why not return simply `None`?
48–50	Unfortunately, this won't work. If you also use updates-testing tag, it can contain newer or older version of package that is now in updates tag. Don't ask me why, I just know it happens quite frequently :) I guess it happens if Bodhi fails to untag something from time to time, of if a packaged is pushed stable before even reaching testing. So, in the end, you have no way of knowing what order the tags should have. You have to compare builds from all the tags and pick the latest version. We have some methods around for that.
67–68	I don't like this. There's no explanation when this happens and why we silently ignore it. There is a documentation in `rpms_to_build()`, so I understand what's going on with the `IndexError`, but still if any error happens in the koji call, it is silently ignored. We can easily return wrong results if just one call fails and one passes. Please do a similar approach as in `rpms_to_build()` - if there is an error (dict returned), raise it. It should be no different from a network error, this is just a server error. Why do you ignore `KeyError`? Should we expect that a build is missing `nvr` key? When?

libtaskotron/koji_utils.py
67	I wonder if we can receive more builds if `latest=True`? I don't see it exactly documented in the API docs. And it always returns a single build in my testing. But an assert here could potentially help us discover some issues in the long run. assert len(build[0]) <= 1, 'More than one build returned with latest=True' It should only crash development environment this way (`python -O` flag removes assertions). Some guidelines here: http://stackoverflow.com/a/945135
70–71	Please document in the docstring that we return `None` if no builds are found.
libtaskotron/rpm_utils.py
104–110	Wouldn't it be easier to just call this code where appropriate instead of defining a new function for it? I.e. in `koji_utils` instead of sorted_nvrs = rpm_utils.sort_by_nvr(nvrs, ascending=False) you would do sorted_nvrs = sorted(nvrs, cmp=rpm_utils.cmpNEVR, reverse=True) What's the point of having a dedicated function?

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	1102	b3f4a38		Jul 7 2015, 8:46 AM	★	★
Diff 2	1120	20f226d	- Optimize latest_by_tag by doing multiCall	Jul 9 2015, 11:34 AM	★	★
Diff 3	1121	93e5ae3	- Optimize latest_by_tag by doing multiCall	Jul 9 2015, 11:38 AM	★	★
Diff 4	1156	93e5ae3	- Fix multiCall processing	Jul 14 2015, 10:59 AM	★	★
Diff 5	1164	93e5ae3	- Assert	Jul 14 2015, 12:29 PM	★	★
Diff 6	1172	3252ef7	Commit rLTRN0bf7b5230b0bfda9acb9c8e7e8d7bf3c271f22d8	Jul 15 2015, 11:55 AM	★	★

Commit	Tree	Parents	Author	Summary	Date
640b18d4efb4	2abeff451af7	eb6f187ce6f4	Martin Krizek	Assert	Jul 14 2015, 12:28 PM
eb6f187ce6f4	04948e5ef255	ccbe8cd0cd7a	Martin Krizek	Fix multiCall processing	Jul 14 2015, 10:58 AM
ccbe8cd0cd7a	86faf088aaa1	606d00aee8fd	Martin Krizek	Optimize latest_by_tag by doing multiCall	Jul 9 2015, 11:33 AM
606d00aee8fd	213ef7e08762	93e5ae393e2a	Martin Krizek	Add downloading latest stable build to koji directive (Show More…)	Jul 2 2015, 12:52 PM

Diff 1172

libtaskotron/directives/koji_directive.py

Show All 12 Lines		10	description: \|
13	a specific build, or you can download all RPMs from all builds belonging	13		a specific build, or you can download all RPMs from all builds belonging
14	to a specific Koji tag.	14		to a specific Koji tag.
15	parameters:	15		parameters:
16	action:	16		action:
17	required: true	17		required: true
18	description: choose whether to download a single build (``download`` value)	18		description: choose whether to download a single build (``download`` value)
19	or all builds belonging to a Koji tag (``download_tag`` value)	19		or all builds belonging to a Koji tag (``download_tag`` value)
20	type: str	20		type: str
21	choices: [download, download_tag]	21		choices: [download, download_tag, download_latest_stable]
22	arch:	22		arch:
23	required: true	23		required: true
24	description: \|	24		description: \|
25	a list of architectures for which to download RPMs for the requested	25		a list of architectures for which to download RPMs for the requested
26	build/tag. If you want to download RPMs for all arches, use ``['all']``.	26		build/tag. If you want to download RPMs for all arches, use ``['all']``.
27		27
28	Note: ``noarch`` RPMs are always automatically downloaded even when not	28		Note: ``noarch`` RPMs are always automatically downloaded even when not
29	requested, unless ``arch=[]`` and ``src=True``.	29		requested, unless ``arch=[]`` and ``src=True``.
30	type: list of str	30		type: list of str
31	choices: [supported architectures, ['all']]	31		choices: [supported architectures, ['all']]
32	koji_build:	32		koji_build:
33	required: true	33		required: true
34	description: \|	34		description: \|
35	N(E)VR of a Koji build to download. Only required when	35		N(E)VR of a Koji build to download (for ``action="download"``) or to search
36	``action="download"``. Example: ``xchat-2.8.8-21.fc20``	36		the latest stable build for (for ``action="download_latest_stable"``). Not
		37		required for ``action="download_tag"``. Example: ``xchat-2.8.8-21.fc20``
			kparalUnsubmitted Not Done N(E)VR of a Koji build to download (for ``action="download"``) or to search the latest stable build for (for ``action="download_latest_stable"``). Not required for ``action="download_tag"``. Example: ``xchat-2.8.8-21.fc20``
37	type: str	38		type: str
38	koji_tag:	39		koji_tag:
39	required: true	40		required: true
40	description: \|	41		description: \|
41	name of a Koji tag to download all builds from. Only required when	42		name of a Koji tag to download all builds from. Only required when
42	``action="download_tag"``. Example: ``f20-updates-pending``	43		``action="download_tag"``. Example: ``f20-updates-pending``
43	type: str	44		type: str
44	src:	45		src:
Show All 38 Lines
83	architecture::	84		architecture::
84		85
85	- name: download koji tag	86		- name: download koji tag
86	koji:	87		koji:
87	action: download_tag	88		action: download_tag
88	koji_tag: ${koji_tag}	89		koji_tag: ${koji_tag}
89	arch: ${arch}	90		arch: ${arch}
90	target_dir: ${workdir}/downloaded_tag/	91		target_dir: ${workdir}/downloaded_tag/
		92
		93		Abidiff needs to download the latest stable build of a package::
		94
		95		- name: download rpms of the latest stable build of a given build from koji
		96		koji:
		97		action: download_latest_stable
		98		koji_build: ${koji_build}
		99		arch: ${arch}
		100		target_dir: ${workdir}/stable/
			kparalUnsubmitted Not Done target_dir: ${workdir}/stable/
91	"""	101		"""
92		102
93	from libtaskotron.koji_utils import KojiClient	103		from libtaskotron.koji_utils import KojiClient
94	from libtaskotron.directives import BaseDirective	104		from libtaskotron.directives import BaseDirective
95	from libtaskotron import rpm_utils	105		from libtaskotron import rpm_utils
96	import libtaskotron.exceptions as exc	106		import libtaskotron.exceptions as exc
97		107
98	directive_class = 'KojiDirective'	108		directive_class = 'KojiDirective'
99		109
		110
100	class KojiDirective(BaseDirective):	111		class KojiDirective(BaseDirective):
101		112
102	def __init__(self, koji_session=None):	113		def __init__(self, koji_session=None):
103	super(KojiDirective, self).__init__()	114		super(KojiDirective, self).__init__()
104	if koji_session is None:	115		if koji_session is None:
105	self.koji = KojiClient(koji_session)	116		self.koji = KojiClient()
106	else:	117		else:
107	self.koji = koji_session	118		self.koji = koji_session
108		119
109	def process(self, input_data, env_data):	120		def process(self, input_data, env_data):
110	# process params	121		# process params
111	valid_actions = ['download', 'download_tag']	122		valid_actions = ['download', 'download_tag', 'download_latest_stable']
112	action = input_data['action']	123		action = input_data['action']
113	if not action in valid_actions:	124		if action not in valid_actions:
114	raise exc.TaskotronDirectiveError('%s is not a valid action for koji '	125		raise exc.TaskotronDirectiveError('%s is not a valid action for koji '
115	'directive' % action)	126		'directive' % action)
116		127
117	if not 'target_dir' in input_data:	128		if 'target_dir' not in input_data:
118	self.target_dir = env_data['workdir']	129		self.target_dir = env_data['workdir']
119	else:	130		else:
120	self.target_dir = input_data['target_dir']	131		self.target_dir = input_data['target_dir']
121		132
122	if 'arch' not in input_data:	133		if 'arch' not in input_data:
123	detected_args = ', '.join(input_data.keys())	134		detected_args = ', '.join(input_data.keys())
124	raise exc.TaskotronDirectiveError(	135		raise exc.TaskotronDirectiveError(
125	"The koji directive requires 'arch' as an argument. Detected "	136		"The koji directive requires 'arch' as an argument. Detected "
Show All 14 Lines		150	if action == 'download':
140	if 'koji_build' not in input_data:	151		if 'koji_build' not in input_data:
141	detected_args = ', '.join(input_data.keys())	152		detected_args = ', '.join(input_data.keys())
142	raise exc.TaskotronDirectiveError(	153		raise exc.TaskotronDirectiveError(
143	"The koji directive requires 'koji_build' for the 'download' "	154		"The koji directive requires 'koji_build' for the 'download' "
144	"action. Detected arguments: %s" % detected_args)	155		"action. Detected arguments: %s" % detected_args)
145		156
146	nvr = rpm_utils.rpmformat(input_data['koji_build'], 'nvr')	157		nvr = rpm_utils.rpmformat(input_data['koji_build'], 'nvr')
147		158
148	output_data['downloaded_rpms'] = self.koji.get_nvr_rpms(nvr,	159		output_data['downloaded_rpms'] = self.koji.get_nvr_rpms(nvr,
149	self.target_dir, arches=self.arches, debuginfo=self.debuginfo, src=self.src)	160		self.target_dir,
150		161		arches=self.arches,
		162		debuginfo=self.debuginfo,
		163		src=self.src)
151	elif action == 'download_tag':	164		elif action == 'download_tag':
			kparalUnsubmitted Not Done Just in case you wanted to format this without the long tiny column on the right and didn't know how, this is also PEP8-compliant: output_data['downloaded_rpms'] = self.koji.get_nvr_rpms( nvr, self.target_dir, arches=self.arches, debuginfo=self.debuginfo, src=self.src)
152	if 'koji_tag' not in input_data:	165		if 'koji_tag' not in input_data:
153	detected_args = ', '.join(input_data.keys())	166		detected_args = ', '.join(input_data.keys())
154	raise exc.TaskotronDirectiveError(	167		raise exc.TaskotronDirectiveError(
155	"The koji directive requires 'koji_tag' for the 'download_tag' "	168		"The koji directive requires 'koji_tag' for the 'download_tag' "
156	"action. Detected arguments: %s" % detected_args)	169		"action. Detected arguments: %s" % detected_args)
157		170
158	koji_tag = input_data['koji_tag']	171		koji_tag = input_data['koji_tag']
159		172
160	output_data['downloaded_rpms'] = self.koji.get_tagged_rpms(koji_tag,	173		output_data['downloaded_rpms'] = self.koji.get_tagged_rpms(koji_tag,
161	self.target_dir, arches=self.arches, debuginfo=self.debuginfo, src=self.src)	174		self.target_dir,
		175		arches=self.arches,
		176		debuginfo=self.debuginfo,
		177		src=self.src)
		178		elif action == 'download_latest_stable':
		179		if 'koji_build' not in input_data:
		180		detected_args = ', '.join(input_data.keys())
		181		raise exc.TaskotronDirectiveError(
		182		"The koji directive requires 'koji_build' for the 'download_latest_stable' "
			kparalUnsubmitted Not Done `'download_latest_stable'`
		183		"action. Detected arguments: %s" % detected_args)
		184
		185		name = rpm_utils.rpmformat(input_data['koji_build'], 'n')
		186		disttag = rpm_utils.get_dist_tag(input_data['koji_build'])
		187		# we need to do 'fc22' -> 'f22' conversion
		188		tag = disttag.replace('c', '')
		189
		190		# first we need to check updates tag and if that fails, the latest
			kparalUnsubmitted Not Done typo: latest
		191		# stable nvr is in the base repo
		192		tags = ['%s-updates' % tag, tag]
		193		latest_stable_nvr = self.koji.latest_by_tag(tags, name)
		194
		195		output_data['downloaded_rpms'] = self.koji.get_nvr_rpms(latest_stable_nvr,
		196		self.target_dir,
		197		arches=self.arches,
		198		debuginfo=self.debuginfo,
		199		src=self.src)
162		200
163	return output_data	201		return output_data

View Options

libtaskotron/koji_utils.py

Show All 36 Lines		35	opts = {
37	# retry behavior can be configured via 'max_retries' (default 30)	37		# retry behavior can be configured via 'max_retries' (default 30)
38	# and 'retry_interval' (default 20) opts keys	38		# and 'retry_interval' (default 20) opts keys
39	'anon_retry': True,	39		'anon_retry': True,
40	# default Koji connection timeout is 12 hours, that's too high.	40		# default Koji connection timeout is 12 hours, that's too high.
41	# let's make it 5 minutes	41		# let's make it 5 minutes
42	'timeout': 60 * 5}	42		'timeout': 60 * 5}
43		43
44	self.session = (koji_session or	44		self.session = (koji_session or
45	koji.ClientSession(config.get_config().koji_url, opts=opts))	45		koji.ClientSession(config.get_config().koji_url, opts=opts))
46		46
		47		def latest_by_tag(self, tags, pkgname):
			kparalUnsubmitted Not Done Does it make sense to raise an exception if the package is not tagged in the list of searched koji tags? Why not return simply `None`?
		48		'''Get the latest Koji build for the given package name in the given tag(s).
		49
		50		:param list tags: list of tags to be searched in
			kparalUnsubmitted Not Done I'd like to change this method to `latest_by_tag(self, pkgname, tags)`, and `tags` would be a non-empty list of searched tags. We would then call it like: latest_by_tag('pidgin', ['f22', 'f22-updates']) The reason to do this (instead of calling the method twice) is that we can now use multicall inside that function and make several calls at once, speeding up the query (and decreasing network error possibility).
			kparalUnsubmitted Not Done Unfortunately, this won't work. If you also use updates-testing tag, it can contain newer or older version of package that is now in updates tag. Don't ask me why, I just know it happens quite frequently :) I guess it happens if Bodhi fails to untag something from time to time, of if a packaged is pushed stable before even reaching testing. So, in the end, you have no way of knowing what order the tags should have. You have to compare builds from all the tags and pick the latest version. We have some methods around for that.
		51		:param str pkgname: name of the package
		52		:return: str containing NVR of the latest build, None if no build was found
			kparalUnsubmitted Not Done Let's make NVR uppercase, thanks :)
		53		'''
		54		self.session.multicall = True
		55		for tag in tags:
			kparalUnsubmitted Not Done Won't this crash if there's no such package?
		56		self.session.listTagged(tag, package=pkgname, latest=True)
		57		builds = self.session.multiCall()
		58
		59		nvrs = []
		60		for build in builds:
		61		# see rpms_to_build for documentation on how multiCall works
		62		if isinstance(build, dict):
		63		raise exc.TaskotronRemoteError('listTagged failed with: %d: %s' %
		64		(build["faultCode"], build["faultString"]))
		65		elif build[0]:
		66		assert len(build[0]) <= 1, 'More than one build returned with latest=True'
		67		nvrs.append(build[0][0]['nvr'])
			kparalUnsubmitted Not Done I wonder if we can receive more builds if `latest=True`? I don't see it exactly documented in the API docs. And it always returns a single build in my testing. But an assert here could potentially help us discover some issues in the long run. assert len(build[0]) <= 1, 'More than one build returned with latest=True' It should only crash development environment this way (`python -O` flag removes assertions). Some guidelines here: http://stackoverflow.com/a/945135
		68
			kparalUnsubmitted Not Done I don't like this. There's no explanation when this happens and why we silently ignore it. There is a documentation in `rpms_to_build()`, so I understand what's going on with the `IndexError`, but still if any error happens in the koji call, it is silently ignored. We can easily return wrong results if just one call fails and one passes. Please do a similar approach as in `rpms_to_build()` - if there is an error (dict returned), raise it. It should be no different from a network error, this is just a server error. Why do you ignore `KeyError`? Should we expect that a build is missing `nvr` key? When?
		69		sorted_nvrs = sorted(nvrs, cmp=rpm_utils.cmpNEVR, reverse=True)
		70		if sorted_nvrs:
		71		return sorted_nvrs[0]
			kparalUnsubmitted Not Done Please document in the docstring that we return `None` if no builds are found.
47		72
48	def rpms_to_build(self, rpms):	73		def rpms_to_build(self, rpms):
49	'''Get list of koji build objects for the rpms. Order of koji objects	74		'''Get list of koji build objects for the rpms. Order of koji objects
50	in this list is the same as order of the respective rpm objects.	75		in this list is the same as order of the respective rpm objects.
51		76
52	:param rpms: list of filenames as either ``/my/path/nvr.a.rpm`` or	77		:param rpms: list of filenames as either ``/my/path/nvr.a.rpm`` or ``nvr.a.rpm``
53	``nvr.a.rpm``
54	:type rpms: list of str	78		:type rpms: list of str
55	:return: list of Koji buildinfo dictionaries (as returned e.g.	79		:return: list of Koji buildinfo dictionaries (as returned e.g.
56	from :meth:`koji.getBuild`) in the same respective order as in``rpms``	80		from :meth:`koji.getBuild`) in the same respective order as in``rpms``
57	:rtype: list	81		:rtype: list
58	:raise TaskotronRemoteError: if rpm or it's related build is not	82		:raise TaskotronRemoteError: if rpm or it's related build is not found
59	found
60	'''	83		'''
61	log.info('Querying Koji to map %d RPMs to their builds...', len(rpms))	84		log.info('Querying Koji to map %d RPMs to their builds...', len(rpms))
62		85
63	self.session.multicall = True	86		self.session.multicall = True
64	for rpm in rpms:	87		for rpm in rpms:
65	# extract filename from the (possible) path	88		# extract filename from the (possible) path
66	rpm = os.path.split(rpm)[-1]	89		rpm = os.path.split(rpm)[-1]
67		90
Show All 27 Lines
95	for build in builds:	118		for build in builds:
96	self.session.getBuild(build)	119		self.session.getBuild(build)
97		120
98	buildinfos = self.session.multiCall()	121		buildinfos = self.session.multiCall()
99		122
100	for i, buildinfo in enumerate(buildinfos):	123		for i, buildinfo in enumerate(buildinfos):
101	# see ^	124		# see ^
102	if isinstance(buildinfo, dict):	125		if isinstance(buildinfo, dict):
103	raise exc.TaskotronRemoteError(	126		raise exc.TaskotronRemoteError('Problem with build %s: %d: %s' %
104	'Problem with build %s: %d: %s' % (	127		(builds[i], buildinfo["faultCode"],
105	builds[i], buildinfo["faultCode"],	128		buildinfo["faultString"]))
106	buildinfo["faultString"]))
107	elif buildinfo[0] is None:	129		elif buildinfo[0] is None:
108	raise exc.TaskotronRemoteError(	130		raise exc.TaskotronRemoteError(
109	'Build %s not found' % builds[i])	131		'Build %s not found' % builds[i])
110		132
111	build_to_buildinfo = dict(zip(builds, buildinfos))	133		build_to_buildinfo = dict(zip(builds, buildinfos))
112	result = []	134		result = []
113	for rpminfo in rpminfos:	135		for rpminfo in rpminfos:
114	build_id = rpminfo[0]['build_id']	136		build_id = rpminfo[0]['build_id']
115	result.append(build_to_buildinfo[build_id][0])	137		result.append(build_to_buildinfo[build_id][0])
116		138
117	return result	139		return result
118		140
119
120	def nvr_to_urls(self, nvr, arches=['all'], debuginfo=False, src=True):	141		def nvr_to_urls(self, nvr, arches=['all'], debuginfo=False, src=True):
121	'''Get list of URLs for RPMs corresponding to a build.	142		'''Get list of URLs for RPMs corresponding to a build.
122		143
123	:param str nvr: build NVR	144		:param str nvr: build NVR
124	:param arches: restrict the arches of builds to provide URLs for. By	145		:param arches: restrict the arches of builds to provide URLs for. By
125	default, all architectures are considered. If you want to consider	146		default, all architectures are considered. If you want to consider
126	just some selected arches, provide their names in a list.	147		just some selected arches, provide their names in a list.
127		148
Show All 16 Lines
144		165
145	# if src is enabled, we need to add it so that it is included in the	166		# if src is enabled, we need to add it so that it is included in the
146	# Koji query	167		# Koji query
147	if src and ('src' not in arches):	168		if src and ('src' not in arches):
148	arches.append('src')	169		arches.append('src')
149		170
150	# if "nothing" is requested, it's probably an error	171		# if "nothing" is requested, it's probably an error
151	if not arches:	172		if not arches:
152	raise exc.TaskotronValueError('Nothing to query for, `arches` was '	173		raise exc.TaskotronValueError('Nothing to query for, `arches` was set to an empty '
153	'set to an empty list and `src` was disabled. At least one of '	174		'list and `src` was disabled. At least one of them '
154	'them need to be non-empty.')	175		'need to be non-empty.')
155		176
156	# find the koji build	177		# find the koji build
157	info = self.session.getBuild(nvr)	178		info = self.session.getBuild(nvr)
158	if info is None:	179		if info is None:
159	raise exc.TaskotronRemoteError(	180		raise exc.TaskotronRemoteError("No such build found in Koji: %s" % nvr)
160	"No such build found in Koji: %s" % nvr)
161		181
162	# list its RPM files	182		# list its RPM files
163	req_arches = None if 'all' in arches else arches	183		req_arches = None if 'all' in arches else arches
164	rpms = self.session.listRPMs(buildID=info['id'], arches=req_arches)	184		rpms = self.session.listRPMs(buildID=info['id'], arches=req_arches)
165	if not debuginfo:	185		if not debuginfo:
166	rpms = [r for r in rpms if (not r['name'].endswith('-debuginfo'))	186		rpms = [r for r in rpms if (not r['name'].endswith('-debuginfo'))
167	and ('-debuginfo-' not in r['name'])]	187		and ('-debuginfo-' not in r['name'])]
168	if not src:	188		if not src:
169	rpms = [r for r in rpms if not r['arch'] == 'src']	189		rpms = [r for r in rpms if not r['arch'] == 'src']
170		190
171	# create URLs	191		# create URLs
172	baseurl = '/'.join((config.get_config().pkg_url, info['package_name'],	192		baseurl = '/'.join((config.get_config().pkg_url, info['package_name'],
173	info['version'], info['release']))	193		info['version'], info['release']))
174	urls = ['%s/%s' % (baseurl, koji.pathinfo.rpm(r)) for r in rpms]	194		urls = ['%s/%s' % (baseurl, koji.pathinfo.rpm(r)) for r in rpms]
175		195
176	return sorted(urls)	196		return sorted(urls)
177		197
178		198		def get_nvr_rpms(self, nvr, dest, arches=['all'], debuginfo=False, src=False):
179	def get_nvr_rpms(self, nvr, dest, arches=['all'], debuginfo=False,
180	src=False):
181	'''Retrieve the RPMs associated with a build NVR into the specified	199		'''Retrieve the RPMs associated with a build NVR into the specified
182	directory.	200		directory.
183		201
184	:param str nvr: build NVR	202		:param str nvr: build NVR
185	:param str dest: location where to store the RPMs	203		:param str dest: location where to store the RPMs
186	:param arches: see :meth:`nvr_to_urls`	204		:param arches: see :meth:`nvr_to_urls`
187	:param debuginfo: see :meth:`nvr_to_urls`	205		:param debuginfo: see :meth:`nvr_to_urls`
188	:param src: see :meth:`nvr_to_urls`	206		:param src: see :meth:`nvr_to_urls`
Show All 18 Lines		224	if config.get_config().download_cache_enabled:
207	cachedir = config.get_config().cachedir	225		cachedir = config.get_config().cachedir
208		226
209	for url in rpm_urls:	227		for url in rpm_urls:
210	rpm_file = file_utils.download(url, dest, cachedir=cachedir)	228		rpm_file = file_utils.download(url, dest, cachedir=cachedir)
211	rpm_files.append(rpm_file)	229		rpm_files.append(rpm_file)
212		230
213	return rpm_files	231		return rpm_files
214		232
215		233		def get_tagged_rpms(self, tag, dest, arches=['all'], debuginfo=False, src=False):
216	def get_tagged_rpms(self, tag, dest, arches=['all'], debuginfo=False,
217	src=False):
218	'''Downloads all RPMs of all NVRs tagged by a specific Koji tag.	234		'''Downloads all RPMs of all NVRs tagged by a specific Koji tag.
219		235
220	Note: This works basically the same as :meth:`get_nvr_rpms`, it just	236		Note: This works basically the same as :meth:`get_nvr_rpms`, it just
221	downloads a lot of builds instead of a single one. For description of	237		downloads a lot of builds instead of a single one. For description of
222	all shared parameters and return values, please see that method.	238		all shared parameters and return values, please see that method.
223		239
224	:param str tag: Koji tag to be queried for available builds, e.g.	240		:param str tag: Koji tag to be queried for available builds, e.g.
225	``f20-updates-pending``	241		``f20-updates-pending``
226	'''	242		'''
227	# always create dest dir, even if nothing gets downloaded	243		# always create dest dir, even if nothing gets downloaded
228	file_utils.makedirs(dest)	244		file_utils.makedirs(dest)
229		245
230	log.info('Querying Koji for tag: %s' % tag)	246		log.info('Querying Koji for tag: %s', tag)
231	tag_data = self.session.listTagged(tag)	247		tag_data = self.session.listTagged(tag)
232		248
233	nvrs = sorted([x['nvr'] for x in tag_data])	249		nvrs = sorted([x['nvr'] for x in tag_data])
234	rpms = []	250		rpms = []
235	log.info("Fetching %s builds for tag: %s", len(nvrs), tag)	251		log.info("Fetching %s builds for tag: %s", len(nvrs), tag)
236	log.debug('Builds to be downloaded:\n %s', '\n '.join(nvrs))	252		log.debug('Builds to be downloaded:\n %s', '\n '.join(nvrs))
237		253
238	for nvr in nvrs:	254		for nvr in nvrs:
239	rpms.extend(self.get_nvr_rpms(nvr, dest, arches, debuginfo, src))	255		rpms.extend(self.get_nvr_rpms(nvr, dest, arches, debuginfo, src))
240		256
241	return rpms	257		return rpms
242		258
243		259
244	def getNEVR(build):	260		def getNEVR(build):
245	'''Extract RPM version identifier in NEVR format from Koji build object	261		'''Extract RPM version identifier in NEVR format from Koji build object
246		262
247	:param dict build: Koji buildinfo dictionary (as returned e.g. from	263		:param dict build: Koji buildinfo dictionary (as returned e.g. from
248	:meth:`koji.getBuild`)	264		:meth:`koji.getBuild`)
249	:return: NEVR string; epoch is included when non-zero, otherwise omitted	265		:return: NEVR string; epoch is included when non-zero, otherwise omitted
250	:raise TaskotronValueError: if ``build`` is of incorrect type	266		:raise TaskotronValueError: if ``build`` is of incorrect type
251	'''	267		'''
252	# validate input	268		# validate input
253	if (not isinstance(build, collections.Mapping) or 'nvr' not in build or	269		if (not isinstance(build, collections.Mapping) or 'nvr' not in build or
254	'epoch' not in build):	270		'epoch' not in build):
255	raise exc.TaskotronValueError("Input argument doesn't look like "	271		raise exc.TaskotronValueError("Input argument doesn't look like "
256	"a Koji build object: %s" % build)	272		"a Koji build object: %s" % build)
257		273
258	rpmver = hawkey.split_nevra(build['nvr'] + '.noarch')	274		rpmver = hawkey.split_nevra(build['nvr'] + '.noarch')
259	rpmver.epoch = build['epoch'] or 0	275		rpmver.epoch = build['epoch'] or 0
260	nevr = '%s-%s' % (rpmver.name, rpmver.evr())	276		nevr = '%s-%s' % (rpmver.name, rpmver.evr())
261	# supress 0 epoch (evr() method always includes epoch, even if 0)	277		# supress 0 epoch (evr() method always includes epoch, even if 0)
262	nevr = rpm_utils.rpmformat(nevr, fmt='nevr')	278		nevr = rpm_utils.rpmformat(nevr, fmt='nevr')
263		279
264	return nevr	280		return nevr

View Options

libtaskotron/rpm_utils.py

Show First 20 Lines • Show All 90 Lines • ▼ Show 20 Line(s)		90	if rpmver1.name != rpmver2.name:
91	raise exc.TaskotronValueError("Name in nevr1 doesn't match name in "	91		raise exc.TaskotronValueError("Name in nevr1 doesn't match name in "
92	"nevr2: %s, %s" % (nevr1, nevr2))	92		"nevr2: %s, %s" % (nevr1, nevr2))
93		93
94	# sack is needed for the comparison, because it can be influence the	94		# sack is needed for the comparison, because it can be influence the
95	# comparison (for example epoch can be set to be ignored). A default empty	95		# comparison (for example epoch can be set to be ignored). A default empty
96	# sack should match Fedora customs	96		# sack should match Fedora customs
97	sack = hawkey.Sack()	97		sack = hawkey.Sack()
98		98
99	return rpmver1.evr_cmp(rpmver2, sack)	99		# we need evr_cmp to return int so we can use it as comparison function
		100		# in python's sorted
		101		return int(rpmver1.evr_cmp(rpmver2, sack))
100		102
101		103
102	def get_dist_tag(rpmstr):	104		def get_dist_tag(rpmstr):
103	'''Parse disttag from an RPM package version string.	105		'''Parse disttag from an RPM package version string.
104		106
105	:param str rpmstr: string to be manipulated in a format of N(E)VR	107		:param str rpmstr: string to be manipulated in a format of N(E)VR
106	(``foo-1.2-3.fc20`` or ``bar-4:1.2-3.fc20``) or N(E)VRA	108		(``foo-1.2-3.fc20`` or ``bar-4:1.2-3.fc20``) or N(E)VRA
107	(``foo-1.2-3.fc20.x86_64`` or ``bar-4:1.2-3.fc20.i686``)	109		(``foo-1.2-3.fc20.x86_64`` or ``bar-4:1.2-3.fc20.i686``)
108	:return: string containing dist tag (``fc20``)	110		:return: string containing dist tag (``fc20``)
			kparalUnsubmitted Not Done Wouldn't it be easier to just call this code where appropriate instead of defining a new function for it? I.e. in `koji_utils` instead of sorted_nvrs = rpm_utils.sort_by_nvr(nvrs, ascending=False) you would do sorted_nvrs = sorted(nvrs, cmp=rpm_utils.cmpNEVR, reverse=True) What's the point of having a dedicated function?
109	:raise TaskotronValueError: if ``rpmstr`` does not contain dist tag	111		:raise TaskotronValueError: if ``rpmstr`` does not contain dist tag
110	'''	112		'''
111		113
112	release = rpmformat(rpmstr, 'r')	114		release = rpmformat(rpmstr, 'r')
113	matches = re.findall(r'\.(fc\d{1,2})\.?', release)	115		matches = re.findall(r'\.(fc\d{1,2})\.?', release)
114	if not matches:	116		if not matches:
115	raise exc.TaskotronValueError('Could not parse disttag from %s' % rpmstr)	117		raise exc.TaskotronValueError('Could not parse disttag from %s' % rpmstr)
116		118
117	# there might be e.g. fc22 in git commit hash as part of the release,	119		# there might be e.g. fc22 in git commit hash as part of the release,
118	# so just take the last match which should be disttag	120		# so just take the last match which should be disttag
119	return matches[-1]	121		return matches[-1]

View Options

testing/test_koji_directive.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	from dingus import Dingus	6		from dingus import Dingus
7		7
8	from libtaskotron.directives import koji_directive	8		from libtaskotron.directives import koji_directive
9		9
10	class TestKojiDirective():	10		class TestKojiDirective():
11	def setup_method(self, method):	11		def setup_method(self, method):
12	self.ref_nvr = 'foo-1.2-3.fc99'	12		self.ref_nvr = 'foo-1.2-3.fc99'
		13		self.ref_previous_nvr = 'foo-1.2-2.fc99'
13	self.ref_arch = ['noarch']	14		self.ref_arch = ['noarch']
14	self.ref_tag = 'tagfoo'	15		self.ref_tag = 'tagfoo'
15	self.ref_name = 'foo'	16		self.ref_name = 'foo'
16	self.ref_version = '1.2'	17		self.ref_version = '1.2'
17	self.ref_release = '3.fc99'	18		self.ref_release = '3.fc99'
18	self.ref_buildid = 123456	19		self.ref_buildid = 123456
19		20
20	self.ref_build = {'package_name': self.ref_name,	21		self.ref_build = {'package_name': self.ref_name,
Show All 29 Lines		44	def test_parse_download_command(self):
50		51
51	test_helper = koji_directive.KojiDirective(stub_koji)	52		test_helper = koji_directive.KojiDirective(stub_koji)
52		53
53	test_helper.process(self.ref_input, self.ref_envdata)	54		test_helper.process(self.ref_input, self.ref_envdata)
54		55
55	getrpm_calls = stub_koji.calls()	56		getrpm_calls = stub_koji.calls()
56	requested_nvr = getrpm_calls[0][1][0]	57		requested_nvr = getrpm_calls[0][1][0]
57	requested_src = getrpm_calls[0][2]['src']	58		requested_src = getrpm_calls[0][2]['src']
		59		requested_debuginfo = getrpm_calls[0][2]['debuginfo']
58		60
59	assert len(getrpm_calls) == 1	61		assert len(getrpm_calls) == 1
60	assert requested_nvr == self.ref_nvr	62		assert requested_nvr == self.ref_nvr
61	assert not requested_src	63		assert not requested_src
		64		assert not requested_debuginfo
62		65
63	def test_parse_download_tag_command(self):	66		def test_parse_download_tag_command(self):
64		67
65	self.ref_input = {'action': 'download_tag',	68		self.ref_input = {'action': 'download_tag',
66	'arch': self.ref_arch,	69		'arch': self.ref_arch,
67	'koji_tag': self.ref_tag}	70		'koji_tag': self.ref_tag}
68		71
69	stub_koji = Dingus(get_tagged_rpms__returns=self.ref_rpms)	72		stub_koji = Dingus(get_tagged_rpms__returns=self.ref_rpms)
70		73
71	test_helper = koji_directive.KojiDirective(stub_koji)	74		test_helper = koji_directive.KojiDirective(stub_koji)
72		75
73	test_helper.process(self.ref_input, self.ref_envdata)	76		test_helper.process(self.ref_input, self.ref_envdata)
74		77
75	getrpm_calls = stub_koji.calls()	78		getrpm_calls = stub_koji.calls()
76	requested_tag = getrpm_calls[0][1][0]	79		requested_tag = getrpm_calls[0][1][0]
77		80
78	assert len(getrpm_calls) == 1	81		assert len(getrpm_calls) == 1
79	assert requested_tag == self.ref_tag	82		assert requested_tag == self.ref_tag
80		83
		84		def test_parse_download_latest_stable_command(self):
		85
		86		self.ref_input = {'action': 'download_latest_stable',
		87		'arch': self.ref_arch,
		88		'koji_build': self.ref_nvr}
		89
		90		stub_koji = Dingus(get_nvr_rpms__returns=self.ref_rpms,
		91		latest_by_tag__returns=self.ref_previous_nvr)
		92
		93		test_helper = koji_directive.KojiDirective(stub_koji)
		94
		95		test_helper.process(self.ref_input, self.ref_envdata)
		96
		97		koji_calls = stub_koji.calls()
		98
		99		getrpm_calls = koji_calls[1]
		100		requested_nvr = getrpm_calls[1][0]
		101		requested_src = getrpm_calls[2]['src']
		102		requested_debuginfo = getrpm_calls[2]['debuginfo']
		103
		104		assert len(koji_calls) == 2
		105		assert requested_nvr == self.ref_previous_nvr
		106		assert not requested_src
		107		assert not requested_debuginfo
		108
81	def test_parse_multiple_arches(self):	109		def test_parse_multiple_arches(self):
82	self.ref_arch = ['noarch', 'i386']	110		self.ref_arch = ['noarch', 'i386']
83		111
84	self.ref_input = {'action': 'download_tag',	112		self.ref_input = {'action': 'download_tag',
85	'arch': self.ref_arch,	113		'arch': self.ref_arch,
86	'koji_tag': self.ref_tag}	114		'koji_tag': self.ref_tag}
87		115
88	stub_koji = Dingus('koji_utils')	116		stub_koji = Dingus('koji_utils')
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines

View Options

testing/test_koji_utils.py

Show All 28 Lines		28	else:
29	return second	29		return second
30		30
31	return multicall	31		return multicall
32		32
33		33
34	class TestKojiClient():	34		class TestKojiClient():
35	def setup_method(self, method):	35		def setup_method(self, method):
36	self.ref_nvr = 'foo-1.2-3.fc99'	36		self.ref_nvr = 'foo-1.2-3.fc99'
		37		self.ref_latest_stable = 'foo-1.2-2.fc99'
37	self.ref_arch = 'noarch'	38		self.ref_arch = 'noarch'
38	self.ref_name = 'foo'	39		self.ref_name = 'foo'
39	self.ref_version = '1.2'	40		self.ref_version = '1.2'
40	self.ref_release = '3.fc99'	41		self.ref_release = '3.fc99'
41	self.ref_buildid = 123456	42		self.ref_buildid = 123456
42	self.ref_filename = "%s.%s.rpm" % (self.ref_nvr, self.ref_arch)	43		self.ref_filename = "%s.%s.rpm" % (self.ref_nvr, self.ref_arch)
		44		self.ref_tags = ['f99-updates', 'f99']
43		45
44	self.ref_build = {'package_name': self.ref_name,	46		self.ref_build = {'package_name': self.ref_name,
45	'version': self.ref_version,	47		'version': self.ref_version,
46	'release': self.ref_release,	48		'release': self.ref_release,
47	'id': self.ref_buildid,	49		'id': self.ref_buildid,
48	'nvr': '%s-%s-%s' % (self.ref_name, self.ref_version,	50		'nvr': '%s-%s-%s' % (self.ref_name, self.ref_version,
49	self.ref_release)}	51		self.ref_release)}
50		52
51	self.ref_rpms = [{'name': self.ref_name, 'version': self.ref_version,	53		self.ref_rpms = [{'name': self.ref_name, 'version': self.ref_version,
52	'release': self.ref_release, 'nvr': self.ref_nvr,	54		'release': self.ref_release, 'nvr': self.ref_nvr,
53	'arch': self.ref_arch, 'build_id': self.ref_buildid,},	55		'arch': self.ref_arch, 'build_id': self.ref_buildid,},
54	{'name': self.ref_name, 'version': self.ref_version,	56		{'name': self.ref_name, 'version': self.ref_version,
55	'release': self.ref_release, 'nvr': self.ref_nvr,	57		'release': self.ref_release, 'nvr': self.ref_nvr,
56	'arch': 'src', 'build_id': self.ref_buildid, },	58		'arch': 'src', 'build_id': self.ref_buildid, },
57	{'name': self.ref_name + '-debuginfo',	59		{'name': self.ref_name + '-debuginfo',
58	'version': self.ref_version,	60		'version': self.ref_version,
59	'release': self.ref_release, 'nvr': self.ref_nvr,	61		'release': self.ref_release, 'nvr': self.ref_nvr,
60	'arch': 'src', 'build_id': self.ref_buildid, },	62		'arch': 'src', 'build_id': self.ref_buildid, },
61	{'name': self.ref_name + '-debuginfo-common',	63		{'name': self.ref_name + '-debuginfo-common',
62	'version': self.ref_version,	64		'version': self.ref_version,
63	'release': self.ref_release, 'nvr': self.ref_nvr,	65		'release': self.ref_release, 'nvr': self.ref_nvr,
64	'arch': 'src', 'build_id': self.ref_buildid, }]	66		'arch': 'src', 'build_id': self.ref_buildid, }]
65		67
		68		def test_latest_by_tag_first_tag_miss(self):
		69		stub_koji = Dingus(listTagged__returns=None,
		70		multiCall__returns=[[[]], [[{'nvr': self.ref_latest_stable}]]])
		71
		72		test_koji = koji_utils.KojiClient(stub_koji)
		73		outcome = test_koji.latest_by_tag(self.ref_tags, self.ref_name)
		74
		75		assert outcome == self.ref_latest_stable
		76
		77		def test_latest_by_tag_second_tag_miss(self):
		78		stub_koji = Dingus(listTagged__returns=None,
		79		multiCall__returns=[[[{'nvr': self.ref_latest_stable}]], [[]]])
		80
		81		test_koji = koji_utils.KojiClient(stub_koji)
		82		outcome = test_koji.latest_by_tag(self.ref_tags, self.ref_name)
		83
		84		assert outcome == self.ref_latest_stable
		85
		86		def test_latest_by_tag_build_not_found(self):
		87		stub_koji = Dingus(listTagged__returns=None,
		88		multiCall__returns=[[[]], [[]]])
		89
		90		test_koji = koji_utils.KojiClient(stub_koji)
		91		assert test_koji.latest_by_tag(self.ref_tags, self.ref_name) is None
		92
66	def test_rpms_to_build(self):	93		def test_rpms_to_build(self):
67	stub_koji = Dingus(getBuild__returns=None,	94		stub_koji = Dingus(getBuild__returns=None,
68	getRPM__returns=None,	95		getRPM__returns=None,
69	multiCall=create_multicall(	96		multiCall=create_multicall(
70	[[self.ref_rpms[0]], [self.ref_rpms[1]]],	97		[[self.ref_rpms[0]], [self.ref_rpms[1]]],
71	[[self.ref_build], [self.ref_build]]))	98		[[self.ref_build], [self.ref_build]]))
72		99
73	test_koji = koji_utils.KojiClient(stub_koji)	100		test_koji = koji_utils.KojiClient(stub_koji)
▲ Show 20 Lines • Show All 276 Lines • Show Last 20 Lines

View Options

testing/test_rpm_utils.py

Show First 20 Lines • Show All 119 Lines • ▼ Show 20 Line(s)		116	def test_some_epoch(self):
120	assert cmpNEVR('foo-0:1.2-3.fc20', 'foo-1.2-3.fc20') == 0	120		assert cmpNEVR('foo-0:1.2-3.fc20', 'foo-1.2-3.fc20') == 0
121	assert cmpNEVR('foo-1.2-3.fc20', 'foo-2:0.1-1.fc19') == -1	121		assert cmpNEVR('foo-1.2-3.fc20', 'foo-2:0.1-1.fc19') == -1
122		122
123	def test_raise(self):	123		def test_raise(self):
124	'''Invalid input param'''	124		'''Invalid input param'''
125	with pytest.raises(exc.TaskotronValueError):	125		with pytest.raises(exc.TaskotronValueError):
126	cmpNEVR('foo-1.2-3.fc20', 'bar-1.2-3.fc20')	126		cmpNEVR('foo-1.2-3.fc20', 'bar-1.2-3.fc20')
127		127
		128
128	class TestGetDistTag(object):	129		class TestGetDistTag(object):
129	def test_nvrs(self):	130		def test_nvrs(self):
130	assert get_dist_tag('foo-1.2-3.fc20') == 'fc20'	131		assert get_dist_tag('foo-1.2-3.fc20') == 'fc20'
131	assert get_dist_tag('foo-1.2-3.fc20.1') == 'fc20'	132		assert get_dist_tag('foo-1.2-3.fc20.1') == 'fc20'
132	assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20') == 'fc20'	133		assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20') == 'fc20'
133	assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20.1') == 'fc20'	134		assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20.1') == 'fc20'
134		135
135	def test_unsupported_dist_tag(self):	136		def test_unsupported_dist_tag(self):
136	with pytest.raises(exc.TaskotronValueError):	137		with pytest.raises(exc.TaskotronValueError):
137	get_dist_tag('foo-1.2-3.el7')	138		get_dist_tag('foo-1.2-3.el7')

			Path	Packages
M			libtaskotron/directives/koji_directive.py (58 lines)
M			libtaskotron/koji_utils.py (64 lines)
M			libtaskotron/rpm_utils.py (4 lines)
M			testing/test_koji_directive.py (28 lines)
M			testing/test_koji_utils.py (27 lines)
M			testing/test_rpm_utils.py (1 line)

Add downloading latest stable build to koji directive
ClosedPublic
Actions

Details

Diff Detail

Revision Contents

Diff 1172

libtaskotron/directives/koji_directive.py

libtaskotron/koji_utils.py

libtaskotron/rpm_utils.py

testing/test_koji_directive.py

testing/test_koji_utils.py

testing/test_rpm_utils.py

Add downloading latest stable build to koji directiveClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 1172

libtaskotron/directives/koji_directive.py

libtaskotron/koji_utils.py

libtaskotron/rpm_utils.py

testing/test_koji_directive.py

testing/test_koji_utils.py

testing/test_rpm_utils.py

Add downloading latest stable build to koji directive
ClosedPublic
Actions