bodhi_utils: make build2update more effective
ClosedPublic
Actions

Authored by kparal on Jul 22 2015, 1:47 PM.

Details

Reviewers

lbrabec
tflink
mkrizek
jskladan

Commits

rLTRNa843192daf66: bodhi_utils: make build2update more effective

Summary

Instead of making many single calls, make use of (somewhat undocumented)
Bodhi method 'get_updates_from_builds'. This speeds up the query time
about 8 times (in a scenario with 50 builds to query).

Test Plan

test suite adjusted. manually tested in ipython, and with upgradepath (depcheck skipped, the downloading is too slow)

Diff Detail

Repository

rLTRN libtaskotron

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

kparal retitled this revision from to bodhi_utils: make build2update more effective.Jul 22 2015, 1:47 PM

kparal updated this object.

kparal edited the test plan for this revision. (Show Details)

kparal added reviewers: lbrabec, jskladan, mkrizek, tflink.

Herald added a subscriber: tflink. · View Herald TranscriptJul 22 2015, 1:47 PM

keep 80 columns for documentation

Neat! I tried build2update() on 350 packages, this code did it in 30 seconds, code from develop branch didn't end in 6 minutes...

Macro doge: such speed very ack

This revision is now accepted and ready to land.Jul 22 2015, 2:25 PM

mkrizek added a dependent revision: D473: bodhi_utils: use internal retry logic.Jul 23 2015, 8:22 AM

jskladan accepted this revision.Jul 23 2015, 8:23 AM

Closed by commit rLTRNa843192daf66: bodhi_utils: make build2update more effective (authored by kparal). · Explain WhyJul 23 2015, 2:53 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents

			Path	Packages
M			libtaskotron/bodhi_utils.py (109 lines)
M			testing/test_bodhi_utils.py (96 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	1230	6d0d0cf		Jul 22 2015, 1:47 PM	★	★
Diff 2	1231	6d0d0cf	- keep 80 columns for documentation	Jul 22 2015, 1:51 PM	★	★
Diff 3	1244	6d0d0cf	Commit rLTRNa843192daf6624cf602ccd5fe96cbf0a2d8bd353	Jul 23 2015, 8:27 AM	★	★

Commit	Tree	Parents	Author	Summary	Date
1aefd3ef42d2	691f352afa6b	2f0dd50c72d8	Kamil Páral	keep 80 columns for documentation	Jul 22 2015, 1:50 PM
2f0dd50c72d8	c604f9f94b73	6d0d0cfa52a4	Kamil Páral	bodhi_utils: make build2update more effective (Show More…)	Jul 22 2015, 1:13 PM

	Status	Author	Revision
	Closed	kparal	D473 bodhi_utils: use internal retry logic
	Closed	kparal	D469 bodhi_utils: make build2update more effective

Diff 1244

View Options

libtaskotron/bodhi_utils.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	'''Utility functions for dealing with Bodhi'''	6		'''Utility functions for dealing with Bodhi'''
7		7
8	from __future__ import absolute_import	8		from __future__ import absolute_import
9	import fedora.client	9		import fedora.client
10		10
11	from .logger import log	11		from .logger import log
12	from . import rpm_utils	12		from . import rpm_utils
13	from . import python_utils	13		from . import python_utils
14	from . import exceptions as exc	14		from . import exceptions as exc
15	from . import config	15		from . import config
16		16
		17
17	class BodhiUtils(object):	18		class BodhiUtils(object):
18	'''Helper Bodhi methods.	19		'''Helper Bodhi methods.
19		20
20	:ivar fedora.client.BodhiClient client: Bodhi client instance	21		:ivar fedora.client.BodhiClient client: Bodhi client instance
21	'''	22		'''
22		23
		24		#: How many requests to make in a single time when using "multicall"-like
		25		#: Bodhi methods. '50' is a good size, there is a 2 minute timeout and it
		26		#: usually takes about 20 seconds for Bodhi to evaluate 50 requests, so
		27		#: there's still a lot of buffer (increasing the size further doesn't
		28		#: seem to speed it up noticeably)
		29		_MULTICALL_REQUEST_SIZE = 50
		30
23	def __init__(self, client=None):	31		def __init__(self, client=None):
24	'''Create a new BodhiUtils instance.	32		'''Create a new BodhiUtils instance.
25		33
26	:param client: custom :class:`BodhiClient` instance. If ``None``, a	34		:param client: custom :class:`BodhiClient` instance. If ``None``, a
27	default BodhiClient instance is used.	35		default BodhiClient instance is used.
28	'''	36		'''
29		37
30	self.taskotron_config = config.get_config()	38		self.taskotron_config = config.get_config()
▲ Show 20 Lines • Show All 80 Lines • ▼ Show 20 Line(s)		105	:return: a tuple of two dictionaries:
111	* The set of keys in both these dictionaries correspond exactly	119		* The set of keys in both these dictionaries correspond exactly
112	to ``builds``. For every build provided in ``builds`` you'll	120		to ``builds``. For every build provided in ``builds`` you'll
113	get an answer in either the first or the second dictionary,	121		get an answer in either the first or the second dictionary,
114	and there will be no extra builds that you haven't specified.	122		and there will be no extra builds that you haven't specified.
115	:raise TaskotronValueError: if ``builds`` type is incorrect	123		:raise TaskotronValueError: if ``builds`` type is incorrect
116	'''	124		'''
117	# validate input params	125		# validate input params
118	if not python_utils.iterable(builds, basestring):	126		if not python_utils.iterable(builds, basestring):
119	raise exc.TaskotronValueError("Param 'builds' must be an iterable "	127		raise exc.TaskotronValueError(
120	"of strings, and yours was: %s" % type(builds))	128		"Param 'builds' must be an iterable of strings, and yours was: %s" % type(builds))
121		129
		130		updates = []
122	build2update = {}	131		build2update = {}
123	failures = {}	132		failures = {}
124	# Bodhi works with NVR only, but we have to ensure we receive and return	133		# Bodhi works with NVR only, but we have to ensure we receive and return
125	# even NEVR format. So we need to convert internally.	134		# even NEVR format. So we need to convert internally.
126	builds_nvr = set([rpm_utils.rpmformat(build, 'nvr')	135		builds_nvr = set([rpm_utils.rpmformat(build, 'nvr') for build in builds])
127	for build in builds])	136		builds_queue = list(builds_nvr)
128		137
129	log.info('Querying Bodhi to map %d builds to their updates...',	138		log.info('Querying Bodhi to map %d builds to their updates...', len(builds))
130	len(builds))	139
131	for index, build in enumerate(sorted(builds)):	140		# retrieve all update data
132	# querying Bodhi takes a long time. We need to display some progress,	141		while builds_queue:
133	# else our tasks might get killed due to inactivity. Print status	142		builds_chunk = builds_queue[:self._MULTICALL_REQUEST_SIZE]
134	# for every 20th build.	143		builds_chunk = ' '.join(builds_chunk)
135	if index % 20 == 0 and index > 0:	144		# this method doesn't seem to be exposed or documented anywhere, it's
136	log.info('Bodhi queries done: %d/%d', index, len(builds))	145		# just available in the Bodhi source code:
137		146		# https://github.com/fedora-infra/bodhi/blob/master/bodhi/controllers.py#L202-L220
138	# if already found answer for this build, skip it	147		# For a space-delimited list of builds in format of NVR, it returns a
139	if build in build2update or build in failures:	148		# dictionary in format of {build: update_data}.
140	continue	149		res = self.client.send_request(method='get_updates_from_builds',
141		150		req_params={'builds': builds_chunk})
142	# get Bodhi update object	151		# there is a bug - it contains an extra item "tg_flash=None"
143	update = self.query_update(rpm_utils.rpmformat(build, 'nvr'))	152		# https://github.com/fedora-infra/bodhi/issues/152
144	if update is None:	153		res.pop('tg_flash', None)
145	# no such Bodhi update	154
146	failures[build] = None	155		updates.extend(res.values())
147	continue	156		builds_queue = builds_queue[self._MULTICALL_REQUEST_SIZE:]
148		157
		158		# don't query for builds which were already found
		159		for update in res.values():
		160		for build in update['builds']:
		161		if build['nvr'] in builds_queue:
		162		builds_queue.remove(build['nvr'])
		163
		164		log.info('Bodhi queries done: %d/%d', len(builds_nvr)-len(builds_queue),
		165		len(builds_nvr))
		166
		167		# separate builds into OK and failures
		168		for update in updates:
149	# all builds listed in the update	169		# all builds listed in the update
150	bodhi_builds = set([build['nvr'] for build in update['builds']])	170		bodhi_builds = set([build['nvr'] for build in update['builds']])
151	# builds not provided in @param builds but part of the update	171		# builds not provided in @param builds but part of the update (NVRs)
152	# (NVRs)
153	missing_builds = bodhi_builds.difference(builds_nvr)	172		missing_builds = bodhi_builds.difference(builds_nvr)
154	# builds provided in @param builds and part of the update	173		# builds provided in @param builds and part of the update
155	matched_builds = [build for build in builds if	174		matched_builds = [build for build in builds if
156	rpm_utils.rpmformat(build, 'nvr') in bodhi_builds]	175		rpm_utils.rpmformat(build, 'nvr') in bodhi_builds]
157		176
158	# reject incomplete updates	177		# reject incomplete updates when strict
159	if missing_builds and strict:	178		if missing_builds and strict:
160	for matched_build in matched_builds:	179		for build in matched_builds:
161	failures[matched_build] = update	180		failures[build] = update
162	continue	181		continue
163		182
164	# create item in build2update	183		# otherwise the update is complete or we don't care
165	for build in matched_builds:	184		for build in matched_builds:
166	build2update[build] = update	185		build2update[build] = update
167		186
168	log.info('Bodhi queries done: %d/%d', len(builds), len(builds))	187		# mark builds without any associated update as a failure
169		188		for build in builds:
170	# some builds might have been added to both @var failures and	189		if build not in build2update and build not in failures:
171	# @var build2update. That's probably a Bodhi fault, because a build	190		failures[build] = None
172	# should not be part of two different updates. If that happens, remove	191
173	# it from @var build2update and keep it in @var failures.
174	for build in failures:
175	if build in build2update:
176	log.warn("The build %s is a part of two different updates: '%s'"
177	" and '%s'. That's probably a bug in Bodhi." % (build,
178	failures[build]['title'], build2update[build]['title']))
179	del build2update[build]
180
181	# a safety check: 'build2update' combined with 'failures' should contain
182	# exactly the same builds as in the input 'builds' param. There's no
183	# reason this should go wrong, ever.
184	diff = set(builds).symmetric_difference(	192		diff = set(builds).symmetric_difference(
185	set(build2update.keys()).union(set(failures.keys())))	193		set(build2update.keys()).union(set(failures.keys())))
186	assert not diff, ("There should be no difference between input and "	194		assert not diff, "Returned NVRs different from input NVRs: %s" % diff
187	"output builds.\nInput: %s\nBuild2update: %s\n"
188	"Failures: %s\nDiff: %s" % (builds, build2update.keys(),
189	failures.keys(), diff))
190		195
191	return (build2update, failures)	196		return (build2update, failures)