implement sudo access and dnf metadata caching
ClosedPublic
Actions

Authored by kparal on Jan 27 2016, 12:54 PM.

Details

Reviewers

lbrabec
tflink
mkrizek
jskladan

Maniphest Tasks

T642: Do not download dnf metadata twice when checking if packages are installed

Commits

rLTRN50550d42edff: implement sudo access and dnf metadata caching

Summary

This allows us to automatically use sudo (if available and running
non-root) for task recipe deps installation. It also uses dnf metadata
caching for increased performance (avoiding metadata refreshes, esp.
imortant on disposable minions) for both task recipe deps installation
and initial remote minion libtaskotron installation. Dnf operations now
always use cache if available, and only refresh it when needed
(cache unavailable or the operation fails).

This does not implement sudo use for initial remote minion installation,
that needs to be dealt with later, but currently that is not a pressing
need for us.

Test Plan

Test suite has not been adjusted yet. If you like the overall direction of this, I'll fix the test suite and add new unit tests. So far, I have tested this manually with both local and remote (persistent and disposable) execution, with root/non-root+sudo/non-root, and dnf cache available/missing, it seems to work. Your testing is welcome.

Diff Detail

Repository

rLTRN libtaskotron

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

kparal retitled this revision from to implement sudo access and dnf metadata caching.Jan 27 2016, 12:54 PM

kparal updated this object.

kparal edited the test plan for this revision. (Show Details)

kparal added reviewers: tflink, mkrizek, lbrabec, jskladan.

kparal added a task: T642: Do not download dnf metadata twice when checking if packages are installed.

Looks good. I'd split it in two commits, before pushing (sudo access, dnf caching), but that's just a technicality IMO.

Looks good.

This revision is now accepted and ready to land.Feb 3 2016, 3:35 PM

I can't test this locally right now but it looks good otherwise

libtaskotron/minion.py
65	can't we do a `--disablerepo=* --enablerepo=taskotron` to achieve the same thing with less hardcoding?

libtaskotron/minion.py
65	We talked about this in person, the problem is that when I do it the other way, I'm hardcoding the name of our taskotron repository, or potentially grab more than intended (with `taskotron`), including currently disabled repos. I'd have to make it configurable. The opposite way seems safer, because Fedora repos names don't change often, and if we refresh more repos than just the taskotron one, they are usually very small so it shouldn't matter that much.

make some tiny changes and add a lot of unit tests

This revision is now accepted and ready to land.Feb 6 2016, 10:46 AM

I'm not sure if I can somehow clear the review flags (trying now), but if somebody could skim through the new changes and just check whether I'm not doing something obviously stupid here, that would be welcome, before I commit this.

I can clear the review flags. That's nice :-)

jskladan accepted this revision.Feb 8 2016, 9:26 AM

This revision is now accepted and ready to land.Feb 8 2016, 9:26 AM

kparal mentioned this in T642: Do not download dnf metadata twice when checking if packages are installed.Feb 8 2016, 2:27 PM

Closed by commit rLTRN50550d42edff: implement sudo access and dnf metadata caching (authored by kparal). · Explain WhyFeb 8 2016, 2:27 PM

This revision was automatically updated to reflect the committed changes.

mkrizek mentioned this in T703: Unicode errors during libtaskotron run when default encoding is different.Feb 9 2016, 9:59 AM

Revision Contents

		Path
M		docs/source/library.rst (6 lines)
M		libtaskotron/config_defaults.py (6 lines)
M		libtaskotron/exceptions.py (12 lines)
M		libtaskotron/executor.py (8 lines)
M		libtaskotron/ext/fedora/rpm_utils.py (132 lines)
M		libtaskotron/minion.py (21 lines)
A	M	libtaskotron/os_utils.py (46 lines)
M		libtaskotron/remote_exec.py (37 lines)
A	M	testing/test_os_utils.py (41 lines)
M		testing/test_remote_exec.py (31 lines)
M		testing/test_rpm_utils.py (202 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	1885	3d79b56		Jan 27 2016, 12:54 PM	★	★
Diff 2	1896	f77af51	make some tiny changes and add a lot of unit tests	Feb 6 2016, 10:44 AM	★	★
Diff 3	1900	c754d6c	Commit rLTRN50550d42edff4aed2bd7417936286dac2f3d240e	Feb 8 2016, 2:25 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
4b48619e88a6	4ed1015bf703	f77af5105fc8	Kamil Páral	implement sudo access and dnf metadata caching (Show More…)	Jan 27 2016, 12:13 PM

Diff 1900

View Options

docs/source/library.rst

Show First 20 Lines • Show All 74 Lines • ▼ Show 20 Line(s)		74	.. automodule:: libtaskotron.main
75	:members:	75		:members:
76		76
77	minion	77		minion
78	------	78		------
79		79
80	.. automodule:: libtaskotron.minion	80		.. automodule:: libtaskotron.minion
81	:members:	81		:members:
82		82
		83		os_utils
		84		--------
		85
		86		.. automodule:: libtaskotron.os_utils
		87		:members:
		88
83	overlord	89		overlord
84	--------	90		--------
85		91
86	.. automodule:: libtaskotron.overlord	92		.. automodule:: libtaskotron.overlord
87	:members:	93		:members:
88		94
89	python_utils	95		python_utils
90	------------	96		------------
▲ Show 20 Lines • Show All 54 Lines • Show Last 20 Lines

View Options

libtaskotron/config_defaults.py

Show First 20 Lines • Show All 74 Lines • ▼ Show 20 Line(s)		31	class Config(object):
75	client_taskdir = '/var/tmp/taskotron/taskdir' #:	75		client_taskdir = '/var/tmp/taskotron/taskdir' #:
76	artifactsdir = '/var/lib/taskotron/artifacts' #:	76		artifactsdir = '/var/lib/taskotron/artifacts' #:
77	cachedir = '/var/cache/taskotron' #:	77		cachedir = '/var/cache/taskotron' #:
78	imagesdir = '/var/lib/taskotron/images' #:	78		imagesdir = '/var/lib/taskotron/images' #:
79	imageurl = 'http://download.fedoraproject.org/pub/fedora/linux/'\	79		imageurl = 'http://download.fedoraproject.org/pub/fedora/linux/'\
80	'releases/23/Cloud/x86_64/Images/Fedora-Cloud-'\	80		'releases/23/Cloud/x86_64/Images/Fedora-Cloud-'\
81	'Base-23-20151030.x86_64.qcow2' #:	81		'Base-23-20151030.x86_64.qcow2' #:
82		82
83	minion_repos = ['https://copr-fe.cloud.fedoraproject.org/coprs/tflink/'	83		minion_repos = [
84	'taskotron/repo/fedora-23/'	84		'https://copr-fe.cloud.fedoraproject.org/coprs/tflink/taskotron/repo/fedora-23/'
85	'tflink-taskotron-fedora-23.repo'] #:	85		'tflink-taskotron-fedora-23.repo'] #:
86		86
87	log_level_stream = 'INFO' #:	87		log_level_stream = 'INFO' #:
88	log_level_file = 'DEBUG' #:	88		log_level_file = 'DEBUG' #:
89		89
90	log_file_enabled = False #:	90		log_file_enabled = False #:
91		91
92	ssh_privkey = '' #:	92		ssh_privkey = '' #:
93		93
Show All 37 Lines

View Options

libtaskotron/exceptions.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	'''This module contains custom Taskotron exceptions'''	6		'''This module contains custom Taskotron exceptions'''
7		7
8	from __future__ import absolute_import	8		from __future__ import absolute_import
9		9
		10
10	class TaskotronError(Exception):	11		class TaskotronError(Exception):
11	'''Common ancestor for Taskotron related exceptions'''	12		'''Common ancestor for Taskotron related exceptions'''
12	pass	13		pass
13		14
14		15
15	class TaskotronValueError(ValueError, TaskotronError):	16		class TaskotronValueError(ValueError, TaskotronError):
16	'''Taskotron-specific :class:`ValueError`'''	17		'''Taskotron-specific :class:`ValueError`'''
17	pass	18		pass
Show All 14 Lines		31	class TaskotronDirectiveError(TaskotronError):
32	pass	33		pass
33		34
34		35
35	class TaskotronRemoteError(TaskotronError):	36		class TaskotronRemoteError(TaskotronError):
36	'''All network and remote-server related errors'''	37		'''All network and remote-server related errors'''
37	pass	38		pass
38		39
39		40
		41		class TaskotronRemoteProcessError(TaskotronRemoteError):
		42		'''A process/command executed remotely returned a non-zero exit code, crashed or failed in a
		43		different way'''
		44		pass
		45
		46
40	class TaskotronNotImplementedError(TaskotronError, NotImplementedError):	47		class TaskotronNotImplementedError(TaskotronError, NotImplementedError):
41	'''NotImplementedError for Taskotron classes, methods and functions'''	48		'''NotImplementedError for Taskotron classes, methods and functions'''
42		49
43		50
44	class TaskotronImportError(TaskotronError):	51		class TaskotronImportError(TaskotronError):
45	'''All issues with Extensions'''	52		'''All issues with Extensions'''
46		53
47	def __init__(self, e):	54		def __init__(self, e):
48	msg = str(e.message)	55		msg = str(e.message)
49	if 'libtaskotron' in msg:	56		if 'libtaskotron' in msg:
50	msg += "\nHint: Make sure that all required sub-packages are installed."	57		msg += "\nHint: Make sure that all required sub-packages are installed."
51	super(TaskotronError, self).__init__(msg)	58		super(TaskotronError, self).__init__(msg)
52		59
53		60
54	class TaskotronImageError(TaskotronError):	61		class TaskotronImageError(TaskotronError):
55	'''All generic image related errors'''	62		'''All generic image related errors'''
56	pass	63		pass
57		64
58		65
59	class TaskotronImageNotFoundError(TaskotronImageError):	66		class TaskotronImageNotFoundError(TaskotronImageError):
60	'''Requested image not found error'''	67		'''Requested image not found error'''
61	pass	68		pass
		69
		70
		71		class TaskotronPermissionError(TaskotronError):
		72		'''Insufficient permissions or privileges'''
		73		pass

View Options

libtaskotron/executor.py

Show First 20 Lines • Show All 55 Lines • ▼ Show 20 Line(s)		49	def execute(self):
56	self._run()	56		self._run()
57		57
58	def _prepare_task(self):	58		def _prepare_task(self):
59	'''Prepare the environment for running a task. If there are packages, requirements or	59		'''Prepare the environment for running a task. If there are packages, requirements or
60	groups specified in the task formula, this method tries to install them (in case of	60		groups specified in the task formula, this method tries to install them (in case of
61	production profile) or checks whether they are installed (in case of developer profile).	61		production profile) or checks whether they are installed (in case of developer profile).
62	'''	62		'''
63	rpms = self.task_data.get('environment', {}).get('rpm', [])	63		rpms = self.task_data.get('environment', {}).get('rpm', [])
64
65	profile = config.get_config().profile	64		profile = config.get_config().profile
66		65
67	# FIXME: change this behaviour in future, so we don't download the metadata twice
68	# temporary hack for systems in production with the required packages installed, but not
69	# permissions to run package installation
70	if not rpm_utils.is_installed(rpms):	66		if not rpm_utils.is_installed(rpms):
71	if profile == config.ProfileName.PRODUCTION or self.arg_data['local']:	67		if profile == config.ProfileName.PRODUCTION or self.arg_data['local']:
72	rpm_utils.install(rpms)	68		rpm_utils.install(rpms)
73	else:	69		else:
74	raise exc.TaskotronError("Some packages are not installed. Please run "	70		raise exc.TaskotronError('Some packages are not installed. Please run '
75	"'dnf install %s' to install all required packages."	71		'"dnf install %s" to install all required packages.'
76	% " ".join([pipes.quote(rpm) for rpm in rpms]))	72		% " ".join([pipes.quote(rpm) for rpm in rpms]))
77		73
78	def _run(self):	74		def _run(self):
79	self._validate_input()	75		self._validate_input()
80		76
81	if not self.workdir: # create temporary workdir if needed	77		if not self.workdir: # create temporary workdir if needed
82	self.workdir = tempfile.mkdtemp(prefix="task-",	78		self.workdir = tempfile.mkdtemp(prefix="task-",
83	dir=config.get_config().tmpdir)	79		dir=config.get_config().tmpdir)
▲ Show 20 Lines • Show All 129 Lines • Show Last 20 Lines

View Options

libtaskotron/ext/fedora/rpm_utils.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	''' Utility methods related to RPM '''	6		''' Utility methods related to RPM '''
7		7
8	from __future__ import absolute_import	8		from __future__ import absolute_import
9	import re	9		import re
10	import hawkey	10		import hawkey
11	import subprocess	11		import subprocess
12	import pipes	12		import pipes
13		13
14	from libtaskotron.logger import log	14		from libtaskotron.logger import log
15	from libtaskotron import exceptions as exc	15		from libtaskotron import exceptions as exc
		16		from libtaskotron import os_utils
16		17
17		18
18	def rpmformat(rpmstr, fmt='nvr', end_arch=False):	19		def rpmformat(rpmstr, fmt='nvr', end_arch=False):
19	'''	20		'''
20	Parse and convert an RPM package version string into a different format.	21		Parse and convert an RPM package version string into a different format.
21	String identifiers: N - name, E - epoch, V - version, R - release, A -	22		String identifiers: N - name, E - epoch, V - version, R - release, A -
22	architecture.	23		architecture.
23		24
▲ Show 20 Lines • Show All 76 Lines • ▼ Show 20 Line(s)		80	def cmpNEVR(nevr1, nevr2):
100	sack = hawkey.Sack()	101		sack = hawkey.Sack()
101		102
102	# we need evr_cmp to return int so we can use it as comparison function	103		# we need evr_cmp to return int so we can use it as comparison function
103	# in python's sorted	104		# in python's sorted
104	return int(rpmver1.evr_cmp(rpmver2, sack))	105		return int(rpmver1.evr_cmp(rpmver2, sack))
105		106
106		107
107	def install(pkgs):	108		def install(pkgs):
108	'''Install packages from system repositories using DNF. You need to have enough permissions	109		'''Install packages from system repositories using DNF. Either root or sudo access required.
109	to run ``dnf`` through ``sudo``.
110		110
111	:param pkgs: packages to be installed, e.g. ``['pidgin']``, or any other specifier	111		Note: This is first tested with ``--cacheonly`` to avoid downloading metadata and speed up the
112	supported by ``dnf install`` command	112		process. If that command fails for whatever reason, we run it again, this time without
		113		``--cacheonly``.
		114
		115		:param pkgs: packages to be installed, e.g. ``['pidgin']``, or any other argument supported by
		116		``dnf install`` command
113	:type pkgs: list of str	117		:type pkgs: list of str
114	:raise TaskotronError: if DNF returncode is not zero	118		:raise TaskotronPermissionError: if we don't have permissions to run DNF as admin
		119		:raise TaskotronError: if DNF return code is not zero
115	'''	120		'''
116	cmd = ['sudo', 'dnf', 'install', '-y']	121		if not pkgs:
117	cmd.extend(pkgs)	122		return
		123		pkglist = ' '.join([pipes.quote(pkg) for pkg in pkgs])
118		124
119	log.info("Running command: %s", " ".join([pipes.quote(c) for c in cmd]))	125		if not os_utils.is_root() and not os_utils.has_sudo():
		126		raise exc.TaskotronPermissionError("Can't install packages without root or sudo access. "
		127		'Packages requested: %s' % pkglist)
120		128
121	proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)	129		cmd = ['dnf', '--assumeyes', 'install']
122	stdout, _ = proc.communicate()	130		cmd.extend(pkgs)
123		131
124	if proc.returncode != 0:	132		if dnf_cache_available():
125	log.error(stdout)	133		cmd.insert(1, '--cacheonly')
126	raise exc.TaskotronError("Unable to install packages: %s" % " ".join(pkgs))
127	else:	134		else:
128	log.debug(stdout)	135		log.warn('No DNF cache available, metadata will be need to be downloaded. If your DNF '
		136		"cache doesn't persist across task executions (e.g. on a disposable VM), "
		137		'consider creating the cache in your base system image to speed up execution.')
		138
		139		if not os_utils.is_root(): # we must have sudo at this point, don't test again needlessly
		140		cmd = ['sudo', '--non-interactive'] + cmd
		141
		142		log.info('Installing %d packages...', len(pkgs))
		143
		144		while True: # we need to call the command twice, if the first run with --cacheonly fails
		145		log.debug('Running: %s', ' '.join([pipes.quote(c) for c in cmd]))
		146
		147		try:
		148		output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
		149		except subprocess.CalledProcessError as e:
		150		if '--cacheonly' in cmd: # maybe just outdated cache
		151		log.debug('✘ Package installation failed, DNF output was:\n%s', e.output.rstrip())
		152		log.debug("Trying again with forced metadata refresh...")
		153		cmd = list(cmd) # make a copy to allow unit tests to intercept both calls easily
		154		cmd.remove('--cacheonly')
		155		cmd.append('--refresh')
		156		continue
		157
		158		log.error('✘ Package installation failed. We tried to install following packages:\n%s'
		159		'\nDNF returned exit code %d and output:\n%s', pkglist, e.returncode,
		160		e.output.rstrip())
		161		raise exc.TaskotronError("Unable to install packages: %s" % pkglist)
		162		else:
		163		log.debug('✔ Package installation completed successfully. DNF output was:\n%s',
		164		output.rstrip())
		165		return
		166
		167		# safeguard against infinite loop, we should never reach this code
		168		log.critical("Infinite loop detected in 'install()'")
		169		assert False, "Infinite loop detected in 'install()'"
		170		return
129		171
130		172
131	def is_installed(pkgs):	173		def is_installed(pkgs):
132	'''Check if packages are installed using DNF. No additional permissions needed.	174		'''Check if packages are installed using DNF. No elevated permissions needed.
133		175
134	:param pkgs: packages to be checked whether they are installed, e.g. ``['pidgin']``,	176		:param pkgs: packages to be checked whether they are installed, e.g. ``['pidgin']`` or any
135	or any other specifier supported by ``dnf install`` command	177		other argument supported by ``dnf install`` command
136	:type pkgs: list of str	178		:type pkgs: list of str
137	:return: True if all specified packages are installed, False otherwise	179		:return: True if all specified packages are installed, False otherwise
138	:rtype: bool	180		:rtype: bool
139	'''	181		'''
140	if not pkgs:	182		if not pkgs:
141	return True	183		return True
142		184
143	cmd = ['dnf', '--assumeno', '--disableplugin=noroot', 'install']	185		cmd = ['dnf', '--assumeno', '--disableplugin=noroot', 'install']
144	cmd.extend(pkgs)	186		cmd.extend(pkgs)
145		187
146	log.debug("Running command: %s", " ".join([pipes.quote(c) for c in cmd]))	188		if dnf_cache_available():
		189		cmd.insert(1, '--cacheonly')
		190		else:
		191		log.warn('No DNF cache available, metadata will be need to be downloaded. If your DNF '
		192		"cache doesn't persist across task executions (e.g. on a disposable VM), "
		193		'consider creating the cache in your base system image to speed up execution.')
		194
		195		if not os_utils.is_root() and os_utils.has_sudo():
		196		cmd = ['sudo', '--non-interactive'] + cmd
		197
		198		log.info('Checking installed state of %d packages...', len(pkgs))
		199		log.debug('Running: %s', ' '.join([pipes.quote(c) for c in cmd]))
		200
		201		try:
		202		subprocess.check_output(cmd, stderr=subprocess.STDOUT)
		203		except subprocess.CalledProcessError as e:
		204		log.debug('✘ Some of specified packages are not installed. DNF returned exit code %d and '
		205		'output:\n%s', e.returncode, e.output.rstrip())
		206		return False
		207		else:
		208		log.debug('✔ All specified packages are installed')
		209		return True
147		210
148	proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
149	stdout, _ = proc.communicate()
150	log.debug(stdout)
151		211
152	return not proc.returncode	212		def dnf_cache_available():
		213		'''Determine whether DNF cache is available.
		214
		215		If it is, we can run read-only actions solely from the cache, and try to use it even for
		216		``install`` action (provided the repos haven't changed much). This way we avoid contacting
		217		DNF repos and downloading new metadata if they changed.
		218
		219		This tests the system cache (if running as root or with sudo access) or the user cache
		220		(otherwise). Since DNF has no direct way of testing this, we simply run an action that should
		221		always pass (``dnf repolist``) with ``--cacheonly`` option and look at the exit code.
		222
		223		:return: bool whether DNF cache is currently available and can be used (system or user cache,
		224		depending on your current admin privileges)
		225		'''
		226		cmd = ['dnf', '--cacheonly', 'repolist']
		227
		228		if not os_utils.is_root() and os_utils.has_sudo():
		229		cmd = ['sudo', '--non-interactive'] + cmd
		230
		231		log.debug('Deciding whether DNF cache is available. Running: %s', ' '.join(cmd))
		232		try:
		233		subprocess.check_output(cmd, stderr=subprocess.STDOUT)
		234		except subprocess.CalledProcessError as e:
		235		log.debug('✘ DNF cache is not available. Received exit code %d and output:\n%s',
		236		e.returncode, e.output.rstrip())
		237		return False
		238		else:
		239		log.debug('✔ DNF cache is available.')
		240		return True
153		241
154		242
155	def get_dist_tag(rpmstr):	243		def get_dist_tag(rpmstr):
156	'''Parse disttag from an RPM package version string.	244		'''Parse disttag from an RPM package version string.
157		245
158	:param str rpmstr: string to be manipulated in a format of N(E)VR	246		:param str rpmstr: string to be manipulated in a format of N(E)VR
159	(``foo-1.2-3.fc20`` or ``bar-4:1.2-3.fc20``) or N(E)VRA	247		(``foo-1.2-3.fc20`` or ``bar-4:1.2-3.fc20``) or N(E)VRA
160	(``foo-1.2-3.fc20.x86_64`` or ``bar-4:1.2-3.fc20.i686``)	248		(``foo-1.2-3.fc20.x86_64`` or ``bar-4:1.2-3.fc20.i686``)
Show All 12 Lines

View Options

libtaskotron/minion.py

Show First 20 Lines • Show All 42 Lines • ▼ Show 20 Line(s)		29	def __init__(self, task_data, arg_data):
43	self.port = arg_data.get('port', None)	43		self.port = arg_data.get('port', None)
44	self.ssh_privkey = arg_data['ssh_privkey'] or config.get_config().ssh_privkey	44		self.ssh_privkey = arg_data['ssh_privkey'] or config.get_config().ssh_privkey
45	#: int exitcode set by exitcode directive.	45		#: int exitcode set by exitcode directive.
46	self.exitcode = None	46		self.exitcode = None
47	self.ssh = None	47		self.ssh = None
48		48
49	def _prepare_task(self):	49		def _prepare_task(self):
50	'''Prepare the environment to run the task.'''	50		'''Prepare the environment to run the task.'''
		51		# TODO: add sudo support
51		52
52	self.taskdir = config.get_config().client_taskdir	53		self.taskdir = config.get_config().client_taskdir
53		54
		55		# create initial DNF cache (but only if needed)
		56		# uses the same logic as in libtaskotron.ext.fedora.rpm_utils.dnf_cache_available()
		57		self.ssh.cmd('dnf --cacheonly repolist \|\| dnf makecache')
		58
54	# add additional repos to minion	59		# add additional repos to minion
55	repos = ['--add-repo %s' % repo for repo in config.get_config().minion_repos]	60		repos = ['--add-repo %s' % pipes.quote(repo) for repo in config.get_config().minion_repos]
56	if repos:	61		if repos:
57	self.ssh.cmd('dnf config-manager %s' % ' '.join(repos))	62		self.ssh.cmd('dnf config-manager %s' % ' '.join(repos))
58		63
		64		# this hard-coding is very suboptimal, but it's hard to find a better way how to refresh
		65		# only taskotron repos (unless we require their names listed in configuration)
			tflinkUnsubmitted Not Done can't we do a `--disablerepo=* --enablerepo=taskotron` to achieve the same thing with less hardcoding?
			kparalAuthorUnsubmitted Not Done We talked about this in person, the problem is that when I do it the other way, I'm hardcoding the name of our taskotron repository, or potentially grab more than intended (with `taskotron`), including currently disabled repos. I'd have to make it configurable. The opposite way seems safer, because Fedora repos names don't change often, and if we refresh more repos than just the taskotron one, they are usually very small so it shouldn't matter that much.
		66		excluded = ['fedora', 'updates', 'updates-testing']
		67		exclude_cmd = ' '.join(['--disablerepo %s' % pipes.quote(repo) for repo in excluded])
		68		self.ssh.cmd('dnf makecache %s' % exclude_cmd)
		69
59	# FIXME this should only need libtaskotron-core T651	70		# FIXME this should only need libtaskotron-core T651
60	# libtaskotron-fedora should be installed as a check dep	71		# libtaskotron-fedora should be installed as a check dep
61	self.ssh.install_pkgs(['libtaskotron-core', 'libtaskotron-fedora'])	72		self.ssh.install_pkgs(['libtaskotron-core', 'libtaskotron-fedora'])
62		73
63	# Copy config files. Go through CONF_DIRS in reversed order to preserve priority.	74		# Copy config files. Go through CONF_DIRS in reversed order to preserve priority.
64	for config_dir in reversed(config.CONF_DIRS):	75		for config_dir in reversed(config.CONF_DIRS):
65	if os.path.exists(config_dir):	76		if os.path.exists(config_dir):
66	self.ssh.put_dir(config_dir, '/etc/taskotron/')	77		self.ssh.put_dir(config_dir, '/etc/taskotron/')
▲ Show 20 Lines • Show All 85 Lines • ▼ Show 20 Line(s)		161	try:
152		163
153	log.info("Creating VM for task")	164		log.info("Creating VM for task")
154	task_vm = vm.TestCloudMachine(self.arg_data['uuid'])	165		task_vm = vm.TestCloudMachine(self.arg_data['uuid'])
155	task_vm.prepare()	166		task_vm.prepare()
156	self.machine = task_vm.ipaddr	167		self.machine = task_vm.ipaddr
157	self.port = 22	168		self.port = 22
158	self.user = task_vm.username	169		self.user = task_vm.username
159		170
160	log.info("Running Task over SSH on machine %s" % task_vm.instancename)	171		log.info('Running task over SSH on machine %s (%s@%s:%s)', task_vm.instancename,
		172		self.user, self.machine, self.port)
161		173
162	task_vm.wait_for_port(self.port)	174		task_vm.wait_for_port(self.port)
163		175
164	with remote_exec.ParamikoWrapper(self.machine, self.port, self.user,	176		with remote_exec.ParamikoWrapper(self.machine, self.port, self.user,
165	self.ssh_privkey, stdio_filename) as self.ssh:	177		self.ssh_privkey, stdio_filename) as self.ssh:
166	self._prepare_task()	178		self._prepare_task()
167	self._run()	179		self._run()
168	self._get_output()	180		self._get_output()
169	execution_ok = True	181		execution_ok = True
170	finally:	182		finally:
171	if self.arg_data['no_destroy']:	183		if self.arg_data['no_destroy']:
172	log.info("--no-destroy mode was set from command line, not tearing VM "	184		log.info('--no-destroy mode was set from command line, not tearing VM '
173	"down for task %s" % self.arg_data['uuid'])	185		'down for task %s: %s@%s:%s', self.arg_data['uuid'], self.user,
		186		self.machine, self.port)
174	else:	187		else:
175	try:	188		try:
176	task_vm.teardown()	189		task_vm.teardown()
177	# Catch 'em all since we can't be sure that every exception trown	190		# Catch 'em all since we can't be sure that every exception trown
178	# is TaskotronRemoteError. If we omitted any exception that would	191		# is TaskotronRemoteError. If we omitted any exception that would
179	# be later thrown in the teardown, we would suppress (effectively	192		# be later thrown in the teardown, we would suppress (effectively
180	# make it silent) any possible exception from execution above	193		# make it silent) any possible exception from execution above
181	# which is more important.	194		# which is more important.
182	except Exception, e:	195		except Exception, e:
183	if execution_ok:	196		if execution_ok:
184	raise e	197		raise e
185	else:	198		else:
186	# a different exception is already being raised, don't override it	199		# a different exception is already being raised, don't override it
187	log.exception(e)	200		log.exception(e)

View Options

libtaskotron/os_utils.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2009-2016, Red Hat, Inc.
		3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
		4		# See the LICENSE file for more details on Licensing
		5
		6		''' Utility methods related to an operating system '''
		7
		8		from __future__ import absolute_import
		9		import os
		10		import subprocess
		11
		12		from libtaskotron.logger import log
		13
		14
		15		def is_root():
		16		'''Determine whether we're currently running under the root account.
		17
		18		:rtype: bool
		19		'''
		20		return os.geteuid() == 0
		21
		22
		23		def has_sudo():
		24		'''Determine whether we currently have a password-less access to sudo.
		25
		26		Note: It's not possible to say whether the access will stay password-less in the future (the
		27		credentials might be set to expire in time), just for this exact moment.
		28
		29		:rtype: bool
		30		'''
		31		# Note: We could run "sudo --reset-timestamp" first to make sure any cached password is
		32		# invalidated and therefore be sure we always have a password-less access. But this might be
		33		# seen as an impolite thing to do (esp. when we're not granted the sudo access, we would still
		34		# reset the cache with every run) and it might obstruct some use cases. Not implemented ATM.
		35		cmd = ['sudo', '--validate', '--non-interactive']
		36		log.debug('Deciding whether we have a password-less sudo access. Running: %s', ' '.join(cmd))
		37
		38		try:
		39		subprocess.check_output(cmd, stderr=subprocess.STDOUT)
		40		except subprocess.CalledProcessError as e:
		41		log.debug("✘ Sudo access is not available. Received exit code %d and output:\n%s",
		42		e.returncode, e.output.rstrip())
		43		return False
		44		else:
		45		log.debug('✔ Sudo access is available')
		46		return True

View Options

libtaskotron/remote_exec.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2015, Red Hat, Inc.	2		# Copyright 2009-2015, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	'''Tools for remote execution primary for disposable clients'''	6		'''Tools for remote execution primary for disposable clients'''
7		7
8	from __future__ import absolute_import	8		from __future__ import absolute_import
9	import os	9		import os
10	import os.path	10		import os.path
11	import sys	11		import sys
12	import socket	12		import socket
13	from stat import S_ISDIR	13		from stat import S_ISDIR
14	from time import sleep	14		from time import sleep
		15		import pipes
15		16
16	import paramiko	17		import paramiko
17		18
18	from .logger import log	19		from .logger import log
19	from . import exceptions as exc	20		from . import exceptions as exc
20		21
21	from libtaskotron.directives import exitcode_directive	22		from libtaskotron.directives import exitcode_directive
22	from libtaskotron.file_utils import Tee	23		from libtaskotron.file_utils import Tee
23		24
24		25
25	class ParamikoWrapper(object):	26		class ParamikoWrapper(object):
26	'''Wrapper for SSH communication using paramiko library'''	27		'''Wrapper for SSH communication using paramiko library'''
27		28
28	# timeout for network operations in seconds (900 seconds = 15 minutes)	29		#: timeout for network operations in seconds (900 seconds = 15 minutes)
29	TIMEOUT = 900	30		TIMEOUT = 900
30		31
31	def __init__(self, hostname, port, username, key_filename, stdio_filename=None):	32		def __init__(self, hostname, port, username, key_filename, stdio_filename=None):
32	self.ssh = None	33		self.ssh = None
33	self.sftp = None	34		self.sftp = None
34	self.hostname = hostname	35		self.hostname = hostname
35	self.port = port	36		self.port = port
36	self.username = username	37		self.username = username
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Line(s)		97	def close(self):
100	self.sftp.close()	101		self.sftp.close()
101	self.ssh.close()	102		self.ssh.close()
102		103
103	def cmd(self, cmd):	104		def cmd(self, cmd):
104	'''Execute a command.	105		'''Execute a command.
105		106
106	:param str cmd: A command to be executed	107		:param str cmd: A command to be executed
107	:returns: returncode of the command	108		:returns: returncode of the command
108	:raise TaskotronRemoteError: If the command has non-zero return code and it isn't a code of	109		:raise TaskotronRemoteProcessError: If the command has non-zero return code and it isn't a
109	exitcode directive failure. Or if the remote hasn't sent any	110		code of the exitcode directive.
110	data for ``TIMEOUT`` seconds.	111		:raise TaskotronRemoteError: If the remote hasn't sent any output for :attr:`TIMEOUT`.
111	'''	112		'''
112		113
113	log.debug('Running command on remote host: %s', cmd)	114		log.debug('Running command on remote host: %s', cmd)
114		115
115	stdin, stdout, stderr = self.ssh.exec_command(cmd, timeout=self.TIMEOUT)	116		stdin, stdout, stderr = self.ssh.exec_command(cmd, timeout=self.TIMEOUT)
116		117
117	# stdout.channel represents channel for both stdout and stderr	118		# stdout.channel represents channel for both stdout and stderr
118	channel = stdout.channel	119		channel = stdout.channel
Show All 11 Lines		130	if channel.recv_stderr_ready():
130	data = channel.recv_stderr(1024)	131		data = channel.recv_stderr(1024)
131	self.outstream.write(data)	132		self.outstream.write(data)
132	alive_counter = 0	133		alive_counter = 0
133		134
134	sleep(0.1)	135		sleep(0.1)
135	alive_counter += 0.1	136		alive_counter += 0.1
136		137
137	if alive_counter >= self.TIMEOUT:	138		if alive_counter >= self.TIMEOUT:
138	raise exc.TaskotronRemoteError('No output received from remote %s@%s:%s for %d '	139		raise exc.TaskotronRemoteError(
139	'seconds while running: %s' %	140		'No output received from machine %s@%s:%s for %d seconds while running: %s' %
140	(self.username, self.hostname, self.port,	141		(self.username, self.hostname, self.port, self.TIMEOUT, cmd))
141	self.TIMEOUT, cmd))
142		142
143	retcode = channel.recv_exit_status()	143		retcode = channel.recv_exit_status()
144		144
145	# read the rest of data that was left in the buffers	145		# read the rest of data that was left in the buffers
146	self.outstream.write(stdout.read())	146		self.outstream.write(stdout.read())
147	self.outstream.write(stderr.read())	147		self.outstream.write(stderr.read())
148		148
149	if retcode != 0 and retcode != exitcode_directive.FAILURE:	149		if retcode != 0 and retcode != exitcode_directive.FAILURE:
150	raise exc.TaskotronRemoteError('Command "%s" on %s@%s exited with code %s' %	150		raise exc.TaskotronRemoteProcessError('Command "%s" on %s@%s exited with code %s' %
151	(cmd, self.username, self.hostname, retcode))	151		(cmd, self.username, self.hostname, retcode))
152	return retcode	152		return retcode
153		153
154	def install_pkgs(self, pkgs):	154		def install_pkgs(self, pkgs):
155	'''Install packages via dnf.	155		'''Install packages via dnf.
156		156
157	:param list pkgs: A list of packages to be installed	157		First tries to install packages using ``dnf --cacheonly --best``. Running from cache
		158		improves performance, while ``--best`` ensures updating to the latest version. If this
		159		doesn't work for some reason, we refresh the cache, drop ``--best`` and try again.
158		160
159	:raise TaskotronRemoteError: If the command has non-zero return code	161		:param list pkgs: A list of packages to be installed (supports any argument that
		162		``dnf install`` accepts)
		163		:raise TaskotronRemoteError: If the command has non-zero return code or times out (see
		164		:meth:`cmd`).
160	'''	165		'''
161		166
		167		pkgs_esc = [pipes.quote(pkg) for pkg in pkgs]
162	log.info('Installing %d packages on remote host...', len(pkgs))	168		log.info('Installing %d packages on remote host...', len(pkgs))
163	self.cmd('dnf -y install %s' % ' '.join(pkgs))	169		try:
		170		self.cmd('dnf --assumeyes --cacheonly --best install %s' % ' '.join(pkgs_esc))
		171		except exc.TaskotronRemoteProcessError as e:
		172		log.debug('Installation failed: %s', e)
		173		log.debug('Trying again with forced metadata refresh...')
		174		self.cmd('dnf --assumeyes --refresh install %s' % ' '.join(pkgs_esc))
164		175
165	def write_file(self, remote_path, data, overwrite=True):	176		def write_file(self, remote_path, data, overwrite=True):
166	'''Write data to a remote file.	177		'''Write data to a remote file.
167		178
168	:param str remote_path: A path to the remote file	179		:param str remote_path: A path to the remote file
169	:param str data: Data to be written	180		:param str data: Data to be written
170	:param bool overwrite: Whether to overwrite remote path. Default is True.	181		:param bool overwrite: Whether to overwrite remote path. Default is True.
171		182
▲ Show 20 Lines • Show All 142 Lines • Show Last 20 Lines

View Options

testing/test_os_utils.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2016, Red Hat, Inc.
		3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
		4		# See the LICENSE file for more details on Licensing
		5
		6		'''Unit tests for libtaskotron.os_utils'''
		7
		8		import os
		9		import subprocess
		10		import mock
		11
		12		from libtaskotron.os_utils import is_root, has_sudo
		13
		14
		15		class TestIsRoot():
		16		'''Test os_utils.is_root()'''
		17
		18		def test_root(self, monkeypatch):
		19		mock_geteuid = mock.Mock(return_value=0)
		20		monkeypatch.setattr(os, 'geteuid', mock_geteuid)
		21		assert is_root()
		22
		23		def test_not_root(self, monkeypatch):
		24		mock_geteuid = mock.Mock(return_value=1000)
		25		monkeypatch.setattr(os, 'geteuid', mock_geteuid)
		26		assert not is_root()
		27
		28
		29		class TestHasSudo():
		30		'''Test os_utils.has_sudo()'''
		31
		32		def test_sudo(self, monkeypatch):
		33		mock_check_output = mock.Mock()
		34		monkeypatch.setattr(subprocess, 'check_output', mock_check_output)
		35		assert has_sudo()
		36
		37		def test_no_sudo(self, monkeypatch):
		38		err = subprocess.CalledProcessError(1, 'cmd', output='')
		39		mock_check_output = mock.Mock(side_effect=err)
		40		monkeypatch.setattr(subprocess, 'check_output', mock_check_output)
		41		assert not has_sudo()

View Options

testing/test_remote_exec.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2015, Red Hat, Inc.	2		# Copyright 2009-2015, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	'''Unit tests for libtaskotron/remote_exec.py'''	6		'''Unit tests for libtaskotron/remote_exec.py'''
7		7
8	import errno	8		import errno
9	import socket	9		import socket
10		10
11	import pytest	11		import pytest
12	import paramiko	12		import paramiko
13	from munch import Munch	13		from munch import Munch
14	from dingus import Dingus	14		from dingus import Dingus
		15		import mock
15		16
16	from libtaskotron import remote_exec	17		from libtaskotron import remote_exec
17	from libtaskotron.exceptions import TaskotronRemoteError	18		from libtaskotron.exceptions import TaskotronRemoteError, TaskotronRemoteProcessError
18		19
19		20
20	STUB_DIR = 'dirname'	21		STUB_DIR = 'dirname'
21	STUB_FILE = 'filename'	22		STUB_FILE = 'filename'
22	STUB_NONEXISTING_FILE = ''	23		STUB_NONEXISTING_FILE = ''
23	# stub for a file that is too large (no space left on a device)	24		# stub for a file that is too large (no space left on a device)
24	STUB_ENOSPC = 1	25		STUB_ENOSPC = 1
25	# file mode	26		# file mode
▲ Show 20 Lines • Show All 202 Lines • ▼ Show 20 Line(s)		145	class TestRemoteExec(object):
228	def test_cmd_success(self):	229		def test_cmd_success(self):
229	assert self.remote.cmd(STUB_CMD_TRUE) == STUB_CMD_TRUE	230		assert self.remote.cmd(STUB_CMD_TRUE) == STUB_CMD_TRUE
230		231
231	def test_cmd_failure(self):	232		def test_cmd_failure(self):
232	with pytest.raises(TaskotronRemoteError):	233		with pytest.raises(TaskotronRemoteError):
233	self.remote.cmd(STUB_CMD_FALSE)	234		self.remote.cmd(STUB_CMD_FALSE)
234		235
235	def test_install_pkgs(self):	236		def test_install_pkgs(self):
236	self.remote.cmd = Dingus()	237		self.remote.cmd = mock.Mock()
237	self.remote.install_pkgs(['libtaskotron', 'rpmlint'])	238		self.remote.install_pkgs(['libtaskotron', 'rpmlint'])
238	assert self.remote.cmd.calls()[0][1][0] == 'dnf -y install libtaskotron rpmlint'	239		assert self.remote.cmd.call_count == 1
		240		cmd = self.remote.cmd.call_args[0][0]
		241		words = ['dnf', 'install', 'libtaskotron', 'rpmlint']
		242		assert all([word in cmd for word in words])
		243
		244		def test_install_pkgs_try_twice(self):
		245		'''If the first execution fails, modify dnf options and try again'''
		246		err = TaskotronRemoteProcessError()
		247		self.remote.cmd = mock.Mock(side_effect=[err, 0])
		248		pkgs = ['libtaskotron', 'rpmlint']
		249		self.remote.install_pkgs(pkgs)
		250
		251		assert self.remote.cmd.call_count == 2
		252		cmd1 = self.remote.cmd.call_args_list[0][0][0]
		253		words1 = ['dnf', 'install', '--cacheonly', '--best'] + pkgs
		254		assert all([word in cmd1 for word in words1])
		255		cmd2 = self.remote.cmd.call_args_list[1][0][0]
		256		words2 = ['dnf', 'install', '--refresh'] + pkgs
		257		assert all([word in cmd2 for word in words2])
		258
		259		def test_install_pkgs_fail(self):
		260		self.remote.cmd = mock.Mock(side_effect=TaskotronRemoteProcessError())
		261		with pytest.raises(TaskotronRemoteProcessError):
		262		self.remote.install_pkgs(['libtaskotron', 'rpmlint'])
		263		assert self.remote.cmd.call_count == 2
239		264
240	def test_mkdir(self):	265		def test_mkdir(self):
241	ref_dirname = 'dirname'	266		ref_dirname = 'dirname'
242	self.remote.cmd = Dingus()	267		self.remote.cmd = Dingus()
243	self.remote.mkdir(ref_dirname)	268		self.remote.mkdir(ref_dirname)
244	assert self.remote.cmd.calls()[0][1][0] == 'mkdir -p -m 777 %s' % ref_dirname	269		assert self.remote.cmd.calls()[0][1][0] == 'mkdir -p -m 777 %s' % ref_dirname
245		270
246	def test_mkdir_mode(self):	271		def test_mkdir_mode(self):
247	ref_dirname = 'dirname'	272		ref_dirname = 'dirname'
248	ref_mode = 644	273		ref_mode = 644
249	self.remote.cmd = Dingus()	274		self.remote.cmd = Dingus()
250	self.remote.mkdir(ref_dirname, ref_mode)	275		self.remote.mkdir(ref_dirname, ref_mode)
251	assert self.remote.cmd.calls()[0][1][0] == 'mkdir -p -m %d %s' % (ref_mode, ref_dirname)	276		assert self.remote.cmd.calls()[0][1][0] == 'mkdir -p -m %d %s' % (ref_mode, ref_dirname)

View Options

testing/test_rpm_utils.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2014, Red Hat, Inc.	2		# Copyright 2009-2014, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	'''Unit tests for libtaskotron/rpm_utils.py'''	6		'''Unit tests for libtaskotron.ext.fedora.rpm_utils'''
7		7
8	import pytest	8		import pytest
9	import subprocess	9		import subprocess
10		10
11	from libtaskotron.ext.fedora.rpm_utils import rpmformat, cmpNEVR, install, get_dist_tag	11		from libtaskotron.ext.fedora.rpm_utils import (cmpNEVR, dnf_cache_available, get_dist_tag, install,
		12		is_installed, rpmformat)
		13		from libtaskotron.ext.fedora import rpm_utils
12	from libtaskotron import exceptions as exc	14		from libtaskotron import exceptions as exc
		15		from libtaskotron import os_utils
13		16
14	from dingus import Dingus	17		import mock
15		18
16		19
17	class TestRpmformat:	20		class TestRpmformat:
18		21
19	def test_nvra(self):	22		def test_nvra(self):
20	'''NVRA as input param'''	23		'''NVRA as input param'''
21	rpmstr = 'foo-1.2-3.fc20.x86_64'	24		rpmstr = 'foo-1.2-3.fc20.x86_64'
22	assert rpmformat(rpmstr, 'nevra', True) == 'foo-1.2-3.fc20.x86_64'	25		assert rpmformat(rpmstr, 'nevra', True) == 'foo-1.2-3.fc20.x86_64'
▲ Show 20 Lines • Show All 102 Lines • ▼ Show 20 Line(s)		95	class TestCmpNEVR(object):
125		128
126	def test_raise(self):	129		def test_raise(self):
127	'''Invalid input param'''	130		'''Invalid input param'''
128	with pytest.raises(exc.TaskotronValueError):	131		with pytest.raises(exc.TaskotronValueError):
129	cmpNEVR('foo-1.2-3.fc20', 'bar-1.2-3.fc20')	132		cmpNEVR('foo-1.2-3.fc20', 'bar-1.2-3.fc20')
130		133
131		134
132	@pytest.mark.usefixtures('setup')	135		@pytest.mark.usefixtures('setup')
133	class TestInstall():	136		class TestInstall(object):
		137		'''Test rpm_utils.install()'''
134		138
135	@pytest.fixture	139		@pytest.fixture
136	def setup(self, monkeypatch):	140		def setup(self, monkeypatch):
137	self.stub_popen = Dingus()	141		self.mock_is_root = mock.Mock(return_value=True)
138	self.stub_popen_proc = Dingus(communicate__returns=['', ''])	142		monkeypatch.setattr(os_utils, 'is_root', self.mock_is_root)
		143		self.mock_has_sudo = mock.Mock(return_value=True)
		144		monkeypatch.setattr(os_utils, 'has_sudo', self.mock_has_sudo)
		145		self.mock_dnf_cache = mock.Mock(return_value=True)
		146		monkeypatch.setattr(rpm_utils, 'dnf_cache_available', self.mock_dnf_cache)
		147
		148		self.mock_check_output = mock.Mock(return_value='')
		149		monkeypatch.setattr(subprocess, 'check_output', self.mock_check_output)
		150		self.err = subprocess.CalledProcessError(1, 'cmd', output='')
		151
		152		def test_install_ok(self):
		153		install(['foo'])
		154
		155		def test_install_fails(self):
		156		self.mock_check_output.side_effect = self.err
		157		with pytest.raises(exc.TaskotronError) as excinfo:
		158		install(['foo'])
139		159
140	self.stub_popen.return_value = self.stub_popen_proc	160		# direct comparison here, because isinstance() would also accept subclasses we throw for
141	monkeypatch.setattr(subprocess, 'Popen', self.stub_popen)	161		# different issues (missing permissions)
		162		assert type(excinfo.value) is exc.TaskotronError
142		163
143	def test_zero_returncode(self):	164		def test_no_permissions(self):
144	self.stub_popen_proc.returncode = 0	165		self.mock_is_root.return_value = False
		166		self.mock_has_sudo.return_value = False
145		167
		168		with pytest.raises(exc.TaskotronPermissionError):
		169		install(['foo'])
		170
		171		def test_add_sudo(self):
		172		self.mock_is_root.return_value = False
146	install(['foo'])	173		install(['foo'])
		174		assert self.mock_check_output.call_args[0][0].index('sudo') == 0
147		175
148	def test_nonzero_returncode(self):	176		def test_dont_add_sudo(self):
149	self.stub_popen_proc.returncode = 1	177		install(['foo'])
		178		assert 'sudo' not in self.mock_check_output.call_args[0][0]
		179
		180		def test_no_pkgs(self):
		181		install([])
		182		assert self.mock_check_output.call_count == 0
		183
		184		def test_special_args(self):
		185		'''Make sure args like 'rpmlint > 1.0' are passed in correctly'''
		186		pkgs = ['foo', 'bar >= 1.0', '@group']
		187		install(pkgs)
		188
		189		call_args = self.mock_check_output.call_args[0][0]
		190		assert all([pkg in call_args for pkg in pkgs])
		191
		192		def test_run_once(self):
		193		'''The command should be executed only once if it passes'''
		194		install(['foo'])
		195		assert self.mock_check_output.call_count == 1
150		196
		197		def test_run_once_fail(self):
		198		'''If there's no cache and install fails, it should not try again'''
		199		self.mock_dnf_cache.return_value = False
		200		self.mock_check_output.side_effect = self.err
		201
		202		with pytest.raises(exc.TaskotronError) as excinfo:
		203		install(['foo'])
		204
		205		assert self.mock_check_output.call_count == 1
		206		# direct comparison here, because isinstance() would also accept subclasses we throw for
		207		# different issues (missing permissions)
		208		assert type(excinfo.value) is exc.TaskotronError
		209
		210		def test_run_twice_on_cache_fail(self):
		211		'''If cache is used and install fails, it should run again without cache'''
		212		self.mock_check_output.side_effect = self.err
151	with pytest.raises(exc.TaskotronError):	213		with pytest.raises(exc.TaskotronError):
152	install(['foo'])	214		install(['foo'])
153		215
		216		assert self.mock_check_output.call_count == 2
		217		assert '--cacheonly' in self.mock_check_output.call_args_list[0][0][0]
		218		assert '--cacheonly' not in self.mock_check_output.call_args_list[1][0][0]
		219
		220		def test_run_twice_second_ok(self):
		221		'''If installation is tried twice and second one succeeds, the whole operation should
		222		succeed.'''
		223		# ideally we would use `(self.err, mock.DEFAULT)`, but there's a bug in mock:
		224		# https://github.com/testing-cabal/mock/issues/190
		225		self.mock_check_output.side_effect = (self.err, '')
		226		install(['foo'])
		227		assert self.mock_check_output.call_count == 2
		228
		229
		230		@pytest.mark.usefixtures('setup')
		231		class TestIsInstalled(object):
		232		'''Test rpm_utils.is_installed()'''
		233
		234		@pytest.fixture
		235		def setup(self, monkeypatch):
		236		self.mock_is_root = mock.Mock(return_value=True)
		237		monkeypatch.setattr(os_utils, 'is_root', self.mock_is_root)
		238		self.mock_has_sudo = mock.Mock(return_value=True)
		239		monkeypatch.setattr(os_utils, 'has_sudo', self.mock_has_sudo)
		240		self.mock_dnf_cache = mock.Mock(return_value=True)
		241		monkeypatch.setattr(rpm_utils, 'dnf_cache_available', self.mock_dnf_cache)
		242
		243		self.mock_check_output = mock.Mock(return_value='')
		244		monkeypatch.setattr(subprocess, 'check_output', self.mock_check_output)
		245		self.err = subprocess.CalledProcessError(1, 'cmd', output='')
		246
		247		def test_installed(self):
		248		assert is_installed(['foo']) is True
		249
		250		def test_not_installed(self):
		251		self.mock_check_output.side_effect = self.err
		252		assert is_installed(['foo']) is False
		253
		254		def test_no_permissions_is_ok(self):
		255		self.mock_is_root.return_value = False
		256		self.mock_has_sudo.return_value = False
		257		assert is_installed(['foo']) is True
		258
		259		def test_add_sudo(self):
		260		self.mock_is_root.return_value = False
		261		is_installed(['foo'])
		262		assert self.mock_check_output.call_args[0][0].index('sudo') == 0
		263
		264		def test_dont_add_sudo(self):
		265		is_installed(['foo'])
		266		assert 'sudo' not in self.mock_check_output.call_args[0][0]
		267
		268		def test_no_pkgs(self):
		269		assert is_installed([]) is True
		270		assert self.mock_check_output.call_count == 0
		271
		272		def test_special_args(self):
		273		'''Make sure args like 'rpmlint > 1.0' are passed in correctly'''
		274		pkgs = ['foo', 'bar >= 1.0', '@group']
		275		is_installed(pkgs)
		276
		277		call_args = self.mock_check_output.call_args[0][0]
		278		assert all([pkg in call_args for pkg in pkgs])
		279
		280		def test_cache_exists(self):
		281		is_installed(['foo'])
		282		assert '--cacheonly' in self.mock_check_output.call_args_list[0][0][0]
		283
		284		def test_cache_doesnt_exist(self):
		285		self.mock_dnf_cache.return_value = False
		286		is_installed(['foo'])
		287		assert '--cacheonly' not in self.mock_check_output.call_args_list[0][0][0]
		288
		289
		290		@pytest.mark.usefixtures('setup')
		291		class TestDnfCacheAvailable(object):
		292		'''Test rpm_utils.dnf_cache_available()'''
		293
		294		@pytest.fixture
		295		def setup(self, monkeypatch):
		296		self.mock_is_root = mock.Mock(return_value=True)
		297		monkeypatch.setattr(os_utils, 'is_root', self.mock_is_root)
		298		self.mock_has_sudo = mock.Mock(return_value=True)
		299		monkeypatch.setattr(os_utils, 'has_sudo', self.mock_has_sudo)
		300
		301		self.mock_check_output = mock.Mock(return_value='')
		302		monkeypatch.setattr(subprocess, 'check_output', self.mock_check_output)
		303		self.err = subprocess.CalledProcessError(1, 'cmd', output='')
		304
		305		def test_cache_exists(self):
		306		assert dnf_cache_available() is True
		307
		308		def test_cache_exists_user(self):
		309		self.mock_is_root.return_value = False
		310		self.mock_has_sudo.return_value = False
		311		assert dnf_cache_available() is True
		312
		313		def test_cache_doesnt_exist(self):
		314		self.mock_check_output.side_effect = self.err
		315		assert dnf_cache_available() is False
		316
		317		def test_cache_doesnt_exist_user(self):
		318		self.mock_is_root.return_value = False
		319		self.mock_has_sudo.return_value = False
		320		self.mock_check_output.side_effect = self.err
		321		assert dnf_cache_available() is False
		322
		323		def test_add_sudo(self):
		324		self.mock_is_root.return_value = False
		325		dnf_cache_available()
		326		assert self.mock_check_output.call_args[0][0].index('sudo') == 0
		327
		328		def test_dont_add_sudo(self):
		329		dnf_cache_available()
		330		assert 'sudo' not in self.mock_check_output.call_args[0][0]
		331
154		332
155	class TestGetDistTag(object):	333		class TestGetDistTag(object):
156	def test_nvrs(self):	334		def test_nvrs(self):
157	assert get_dist_tag('foo-1.2-3.fc20') == 'fc20'	335		assert get_dist_tag('foo-1.2-3.fc20') == 'fc20'
158	assert get_dist_tag('foo-1.2-3.fc20.1') == 'fc20'	336		assert get_dist_tag('foo-1.2-3.fc20.1') == 'fc20'
159	assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20') == 'fc20'	337		assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20') == 'fc20'
160	assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20.1') == 'fc20'	338		assert get_dist_tag('foo-1.2-3.fc22hashgit.fc20.1') == 'fc20'
161		339
162	def test_unsupported_dist_tag(self):	340		def test_unsupported_dist_tag(self):
163	with pytest.raises(exc.TaskotronValueError):	341		with pytest.raises(exc.TaskotronValueError):
164	get_dist_tag('foo-1.2-3.el7')	342		get_dist_tag('foo-1.2-3.el7')

implement sudo access and dnf metadata cachingClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 1900

docs/source/library.rst

libtaskotron/config_defaults.py

libtaskotron/exceptions.py

libtaskotron/executor.py

libtaskotron/ext/fedora/rpm_utils.py

libtaskotron/minion.py

libtaskotron/os_utils.py

libtaskotron/remote_exec.py

testing/test_os_utils.py

testing/test_remote_exec.py

testing/test_rpm_utils.py

implement sudo access and dnf metadata caching
ClosedPublic
Actions