libtaskotron/minion.py
83–85	Let's add `tar` as a dependency of `libtaskotron-core` instead?
libtaskotron/remote_exec.py
332	If this works well, I have no issues replacing the original `put_dir` method (we can always use git revert). If you think there are still some uses for a standard recursive copy, let's name this `put_dir_tarred()`.
334	Using ``local_path`` and ``remote_path`` helps readability in docs.
362	It seems you don't delete this file afterwards.
363	What if `task.tar` is already there? What if `task.tar` is inside the tar? :-) We own `/var/tmp/taskotron`, maybe we could use it for this?
365	Maybe `with` statement could be useful here.
369	Does this work for nested directories? Please try with: dir file dir/file dir/dir/file (using these exact names)
testing/test_remote_exec.py
229	I might be slow today, but I think the `assert` and `is None` do nothing here, because the method raises the error, and therefore neither the assert not the comparison is executed at all.

libtaskotron.spec
34	F23 contains 1.28-6.
libtaskotron/remote_exec.py
322–325	Those variables should get bash-escaped, IIUIC.

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	2464	0888cc3		Aug 10 2016, 11:25 AM	★	★
Diff 2	2469	0888cc3	- polishing	Aug 11 2016, 11:10 AM	★	★
Diff 3	2471	0888cc3	- moar polish	Aug 11 2016, 1:09 PM	★	★
Diff 4	2472	0888cc3	- version fix in specfile	Aug 11 2016, 1:10 PM	★	★
Diff 5	2473	0888cc3	rLTRN57d507d6d966cf29594bc3a137f192b6b6a35c91	Aug 11 2016, 1:13 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
4aa2b941fde0	7fe39c3811fa	18c74fc8801b	Lukas Brabec	version fix in specfile	Aug 11 2016, 1:10 PM
18c74fc8801b	d0bed9edaea9	2ec35ffa88fc	Lukas Brabec	moar polish	Aug 11 2016, 1:08 PM
2ec35ffa88fc	08a10ccc46fc	75a337aa6b25	Lukas Brabec	polishing	Aug 11 2016, 10:48 AM
75a337aa6b25	2db7155802f2	0888cc3bb1df	Lukas Brabec	copy dir to minion via tarball (Show More…)	Aug 10 2016, 11:20 AM

Diff 2473

libtaskotron.spec

Show All 25 Lines
26	Requires: python-configparser >= 3.5.0b2	26		Requires: python-configparser >= 3.5.0b2
27	Requires: python-jinja2 >= 2.8	27		Requires: python-jinja2 >= 2.8
28	Requires: python-progressbar >= 2.3	28		Requires: python-progressbar >= 2.3
29	Requires: python-resultsdb_api >= 1.2.1	29		Requires: python-resultsdb_api >= 1.2.1
30	Requires: python-requests >= 2.7.0	30		Requires: python-requests >= 2.7.0
31	Requires: python-setuptools	31		Requires: python-setuptools
32	Requires: python-xunitparser >= 1.3.3	32		Requires: python-xunitparser >= 1.3.3
33	Requires: PyYAML >= 3.11	33		Requires: PyYAML >= 3.11
		34		Requires: tar >= 1.28-6
			kparalUnsubmitted Not Done F23 contains 1.28-6.
34	BuildRequires: pytest >= 2.7.3	35		BuildRequires: pytest >= 2.7.3
35	BuildRequires: python2-devel	36		BuildRequires: python2-devel
36	BuildRequires: python-configparser >= 3.5.0b2	37		BuildRequires: python-configparser >= 3.5.0b2
37	BuildRequires: python-dingus >= 0.3.4	38		BuildRequires: python-dingus >= 0.3.4
38	BuildRequires: python-mock >= 1.0.1	39		BuildRequires: python-mock >= 1.0.1
39	BuildRequires: python-progressbar >= 2.3	40		BuildRequires: python-progressbar >= 2.3
40	BuildRequires: python-resultsdb_api >= 1.2.1	41		BuildRequires: python-resultsdb_api >= 1.2.1
41	BuildRequires: python-requests >= 2.7.0	42		BuildRequires: python-requests >= 2.7.0
▲ Show 20 Lines • Show All 357 Lines • Show Last 20 Lines

View Options

libtaskotron/minion.py

Show First 20 Lines • Show All 74 Lines • ▼ Show 20 Line(s)		57	def _prepare_task(self):
75	# only taskotron repos (unless we require their names listed in configuration)	75		# only taskotron repos (unless we require their names listed in configuration)
76	excluded = ['fedora', 'updates', 'updates-testing']	76		excluded = ['fedora', 'updates', 'updates-testing']
77	exclude_cmd = ' '.join(['--disablerepo %s' % pipes.quote(repo) for repo in excluded])	77		exclude_cmd = ' '.join(['--disablerepo %s' % pipes.quote(repo) for repo in excluded])
78	self.ssh.cmd('dnf makecache %s' % exclude_cmd)	78		self.ssh.cmd('dnf makecache %s' % exclude_cmd)
79		79
80	# install libtaskotron	80		# install libtaskotron
81	# FIXME this should only need libtaskotron-core T651. libtaskotron-fedora should be	81		# FIXME this should only need libtaskotron-core T651. libtaskotron-fedora should be
82	# installed as a check dep.	82		# installed as a check dep.
83	self.ssh.install_pkgs(['libtaskotron-core', 'libtaskotron-fedora'])	83		# TODO: remove tar dep once libtaskotron 0.4.16 has been released for long enough
		84		# (tar is here just for backward compatibility, see D965)
		85		self.ssh.install_pkgs(['libtaskotron-core', 'libtaskotron-fedora', 'tar'])
			kparalUnsubmitted Not Done Let's add `tar` as a dependency of `libtaskotron-core` instead?
84		86
85	log.info('Copying task files onto the minion...')	87		log.info('Copying task files onto the minion...')
86		88
87	# Copy config files. Go through CONF_DIRS in reversed order to preserve priority.	89		# Copy config files. Go through CONF_DIRS in reversed order to preserve priority.
88	for config_dir in reversed(config.CONF_DIRS):	90		for config_dir in reversed(config.CONF_DIRS):
89	if os.path.exists(config_dir):	91		if os.path.exists(config_dir):
90	self.ssh.put_dir(config_dir, '/etc/taskotron/')	92		self.ssh.put_dir(config_dir, '/etc/taskotron/')
91		93
▲ Show 20 Lines • Show All 164 Lines • Show Last 20 Lines

View Options

libtaskotron/remote_exec.py

1	# -- coding: utf-8 --	1		# -- coding: utf-8 --
2	# Copyright 2009-2015, Red Hat, Inc.	2		# Copyright 2009-2015, Red Hat, Inc.
3	# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>	3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4	# See the LICENSE file for more details on Licensing	4		# See the LICENSE file for more details on Licensing
5		5
6	'''Tools for remote execution primary for disposable clients'''	6		'''Tools for remote execution primary for disposable clients'''
7		7
8	from __future__ import absolute_import	8		from __future__ import absolute_import
9	import os	9		import os
10	import os.path	10		import os.path
11	import sys	11		import sys
12	import socket	12		import socket
13	import stat	13		import stat
14	import time	14		import time
15	import pipes	15		import pipes
		16		import tarfile
		17		import tempfile
16		18
17	import paramiko	19		import paramiko
18		20
19	from .logger import log	21		from .logger import log
20	from . import exceptions as exc	22		from . import exceptions as exc
21		23
22	from libtaskotron.directives import exitcode_directive	24		from libtaskotron.directives import exitcode_directive
23	from libtaskotron import file_utils	25		from libtaskotron import file_utils
▲ Show 20 Lines • Show All 72 Lines • ▼ Show 20 Line(s)		28	class ParamikoWrapper(object):
96		98
97	def close(self):	99		def close(self):
98	'''Close open connections and files.'''	100		'''Close open connections and files.'''
99		101
100	self.outstream.close()	102		self.outstream.close()
101	self.sftp.close()	103		self.sftp.close()
102	self.ssh.close()	104		self.ssh.close()
103		105
104	def cmd(self, cmd):	106		def cmd(self, cmd, debug=True):
105	'''Execute a command.	107		'''Execute a command.
106		108
107	:param str cmd: A command to be executed. Make sure you escape it properly to prevent shell	109		:param str cmd: A command to be executed. Make sure you escape it properly to prevent shell
108	expansion, in case it is not desired.	110		expansion, in case it is not desired.
		111		:param bool debug: Whether print out debugging log messages about what's happening.
109	:returns: returncode of the command	112		:returns: returncode of the command
110	:raise TaskotronRemoteProcessError: If the command has non-zero return code and it isn't a	113		:raise TaskotronRemoteProcessError: If the command has non-zero return code and it isn't a
111	code of the exitcode directive.	114		code of the exitcode directive.
112	:raise TaskotronRemoteTimeoutError: If the remote hasn't sent any output for	115		:raise TaskotronRemoteTimeoutError: If the remote hasn't sent any output for
113	:attr:`TIMEOUT`.	116		:attr:`TIMEOUT`.
114	'''	117		'''
115		118
116	log.debug('Running command on remote host: %s', cmd)	119		if debug:
117	# write the executed command also to stdio log, so that it's clear what's happening when	120		log.debug('Running command on remote host: %s', cmd)
118	# reading it	121		# write the executed command also to stdio log, so that it's clear what's happening
119	self.outstream.write('$ %s\n' % cmd)	122		# when reading it
		123		self.outstream.write('$ %s\n' % cmd)
120		124
121	# bufsize=1 means line buffering, which should improve the chance of receiving complete	125		# bufsize=1 means line buffering, which should improve the chance of receiving complete
122	# lines. get_pty=True allocates a pseudo-terminal, which is needed to have stdout and	126		# lines. get_pty=True allocates a pseudo-terminal, which is needed to have stdout and
123	# stderr interlaced properly (in the exact chronological order)	127		# stderr interlaced properly (in the exact chronological order)
124	stdin, stdout, _ = self.ssh.exec_command(cmd, timeout=self.TIMEOUT, bufsize=1,	128		stdin, stdout, _ = self.ssh.exec_command(cmd, timeout=self.TIMEOUT, bufsize=1,
125	get_pty=True)	129		get_pty=True)
126		130
127	# stdout.channel represents channel for both stdout and stderr	131		# stdout.channel represents channel for both stdout and stderr
▲ Show 20 Lines • Show All 143 Lines • ▼ Show 20 Line(s)		273	def _remote_isdir(self, remote_path):
271	e.g. even if it exists, but is a file).'''	275		e.g. even if it exists, but is a file).'''
272	try:	276		try:
273	return stat.S_ISDIR(self.sftp.lstat(remote_path).st_mode)	277		return stat.S_ISDIR(self.sftp.lstat(remote_path).st_mode)
274	except IOError:	278		except IOError:
275	# when the dir doesn't exist: IOError: [Errno 2] No such file	279		# when the dir doesn't exist: IOError: [Errno 2] No such file
276	return False	280		return False
277		281
278	def put_dir(self, local_path, remote_path, overwrite=True, debug=True):	282		def put_dir(self, local_path, remote_path, overwrite=True, debug=True):
279	'''Copy a directory to a remote path. File permissions are not preserved.	283		'''Copy a directory to a remote path. This method creates a tarball from contents of
		284		``local_path``, copies the tarball to remote machine and extracts it to ``remote_path``.
		285		File permissions and symlinks are preserved.
280		286
281	:param str remote_path: A path to the remote directory. This directory will be created, if	287		:param str remote_path: A path to the remote directory. This directory will be created, if
282	its parent exists. Otherwise you need to create the full tree	288		its parent exists. Otherwise you need to create the full tree
283	structure manually beforehand.	289		structure manually beforehand.
284	:param str local_path: A path to the local directory	290		:param str local_path: A path to the local directory
285	:param bool overwrite: Whether to overwrite remote path (merge local dir with the remote	291		:param bool overwrite: Whether to overwrite remote path (merge local dir with the remote
286	dir). If you choose to not overwrite and ``remote_path`` exists,	292		dir). If you choose to not overwrite and ``remote_path`` exists,
287	this method with just immediately return.	293		this method with just immediately return.
288	:param bool debug: Whether print out debugging log messages about what's happening.	294		:param bool debug: Whether print out debugging log messages about what's happening.
289	:raise TaskotronRemoteError: If the directory could not be uploaded	295		:raise TaskotronRemoteError: If the directory could not be copied
290	'''	296		'''
291
292	if debug:	297		if debug:
293	log.debug('Recursively copying %s to %s@%s:%s ...', local_path, self.username,	298		log.debug('Copying %s to %s@%s:%s ...', local_path, self.username,
294	self.hostname, remote_path)	299		self.hostname, remote_path)
295		300
296	# keep the conditionals in this order to avoid unnecessary remote calls	301		# keep the conditionals in this order to avoid unnecessary remote calls
297	if not overwrite and self._remote_file_exists(remote_path):	302		if not overwrite and self._remote_file_exists(remote_path):
298	if debug:	303		if debug:
299	log.debug('Remote path %s already exists, not overwriting.', remote_path)	304		log.debug('Remote path %s already exists, not overwriting.', remote_path)
300	return	305		return
301		306
302	try:	307		try:
303	if not self._remote_isdir(remote_path):	308		if not self._remote_isdir(remote_path):
304	self.sftp.mkdir(remote_path)	309		self.sftp.mkdir(remote_path)
305		310
306	files = os.listdir(local_path)	311		files_to_copy = os.listdir(local_path)
		312		tar_local_path = tempfile.mktemp()
		313		tar_remote_path = os.path.join(remote_path, 'tarred_files.tar')
		314
		315		with tarfile.open(tar_local_path, 'w') as tar:
		316		for filename in files_to_copy:
		317		tar.add(os.path.join(local_path, filename), arcname=filename)
		318
		319		self.put_file(tar_local_path, tar_remote_path, debug=False)
		320		self.cmd('tar xf %s -C %s' % (pipes.quote(tar_remote_path), pipes.quote(remote_path)),
		321		debug=False)
		322
		323		os.remove(tar_local_path)
		324		self.cmd('rm -f %s' % pipes.quote(tar_remote_path), debug=False)
307		325
			kparalUnsubmitted Not Done Those variables should get bash-escaped, IIUIC.
308	for file_ in files:
309	remote_file = os.path.join(remote_path, file_)
310	local_file = os.path.join(local_path, file_)
311	if os.path.isdir(local_file):
312	self.put_dir(local_file, remote_file, debug=False)
313	else:
314	self.put_file(local_file, remote_file, debug=False)
315	except (OSError, IOError) as e:	326		except (OSError, IOError) as e:
316	log.exception(e)	327		log.exception(e)
317	raise exc.TaskotronRemoteError('Could not copy dir %s to %s@%s:%s: %s' %	328		raise exc.TaskotronRemoteError('Could not copy dir %s to %s@%s:%s: %s' %
318	(local_path, self.username, self.hostname,	329		(local_path, self.username, self.hostname,
319	remote_path, e))	330		remote_path, e))
320		331
321	def get_dir(self, remote_path, local_path, debug=True):	332		def get_dir(self, remote_path, local_path, debug=True):
			kparalUnsubmitted Not Done If this works well, I have no issues replacing the original `put_dir` method (we can always use git revert). If you think there are still some uses for a standard recursive copy, let's name this `put_dir_tarred()`.
322	'''Get a directory from a remote path. File permissions are not preserved.	333		'''Get a directory from a remote path. File permissions are not preserved.
323		334
			kparalUnsubmitted Not Done Using ``local_path`` and ``remote_path`` helps readability in docs.
324	:param str remote_path: A path to the remote directory	335		:param str remote_path: A path to the remote directory
325	:param str local_path: A path to the local directory. This directory will be created, if	336		:param str local_path: A path to the local directory. This directory will be created, if
326	its parent exists. Otherwise you need to create the full tree	337		its parent exists. Otherwise you need to create the full tree
327	structure manually beforehand.	338		structure manually beforehand.
328	:param bool debug: Whether print out debugging log messages about what's happening.	339		:param bool debug: Whether print out debugging log messages about what's happening.
329	:raise TaskotronRemoteError: If the directory could not be downloaded	340		:raise TaskotronRemoteError: If the directory could not be downloaded
330	'''	341		'''
331		342
Show All 11 Lines		353	for fileattr in fileattrs:
343	remote_file = os.path.join(remote_path, fileattr.filename)	354		remote_file = os.path.join(remote_path, fileattr.filename)
344	local_file = os.path.join(local_path, fileattr.filename)	355		local_file = os.path.join(local_path, fileattr.filename)
345	if stat.S_ISDIR(fileattr.st_mode):	356		if stat.S_ISDIR(fileattr.st_mode):
346	self.get_dir(remote_file, local_file, debug=False)	357		self.get_dir(remote_file, local_file, debug=False)
347	else:	358		else:
348	self.get_file(remote_file, local_file, debug=False)	359		self.get_file(remote_file, local_file, debug=False)
349	except (OSError, IOError) as e:	360		except (OSError, IOError) as e:
350	raise exc.TaskotronRemoteError('Could not copy dir %s@%s:%s to %s: %s' %	361		raise exc.TaskotronRemoteError('Could not copy dir %s@%s:%s to %s: %s' %
351	(self.username, self.hostname, remote_path,	362		(self.username, self.hostname, remote_path,
			kparalUnsubmitted Not Done It seems you don't delete this file afterwards.
352	local_path, e))	363		local_path, e))
			kparalUnsubmitted Not Done What if `task.tar` is already there? What if `task.tar` is inside the tar? :-) We own `/var/tmp/taskotron`, maybe we could use it for this?
			kparalUnsubmitted Not Done Maybe `with` statement could be useful here.
			kparalUnsubmitted Not Done Does this work for nested directories? Please try with: dir file dir/file dir/dir/file (using these exact names)

View Options

testing/test_remote_exec.py

Show First 20 Lines • Show All 216 Lines • ▼ Show 20 Line(s)		216	with pytest.raises(TaskotronRemoteError):
217	self.remote.write_file(STUB_NONWRITABLE_FILE, ref_data)	217		self.remote.write_file(STUB_NONWRITABLE_FILE, ref_data)
218		218
219	def test_write_file_dont_overwrite(self):	219		def test_write_file_dont_overwrite(self):
220	ref_data = 'foobar'	220		ref_data = 'foobar'
221	assert self.remote.write_file(STUB_FILE, ref_data, overwrite=False) is None	221		assert self.remote.write_file(STUB_FILE, ref_data, overwrite=False) is None
222		222
223	def test_put_dir_not_dir(self):	223		def test_put_dir_not_dir(self):
224	with pytest.raises(TaskotronRemoteError):	224		with pytest.raises(TaskotronRemoteError):
225	assert self.remote.put_dir(STUB_FILE, STUB_DIR) is None	225		self.remote.put_dir(STUB_FILE, STUB_DIR)
226		226
227	def test_put_dir_not_dir_dont_overwrite(self):	227		def test_put_dir_not_dir_dont_overwrite(self):
228	assert self.remote.put_dir(STUB_DIR, STUB_DIR, overwrite=False) is None	228		assert self.remote.put_dir(STUB_DIR, STUB_DIR, overwrite=False) is None
229		229
			kparalUnsubmitted Not Done I might be slow today, but I think the `assert` and `is None` do nothing here, because the method raises the error, and therefore neither the assert not the comparison is executed at all.
230	def test_get_dir_not_dir(self):	230		def test_get_dir_not_dir(self):
231	with pytest.raises(TaskotronRemoteError):	231		with pytest.raises(TaskotronRemoteError):
232	self.remote.get_dir(STUB_FILE, STUB_DIR)	232		self.remote.get_dir(STUB_FILE, STUB_DIR)
233		233
234	def test_cmd_success(self):	234		def test_cmd_success(self):
235	assert self.remote.cmd(STUB_CMD_TRUE) == STUB_CMD_TRUE	235		assert self.remote.cmd(STUB_CMD_TRUE) == STUB_CMD_TRUE
236		236
237	def test_cmd_failure(self):	237		def test_cmd_failure(self):
▲ Show 20 Lines • Show All 48 Lines • Show Last 20 Lines

			Path	Packages
M			libtaskotron.spec (1 line)
M			libtaskotron/minion.py (4 lines)
M			libtaskotron/remote_exec.py (45 lines)
M			testing/test_remote_exec.py (2 lines)

copy dir to minion via tarball
ClosedPublic
Actions

Details

Diff Detail

Revision Contents

Diff 2473

libtaskotron.spec

libtaskotron/minion.py

libtaskotron/remote_exec.py

testing/test_remote_exec.py

copy dir to minion via tarballClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 2473

libtaskotron.spec

libtaskotron/minion.py

libtaskotron/remote_exec.py

testing/test_remote_exec.py

copy dir to minion via tarball
ClosedPublic
Actions