libtaskotron/directives/shell_directive.py
3	One would think we could have a script for that :-)
20	What about: execute one or more shell commands or direct processes
23–24	Let's be a bit more verbose. What about this: Execute a command in shell or as a direct process. There are no parameters for this directive, it uses a slightly different format: - Provide a list of commands directly under the directive name, each item is one command. - If you use a string, it will be executed in the default shell (including all shell expansions). - If you use a list of strings, it will be executed as a direct process (no shell expansions apply, which also means you don't need to worry about escaping special characters). - If start the string with ``ignorereturn:`` key and create a list sub-section under it, all those commands will ignore their exit code and not raise an error. See the example section for a better idea. This directive is intended for basic simple scripting or running commands. If you need anything more advanced, it's recommended to create a standalone script (bash, perl, etc) and execute it instead.
26–29	This is probably confusing, because we don't really require any parameter (as understood by reading all other directives). I'd keep the `parameters` section empty and describe the behavior in detail in the description. But it seems we'll need to fix `generate_directive_docs.py` to allow `parameters` to be empty. If that's too much work, let's keep it as it is and create a ticket (please link it here) requesting changes in `generate_directive_docs.py`.
34–36	I'd probably remove this, I find it unnecessary/superfluous. It should be clear how to write a task formula from our formula howtos.
60	multiple
61	``ignorereturn``
69–73	Speaking about `shell=False` is a bit implementation specific, what about: You can run a process directly, without shell processing, which means you don't need to worry about escaping special characters or e.g. spaces in file names:: - name: execute command directly, without shell shell: - [echo, "don't worry about code injection"] - [rm, a file with # funny & characters] - [mkdir, ${variable_with_unknown_contents}]
82	file loader is unclear to me, what about: Shell directive can be used to copy file contents into a variable. With this, you can later use ``${loaded_file}`` in other directives::
97	Let's log this instead of printing.
98	"a list", very nitpicky, sorry. Makes it easier to read. Also, this error is not documented.
107–112	Would this be shorter, safer and easier to read? if isinstance(cmd, dict) and 'ignorereturn' in cmd: self.process(cmd['ignorereturn'], arg_data, working_data, ignorereturn=True) continue
119	Could we print `Executing shell` for `shell=True` and `Executing process` for `shell=False`? That will help us debug potential issues in the future.
134	Could we please get a real-time output of the subprocess? This is how: proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, close_fds=True) output = [] for line in iter(proc.stdout.readline, ''): print line, output.append(line) return ''.join(output)
libtaskotron/executor.py
148–154 ↗	(On Diff #2397)	This would probably deserve an explanatory comment.
testing/test_shell_directive.py
45–53	Tests like these, where you actually seem to execute system shell, should probably go to a `functest_` file. I guess it is similar to working with a filesystem - it's not immediate.
66	What if there is `ocho` on the system? Would it be better to mock the subprocess call to simply raise the same error it would raise for a non-existent command?

libtaskotron/directives/shell_directive.py
125	Let's at least say what this is: log.debug('Shell directive received invalid input: %r', params)
148	This line is probably not useful anymore.
testing/test_shell_directive.py
66	I don't even

libtaskotron/directives/shell_directive.py
26–29	Created T821.

		Path
M		docs/source/docker.rst (20 lines)
M		docs/source/taskyaml.rst (5 lines)
D	M	libtaskotron/directives/bash_directive.py (88 lines)
A	M	libtaskotron/directives/shell_directive.py (151 lines)
A	M	testing/functest_shell_directive.py (93 lines)
D	M	testing/test_bash_directive.py (74 lines)
A	M	testing/test_shell_directive.py (39 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	2328	7a998b0		Jun 30 2016, 1:14 PM	★	★
Diff 2	2329	7a998b0	- fix	Jun 30 2016, 1:38 PM	★	★
Diff 3	2397	c1b6392	- reflect new format in docs	Jul 21 2016, 8:50 AM	★	★
Diff 4	2413	c1b6392	- polishing, escaping removed	Jul 25 2016, 9:09 AM	★	★
Diff 5	2416	d9b7a64	rLTRN222a68f14628d950f2fb9e08be2f19373ad686aa	Jul 25 2016, 11:24 AM	★	★

Commit	Tree	Parents	Author	Summary	Date
affe4210f4c1	2a40c096b348	7deb52706689	Lukas Brabec	polishing, escaping removed	Jul 25 2016, 8:55 AM
7deb52706689	0fe730e72b5f	127abda19f32	Lukas Brabec	moar docs	Jul 21 2016, 8:49 AM
127abda19f32	c75ccc65b36e	e3eaa4b9ab19	Lukas Brabec	moar tests	Jul 21 2016, 8:09 AM
e3eaa4b9ab19	f02adb513823	16c43387de99	Lukas Brabec	reflect new format in docs	Jul 13 2016, 1:16 PM
16c43387de99	0c1a0b6473cb	17d5cb47de13	Lukas Brabec	fix	Jun 30 2016, 1:38 PM
17d5cb47de13	1b5f8380fe7d	c1b6392b7235	Lukas Brabec	bash to shell directive (Show More…)	Jun 8 2016, 8:44 AM

Diff 2416

docs/source/docker.rst

Show First 20 Lines • Show All 99 Lines • ▼ Show 20 Line(s)		99	.. code-block:: yaml
100		100
101	name: httpd_sanity	101		name: httpd_sanity
102		102
103	environment:	103		environment:
104	rpm:	104		rpm:
105	- docker	105		- docker
106		106
107	actions:	107		actions:
108	- bash:	108		- shell:
109	command: bash test.sh ${item} ${workdir}/test_output	109		- bash test.sh ${item} ${workdir}/test_output
110		110
111	- bash:	111		- shell:
112	command: cat ${workdir}/test_output	112		- cat ${workdir}/test_output
113	export: test_results	113		export: test_results
114		114
115	- resultsdb:	115		- resultsdb:
116	results: ${test_results}	116		results: ${test_results}
117		117
118		118
119	Pytest	119		Pytest
120	======	120		======
Show All 34 Lines		153	.. code-block:: yaml
155	name: httpd_sanity	155		name: httpd_sanity
156		156
157	environment:	157		environment:
158	rpm:	158		rpm:
159	- docker	159		- docker
160	- python2-pytest	160		- python2-pytest
161		161
162	actions:	162		actions:
163	- bash:	163		- shell:
164	command: systemctl start docker	164		- systemctl start docker
165		165		- docker pull ${item}
166	- bash:	166		- py.test dockertest.py --junitxml=${artifactsdir}/output.xml
167	command: docker pull ${item}
168
169	- bash:
170	command: py.test dockertest.py --junitxml=${artifactsdir}/output.xml
171		167
172	- xunit:	168		- xunit:
173	file: ${artifactsdir}/output.xml	169		file: ${artifactsdir}/output.xml
174	aggregation: none	170		aggregation: none
175	export: test_results	171		export: test_results
176		172
177	- resultsdb:	173		- resultsdb:
178	results: ${test_results}	174		results: ${test_results}
179		175
180		176
181	Other possibilities	177		Other possibilities
182	===================	178		===================
183	Taskotron is capable of running arbitrary code (e.g. via bash directive) in any language. If you	179		Taskotron is capable of running arbitrary code (e.g. via bash directive) in any language. If you
184	cannot choose between shell script and pytest, you can always write your docker tests in any other	180		cannot choose between shell script and pytest, you can always write your docker tests in any other
185	language you prefer. Just make sure that the test results are in format we understand --	181		language you prefer. Just make sure that the test results are in format we understand --
186	:ref:`resultyaml-format` or xUnit.	182		:ref:`resultyaml-format` or xUnit.

View Options

docs/source/taskyaml.rst

Show First 20 Lines • Show All 281 Lines • ▼ Show 20 Line(s)
282	Creates repositories from locally downloaded rpms.	282	Creates repositories from locally downloaded rpms.
283		283
284	python	284	python
285	------	285	------
286		286
287	Allows the execution of a python script as part of task execution.	287	Allows the execution of a python script as part of task execution.
288	Offers several methods of executing python code for a task.	288	Offers several methods of executing python code for a task.
289		289
		290	shell
		291	-----
		292
		293	Allows the execution of a shell command(s) in subprocess call.
		294
290	resultsdb	295	resultsdb
291	---------	296	---------
292		297
293	Provides an interface to `resultsdb <https://bitbucket.org/fedoraqa/resultsdb>`_	298	Provides an interface to `resultsdb <https://bitbucket.org/fedoraqa/resultsdb>`_
294	for storing results of executed tasks.	299	for storing results of executed tasks.

View Options

libtaskotron/directives/bash_directive.py

This file was deleted.

This file was completely deleted. Show File Contents

View Options

libtaskotron/directives/shell_directive.py

This file was added.

1		# -- coding: utf-8 --
2		# Copyright 2009-2016, Red Hat, Inc.
3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
	kparalUnsubmitted Not Done One would think we could have a script for that :-)
4		# See the LICENSE file for more details on Licensing
5
6		from __future__ import absolute_import
7
8		import os
9		import subprocess
10
11		from libtaskotron.directives import BaseDirective
12		from libtaskotron.exceptions import TaskotronDirectiveError
13		from libtaskotron.logger import log
14
15		DOCUMENTATION = """
16		module: shell_directive
17		short_description: execute one ore more shell commands or direct processes
18
19		description: \|
20		Execute a command in shell or as a direct process. There are no parameters for this directive,
	kparalUnsubmitted Not Done What about: execute one or more shell commands or direct processes
21		it uses a slightly different format:
22
23		- Provide a list of commands directly under the directive name, each item is one command.
24		- If you use a string, it will be executed in the default shell (including all shell expansions).
	kparalUnsubmitted Not Done Let's be a bit more verbose. What about this: Execute a command in shell or as a direct process. There are no parameters for this directive, it uses a slightly different format: - Provide a list of commands directly under the directive name, each item is one command. - If you use a string, it will be executed in the default shell (including all shell expansions). - If you use a list of strings, it will be executed as a direct process (no shell expansions apply, which also means you don't need to worry about escaping special characters). - If start the string with ``ignorereturn:`` key and create a list sub-section under it, all those commands will ignore their exit code and not raise an error. See the example section for a better idea. This directive is intended for basic simple scripting or running commands. If you need anything more advanced, it's recommended to create a standalone script (bash, perl, etc) and execute it instead.
25		- If you use a list of strings, it will be executed as a direct process (no shell expansions
26		apply, which also means you don't need to worry about escaping special characters).
27		- If start the string with ``ignorereturn:`` key and create a list sub-section under it, all
28		those commands will ignore their exit code and not raise an error.
29
	kparalUnsubmitted Not Done This is probably confusing, because we don't really require any parameter (as understood by reading all other directives). I'd keep the `parameters` section empty and describe the behavior in detail in the description. But it seems we'll need to fix `generate_directive_docs.py` to allow `parameters` to be empty. If that's too much work, let's keep it as it is and create a ticket (please link it here) requesting changes in `generate_directive_docs.py`.
	kparalUnsubmitted Not Done Created T821.
30		See the example section for a better idea.
31
32		This directive is intended for basic simple scripting or running commands. If you need anything
33		more advanced, it's recommended to create a standalone script (bash, perl, etc) and execute it
34		instead.
35		parameters:
36		directly_list_of_cmds:
	kparalUnsubmitted Not Done I'd probably remove this, I find it unnecessary/superfluous. It should be clear how to write a task formula from our formula howtos.
37		required: true
38		description: shell command(s) to be executed
39		type: list
40		returns: \|
41		The stdout of executed shell command, in case of multiple commands, output of the last one is
42		returned.
43
44		raises: \|
45		* :class:`.TaskotronDirectiveError`: when parameter of shell directive is not a list, if command
46		to be executed does not exist or if command returned non-zero returncode (returncode can be
47		ignored, see examples).
48
49		version_added: 0.4.15
50		"""
51
52		EXAMPLES = """
53		This executes the shell command in subprocess::
54
55		- name: pull docker image
56		shell:
57		- docker pull httpd
58
59		You can also execute multiple commands::
60
	kparalUnsubmitted Not Done multiple
61		- name: do some stuff
	kparalUnsubmitted Not Done ``ignorereturn``
62		shell:
63		- docker pull httpd
64		- docker run -d httpd
65		- docker ps
66
67		Shell directive will raise if command (or one of them if multiple) returns non-zero returncode,
68		you can bypass this by using ``ignorereturn``::
69
70		- name: do more stuff
71		shell:
72		- ignorereturn:
73		- docker pull thiswontwork!1!1!!
	kparalUnsubmitted Not Done Speaking about `shell=False` is a bit implementation specific, what about: You can run a process directly, without shell processing, which means you don't need to worry about escaping special characters or e.g. spaces in file names:: - name: execute command directly, without shell shell: - [echo, "don't worry about code injection"] - [rm, a file with # funny & characters] - [mkdir, ${variable_with_unknown_contents}]
74		- echo 'No prob, continuing'
75
76		You can run a process directly, without shell processing, which means you don't need to worry
77		about escaping special characters or e.g. spaces in file names::
78
79		- name: execute command directly, without shell
80		shell:
81		- [echo, "don't worry about code injection"]
82		- [rm, a file with # funny & characters]
	kparalUnsubmitted Not Done file loader is unclear to me, what about: Shell directive can be used to copy file contents into a variable. With this, you can later use ``${loaded_file}`` in other directives::
83		- [mkdir, ${variable_with_unknown_contents}]
84
85		You can also run bash script::
86
87		- name: execute bash script
88		shell:
89		- bash test.sh
90		export: test_output
91
92		Shell directive can be used to copy file contents into a variable. With this, you can later use
93		``${loaded_file}`` in other directives::
94
95		- name: load file to formula variable
96		shell:
97		- cat some_file
	kparalUnsubmitted Not Done Let's log this instead of printing.
98		export: loaded_file
	kparalUnsubmitted Not Done "a list", very nitpicky, sorry. Makes it easier to read. Also, this error is not documented.
99		"""
100
101		directive_class = 'ShellDirective'
102
103
104		def check_output(cmd, **kwargs):
105		"""Custom check_output that prints output in real-time"""
106
107		proc = subprocess.Popen(cmd, **kwargs)
108		output = []
109		for line in iter(proc.stdout.readline, ''):
110		print line,
111		output.append(line)
112
	kparalUnsubmitted Not Done Would this be shorter, safer and easier to read? if isinstance(cmd, dict) and 'ignorereturn' in cmd: self.process(cmd['ignorereturn'], arg_data, working_data, ignorereturn=True) continue
113		retcode = proc.poll()
114		if retcode:
115		raise subprocess.CalledProcessError(retcode, cmd, ''.join(output))
116
117		return ''.join(output)
118
119
	kparalUnsubmitted Not Done Could we print `Executing shell` for `shell=True` and `Executing process` for `shell=False`? That will help us debug potential issues in the future.
120		class ShellDirective(BaseDirective):
121		def process(self, params, arg_data, ignorereturn=False):
122
123		if not isinstance(params, list):
124		log.debug('Shell directive received invalid input: %r', params)
125		raise TaskotronDirectiveError("Shell directive must contain a list of commands")
	kparalUnsubmitted Not Done Let's at least say what this is: log.debug('Shell directive received invalid input: %r', params)
126
127		basedir = os.path.dirname(os.path.abspath(arg_data['task']))
128		output = ""
129
130		for cmd in params:
131		if isinstance(cmd, dict) and 'ignorereturn' in cmd:
132		output = self.process(cmd['ignorereturn'], arg_data, ignorereturn=True)
133		continue
134
	kparalUnsubmitted Not Done Could we please get a real-time output of the subprocess? This is how: proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, close_fds=True) output = [] for line in iter(proc.stdout.readline, ''): print line, output.append(line) return ''.join(output)
135		try:
136		if isinstance(cmd, list):
137		log.info("Executing process: %s", cmd)
138		output = check_output(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
139		bufsize=1, close_fds=True, cwd=basedir)
140		else:
141		log.info("Executing shell: %s", cmd)
142		output = check_output(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
143		bufsize=1, close_fds=True, cwd=basedir, shell=True)
144
145		except subprocess.CalledProcessError as e:
146		if not ignorereturn:
147		raise TaskotronDirectiveError(e)
148		except OSError as e:
	kparalUnsubmitted Not Done This line is probably not useful anymore.
149		raise TaskotronDirectiveError(e)
150
151		return output

View Options

testing/functest_shell_directive.py

This file was added.

		1		# -- coding: utf-8 --
		2		# Copyright 2009-2016, Red Hat, Inc.
		3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
		4		# See the LICENSE file for more details on Licensing
		5
		6		from libtaskotron.directives import shell_directive
		7		from libtaskotron.exceptions import TaskotronDirectiveError
		8
		9		import os
		10		import pytest
		11
		12		OUTPUT = "Piko a Rybi prsty"
		13		PATH = os.path.realpath(__file__)
		14
		15
		16		class TestCheckCommand(object):
		17
		18		def test_existing_command(self):
		19		test_params = ['echo -n %s' % OUTPUT]
		20		test_arg_data = {'task': PATH}
		21		test_directive = shell_directive.ShellDirective()
		22
		23		output = test_directive.process(test_params, test_arg_data)
		24
		25		assert output == OUTPUT
		26
		27		def test_shell_false(self):
		28		test_params = [['echo', '-n', OUTPUT]]
		29		test_arg_data = {'task': PATH}
		30		test_directive = shell_directive.ShellDirective()
		31
		32		output = test_directive.process(test_params, test_arg_data)
		33
		34		assert output == OUTPUT
		35
		36		def test_non_existing_command(self):
		37		test_params = ['if_this_exists_i_will_go_to_hell_and_i_will_take_kparal_with_me']
		38		test_arg_data = {'task': PATH}
		39		test_directive = shell_directive.ShellDirective()
		40
		41		with pytest.raises(TaskotronDirectiveError):
		42		test_directive.process(test_params, test_arg_data)
		43
		44		def test_zero_returncode(self):
		45		test_params = ['true']
		46		test_arg_data = {'task': PATH}
		47		test_directive = shell_directive.ShellDirective()
		48
		49		output = test_directive.process(test_params, test_arg_data)
		50
		51		assert output == ""
		52
		53		def test_nonzero_returncode(self):
		54		test_params = ['false']
		55		test_arg_data = {'task': PATH}
		56		test_directive = shell_directive.ShellDirective()
		57
		58		with pytest.raises(TaskotronDirectiveError):
		59		test_directive.process(test_params, test_arg_data)
		60
		61		def test_multi_command(self):
		62		test_params = ['true',
		63		'echo -n %s' % OUTPUT[::-1],
		64		'echo -n %s' % OUTPUT]
		65		test_arg_data = {'task': PATH}
		66		test_directive = shell_directive.ShellDirective()
		67
		68		output = test_directive.process(test_params, test_arg_data)
		69
		70		assert output == OUTPUT
		71
		72		def test_multi_command_error(self):
		73		test_params = ['true',
		74		'echo -n %s' % OUTPUT[::-1],
		75		'false',
		76		'echo -n %s' % OUTPUT]
		77		test_arg_data = {'task': PATH}
		78		test_directive = shell_directive.ShellDirective()
		79
		80		with pytest.raises(TaskotronDirectiveError):
		81		test_directive.process(test_params, test_arg_data)
		82
		83		def test_multi_command_ignore_error(self):
		84		test_params = ['true',
		85		'echo -n %s' % OUTPUT[::-1],
		86		{'ignorereturn': ['false']},
		87		'echo -n %s' % OUTPUT]
		88		test_arg_data = {'task': PATH}
		89		test_directive = shell_directive.ShellDirective()
		90
		91		output = test_directive.process(test_params, test_arg_data)
		92
		93		assert output == OUTPUT

View Options

testing/test_bash_directive.py

This file was deleted.

This file was completely deleted. Show File Contents

View Options

testing/test_shell_directive.py

This file was added.

1		# -- coding: utf-8 --
2		# Copyright 2009-2016, Red Hat, Inc.
3		# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4		# See the LICENSE file for more details on Licensing
5
6		from libtaskotron.directives import shell_directive
7		from libtaskotron.exceptions import TaskotronDirectiveError
8
9		import os
10		import pytest
11		import subprocess
12		import mock
13
14		OUTPUT = "Piko a Rybi prsty"
15		PATH = os.path.realpath(__file__)
16
17
18		class TestCheckCommand(object):
19		def test_a_list(self, monkeypatch):
20		test_params = ['cmd.exe']
21		test_arg_data = {'task': PATH}
22		test_directive = shell_directive.ShellDirective()
23
24		stub_check_output = mock.Mock(return_value=OUTPUT)
25		monkeypatch.setattr(shell_directive, 'check_output', stub_check_output)
26
27		output = test_directive.process(test_params, test_arg_data)
28
29		assert output == OUTPUT
30
31		def test_not_a_list(self, monkeypatch):
32		test_params = {'a dict': 'instead of a list'}
33		test_arg_data = {'task': PATH}
34		test_directive = shell_directive.ShellDirective()
35
36		monkeypatch.setattr(subprocess, 'check_output', mock.Mock())
37
38		with pytest.raises(TaskotronDirectiveError):
39		test_directive.process(test_params, test_arg_data)
	kparalUnsubmitted Not Done What if there is `ocho` on the system? Would it be better to mock the subprocess call to simply raise the same error it would raise for a non-existent command?
	kparalUnsubmitted Not Done Tests like these, where you actually seem to execute system shell, should probably go to a `functest_` file. I guess it is similar to working with a filesystem - it's not immediate.
	kparalUnsubmitted Not Done I don't even

bash to shell directive
ClosedPublic
Actions

Details

Diff Detail

Revision Contents

Diff 2416

docs/source/docker.rst

docs/source/taskyaml.rst

libtaskotron/directives/bash_directive.py

libtaskotron/directives/shell_directive.py

testing/functest_shell_directive.py

testing/test_bash_directive.py

testing/test_shell_directive.py

bash to shell directiveClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 2416

docs/source/docker.rst

docs/source/taskyaml.rst

libtaskotron/directives/bash_directive.py

libtaskotron/directives/shell_directive.py

testing/functest_shell_directive.py

testing/test_bash_directive.py

testing/test_shell_directive.py

bash to shell directive
ClosedPublic
Actions