reworking runner to make it capable of running more than rpmlint. removed command arg, moved everything to key-value args and updated existing directives. Fixes T71
ClosedPublic
Actions

Authored by tflink on Feb 10 2014, 8:41 PM.

Details

Reviewers

kparal
mkrizek

Maniphest Tasks

T71: make runner more flexible so that it can run more than rpmlint

Commits

rLTRN3fb8c35b2eb0: reworking runner to make it capable of running more than rpmlint. removed…

Summary

reworking runner and action handling to be more useful, output still has issues, will be fixed in another review - this one is large enough as it is

Test Plan

tested locally with rpmlint (will update that repo shortly), added unit tests and all are passing

Diff Detail

Lint

Lint Skipped

Unit

Unit Tests Skipped

task-rpmlint on bitbucket has been updated for these changes. I wanted to push those changes to a non-master branch but that didn't work. We're not using that repo for running anything ATM so I'm just going to leave it as-is for now.

Looks good to me overall, just a few nitpicks.

libtaskotron/directives/dummy_directive.py
9	Do you think it would be useful to list required input_data's keys for each directive in its class in pydoc?
libtaskotron/directives/koji_directive.py
103	Shouldn't this line be: "if 'envr' and 'arch' in input_data:"?
libtaskotron/directives/python_directive.py
91	file=<python file>
103	Again, should we check for 'callable' and 'file' if they are present in input_data?
libtaskotron/runner.py
5	jinja2 is not included in requirements.txt
51–53	I wonder, should we use term 'subtask' instead of 'action'? It might be a little confusing to use 'action' since not all directives have 'action' (the python directive) and if I am correct this is not action as in arg of a directive but action as in subtask, right?
readme.rst
32 ↗	(On Diff #12)	This change is already in develop.

Thanks for the comments, I'll get the stuff changed

libtaskotron/directives/dummy_directive.py
9	yeah, I'll add that for the two directives
libtaskotron/directives/koji_directive.py
103	yeah, there should be some input validation here and in all the directives
libtaskotron/runner.py
51–53	I'd prefer changing the parameter name for koji if there's potential for confusion. To me, the idea of a task made up of actions flows well in a linguistic sense while the directive arg could be changed from 'action' to 'command' without any loss in how natural it sounds.

libtaskotron/runner.py
51–53	That works for me.

Fixed documentation, input arg checking and requirements.txt from comments in review

I'm not sure why the readme.rst changes still show up here - there is no difference between the branches on my local system. Either way, I'll make sure that the file doesn't change when I merge to develop

libtaskotron/runner.py
79	Why don't we run the yaml file through jinja first and then load it so we do jinja parsing just once. Is there a reason we do jinja parsing for each action separately? Unless I am missing something.

That was the best solution that I came up with, anyways. I'm open to other suggestions, though

libtaskotron/runner.py
79	because not all the variables exist when the yaml file is loaded. The biggest example of this is for values which are exported - the actual values don't exist until after the action has been run. If we render the template before that execution, the value will be empty (at best) and the data won't be transferred between actions

LGTM

libtaskotron/directives/koji_directive.py
98	It would be much better to use some custom Exception - maybe some libtaskotron-wide but libtaskotron specific one
libtaskotron/directives/python_directive.py
100	It would be much better to use some custom Exception - maybe some libtaskotron-wide but libtaskotron specific one

jskladan - yeah, error handling is something that needs to be improved and made consistent throughout the codebase. Just raising bare Exceptions isn't the best solution but I'd rather handle that in a different ticket/patch. This one's big enough as it is.

Closed by commit rLTRN3fb8c35b2eb0 (authored by @tflink).

Revision Contents

		Path
M		libtaskotron/directives/__init__.py (2 lines)
A	M	libtaskotron/directives/dummy_directive.py (26 lines)
M		libtaskotron/directives/koji_directive.py (54 lines)
M		libtaskotron/directives/python_directive.py (23 lines)
M		libtaskotron/runner.py (141 lines)
M		requirements.txt (5 lines)
V		testing/{test_koji.py → test_koji_directive.py}
V		testing/{test_koji_directive.py ← test_koji.py} (7 lines)
M		testing/test_python_directive.py (20 lines)
A	M	testing/test_runner.py (227 lines)
D	M	testing/test_yamlrunner.py (83 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	12	a196da5		Feb 10 2014, 8:41 PM	★	★
Diff 2	18	a196da5	- Fixed documentation, input arg checking and requirements.txt from comments in review	Feb 14 2014, 5:23 AM	★	★
Diff 3	21	a1de947	Commit rLTRN3fb8c35b2eb0a18f39eed2bb5749c2deffab5040	Feb 20 2014, 7:25 AM	★	★

Commit	Tree	Parents	Author	Summary	Date
b4e961f31e77	dd7b9654f2d2	9d996f309698	Tim Flink	Fixed documentation, input arg checking and requirements.txt from comments in… (Show More…)	Feb 14 2014, 5:23 AM
9d996f309698	7643fed11955	a196da5d0174	Tim Flink	reworking runner to make it capable of running more than rpmlint. removed… (Show More…)	Feb 6 2014, 5:09 PM

Diff 21

View Options

libtaskotron/directives/init.py

1	class BaseDirective(object):	1	class BaseDirective(object):
2		2
3	def __init__(self):	3	def __init__(self):
4	pass	4	pass
5		5
6	def process(self, command, input_data, env_data):	6	def process(self, input_data, env_data):
7	raise NotImplementedError	7	raise NotImplementedError
8		8

View Options

libtaskotron/directives/dummy_directive.py

This file was added.

1		from libtaskotron.directives import BaseDirective
2
3		directive_class = 'DummyDirective'
4
5
6		class DummyDirective(BaseDirective):
7		"""The dummy directive is just what it sounds like - a directive that
8		doesn't do anything other than return a status and (optionally) a message.
9
	mkrizekUnsubmitted Not Done Do you think it would be useful to list required input_data's keys for each directive in its class in pydoc?
	tflinkAuthorUnsubmitted Not Done yeah, I'll add that for the two directives
10		It is primarially meant for testing the runner or as a placeholder while
11		writing new tasks.
12
13		format:
14		dummy: result=<result> [msg=<some string message>]
15		required params: result
16		optional params: msg
17		"""
18
19		def process(self, input_data, env_data):
20		expected_result = input_data['result']
21
22		if expected_result.lower() == 'fail':
23		raise Exception("This is a dummy directive and configured to fail")
24
25		if 'msg' in input_data:
26		return input_data['msg']

View Options

libtaskotron/directives/koji_directive.py

1	import koji	1		import koji
2		2
3	from libtaskotron import file_utils	3		from libtaskotron import file_utils
4	from libtaskotron.directives import BaseDirective	4		from libtaskotron.directives import BaseDirective
5		5
		6		directive_class = 'KojiDirective'
		7
6	koji_url = 'http://koji.fedoraproject.org/kojihub'	8		koji_url = 'http://koji.fedoraproject.org/kojihub'
7	pkg_url = 'http://kojipkgs.fedoraproject.org/packages'	9		pkg_url = 'http://kojipkgs.fedoraproject.org/packages'
8		10
		11
9	class KojiDirective(BaseDirective):	12		class KojiDirective(BaseDirective):
		13		"""The koji directive interfaces with Koji to facilitate various koji actions.
		14
		15		format: "koji: command=[download] <command args>
		16
		17		Valid Commands:
		18
		19		download
		20		Downloads the envr specified as "envr"
		21		format: "koji: command=download envr=<envr to download> arch=<desired arch>
		22		"""
		23
10	def __init__(self, koji_session=None):	24		def __init__(self, koji_session=None):
11	self.session = koji_session	25		self.session = koji_session
12		26
13	def get_koji_session(self):	27		def get_koji_session(self):
14	if self.session == None:	28		if self.session == None:
15	self.session = koji.ClientSession(koji_url)	29		self.session = koji.ClientSession(koji_url)
16		30
17	return self.session	31		return self.session
Show All 36 Lines		67	'''Retrieve the RPMs associated with a build NVR into the specified
54	directory.	68		directory.
55		69
56	@param nvr build NVR	70		@param nvr build NVR
57	@param rpm_dir location where to store the RPMs	71		@param rpm_dir location where to store the RPMs
58	@return list of local filenames of the grabbed RPMs	72		@return list of local filenames of the grabbed RPMs
59	@raise urlgrabber.grabber.URLGrabError if downloading failed	73		@raise urlgrabber.grabber.URLGrabError if downloading failed
60	'''	74		'''
61		75
62	rpm_urls = self.nvr_to_urls(nvr, arches=arches, debuginfo=debuginfo, src=src)	76		rpm_urls = self.nvr_to_urls(nvr,
		77		arches=arches,
		78		debuginfo=debuginfo,
		79		src=src)
		80
63	rpm_files = []	81		rpm_files = []
64	print 'Fetching RPMs for: %s' % nvr	82		print 'Fetching RPMs for: %s' % nvr
65	for url in rpm_urls:	83		for url in rpm_urls:
66	print ' fetching %s' % url	84		print ' fetching %s' % url
67	rpm_file = file_utils. download(url, rpm_dir)	85		rpm_file = file_utils. download(url, rpm_dir)
68	rpm_files.append(rpm_file)	86		rpm_files.append(rpm_file)
69		87
70	return rpm_files	88		return rpm_files
71		89
72	def process(self, command, input_data, env_data):	90		def process(self, input_data, env_data):
73	valid_commands = ['download']	91		valid_actions = ['download']
74	valid_args = ['envr']
75		92
76	output_data = {}	93		output_data = {}
77		94
78	# assume that the command input is in the form '<command> [<arg1>[<arg2>]]'	95		action = input_data['action']
79	commands = command.split()
80
81	# for now, strip off any $ characters, they will eventually be vars but that
82	# functionality isnt implemented yet
83	commands = [c.replace('$', '') for c in commands]
84		96
85		97		if not action in valid_actions:
86	if not commands[0] in valid_commands:	98		raise Exception('%s is not a valid command for koji helper' % action)
			jskladanUnsubmitted Not Done It would be much better to use some custom Exception - maybe some libtaskotron-wide but libtaskotron specific one
87	raise Exception('%s is not a valid command for koji helper' % commands[0])
88		99
89	# for now, the only valid command is download, can simplify code	100		# for now, the only valid command is download, can simplify code
90		101
91	for arg in commands[1:]:	102		if action == 'download':
92	if not arg in valid_args:	103		if 'envr' in input_data and 'arch' in input_data:
			mkrizekUnsubmitted Not Done Shouldn't this line be: "if 'envr' and 'arch' in input_data:"?
			tflinkAuthorUnsubmitted Not Done yeah, there should be some input validation here and in all the directives
93	raise Exception('%s is not a valid arg for koji helper.' % arg)
94
95	if commands[0] == 'download':
96	if commands[1] == 'envr':
97	# we need to find the tmpdir and nvr from input data	104		# we need to find the tmpdir and nvr from input data
98	tmpdir = env_data['tmpdir']	105		tmpdir = env_data['workdir']
99	envr = input_data['envr']	106		envr = input_data['envr']
100	arches = [input_data['arch']]	107		arches = [input_data['arch']]
101		108
102	print "getting koji builds for %s (%s) and downloading to %s" % (envr, arches, tmpdir)	109		print "getting koji builds for %s (%s) and downloading to %s" %\
103	output_data['downloaded_rpms'] = self.get_nvr_rpms(envr, tmpdir, arches)	110		(envr, arches, tmpdir)
		111		output_data['downloaded_rpms'] = self.get_nvr_rpms(envr,
		112		tmpdir,
		113		arches)
104		114
105	return output_data	115		return output_data

View Options

libtaskotron/directives/python_directive.py

Show All 28 Lines		9	class PythonDirective(BaseDirective):
29		29
30	In this case, you could pass in 'embedded_task_target' as the callable arg	30		In this case, you could pass in 'embedded_task_target' as the callable arg
31		31
32	def task_method():	32		def task_method():
33	return "I'm a task method!"	33		return "I'm a task method!"
34		34
35	And in this third case, you would pass in 'task_method' as the callable arg	35		And in this third case, you would pass in 'task_method' as the callable arg
36		36
37	format: "python: <python filename> callable=<method name> [<kwargname>=<kwargvalue>'	37		format:
38	required params: <python filename>, callable	38		python: file=<python filename> callable=<method name> [<kwargname>=<kwargvalue>'
39	optional params: farther key/value pairs, to be passed into the callable as kwargs	39		required params: <python filename>, callable
		40		optional params: farther key/value pairs, to be passed into the callable as kwargs
40	"""	41		"""
41		42
42	def _do_getattr(self, module, name):	43		def _do_getattr(self, module, name):
43	"""Isolation of getattr to make the code more easily testable	44		"""Isolation of getattr to make the code more easily testable
44		45
45	@param module module from which to getattr from	46		@param module module from which to getattr from
46	@param name name of object to getattr	47		@param name name of object to getattr
47	@returns object retrieved from module	48		@returns object retrieved from module
Show All 32 Lines		79	def checkfile(self, filename):
80		81
81	@param filename abspath to the filename	82		@param filename abspath to the filename
82	@throws Exception if the file does not exist	83		@throws Exception if the file does not exist
83	"""	84		"""
84		85
85	if not os.path.exists(filename):	86		if not os.path.exists(filename):
86	raise Exception("The file %s does not exist!" % filename)	87		raise Exception("The file %s does not exist!" % filename)
87		88
88	def process(self, command, input_data, env_data):	89		def process(self, input_data, env_data):
89	"""For the python directive, we're expecting a yaml declaration of the form	90		"""For the python directive, we're expecting a yaml declaration of the form
90	"python: <python file> callable=<method name> [kwarg1=something ...]"	91		"python: file=<python file> callable=<method name> [kwarg1=something ...]"
			mkrizekUnsubmitted Not Done file=<python file>
91		92
92	@param command full path to the python module	93		@param input_data dictionary of all kwargs from yaml declaration, including callable and file
93	@param input_data dictionary of all kwargs from yaml declaration, including method_name
94	@param env_data information on the environment in which we're running	94		@param env_data information on the environment in which we're running
95	@return output from the method specified, this is expected to be TAP output	95		@return output from the method specified, this is expected to be TAP output
96	"""	96		"""
97		97
		98		if 'callable' not in input_data or 'file' not in input_data:
		99		detected_args = ', '.join(input_data.keys())
		100		raise Exception("The python directive requires both 'callable' and \
			jskladanUnsubmitted Not Done It would be much better to use some custom Exception - maybe some libtaskotron-wide but libtaskotron specific one
		101		'file' arguments. Detected arguments: %s" % detected_args)
		102
98	method_name = input_data.pop('callable')	103		method_name = input_data.pop('callable')
			mkrizekUnsubmitted Not Done Again, should we check for 'callable' and 'file' if they are present in input_data?
99	basedir = env_data['basedir']	104		basedir = os.path.dirname(env_data['taskfile'])
100		105
101	pyfile = os.path.join(basedir, command)	106		pyfile = os.path.join(basedir, input_data.pop('file'))
102		107
103	self.checkfile(pyfile)	108		self.checkfile(pyfile)
104		109
105	task_name, task = self.load_pyfile(pyfile)	110		task_name, task = self.load_pyfile(pyfile)
106		111
107	output = self.execute(task, method_name, input_data)	112		output = self.execute(task, method_name, input_data)
108		113
109	return output	114		return output

View Options

libtaskotron/runner.py

1	import tempfile	1		import tempfile
2	from libtaskotron import taskyaml
3	import os.path	2		import os.path
4	import argparse	3		import argparse
5	import imp	4		import imp
		5		from jinja2 import Template, Environment
			mkrizekUnsubmitted Not Done jinja2 is not included in requirements.txt
		6		import copy
		7
		8		from libtaskotron import taskyaml
6		9
7	class Runner:	10		class Runner:
8	def __init__(self, taskdata, inputdata, workdir=None):	11		def __init__(self, taskdata, argdata, workdir=None):
		12		self.taskdata = taskdata
		13		self.envdata = argdata
		14		self.working_data = {}
		15		self.directives = {}
		16		self.jinja_env = None
		17
9	if workdir is None:	18		if workdir is None:
10	self.workdir = tempfile.mkdtemp(dir='/var/tmp')	19		self.workdir = tempfile.mkdtemp(dir='/var/tmp')
11	else:	20		else:
12	self.workdir = workdir	21		self.workdir = workdir
13		22
14	self.taskdata = taskdata
15	self.inputdata = inputdata
16	self.working_data = {}
17	self.directives = {}
18
19	def run(self):	23		def run(self):
20	# this isn't very flexible yet, just a proof of concept	24		self._validate_input()
21	self.do_prep()	25		self.envdata['workdir'] = self.workdir
22		26
23	self.do_execution()	27		self.do_actions()
24		28
25	def _load_directive(self, directive_name, directive_dir=None):	29		def _load_directive(self, directive_name, directive_dir=None):
26	# look in default path if nothing is specified	30		# look in default path if nothing is specified
27	if not directive_dir:	31		if not directive_dir:
28	directive_dir = os.path.join(os.path.dirname(__file__), 'directives')	32		directive_dir = os.path.join(os.path.dirname(__file__),
		33		'directives')
29		34
30	real_name = "%s_directive" % directive_name	35		real_name = "%s_directive" % directive_name
31	directive_file = os.path.join(directive_dir, '%s.py' % real_name)	36		directive_file = os.path.join(directive_dir, '%s.py' % real_name)
32		37
33	if not os.path.exists(directive_file):	38		if not os.path.exists(directive_file):
34	raise Exception("Directive %s not found in directory %s" % (directive_name, directive_dir))	39		raise Exception("Directive %s not found in directory %s" %
		40		(directive_name, directive_dir))
35		41
36	loaded_directive = imp.load_source(real_name, directive_file)	42		loaded_directive = imp.load_source(real_name, directive_file)
37	self.directives[directive_name] = loaded_directive	43		self.directives[directive_name] = loaded_directive
38		44
		45		def _get_jinja_env(self):
		46		if not self.jinja_env:
		47		self.jinja_env = Environment()
		48		self.jinja_env.globals = self.envdata
		49		return self.jinja_env
39		50
40	def do_prep(self):	51		def _render_action(self, action):
41	commands = self.taskdata['preparation']	52		# copy the input so that we don't disrupt what we're processing
42		53		rendered_action = copy.deepcopy(action)
			mkrizekUnsubmitted Not Done I wonder, should we use term 'subtask' instead of 'action'? It might be a little confusing to use 'action' since not all directives have 'action' (the python directive) and if I am correct this is not action as in arg of a directive but action as in subtask, right?
			tflinkAuthorUnsubmitted Not Done I'd prefer changing the parameter name for koji if there's potential for confusion. To me, the idea of a task made up of actions flows well in a linguistic sense while the directive arg could be changed from 'action' to 'command' without any loss in how natural it sounds.
			mkrizekUnsubmitted Not Done That works for me.
43	supported_prep = ['koji']
44	for command_type in commands:
45	if command_type in supported_prep:
46	self._load_directive(command_type)
47	else:
48	raise Exception("Prep command %s not currently supported" % command_type)
49
50	# the only supported prep right now is koji, makes for simple code :)
51		54
52	helper = self.directives['koji'].KojiDirective()	55		# any action is going to be a dict with 0 or more embedded dicts
		56		for action_key in action:
		57		jinja_env = self._get_jinja_env()
53		58
54	# instead, merge dict coming out of prep - let prep decide what data is named	59		if isinstance(action[action_key], dict):
55	prep_output = helper.process(commands['koji'], self.inputdata, {'tmpdir':self.workdir})	60		action_line = action[action_key]
56	self.working_data.update(prep_output)	61		for keyval in action_line:
		62		input_action = action[action_key][keyval]
		63		arg_template = jinja_env.from_string(input_action)
57		64
58	def do_deps(self, deps):	65		rendered_line = arg_template.render(self.working_data)
59	print "fake verifying deps: %s" % str(deps)	66		rendered_action[action_key][keyval] = rendered_line
60		67
61	def do_execution(self):	68		return rendered_action
62	commands = self.taskdata['execution']
63	print "doing execution with commands: %s" % str(commands)
64		69
65	supported_execution = ['python']	70		def _extract_directive_from_action(self, action):
		71		for key in action:
		72		if key not in ['name', 'export']:
		73		return key
		74		raise Exception('no directive found in action %s' % str(action))
66		75
67	for command_type in commands:	76		def do_single_action(self, action):
68	if command_type in supported_execution:	77		directive_name = self._extract_directive_from_action(action)
69	self._load_directive(command_type)
70	else:
71	raise Exception("Execution command %s not currently supported" % command_type)
72		78
73	# the only supported execution right now is rpmlint as a python file	79		rendered_action = self._render_action(action)
			mkrizekUnsubmitted Not Done Why don't we run the yaml file through jinja first and then load it so we do jinja parsing just once. Is there a reason we do jinja parsing for each action separately? Unless I am missing something.
			tflinkAuthorUnsubmitted Not Done because not all the variables exist when the yaml file is loaded. The biggest example of this is for values which are exported - the actual values don't exist until after the action has been run. If we render the template before that execution, the value will be empty (at best) and the data won't be transferred between actions
74	args = commands['python'].split()
75		80
76	# for now, ignore any args since we're just supporting pylint	81		self._load_directive(directive_name)
77	pyfilename = args[0]
78		82
79	basedir = os.path.dirname(os.path.abspath(self.inputdata['taskfile']))	83		directive_object = self.directives[directive_name]
		84		directive_classname = directive_object.directive_class
80		85
81	helper = self.directives['python'].PythonDirective()	86		directive_callable = getattr(directive_object, directive_classname)()
82		87
83	# prepare input for running the python directive	88		output = directive_callable.process(rendered_action[directive_name],
84	input_args = {'callable': 'run', 'workingdir': self.workdir}	89		self.envdata)
85	env_args = {'basedir': basedir}
86		90
87	exec_output = helper.process(pyfilename, input_args, env_args)	91		if 'export' in action:
		92		self.working_data[action['export']] = output
88		93
89	self.working_data.update({'exec_output':exec_output})	94		def do_actions(self):
		95		for action in self.taskdata['task']:
		96		self.do_single_action(action)
90		97
91	def do_post(self, commands):	98		def _validate_input(self):
92	print "doing post with commands: %s" % str(commands)	99		required_input = self.taskdata['input']['args'].split(',')
93		100
94	def process_input(input_spec, args):	101		for arg in required_input:
95	print "verifying input: %s" % str(input_spec)	102		if not arg in self.envdata:
		103		raise Exception('Required input arg %s was not defined' % arg)
96		104
97	required_input = input_spec['args'].split(',')	105		def _validate_env(self):
		106		raise NotImplementedError("Environment validation is not yet implemented")
98		107
99	data = {}
100	for arg in required_input:
101	if args.__getattribute__(arg) is None:
102	raise Exception('expected arg %s was not defined' % arg)
103	else:
104	data[arg] = args.__getattribute__(arg)
105
106	data['taskfile'] = args.task[0]
107	return data
108		108
109	def get_argparser():	109		def get_argparser():
110	parser = argparse.ArgumentParser()	110		parser = argparse.ArgumentParser()
111	parser.add_argument("task", nargs=1, help="task to run")	111		parser.add_argument("task", nargs=1, help="task to run")
112	parser.add_argument("-a", "--arch", choices=["i386", "x86_64", "armhfp", "noarch"], help="""	112		parser.add_argument("-a", "--arch", choices=["i386", "x86_64", "armhfp", "noarch"], help="""
113	Architecture to be used; support arches are x86, x86_64, and armhfp.	113		Architecture to be used; support arches are x86, x86_64, and armhfp.
114	If omitted, defaults to noarch.	114		If omitted, defaults to noarch.
115	""")	115		""")
116	parser.add_argument("-e", "--envr", help="envr to be used as input")	116		parser.add_argument("-e", "--envr", help="envr to be used as input")
117	return parser	117		return parser
118		118
119	def main():	119		def main():
120	parser = get_argparser()	120		parser = get_argparser()
121	args = parser.parse_args()	121		args = parser.parse_args()
122		122
123	task_data = taskyaml.parse_yaml_from_file(args.task[0])	123		arg_data = vars(args)
124		124		arg_data['taskfile'] = args.task[0]
125	print "task data: %s" % str(task_data)
126
127	input_data = process_input(task_data['input'], args)
128		125
129	print "input data: %s" % str(input_data)	126		task_data = taskyaml.parse_yaml_from_file(arg_data['taskfile'])
130		127
131	task_runner = Runner(task_data, input_data)	128		task_runner = Runner(task_data, arg_data)
132	task_runner.run()	129		task_runner.run()
133		130
134	for result in task_runner.working_data['exec_output']:	131		for output in task_runner.working_data:
135	print result	132		print task_runner.working_data[output]
		133		No newline at end of file

View Options

requirements.txt

1	PyYAML==3.10	1	PyYAML==3.10
2	dingus==0.3.4	2	dingus==0.3.4
3	py==1.4.18	3	py>=1.4.18
4	pyaml==13.07.1	4	pyaml==13.07.1
5	pytest==2.4.2	5	pytest>=2.4.2
		6	jinja2>=2.7.2
6		7
7	# these are bundled in libtaskotron for now	8	# these are bundled in libtaskotron for now
8	#bayeux==0.7	9	#bayeux==0.7
9	#yamlish==0.10	10	#yamlish==0.10

View Options

testing/test_koji.py

This file was moved to testing/test_koji_directive.py.

View Options

testing/test_koji_directive.py

This file was moved from testing/test_koji.py.

Show First 20 Lines • Show All 50 Lines • ▼ Show 20 Line(s)		38	def should_i386_rpmurls_query_i686(self):
51		51
52	listrpm_calls = stub_koji.calls('listRPMs')	52		listrpm_calls = stub_koji.calls('listRPMs')
53	requested_arches = listrpm_calls[0][2]['arches']	53		requested_arches = listrpm_calls[0][2]['arches']
54		54
55	assert 'i686' in requested_arches	55		assert 'i686' in requested_arches
56	assert 'i386' in requested_arches	56		assert 'i386' in requested_arches
57		57
58	def test_parse_download_command(self, monkeypatch):	58		def test_parse_download_command(self, monkeypatch):
59	ref_command = 'download envr'	59		ref_input = {'action': 'download', 'arch': self.ref_arch, 'envr': self.ref_nvr}
60	ref_input = {'arch':self.ref_arch, 'envr':self.ref_nvr}	60		ref_envdata = {'workdir':'/var/tmp/foo'}
61	ref_envdata = {'tmpdir':'/var/tmp/foo'}
62		61
63	stub_koji = Dingus(getBuild__returns = self.ref_build, listRPMs__returns = self.ref_rpms)	62		stub_koji = Dingus(getBuild__returns = self.ref_build, listRPMs__returns = self.ref_rpms)
64	ref_rpms = ['/var/tmp/fake/%s.%s.rpm' % (self.ref_nvr, self.ref_arch)]	63		ref_rpms = ['/var/tmp/fake/%s.%s.rpm' % (self.ref_nvr, self.ref_arch)]
65		64
66	stub_get_rpms = Dingus(get_nvr_rpms__returns=ref_rpms)	65		stub_get_rpms = Dingus(get_nvr_rpms__returns=ref_rpms)
67	test_helper = koji_directive.KojiDirective(stub_koji)	66		test_helper = koji_directive.KojiDirective(stub_koji)
68		67
69	monkeypatch.setattr(test_helper, 'get_nvr_rpms', stub_get_rpms)	68		monkeypatch.setattr(test_helper, 'get_nvr_rpms', stub_get_rpms)
70	test_helper.process(ref_command, ref_input, ref_envdata)	69		test_helper.process(ref_input, ref_envdata)
71		70
72	getrpm_calls = stub_get_rpms.calls()	71		getrpm_calls = stub_get_rpms.calls()
73	print getrpm_calls	72		print getrpm_calls
74	requested_nvr = getrpm_calls[0][1][0]	73		requested_nvr = getrpm_calls[0][1][0]
75		74
76	assert len(getrpm_calls) == 1	75		assert len(getrpm_calls) == 1
77	assert requested_nvr == self.ref_nvr	76		assert requested_nvr == self.ref_nvr
78		77
79	def should_throw_exception_norpms(self):	78		def should_throw_exception_norpms(self):
80	stub_koji = Dingus(getBuild__returns = self.ref_build)	79		stub_koji = Dingus(getBuild__returns = self.ref_build)
81		80
82	test_koji = koji_directive.KojiDirective(stub_koji)	81		test_koji = koji_directive.KojiDirective(stub_koji)
83		82
84	with pytest.raises(Exception):	83		with pytest.raises(Exception):
85	test_koji.nvr_to_urls(self.ref_nvr, arches = [self.ref_arch])	84		test_koji.nvr_to_urls(self.ref_nvr, arches = [self.ref_arch])
86		85

View Options

testing/test_python_directive.py

Show First 20 Lines • Show All 71 Lines • ▼ Show 20 Line(s)		62	def test_task_method_is_executed(self, monkeypatch):
72	test_directive.execute(ref_modulename, ref_methodname, ref_kwargs)	72		test_directive.execute(ref_modulename, ref_methodname, ref_kwargs)
73		73
74	assert (stub_getattr.calls()[1].name, stub_getattr.calls()[1].args, stub_getattr.calls()[1].kwargs) == ('()', (), (ref_kwargs))	74		assert (stub_getattr.calls()[1].name, stub_getattr.calls()[1].args, stub_getattr.calls()[1].kwargs) == ('()', (), (ref_kwargs))
75		75
76		76
77	class TestProcess(object):	77		class TestProcess(object):
78		78
79	def setup_method(self, method):	79		def setup_method(self, method):
80	self.ref_command = 'foo.py'	80
81	self.ref_input = {'callable': 'bar', 'kwarg1': 'baz'}	81		self.ref_taskyaml = '/path/to/foo/footask.yml'
82	self.ref_env = {'basedir': '/path/to/foo/'}	82		self.ref_input = {'file': 'foo.py', 'callable': 'bar', 'kwarg1': 'baz'}
		83		self.ref_env = {'taskfile': self.ref_taskyaml}
		84		self.ref_pyfile = os.path.join(os.path.dirname(self.ref_taskyaml), self.ref_input['file'])
83		85
84	self.stub_checkfile = Dingus('checkfile')	86		self.stub_checkfile = Dingus('checkfile')
85	self.stub_imported = Dingus('imported')	87		self.stub_imported = Dingus('imported')
86	self.stub_pyload = Dingus('pyload', dostuff__returns=('name', self.stub_imported))	88		self.stub_pyload = Dingus('pyload', dostuff__returns=('name', self.stub_imported))
87	self.stub_execute = Dingus('execute')	89		self.stub_execute = Dingus('execute')
88		90
89	def test_process_checks_file(self, monkeypatch):	91		def test_process_checks_file(self, monkeypatch):
90		92
		93
91	test_directive = python_directive.PythonDirective()	94		test_directive = python_directive.PythonDirective()
92	monkeypatch.setattr(test_directive, 'checkfile', self.stub_checkfile)	95		monkeypatch.setattr(test_directive, 'checkfile', self.stub_checkfile)
93	monkeypatch.setattr(test_directive, 'load_pyfile', self.stub_pyload.dostuff)	96		monkeypatch.setattr(test_directive, 'load_pyfile', self.stub_pyload.dostuff)
94	monkeypatch.setattr(test_directive, 'execute', self.stub_execute)	97		monkeypatch.setattr(test_directive, 'execute', self.stub_execute)
95		98
96	test_output = test_directive.process(self.ref_command, copy.copy(self.ref_input), self.ref_env)	99		test_output = test_directive.process(copy.copy(self.ref_input), self.ref_env)
97		100
98	checkfile_call = self.stub_checkfile.calls[0]	101		checkfile_call = self.stub_checkfile.calls[0]
99		102
100	assert (checkfile_call.name, checkfile_call.args, checkfile_call.kwargs) == ('()', (os.path.join(self.ref_env['basedir'], self.ref_command),), {})	103
		104		assert (checkfile_call.name, checkfile_call.args, checkfile_call.kwargs) == ('()', (self.ref_pyfile,), {})
101		105
102	def test_process_loads_file(self, monkeypatch):	106		def test_process_loads_file(self, monkeypatch):
103		107
104	test_directive = python_directive.PythonDirective()	108		test_directive = python_directive.PythonDirective()
105	monkeypatch.setattr(test_directive, 'checkfile', self.stub_checkfile)	109		monkeypatch.setattr(test_directive, 'checkfile', self.stub_checkfile)
106	monkeypatch.setattr(test_directive, 'load_pyfile', self.stub_pyload.dostuff)	110		monkeypatch.setattr(test_directive, 'load_pyfile', self.stub_pyload.dostuff)
107	monkeypatch.setattr(test_directive, 'execute', self.stub_execute)	111		monkeypatch.setattr(test_directive, 'execute', self.stub_execute)
108		112
109	test_output = test_directive.process(self.ref_command, copy.copy(self.ref_input), self.ref_env)	113		test_output = test_directive.process(copy.copy(self.ref_input), self.ref_env)
110		114
111	pyload_call = self.stub_pyload.calls[0]	115		pyload_call = self.stub_pyload.calls[0]
112		116
113	assert (pyload_call.name, pyload_call.args, pyload_call.kwargs) == ('dostuff', (os.path.join(self.ref_env['basedir'], self.ref_command),), {})	117		assert (pyload_call.name, pyload_call.args, pyload_call.kwargs) == ('dostuff', (self.ref_pyfile,), {})
114		118
115	def test_process_executes(self, monkeypatch):	119		def test_process_executes(self, monkeypatch):
116		120
117	ref_kwargs = {'kwarg1': 'baz'}	121		ref_kwargs = {'kwarg1': 'baz'}
118		122
119	test_directive = python_directive.PythonDirective()	123		test_directive = python_directive.PythonDirective()
120	monkeypatch.setattr(test_directive, 'checkfile', self.stub_checkfile)	124		monkeypatch.setattr(test_directive, 'checkfile', self.stub_checkfile)
121	monkeypatch.setattr(test_directive, 'load_pyfile', self.stub_pyload.dostuff)	125		monkeypatch.setattr(test_directive, 'load_pyfile', self.stub_pyload.dostuff)
122	monkeypatch.setattr(test_directive, 'execute', self.stub_execute)	126		monkeypatch.setattr(test_directive, 'execute', self.stub_execute)
123		127
124	test_output = test_directive.process(self.ref_command, copy.copy(self.ref_input), self.ref_env)	128		test_output = test_directive.process(copy.copy(self.ref_input), self.ref_env)
125		129
126	execute_call = self.stub_execute.calls[0]	130		execute_call = self.stub_execute.calls[0]
127		131
128	assert (execute_call.name, execute_call.args, execute_call.kwargs) == ('()', (self.stub_imported, self.ref_input['callable'], ref_kwargs), {})	132		assert (execute_call.name, execute_call.args, execute_call.kwargs) == ('()', (self.stub_imported, self.ref_input['callable'], ref_kwargs), {})
129	No newline at end of file	133		No newline at end of file

View Options

testing/test_runner.py

This file was added.

		1		import pytest
		2		import os
		3		import sys
		4
		5		from libtaskotron import runner
		6
		7
		8
		9		class TestRunnerInputVerify():
		10		def test_yamlrunner_valid_input(self):
		11		ref_cmd = '-e foo-1.2-3.fc99 -a x86_64 footask.yml'
		12		ref_argdata = {'input': {'args': 'envr,arch'}}
		13		ref_data = {'envr': 'foo-1.2-3.fc99', 'arch': 'x86_64', 'taskfile': 'footask.yml'}
		14
		15		test_parser = runner.get_argparser()
		16		test_args = vars(test_parser.parse_args(ref_cmd.split()))
		17
		18		test_runner = runner.Runner(ref_argdata, test_args)
		19
		20		test_runner._validate_input()
		21
		22		def test_yamlrunner_fails_missing_arg(self):
		23		ref_cmd = '-e foo-1.2-3.fc99 footask.yml'
		24		ref_inputspec = {'args':'envr,arch'}
		25
		26		test_parser = runner.get_argparser()
		27		test_args = vars(test_parser.parse_args(ref_cmd.split()))
		28
		29		test_runner = runner.Runner(ref_inputspec, test_args)
		30
		31		with pytest.raises(Exception):
		32		test_runner._validate_input()
		33
		34		class TestRunnerSetup():
		35		def test_trivial_creation(self, tmpdir):
		36		ref_taskdata = {'preparation': {'koji': 'download envr'},
		37		'input': {'args': 'envr,arch'}, 'post': {'shell': 'clean'},
		38		'execution': {'python': 'run_rpmlint.py'},
		39		'dependencies': ['rpmlint', 'libtaskbot']}
		40		ref_inputdata = {}
		41
		42		test_runner = runner.Runner(ref_taskdata, ref_inputdata, tmpdir)
		43
		44		assert test_runner.taskdata == ref_taskdata
		45
		46		class TestRunnerPrep():
		47		def test_prep_invalid_commands(self, tmpdir):
		48		ref_taskdata = {'preparation': {'turboencabulator':'reduce sinusoidal repleneration'}}
		49		ref_inputdata = {}
		50
		51		test_runner = runner.Runner(ref_taskdata, ref_inputdata, tmpdir)
		52
		53		with pytest.raises(Exception):
		54		test_runner.do_prep()
		55
		56		class TestRunnerDirectiveLoader():
		57
		58		def setup_method(self, method):
		59		self.ref_dir = os.path.abspath('testing/test_directives/')
		60		self.ref_taskdata = {}
		61		self.ref_inputdata = {}
		62		self.ref_directivename = 'test'
		63
		64		def test_runner_load_directive(self, tmpdir):
		65		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata,
		66		tmpdir)
		67
		68		test_runner._load_directive(self.ref_directivename, self.ref_dir)
		69
		70		assert '%s_directive' % self.ref_directivename in sys.modules.keys()
		71
		72		def test_runner_load_directive_indict(self, tmpdir):
		73		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata,
		74		tmpdir)
		75
		76		test_runner._load_directive(self.ref_directivename, self.ref_dir)
		77
		78		assert self.ref_directivename in test_runner.directives.keys()
		79
		80		def test_runner_load_directive_notexist(self, tmpdir):
		81		self.ref_directivename = 'noexist'
		82
		83		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata,
		84		tmpdir)
		85
		86		with pytest.raises(Exception):
		87		test_runner._load_directive(self.ref_directivename, self.ref_dir)
		88
		89		class TestRunnerSingleAction():
		90
		91		def setup_method(self, method):
		92		self.ref_taskdata = {}
		93		self.ref_inputdata = {}
		94
		95		def test_runner_extract_directive_name(self):
		96		ref_directivename = 'dummy'
		97		ref_action = {'name': 'test action',
		98		ref_directivename: {'result': 'pass', 'msg': 'dummy message'}}
		99
		100		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		101
		102		test_directivename = test_runner._extract_directive_from_action(ref_action)
		103
		104		assert test_directivename == ref_directivename
		105
		106		def test_runner_single_dummy_action_pass(self):
		107		ref_action = {'dummy': {'result': 'pass'}}
		108		ref_action.update({'name': 'Dummy Action' })
		109
		110		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		111
		112		test_runner.do_single_action(ref_action)
		113
		114
		115		def test_runner_single_dummy_action_fail(self):
		116		ref_action = {'name': 'Dummy Action',
		117		'dummy': {'result': 'fail'}}
		118
		119		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		120
		121		with pytest.raises(Exception):
		122		test_runner.do_single_action(ref_action)
		123
		124		def test_runner_do_single_dummy_action_pass_export(self):
		125		ref_exportname = 'dummy_output'
		126		ref_output = 'this is example output'
		127		ref_action = {'name': 'Dummy Action',
		128		'dummy': {'result': 'pass', 'msg': ref_output},
		129		'export': '%s' % ref_exportname}
		130
		131
		132		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		133
		134		test_runner.do_single_action(ref_action)
		135
		136		assert test_runner.working_data[ref_exportname] == ref_output
		137
		138		def test_runner_do_single_dummy_action_pass_export_rendered_msg(self):
		139		ref_exportname = 'dummy_output'
		140		ref_message = 'This is a variable message'
		141		ref_messagename = 'variable_messsage'
		142		ref_action = {'name': 'Dummy Action',
		143		'dummy': {'result': 'pass',
		144		'msg': '{{ %s }}' % ref_messagename},
		145		'export': '%s' % ref_exportname}
		146
		147		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		148		test_runner.working_data[ref_messagename] = ref_message
		149
		150		test_runner.do_single_action(ref_action)
		151
		152		assert test_runner.working_data[ref_exportname] == ref_message
		153
		154
		155		class TestRunnerDoActions():
		156
		157		def setup_method(self, method):
		158		self.ref_taskdata = {}
		159		self.ref_inputdata = {}
		160
		161		def test_runner_do_single_dummy_pass(self):
		162		ref_action = {'name': 'Dummy Action',
		163		'dummy': {'result': 'pass'}}
		164		self.ref_taskdata = {'task': [ref_action]}
		165
		166		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		167
		168		test_runner.do_actions()
		169
		170		def test_runner_do_single_dummy_fail(self):
		171		ref_action = {'name': 'Dummy Action',
		172		'dummy': {'result': 'fail'}}
		173		self.ref_taskdata = {'task': [ref_action]}
		174
		175		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		176
		177		with pytest.raises(Exception):
		178		test_runner.do_actions()
		179
		180		def test_runner_do_multiple_dummy_fail(self):
		181		ref_action = {'name': 'Dummy Action',
		182		'dummy': {'result': 'fail'}}
		183		self.ref_taskdata = {'task': [ref_action, ref_action]}
		184
		185		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		186
		187		with pytest.raises(Exception):
		188		test_runner.do_actions()
		189
		190		def test_runner_do_multiple_dummy_pass_export_rendered_msg(self):
		191		ref_exportname = 'dummy_output'
		192		ref_message = 'This is a variable message'
		193		ref_messagename = 'variable_messsage'
		194		ref_action = {'name': 'Dummy Action',
		195		'dummy': {'result': 'pass',
		196		'msg': '{{ %s }}' % ref_messagename},
		197		'export': '%s' % ref_exportname}
		198
		199		self.ref_taskdata = {'task': [ref_action, ref_action]}
		200
		201		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		202		test_runner.working_data[ref_messagename] = ref_message
		203
		204		test_runner.do_actions()
		205
		206		assert test_runner.working_data[ref_exportname] == ref_message
		207
		208
		209		class TestRunnerRenderAction():
		210
		211		def setup_method(self, method):
		212		self.ref_taskdata = {}
		213		self.ref_inputdata = {}
		214
		215		def test_runner_render_single_variable(self):
		216		ref_message = 'This is a variable message'
		217		ref_messagename = 'variable_messsage'
		218		ref_action = {'name': 'test dummy action', 'dummy': {'result': 'pass', 'msg': '{{ %s }}' % ref_messagename}}
		219
		220		test_runner = runner.Runner(self.ref_taskdata, self.ref_inputdata)
		221		test_runner.working_data[ref_messagename] = ref_message
		222
		223		test_rendered_action = test_runner._render_action(ref_action)
		224
		225		assert test_rendered_action['dummy']['msg'] == ref_message
		226
		227

View Options

testing/test_yamlrunner.py

This file was deleted.

This file was completely deleted. Show File Contents

reworking runner to make it capable of running more than rpmlint. removed command arg, moved everything to key-value args and updated existing directives. Fixes T71ClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 21

libtaskotron/directives/__init__.py

libtaskotron/directives/dummy_directive.py

libtaskotron/directives/koji_directive.py

libtaskotron/directives/python_directive.py

libtaskotron/runner.py

requirements.txt

testing/test_koji.py

testing/test_koji_directive.py

testing/test_python_directive.py

testing/test_runner.py

testing/test_yamlrunner.py

reworking runner to make it capable of running more than rpmlint. removed command arg, moved everything to key-value args and updated existing directives. Fixes T71
ClosedPublic
Actions

libtaskotron/directives/init.py