Initial Skeleton for disposable clients
ClosedPublic
Actions

Authored by tflink on Apr 1 2015, 2:10 PM.

Details

Reviewers

kparal
mkrizek
jskladan

Commits

rLTRN75bd9cc06ce7: trivial skeleton for starting disposable client development

Summary

This is a very simple start to the disposable client work. It's by no means set in stone - meant to be a starting point that can (and should) be altered as we continue the work.

Test Plan

No testing needed at this point, meant to be a starting point with little working code.

Diff Detail

Repository

rLTRN libtaskotron

Branch

disposable-develop

Lint

Lint OK

Unit

No Unit Test Coverage

tflink retitled this revision from to Initial Skeleton for disposable clients.Apr 1 2015, 2:10 PM

tflink updated this object.

tflink edited the test plan for this revision. (Show Details)

tflink added reviewers: kparal, mkrizek, jskladan.

tflink set the repository for this revision to rLTRN libtaskotron.

tflink added a project: libtaskotron.

Herald added a subscriber: tflink. · View Herald TranscriptApr 1 2015, 2:10 PM

Thanks, looks reasonable.

This revision is now accepted and ready to land.Apr 1 2015, 3:04 PM

LGTM

Looks good. I have probably missed something so - what is the thought behind the --patch argument?

In D321#5845, @jskladan wrote:

Looks good. I have probably missed something so - what is the thought behind the --patch argument?

When I was writing the PoC, I ran into a problem - the VMs are created from scratch using packages. If I wanted to test a small change, I had to build a new package and get in a repo used during client creation.

It's a huge PITA, so I added the --patch argument as a sort of hotfix on the clients, enabling me to test changes in the libtaskotron code running on the clients without too much effort. It's mostly a dev convenience option but I'm open to other solutions or taking it out if it's considered too hacky

This was pushed a while ago and the change wasn't detected. Closing

Revision Contents

			Path	Packages
M			libtaskotron/runner.py (45 lines)
A	M		libtaskotron/virtualmachine.py (19 lines)

Commit	Tree	Parents	Author	Summary	Date
75bd9cc06ce7	7145b7f95069	326875055694	Tim Flink	trivial skeleton for starting disposable client development	Apr 1 2015, 1:57 PM

Diff 876

View Options

libtaskotron/runner.py

Show First 20 Lines • Show All 163 Lines • ▼ Show 20 Line(s)		163	if not arg in self.envdata:
164	raise TaskotronYamlError("Required input arg '%s' "	164		raise TaskotronYamlError("Required input arg '%s' "
165	"was not defined" % arg)	165		"was not defined" % arg)
166		166
167	def _validate_env(self):	167		def _validate_env(self):
168	# TODO: implement this	168		# TODO: implement this
169	raise NotImplementedError("Environment validation is not"	169		raise NotImplementedError("Environment validation is not"
170	" yet implemented")	170		" yet implemented")
171		171
		172		class RemoteRunner(object):
		173		def __init__(self, task_data, arg_data):
		174		self.ssh = None
		175		self.task_data = task_data
		176		self.arg_data = arg_data
		177		self.taskdir = '/home/fedora/taskdir'
		178		self.taskotron_basedir = os.path.abspath(os.path.dirname(libtaskotron.__file__) + '/../')
		179		self.output_basedir = '{}/logfiles'.format(self.taskotron_basedir)
		180		output_dirname = datetime.datetime.utcnow().strftime('%Y%d%m-%H%M%S')
		181		self.output_dirname = os.path.join(self.output_basedir, output_dirname)
		182		self.ssh_keyfile = '{}/conf/id_taskotron'.format(self.taskotron_basedir)
		183
		184
		185		def prepare_task():
		186		# add ssh keys if needed for controlling other machines
		187		# install required packages
		188		# patch remote libtaskotron if needed (for dev, shouldn't be used in production)
		189		# create dirs required for execution
		190		# put files needed for execution (task, input files, etc)
		191		pass
		192
		193		def run():
		194		# effectively override the local run method to encapsulate the running
		195		# in a remote VM
		196		pass
		197
172		198
173	def get_argparser():	199		def get_argparser():
174	parser = argparse.ArgumentParser()	200		parser = argparse.ArgumentParser()
175	parser.add_argument("task", nargs=1, help="task to run")	201		parser.add_argument("task", nargs=1, help="task to run")
176	parser.add_argument("-a", "--arch",	202		parser.add_argument("-a", "--arch",
177	choices=["i386", "x86_64", "armhfp", "noarch"],	203		choices=["i386", "x86_64", "armhfp", "noarch"],
178	action='append',	204		action='append',
179	help="architecture specifying the item to be checked. If omitted, "	205		help="architecture specifying the item to be checked. If omitted, "
Show All 12 Lines		217	help="Unique job identifier for the execution"
192	"status tracking purposes. If unset, defaults to"	218		"status tracking purposes. If unset, defaults to"
193	"current datetime in UTC")	219		"current datetime in UTC")
194	parser.add_argument("--override", action="append", default=[],	220		parser.add_argument("--override", action="append", default=[],
195	metavar='VAR=VALUE',	221		metavar='VAR=VALUE',
196	help="override internal variable values used in runner "	222		help="override internal variable values used in runner "
197	"and the task formula. Value itself is evaluated "	223		"and the task formula. Value itself is evaluated "
198	"by eval(). This option can be used multiple times. "	224		"by eval(). This option can be used multiple times. "
199	"Example: --override \"workdir='/some/dir/'\"")	225		"Example: --override \"workdir='/some/dir/'\"")
		226		parser.add_argument("-p", "--patch", help="patch to apply to remote system"
		227		"before execution of task")
200		228
201	return parser	229		return parser
202		230
203		231
204	def process_args(args):	232		def process_args(args):
205	""" Processes raw input args and converts them into specific data types that	233		""" Processes raw input args and converts them into specific data types that
206	can be used by tasks. This includes e.g. creating new args based on	234		can be used by tasks. This includes e.g. creating new args based on
207	(item, item_type) pairs, or adjusting selected architecture.	235		(item, item_type) pairs, or adjusting selected architecture.
▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Line(s)		261	def main():
253	arg_data = process_args(vars(args))	281		arg_data = process_args(vars(args))
254	arg_data['taskfile'] = args.task[0]	282		arg_data['taskfile'] = args.task[0]
255		283
256	# parse task formula	284		# parse task formula
257	task_data = taskformula.parse_yaml_from_file(arg_data['taskfile'])	285		task_data = taskformula.parse_yaml_from_file(arg_data['taskfile'])
258	if not task_data:	286		if not task_data:
259	raise TaskotronYamlError('Input file should not be empty')	287		raise TaskotronYamlError('Input file should not be empty')
260		288
261	# run the task	289		# if environment/machine is defined in the task, we can tell that it is
262	task_runner = Runner(task_data, arg_data)	290		# supposed to be run in a virtual machine as a disposable client
263	task_runner.run()	291		if 'environment' in task_data.keys() and task_data['environment']['machine'] is not None and not args.local:
		292		log.info("Running Task in VM")
		293		task_runner = RemoteRunner(task_data, arg_data)
		294		task_runner.prepare_task()
		295		output = task_runner.run()
		296		log.info("remote stdout: %s" % output[0])
		297		log.info("remote stderr: %s" % output[1])
		298
		299		else:
		300		# run the task locally
		301		task_runner = Runner(task_data, arg_data)
		302		task_runner.run()
264		303
265	# finalization	304		# finalization
266	log.info('Execution finished at: %s',	305		log.info('Execution finished at: %s',
267	datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC'))	306		datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC'))