Differential D489

adjust --ssh implementation
ClosedPublic
Actions

Authored by kparal on Aug 6 2015, 4:12 PM.

Details

Reviewers
lbrabec
tflink
Maniphest Tasks
T408: refine formula syntax from PoC
Commits
rLTRN8621baa4b112: adjust --ssh implementation and implement --libvirt
Summary
  • Slight improvements in taskotron.yaml documentation.
  • Removal of SSH section from taskotron.yaml, because using it from the cmdline should be enough for now, and we can add more features if somebody really needs it.
  • Moved ssh_privkey to Secrets section and made sure this key is named the same way everywhere in the code.
  • Moved checking valid config values into _check_sanity() function.
  • Throw an error when invalid profile name is specified, to make it in line with invalid runtask mode specified.
  • Rework a bit local/remote execution decision code. Move closer to our final state, where --ssh execution is a bit different from --libvirt execution (throw 'not implemented' for --libvirt for now).
  • Add a few tests.
Test Plan

test suite works, remote execution works

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to adjust --ssh implementation.Aug 6 2015, 4:12 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added reviewers: tflink, lbrabec.
lbrabec accepted this revision.Aug 7 2015, 8:12 AM

Besides the comment, looks good to me.

libtaskotron/runner.py
390–400

Just an idea, changing metavar to /path/to/private.key would be probably more readable and consistent with comments in config

This revision is now accepted and ready to land.Aug 7 2015, 8:12 AM
kparal marked an inline comment as done.Aug 7 2015, 12:57 PM

I've implemented also --libvirt so that we can easily plug it in into D482. I suppose we might go with --remote in the future, but it was easier this way in the short timeframe.

libtaskotron/runner.py
390–400

Thanks, that's better.

kparal updated this revision to Diff 1287.Aug 7 2015, 12:58 PM
kparal marked an inline comment as done.
  • implement --libvirt
Closed by commit rLTRN8621baa4b112: adjust --ssh implementation and implement --libvirt (authored by kparal). · Explain WhyAug 7 2015, 1:10 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 1288

View Options

conf/taskotron.yaml.example

1## Main configuration file for Taskotron 1​## Main configuration file for Taskotron
2## The file is in YAML syntax, read more about it at: 2​## The file is in YAML syntax, read more about it at:
3## http://en.wikipedia.org/wiki/Yaml 3​## http://en.wikipedia.org/wiki/Yaml
4## FIXME: add a link to Taskotron docs 4​## FIXME: add a link to Taskotron docs
5 5​
6## ==== GENERAL section ==== 6​## ==== GENERAL section ====
7​## Config profile selection.
7## There are two major config profiles in Taskotron - development and 8​## There are two major config profiles in Taskotron - development and
8## production. 9​## production.
9## /Development/ profile is used for developing libtaskotron, developing checks 10​## Development profile is used for developing libtaskotron, developing checks
10## based on libtaskotron and local execution of these checks. 11​## based on libtaskotron and local execution of these checks.
11## /Production/ profile is used for deploying Taskotron as a service on a 12​## Production profile is used for deploying Taskotron as a service on a
12## server, periodically executing the checks and reporting results to relevant 13​## server, periodically executing the checks and reporting results to relevant
13## result consumers. 14​## result consumers.
14## *The default profile is /development/*. If you want to switch to the 15​## The default profile is 'development'. If you want to switch to the
15## /production/ profile, uncomment the following line. 16​## 'production' profile, uncomment the following line.
16## You can also switch a profile temporarily by using TASKOTRON_PROFILE=name 17​## You can also switch a profile temporarily by using TASKOTRON_PROFILE=name
17## environment variable, it has a higher priority. All other options set in this 18​## environment variable, it has a higher priority. All other options set in this
18## file still apply of course. 19​## file still apply of course.
19## [choices: production, development; default: development] 20​## [choices: production, development; default: development]
20#profile: production 21​#profile: production
21 22​
22## ==== RUNTASK MODE section ==== 23​## Task execution mode. The tasks can be executed locally (on the current
23## Taskotron supports local task execution as well as execution in remote 24​## machine) or remotely in a disposable virtual machine. Local execution is
24## (virtual) machine. 25​## dangerous with destructive or untrusted tasks, remote execution requires
25## /Local/ runtask mode denotes local execution 26​## some additional setup (see Taskotron documentation).
26## /Libvirt/ runtask mode denotes that task execution is done in virtual 27​## Remote execution is done through libvirt, it creates a throwaway virtual
27## machine, that is created just for purpose of running specified task and 28​## machine from a specified disk image, executes the task and deletes the
28## upon finishing it is disposed. 29​## machine.
29## *The default mode for development profile is /local/ and /libvirt/ for 30​## Local execution is the default mode for development profile and remote
30## production profile*. 31​## execution for production profile.
31## [choices: local, libvirt; default: local for development, libvirt for production] 32​## [choices: local, libvirt; default: local for development, libvirt for production]
32#runtask_mode: libvirt 33​#runtask_mode: libvirt
33 34​
34## ==== SSH CONNECTION section ==== ​
35## Taskotron connects to remote machine using ssh connection. ​
36## Connection information can be specified with command line argument ​
37## --ssh [user@machine[:port]] or here in format /user@machine/ or /user@machine:port/. ​
38## If -ssh is passed to taskotron without any additional argument, the values ​
39## are loaded from config. *The default setting is /root@localhost:22/*. ​
40## You can also specify path to private key, *the default path for ​
41## private key is conf/id_taskotron * or you can use command line argument ​
42## --ssh-privkey in the same way as --ssh. ​
43#ssh: root@localhost:22 ​
44#privkey: conf/id_taskotron ​
45 35​
46## ==== REPORTING section ==== 36​## ==== REPORTING section ====
47## This section controls which result reports you want to send after the test 37​## This section controls which result reports you want to send after the test
48## execution is complete 38​## execution is complete.
49 39​
50## Overall setting whether to send any reports at all. If this is False, no 40​## Overall setting whether to send any reports at all. If this is False, no
51## reports will be sent anywhere, regardless of your configuration for 41​## reports will be sent anywhere, regardless of your configuration for
52## individual recipient systems. 42​## individual recipient systems.
53## [default: True for production, False for development] 43​## [default: True for production, False for development]
54#reporting_enabled: True 44​#reporting_enabled: True
55 45​
56## If True, test results (for selected tests) will be sent as comments to 46​## If True, test results (for selected tests) will be sent as comments to
▲ Show 20 Lines • Show All 80 Lines • ▼ Show 20 Line(s)
137## The location of artifacts produced by checks 127​## The location of artifacts produced by checks
138#artifactsdir: /var/lib/taskotron/artifacts 128​#artifactsdir: /var/lib/taskotron/artifacts
139 129​
140## The location of cached files downloaded by Taskotron 130​## The location of cached files downloaded by Taskotron
141#cachedir: /var/cache/taskotron 131​#cachedir: /var/cache/taskotron
142 132​
143 133​
144## ==== LOGGING section ==== 134​## ==== LOGGING section ====
145## This section contains configuration of logging 135​## This section contains configuration of logging.
146 136​
147## Configuration of logging level. Here can be configured which messages 137​## Configuration of logging level. Here can be configured which messages
148## will be logged. You can specify different level for logging to standard 138​## will be logged. You can specify different level for logging to standard
149## output (option log_level_stream) and logging to file (log_level_file). 139​## output (option log_level_stream) and logging to file (log_level_file).
150## Possible values can be found here: 140​## Possible values can be found here:
151## https://docs.python.org/2.7/library/logging.html#logging-levels 141​## https://docs.python.org/2.7/library/logging.html#logging-levels
152#log_level_stream: INFO 142​#log_level_stream: INFO
153#log_level_file: DEBUG 143​#log_level_file: DEBUG
154 144​
155## If True, logging to file will be enabled. 145​## If True, logging to file will be enabled.
156## [default: True for production, False for development] 146​## [default: True for production, False for development]
157#log_file_enabled: True 147​#log_file_enabled: True
158 148​
159 149​
160## ==== SECRETS section ==== 150​## ==== SECRETS section ====
161## All login credentials and other secrets are here. If you add some secret 151​## All login credentials and other secrets are here. If you add some secret
162## here, make sure you make this file readable just for the right user accounts. 152​## here, make sure you make this file readable just for the right user accounts.
163 153​
164## FAS (Fedora Accounts System) credentials 154​## FAS (Fedora Accounts System) credentials
165## These credentials are used when reporting results into Bodhi. 155​## These credentials are used when reporting results into Bodhi.
166#fas_username: taskotron 156​#fas_username: taskotron
167#fas_password: password 157​#fas_password: password
158​
159​## SSH private key location. Used for remote task execution, when connecting to
160​## VMs and remote machines. If your systems are not configured for automatic
161​## connection (private keys located in SSH's standard search path), specify a
162​## path to the private key here. An empty value (the default) means to rely on
163​## the standard search path only.
164​#ssh_privkey: /path/to/private.key
View Options

libtaskotron/config.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2014, Red Hat, Inc. 2​# Copyright 2009-2014, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6'''Global configuration for Taskotron and relevant helper methods.''' 6​'''Global configuration for Taskotron and relevant helper methods.'''
7 7​
8from __future__ import absolute_import 8​from __future__ import absolute_import
9import os 9​import os
10import yaml 10​import yaml
11import collections 11​import collections
12 12​
13import libtaskotron 13​import libtaskotron
14from libtaskotron import exceptions as exc 14​from libtaskotron import exceptions as exc
15from libtaskotron.logger import log 15​from libtaskotron.logger import log
16from libtaskotron.config_defaults import (Config, ProductionConfig, 16​from libtaskotron.config_defaults import (Config, ProductionConfig, TestingConfig, ProfileName,
17 TestingConfig, ProfileName, RuntaskModeName) 17​ RuntaskModeName)
18from libtaskotron import file_utils 18​from libtaskotron import file_utils
19 19​
20CONF_DIRS = [ # local checkout dir first, then system wide dir 20​CONF_DIRS = [ # local checkout dir first, then system wide dir
21 os.path.abspath(os.path.dirname(libtaskotron.__file__) + '/../conf'), 21​ os.path.abspath(os.path.dirname(libtaskotron.__file__) + '/../conf'),
22 '/etc/taskotron'] 22​ '/etc/taskotron']
23'''A list of directories where config files are stored. The config files are 23​'''A list of directories where config files are stored. The config files are
24loaded from these locations in the specified order, and only the first config 24​loaded from these locations in the specified order, and only the first config
25file found is used (further locations are ignored). The first location is 25​file found is used (further locations are ignored). The first location is
Show All 29 Lines
55def _get_instance(): 55​def _get_instance():
56 '''Do everything necessary to fully initialize Config instance, update it 56​ '''Do everything necessary to fully initialize Config instance, update it
57 with external configuration, make all final touches. 57​ with external configuration, make all final touches.
58 58​
59 :return: :class:`.Config` instance ready to be used 59​ :return: :class:`.Config` instance ready to be used
60 :raise TaskotronConfigError: if file config parsing and handling failed 60​ :raise TaskotronConfigError: if file config parsing and handling failed
61 ''' 61​ '''
62 config = _load() 62​ config = _load()
63​ _check_sanity(config)
63 log.debug('Using config profile: %s', config.profile) 64​ log.debug('Using config profile: %s', config.profile)
64 65​
65 # make sure required directories exist 66​ # make sure required directories exist
66 _create_dirs(config) 67​ _create_dirs(config)
67 68​
68 return config 69​ return config
69 70​
70 71​
Show All 25 Lines72def _load():
96 # reload the config defaults if the profile was specified in the config file 97​ # reload the config defaults if the profile was specified in the config file
97 # (and not present in env variable) 98​ # (and not present in env variable)
98 if file_profile and not env_profile: 99​ if file_profile and not env_profile:
99 config = _load_defaults(file_profile) 100​ config = _load_defaults(file_profile)
100 101​
101 # merge the changes from the config file 102​ # merge the changes from the config file
102 _merge_config(config, file_config) 103​ _merge_config(config, file_config)
103 104​
104 # check if runtaskmode is set properly ​
105 if config.runtask_mode not in [RuntaskModeName.LOCAL, RuntaskModeName.LIBVIRT]: ​
106 if config.profile == ProfileName.PRODUCTION: ​
107 runtask_mode = RuntaskModeName.LIBVIRT ​
108 else: ​
109 runtask_mode = RuntaskModeName.LOCAL ​
110 log.warn('Unknown runtask_mode "%s", selecting "%s"', config.runtask_mode, ​
111 runtask_mode) ​
112 config.runtask_mode = runtask_mode ​
113 ​
114 # set config filename used, this is set after merging 105​ # set config filename used, this is set after merging
115 # so it doesn't get overridden 106​ # so it doesn't get overridden
116 config.config_filename = filename 107​ config.config_filename = filename
117 108​
118 return config 109​ return config
119 110​
120 111​
112​def _check_sanity(config):
113​ '''Check important config keys for sane/allowed values. Raise exception
114​ if config file is not correct.
115​
116​ :param config: :class:`.Config` instance
117​ :raise TaskotronConfigError: when important config key has invalid value
118​ '''
119​
120​ # runtask_mode value must be from its enum
121​ if config.runtask_mode not in [RuntaskModeName.LOCAL, RuntaskModeName.LIBVIRT]:
122​ raise exc.TaskotronConfigError('Invalid runtask mode name: %s' % config.runtask_mode)
123​
124​
121def _load_defaults(profile): 125​def _load_defaults(profile):
122 '''Load and return Config (or its subclass) based on chosen profile. 126​ '''Load and return Config (or its subclass) based on chosen profile.
123 127​
124 :param str profile: profile name as defined in :class:`.ProfileName` 128​ :param str profile: profile name as defined in :class:`.ProfileName`
125 :return: :class:`.Config` instance or its subclass 129​ :return: :class:`.Config` instance or its subclass
130​ :raise TaskotronConfigError: if unknown profile name is requested
126 ''' 131​ '''
127 132​
128 if profile == ProfileName.PRODUCTION: 133​ if profile == ProfileName.PRODUCTION:
129 return ProductionConfig() 134​ return ProductionConfig()
130 elif profile == ProfileName.TESTING: 135​ elif profile == ProfileName.TESTING:
131 return TestingConfig() 136​ return TestingConfig()
132 else: 137​ elif profile == ProfileName.DEVELOPMENT or not profile:
133 if profile and profile != ProfileName.DEVELOPMENT: ​
134 log.warn('Unknown profile "%s", selecting "%s"', profile, ​
135 ProfileName.DEVELOPMENT) ​
136 # the default is the development profile 138​ # the default is the development profile
137 return Config() 139​ return Config()
140​ else:
141​ raise exc.TaskotronConfigError('Invalid profile name: %s' % profile)
138 142​
139 143​
140def _search_dirs(conf_dirs, conf_file): 144​def _search_dirs(conf_dirs, conf_file):
141 '''Find the configuration file in a set of directories. The first match 145​ '''Find the configuration file in a set of directories. The first match
142 is returned. 146​ is returned.
143 147​
144 :param conf_dirs: a list of directories to search through 148​ :param conf_dirs: a list of directories to search through
145 :type conf_dirs: list of str 149​ :type conf_dirs: list of str
▲ Show 20 Lines • Show All 111 Lines • Show Last 20 Lines
View Options

libtaskotron/config_defaults.py

Show All 16 Lines14class ProfileName(object):
17 17​
18 DEVELOPMENT = 'development' #: 18​ DEVELOPMENT = 'development' #:
19 PRODUCTION = 'production' #: 19​ PRODUCTION = 'production' #:
20 TESTING = 'testing' #: 20​ TESTING = 'testing' #:
21 21​
22 22​
23class RuntaskModeName(object): 23​class RuntaskModeName(object):
24 '''Enum of available runtask mode names. These can be specified in 24​ '''Enum of available runtask mode names. These can be specified in
25 the config file''' 25​ the config file.'''
26 26​
27 LIBVIRT = 'libvirt' #: 27​ LIBVIRT = 'libvirt' #:
28 LOCAL = 'local' #: 28​ LOCAL = 'local' #:
29 29​
30 30​
31class Config(object): 31​class Config(object):
32 '''Global configuration for Taskotron (development profile). 32​ '''Global configuration for Taskotron (development profile).
33 33​
Show All 15 Lines48 '''Filename of the loaded config file. To be set after an external
49 config file is loaded from the disk and its values merged with the 49​ config file is loaded from the disk and its values merged with the
50 default values. (If no config file is found, this is going to 50​ default values. (If no config file is found, this is going to
51 stay empty). *Do not* set this value manually in a config file 51​ stay empty). *Do not* set this value manually in a config file
52 itself - it is for internal use only.''' 52​ itself - it is for internal use only.'''
53 config_filename = '' #: 53​ config_filename = '' #:
54 54​
55 profile = ProfileName.DEVELOPMENT #: 55​ profile = ProfileName.DEVELOPMENT #:
56 56​
57 ssh = 'root@localhost:22' #: 57​ runtask_mode = RuntaskModeName.LOCAL #:
58 privkey = 'conf/id_taskotron' #: ​
59 ​
60 runtask_mode = 'local' #: ​
61 58​
62 reporting_enabled = False #: 59​ reporting_enabled = False #:
63 report_to_bodhi = True #: 60​ report_to_bodhi = True #:
64 report_to_resultsdb = True #: 61​ report_to_resultsdb = True #:
65 62​
66 koji_url = 'http://koji.fedoraproject.org/kojihub' #: 63​ koji_url = 'http://koji.fedoraproject.org/kojihub' #:
67 pkg_url = 'http://kojipkgs.fedoraproject.org/packages' #: 64​ pkg_url = 'http://kojipkgs.fedoraproject.org/packages' #:
68 bodhi_server = 'https://admin.fedoraproject.org/updates' #: 65​ bodhi_server = 'https://admin.fedoraproject.org/updates' #:
Show All 16 Lines
85 82​
86 log_level_stream = 'INFO' #: 83​ log_level_stream = 'INFO' #:
87 log_level_file = 'DEBUG' #: 84​ log_level_file = 'DEBUG' #:
88 85​
89 log_file_enabled = False #: 86​ log_file_enabled = False #:
90 87​
91 fas_username = 'taskotron' #: 88​ fas_username = 'taskotron' #:
92 fas_password = 'password' #: 89​ fas_password = 'password' #:
90​ ssh_privkey = None #:
93 91​
94 def __str__(self): 92​ def __str__(self):
95 ''' Make this object more readable when printing ''' 93​ ''' Make this object more readable when printing '''
96 return '<%s: %s>' % (self.__class__.__name__, 94​ return '<%s: %s>' % (self.__class__.__name__,
97 pprint.pformat(vars(self))) 95​ pprint.pformat(vars(self)))
98 96​
99 97​
100class ProductionConfig(Config): 98​class ProductionConfig(Config):
101 '''Configuration for production profile. Inherits values from 99​ '''Configuration for production profile. Inherits values from
102 :class:`Config` and overrides some. Read Config documentation.''' 100​ :class:`Config` and overrides some. Read Config documentation.'''
103 101​
104 profile = ProfileName.PRODUCTION #: 102​ profile = ProfileName.PRODUCTION #:
105 103​
106 runtask_mode = 'libvirt' #: 104​ runtask_mode = RuntaskModeName.LIBVIRT #:
107 105​
108 reporting_enabled = True #: 106​ reporting_enabled = True #:
109 107​
110 download_cache_enabled = False #: 108​ download_cache_enabled = False #:
111 109​
112 log_level_stream = 'INFO' #: 110​ log_level_stream = 'INFO' #:
113 log_level_file = 'DEBUG' #: 111​ log_level_file = 'DEBUG' #:
114 112​
Show All 16 Lines
View Options

libtaskotron/runner.py

Show All 34 Lines
35 35​
36class Runner(): 36​class Runner():
37 '''Runner class that decides whether the task should be run locally or remotely, 37​ '''Runner class that decides whether the task should be run locally or remotely,
38 intializes variables (such as logger and ssh), instantiates Local or Remote Runner 38​ intializes variables (such as logger and ssh), instantiates Local or Remote Runner
39 and runs the task. 39​ and runs the task.
40 ''' 40​ '''
41 41​
42 def __init__(self, task_data, arg_data, uuid_dir): 42​ def __init__(self, task_data, arg_data, uuid_dir):
43 ''' 43​ '''Create a new instance.
44​
44 :param dict task_data: data loaded from task formula 45​ :param dict task_data: data loaded from task formula
45 :param dict arg_data: data from command line arguments 46​ :param dict arg_data: data from command line arguments
46 :param str uuid_dir: path to artifacts directory 47​ :param str uuid_dir: path to artifacts directory
47 ''' 48​ '''
48 49​
49 self.task_data = task_data 50​ self.task_data = task_data
50 self.arg_data = arg_data 51​ self.arg_data = arg_data
51 self.uuid_dir = uuid_dir 52​ self.uuid_dir = uuid_dir
52 53​
53 # decide whether to run on disposable client 54​ #: run locally or remotely
54 # TODO: at the moment, runtask_mode='libvirt' actually means execution over ssh, this 55​ self.run_remotely = False
55 # will change when T408 is done 56​ #: when running remotely, run directly over ssh, instead of using libvirt
57​ self.direct_ssh = False
58​ self.user = self.machine = self.port = None
59​ self.ssh_privkey = arg_data['ssh_privkey'] or config.get_config().ssh_privkey
60​
61​ # decide whether to run locally or remotely
56 runtask_mode = config.get_config().runtask_mode 62​ runtask_mode = config.get_config().runtask_mode
57 if runtask_mode == config.RuntaskModeName.LIBVIRT: 63​ if runtask_mode == config.RuntaskModeName.LOCAL:
58 self.run_remotely = True ​
59 elif runtask_mode == config.RuntaskModeName.LOCAL: ​
60 self.run_remotely = False 64​ self.run_remotely = False
65​ elif runtask_mode == config.RuntaskModeName.LIBVIRT:
66​ self.run_remotely = True
61 else: 67​ else:
62 assert False, 'This should never occur' 68​ assert False, 'This should never occur'
63 69​
64 # if --local was passed, override local/disposable decision 70​ if arg_data['local']:
65 if self.arg_data['local']: 71​ log.debug("Forcing local execution (option --local)")
66 self.run_remotely = False 72​ self.run_remotely = False
67 log.debug("Overriding local/disposable decision, running locally (flag --local)") ​
68 73​
69 # if --ssh was passed, override local/disposable decision 74​ elif arg_data['libvirt']:
70 elif self.arg_data['ssh']: 75​ log.debug("Forcing remote execution (option --libvirt)")
76​ self.run_remotely = True
77​ self.direct_ssh = False
78​
79​ elif arg_data['ssh']:
80​ log.debug('Forcing remote execution (option --ssh)')
71 self.run_remotely = True 81​ self.run_remotely = True
82​ self.direct_ssh = True
72 83​
73 self.user, machine = self.arg_data['ssh'].split('@') 84​ self.user, machine = arg_data['ssh'].split('@')
74 if ':' in machine: 85​ if ':' in machine:
75 self.machine, port = machine.split(':') 86​ self.machine, port = machine.split(':')
76 self.port = int(port) 87​ self.port = int(port)
77 else: 88​ else:
78 self.machine = machine 89​ self.machine = machine
79 self.port = 22 90​ self.port = 22
80 91​
81 self.privkey = self.arg_data['ssh_privkey'] 92​ log.debug('Execution mode: %s', 'remote' if self.run_remotely else 'local')
82 ​
83 log.debug("Overriding local/disposable decision, running over ssh (arg --ssh)") ​
84 93​
85 def run(self): 94​ def run(self):
86 '''Run the apropriate code based on previous local/remote decision.''' 95​ '''Run the apropriate code based on previous local/remote decision.'''
87 if self.run_remotely: 96​ if self.run_remotely:
88 self._run_remotely() 97​ self._run_remotely()
89 else: 98​ else:
90 self._run_locally() 99​ self._run_locally()
91 100​
92 def _run_locally(self): 101​ def _run_locally(self):
93 '''Init environment, instantiate LocalRunner and run it.''' 102​ '''Init environment, instantiate LocalRunner and run it.'''
94 103​
95 filelog_path = os.path.join(self.uuid_dir, 'taskotron.log') 104​ filelog_path = os.path.join(self.uuid_dir, 'taskotron.log')
96 logger.add_filehandler(filelog_path=filelog_path, remove_mem_handler=True) 105​ logger.add_filehandler(filelog_path=filelog_path, remove_mem_handler=True)
97 # run the task locally 106​
98 task_runner = LocalRunner(self.task_data, self.arg_data) 107​ task_runner = LocalRunner(self.task_data, self.arg_data)
99 task_runner.prepare_task() 108​ task_runner.prepare_task()
100 task_runner.run() 109​ task_runner.run()
101 110​
102 def _run_remotely(self): 111​ def _run_remotely(self):
103 '''Init environment, instantiate RemoteRunner and run it.''' 112​ '''Init environment, instantiate RemoteRunner and run it.'''
104 113​
105 filelog_path = os.path.join(self.uuid_dir, 'taskotron-initiator.log') 114​ filelog_path = os.path.join(self.uuid_dir, 'taskotron-initiator.log')
106 logger.add_filehandler(filelog_path=filelog_path, remove_mem_handler=True) 115​ logger.add_filehandler(filelog_path=filelog_path, remove_mem_handler=True)
107 log.info("Running Task over SSH on machine %s" % self.machine) ​
108 # FIXME change this when the vm code is done ​
109 stdio_filename = os.path.join(self.uuid_dir, 'taskotron-stdio.log') 116​ stdio_filename = os.path.join(self.uuid_dir, 'taskotron-stdio.log')
110 with remote_exec.RemoteExec(self.machine, self.port, 117​
111 self.user, self.privkey, stdio_filename) as ssh: 118​ if self.direct_ssh:
119​ log.info("Running task over SSH on machine '%s'...", self.machine)
120​ else:
121​ log.info("Running task on a disposable client using libvirt...")
122​ # FIXME: create disposable client here
123​ raise exc.TaskotronError('libvirt execution not yet implemented')
124​
125​ with remote_exec.RemoteExec(self.machine, self.port, self.user,
126​ self.ssh_privkey, stdio_filename) as ssh:
112 task_runner = RemoteRunner(ssh, self.task_data, self.arg_data) 127​ task_runner = RemoteRunner(ssh, self.task_data, self.arg_data)
113 try: 128​ try:
114 task_runner.prepare_task() 129​ task_runner.prepare_task()
115 task_runner.run() 130​ task_runner.run()
116 except exc.TaskotronRemoteError, e: 131​ except exc.TaskotronRemoteError, e:
117 log.exception(e) 132​ log.exception(e)
118 task_runner.get_output() 133​ task_runner.get_output()
119 134​
▲ Show 20 Lines • Show All 176 Lines • ▼ Show 20 Line(s)305 def prepare_task(self):
296 self.ssh.install_pkgs('libtaskotron') 311​ self.ssh.install_pkgs('libtaskotron')
297 312​
298 # configure libtaskotron 313​ # configure libtaskotron
299 if config.get_config().config_filename: 314​ if config.get_config().config_filename:
300 self.ssh.put_file(config.get_config().config_filename, '/etc/taskotron/taskotron.yaml') 315​ self.ssh.put_file(config.get_config().config_filename, '/etc/taskotron/taskotron.yaml')
301 316​
302 # patch remote libtaskotron if needed (for dev, shouldn't be used in production) 317​ # patch remote libtaskotron if needed (for dev, shouldn't be used in production)
303 if self.arg_data['patch'] is not None: 318​ if self.arg_data['patch'] is not None:
304 self.ssh.put_file(os.path.abspath(self.arg_data['patch']), '%s/%s' % (self.taskdir, self.arg_data['patch'])) 319​ self.ssh.put_file(os.path.abspath(self.arg_data['patch']), '%s/%s' % (
320​ self.taskdir, self.arg_data['patch']))
305 self.ssh.install_pkgs('patch') 321​ self.ssh.install_pkgs('patch')
306 self.ssh.cmd('patch -d /usr/lib/python2.7/site-packages/ -p1 -i %s/%s' % (self.taskdir, self.arg_data['patch'])) 322​ self.ssh.cmd('patch -d /usr/lib/python2.7/site-packages/ -p1 -i %s/%s' % (
323​ self.taskdir, self.arg_data['patch']))
307 324​
308 # create needed dirs 325​ # create needed dirs
309 self.ssh.mkdir(self.taskdir) 326​ self.ssh.mkdir(self.taskdir)
310 327​
311 # put files needed for execution (task, input files, etc) 328​ # put files needed for execution (task, input files, etc)
312 self.ssh.put_dir(os.path.dirname(self.arg_data['taskfile']), self.taskdir) 329​ self.ssh.put_dir(os.path.dirname(self.arg_data['taskfile']), self.taskdir)
313 330​
314 # need to have default_flow_style false to get valid yaml w/ nested dicts 331​ # need to have default_flow_style false to get valid yaml w/ nested dicts
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Line(s)373 parser.add_argument("-j", "--jobid", default="-1",
357 help="optional job identifier used to render log urls") 374​ help="optional job identifier used to render log urls")
358 parser.add_argument("-d", "--debug", action="store_true", 375​ parser.add_argument("-d", "--debug", action="store_true",
359 help="Enable debug output " 376​ help="Enable debug output "
360 "(set logging level to 'DEBUG')") 377​ "(set logging level to 'DEBUG')")
361 parser.add_argument("--uuid", default=datetime.datetime.utcnow().strftime("%Y%m%d_%H%M%S_%f"), 378​ parser.add_argument("--uuid", default=datetime.datetime.utcnow().strftime("%Y%m%d_%H%M%S_%f"),
362 help="Unique job identifier for the execution" 379​ help="Unique job identifier for the execution"
363 "status tracking purposes. If unset, defaults to" 380​ "status tracking purposes. If unset, defaults to"
364 "current datetime in UTC") 381​ "current datetime in UTC")
365 parser.add_argument("--override", action="append", default=[], 382​ parser.add_argument("--override", action="append", default=[], metavar='VAR=VALUE',
366 metavar='VAR=VALUE', ​
367 help="override internal variable values used in runner " 383​ help="override internal variable values used in runner "
368 "and the task formula. Value itself is evaluated " 384​ "and the task formula. Value itself is evaluated "
369 "by eval(). This option can be used multiple times. " 385​ "by eval(). This option can be used multiple times. "
370 "Example: --override \"workdir='/some/dir/'\"") 386​ "Example: --override \"workdir='/some/dir/'\"")
371 parser.add_argument("-p", "--patch", help="patch to apply to remote system" 387​ parser.add_argument("-p", "--patch", help="patch to apply to remote system"
372 "before execution of task") 388​ "before execution of task")
373 parser.add_argument("--local", action="store_true", 389​ parser.add_argument("--local", action="store_true",
374 help="make the task run on this particular machine, regardless of " 390​ help="make the task run locally on this very machine (the default "
375 "the default execution mode defined in the configuration file. " 391​ "behavior for development profile). This also approves any required "
376 "This also approves any required system-wide changes to be performed " 392​ "system-wide changes to be performed (automatic installation of "
377 "(automatic installation of package dependencies, destructive tasks " 393​ "package dependencies, destructive tasks allowed, etc).")
378 "allowed, etc).") 394​ parser.add_argument('--libvirt', action='store_true',
379 parser.add_argument("--ssh", nargs='?', 395​ help="make the task run remotely in a disposable client spawned using "
380 metavar='user@machine[:port]', 396​ "libvirt (the default behavior for production profile).")
381 default=None, const=config.get_config().ssh, 397​ parser.add_argument("--ssh", metavar='user@machine[:port]',
382 help="make the task run over ssh on remote machine, regardless of " 398​ help="make the task run on a remote machine over ssh")
383 "the default execution mode defined in the configuration file.") 399​ parser.add_argument("--ssh-privkey", metavar="</path/to/private.key>",
384 parser.add_argument("--ssh-privkey", nargs='?', 400​ help="path to private key for remote connections over ssh")
lbrabecUnsubmitted
Done

Just an idea, changing metavar to /path/to/private.key would be probably more readable and consistent with comments in config

kparalAuthorUnsubmitted
Not Done

Thanks, that's better.

385 metavar="<path to private key>", ​
386 default=None, const=config.get_config().privkey, ​
387 help="provide path to private key for ssh connection to remote machine.") ​
388 401​
389 return parser 402​ return parser
390 403​
391 404​
392def check_args(parser, args): 405​def check_args(parser, args):
393 """ Check if passed args doesn't have conflicts and have proper format. In case of error, this 406​ """ Check if passed args doesn't have conflicts and have proper format. In case of error, this
394 function prints error message and exits the program. 407​ function prints error message and exits the program.
395 408​
396 :param ArgumentParser parser: parser object used to show error message and exit the program 409​ :param ArgumentParser parser: parser object used to show error message and exit the program
397 :param dict args: arguments previously returned by argument parser converted to dict 410​ :param dict args: arguments previously returned by argument parser converted to dict
398 """ 411​ """
399 412​
400 if args['ssh'] and args['local']: 413​ if len([arg for arg in [args['local'], args['libvirt'], args['ssh']] if arg]) > 1:
401 parser.error("Args --ssh and --local cannot be passed together") 414​ parser.error("Args --local, --libvirt and --ssh are mutually exclusive")
402 415​
403 if args['ssh']: 416​ if args['ssh']:
404 if '@' not in args['ssh']: 417​ if '@' not in args['ssh']:
405 parser.error("User and machine not in format user@machine or user@machine:port") 418​ parser.error("SSH connection info not in format 'user@machine' or 'user@machine:port'")
406 419​
407 420​
408def process_args(args): 421​def process_args(args):
409 """ Processes raw input args and converts them into specific data types that 422​ """ Processes raw input args and converts them into specific data types that
410 can be used by tasks. This includes e.g. creating new args based on 423​ can be used by tasks. This includes e.g. creating new args based on
411 (item, item_type) pairs, or adjusting selected architecture. 424​ (item, item_type) pairs, or adjusting selected architecture.
412 425​
413 :param dict args: dictionary of raw arguments 426​ :param dict args: dictionary of raw arguments
▲ Show 20 Lines • Show All 74 Lines • Show Last 20 Lines
View Options

testing/test_config.py

Show First 20 Lines • Show All 54 Lines • ▼ Show 20 Line(s)54 def test_load_defaults(self):
55 '''Test _load_defaults() function''' 55​ '''Test _load_defaults() function'''
56 for attr, value in vars(config.ProfileName).items(): 56​ for attr, value in vars(config.ProfileName).items():
57 print attr, value 57​ print attr, value
58 if attr.startswith('_'): 58​ if attr.startswith('_'):
59 continue 59​ continue
60 conf = config._load_defaults(value) 60​ conf = config._load_defaults(value)
61 assert conf.profile == value 61​ assert conf.profile == value
62 62​
63​ def test_load_defaults_invalid_name(self):
64​ '''Should crash for invalid profile name'''
65​ with pytest.raises(exc.TaskotronConfigError):
66​ config._load_defaults('invalid profile name')
67​
63 def test_load_file_empty(self): 68​ def test_load_file_empty(self):
64 '''Test _load_file() function with empty file''' 69​ '''Test _load_file() function with empty file'''
65 contents = StringIO('') 70​ contents = StringIO('')
66 conf_object = config._load_file(contents) 71​ conf_object = config._load_file(contents)
67 assert conf_object == {} 72​ assert conf_object == {}
68 73​
69 def test_load_file_commented(self): 74​ def test_load_file_commented(self):
70 '''Test _load_file() function with fully commented out file''' 75​ '''Test _load_file() function with fully commented out file'''
▲ Show 20 Lines • Show All 74 Lines • ▼ Show 20 Line(s)149 def test_devel_profile_no_config(self, monkeypatch):
145 '''When there are no config files and no envvar, the profile should be 150​ '''When there are no config files and no envvar, the profile should be
146 set to 'development' ''' 151​ set to 'development' '''
147 self.unset_profile(monkeypatch) 152​ self.unset_profile(monkeypatch)
148 self.disable_create_dirs(monkeypatch) 153​ self.disable_create_dirs(monkeypatch)
149 # make sure we don't find any config files 154​ # make sure we don't find any config files
150 monkeypatch.setattr(config, 'CONF_DIRS', []) 155​ monkeypatch.setattr(config, 'CONF_DIRS', [])
151 conf = config.get_config() 156​ conf = config.get_config()
152 assert conf.profile == config.ProfileName.DEVELOPMENT 157​ assert conf.profile == config.ProfileName.DEVELOPMENT
158​
159​ def test_check_sanity_runtask_mode(self):
160​ '''Should crash for unknown runtask mode names'''
161​ conf = config.get_config()
162​
163​ for attr, value in vars(config.RuntaskModeName).items():
164​ if attr.startswith('_'):
165​ continue
166​ conf.runtask_mode = value
167​ config._check_sanity(conf)
168​
169​ conf.runtask_mode = 'invalid runtask mode name'
170​ with pytest.raises(exc.TaskotronConfigError):
171​ config._check_sanity(conf)
View Options

testing/test_runner.py

Show First 20 Lines • Show All 579 Lines • ▼ Show 20 Line(s)575 def setup(self, monkeypatch):
580 monkeypatch.setattr(runner.Runner, '_run_remotely', self.stub_remote) 580​ monkeypatch.setattr(runner.Runner, '_run_remotely', self.stub_remote)
581 581​
582 self.stub_get_config = Dingus() 582​ self.stub_get_config = Dingus()
583 self.stub_config_local = Dingus() 583​ self.stub_config_local = Dingus()
584 self.stub_config_local.runtask_mode = config.RuntaskModeName.LOCAL 584​ self.stub_config_local.runtask_mode = config.RuntaskModeName.LOCAL
585 self.stub_config_remote = Dingus() 585​ self.stub_config_remote = Dingus()
586 self.stub_config_remote.runtask_mode = config.RuntaskModeName.LIBVIRT 586​ self.stub_config_remote.runtask_mode = config.RuntaskModeName.LIBVIRT
587 587​
588 def test_bare_no_local(self, monkeypatch): 588​ self.empty_args = {'local': None, 'libvirt': None, 'ssh': None, 'ssh_privkey': None}
589 ref_inputdata = {'local': False, 'ssh': None, 'ssh_privkey': None} 589​ self.fake_taskdata = {'environment': {'rpm': ['foo', 'bar']}}
590 ref_taskdata = {'environment': {'rpm': ['foo', 'bar']}} 590​
591​ def test_local_no_cli(self, monkeypatch):
592​ '''run locally with local runmode in config and no override from cli'''
593​ ref_inputdata = self.empty_args.copy()
591 self.stub_get_config.return_value = self.stub_config_local 594​ self.stub_get_config.return_value = self.stub_config_local
592 monkeypatch.setattr(config, 'get_config', self.stub_get_config) 595​ monkeypatch.setattr(config, 'get_config', self.stub_get_config)
593 596​
594 test_runner = runner.Runner(ref_taskdata, ref_inputdata, self.uuid_dir) 597​ test_runner = runner.Runner(self.fake_taskdata, ref_inputdata, self.uuid_dir)
598​ assert test_runner.run_remotely is False
595 599​
596 test_runner.run() 600​ test_runner.run()
597 ​
598 assert len(self.stub_local.calls()) == 1 and len(self.stub_remote.calls()) == 0 601​ assert len(self.stub_local.calls()) == 1 and len(self.stub_remote.calls()) == 0
599 602​
600 def test_remote_no_local(self, monkeypatch): 603​ def test_remote_no_cli(self, monkeypatch):
601 ref_inputdata = {'local': False, 'ssh': None, 'ssh_privkey': None} 604​ '''run remotely with remote runmode in config and no override from cli'''
602 ref_taskdata = {'environment': {'rpm': ['foo', 'bar']}} 605​ ref_inputdata = self.empty_args.copy()
603 self.stub_get_config.return_value = self.stub_config_remote 606​ self.stub_get_config.return_value = self.stub_config_remote
604 monkeypatch.setattr(config, 'get_config', self.stub_get_config) 607​ monkeypatch.setattr(config, 'get_config', self.stub_get_config)
605 608​
606 test_runner = runner.Runner(ref_taskdata, ref_inputdata, self.uuid_dir) 609​ test_runner = runner.Runner(self.fake_taskdata, ref_inputdata, self.uuid_dir)
610​ assert test_runner.run_remotely is True
611​ assert test_runner.direct_ssh is False
607 612​
608 test_runner.run() 613​ test_runner.run()
609 ​
610 assert len(self.stub_local.calls()) == 0 and len(self.stub_remote.calls()) == 1 614​ assert len(self.stub_local.calls()) == 0 and len(self.stub_remote.calls()) == 1
611 615​
612 def test_remote_override_local(self, monkeypatch): 616​ def test_remote_override_local(self, monkeypatch):
613 ref_inputdata = {'local': True, 'ssh': None, 'ssh_privkey': None} 617​ '''run locally with remote runmode in config and local override from cli'''
614 ref_taskdata = {'environment': {'rpm': ['foo', 'bar']}} 618​ ref_inputdata = self.empty_args.copy()
619​ ref_inputdata.update({'local': True})
615 self.stub_get_config.return_value = self.stub_config_remote 620​ self.stub_get_config.return_value = self.stub_config_remote
616 monkeypatch.setattr(config, 'get_config', self.stub_get_config) 621​ monkeypatch.setattr(config, 'get_config', self.stub_get_config)
617 622​
618 test_runner = runner.Runner(ref_taskdata, ref_inputdata, self.uuid_dir) 623​ test_runner = runner.Runner(self.fake_taskdata, ref_inputdata, self.uuid_dir)
624​ assert test_runner.run_remotely is False
619 625​
620 test_runner.run() 626​ test_runner.run()
621 ​
622 assert len(self.stub_local.calls()) == 1 and len(self.stub_remote.calls()) == 0 627​ assert len(self.stub_local.calls()) == 1 and len(self.stub_remote.calls()) == 0
623 628​
624 def test_bare_override_ssh(self, monkeypatch): 629​ def test_local_override_ssh(self, monkeypatch):
625 ref_inputdata = {'local': False, 'ssh': 'david@HAL9000', 'ssh_privkey': None} 630​ '''run remotely with local runmode in config and ssh override from cli'''
626 ref_taskdata = {'environment': {'rpm': ['foo', 'bar']}} 631​ ref_inputdata = self.empty_args.copy()
632​ ref_inputdata.update({'ssh': 'david@HAL9000'})
627 self.stub_get_config.return_value = self.stub_config_local 633​ self.stub_get_config.return_value = self.stub_config_local
628 monkeypatch.setattr(config, 'get_config', self.stub_get_config) 634​ monkeypatch.setattr(config, 'get_config', self.stub_get_config)
629 635​
630 test_runner = runner.Runner(ref_taskdata, ref_inputdata, self.uuid_dir) 636​ test_runner = runner.Runner(self.fake_taskdata, ref_inputdata, self.uuid_dir)
637​ assert test_runner.run_remotely is True
638​ assert test_runner.direct_ssh is True
631 639​
632 test_runner.run() 640​ test_runner.run()
641​ assert len(self.stub_local.calls()) == 0 and len(self.stub_remote.calls()) == 1
633 642​
643​ def test_local_override_libvirt(self, monkeypatch):
644​ '''run remotely with local runmode in config and libvirt override from cli'''
645​ ref_inputdata = self.empty_args.copy()
646​ ref_inputdata.update({'libvirt': True})
647​ self.stub_get_config.return_value = self.stub_config_local
648​ monkeypatch.setattr(config, 'get_config', self.stub_get_config)
649​
650​ test_runner = runner.Runner(self.fake_taskdata, ref_inputdata, self.uuid_dir)
651​ assert test_runner.run_remotely is True
652​ assert test_runner.direct_ssh is False
653​
654​ test_runner.run()
634 assert len(self.stub_local.calls()) == 0 and len(self.stub_remote.calls()) == 1 655​ assert len(self.stub_local.calls()) == 0 and len(self.stub_remote.calls()) == 1
635 656​
636 657​
637class TestRunnerLocalRunner(): 658​class TestRunnerLocalRunner():
638 def test_local_runner_used(self, monkeypatch): 659​ def test_local_runner_used(self, monkeypatch):
639 ref_inputdata = {'local': True, 'ssh': None, 'ssh_privkey': None} 660​ ref_inputdata = {'local': True, 'ssh': None, 'ssh_privkey': None}
640 ref_taskdata = {'environment': {'rpm': ['foo', 'bar']}} 661​ ref_taskdata = {'environment': {'rpm': ['foo', 'bar']}}
641 662​
Show All 22 Lines674 def test_local_runner_properly_used(self, monkeypatch):
664 685​
665 test_runner.run() 686​ test_runner.run()
666 687​
667 methods_called = [call[0] for call in stub_local_runner_object.calls()] 688​ methods_called = [call[0] for call in stub_local_runner_object.calls()]
668 689​
669 assert len(methods_called) == 2 690​ assert len(methods_called) == 2
670 assert 'prepare_task' in methods_called and 'run' in methods_called 691​ assert 'prepare_task' in methods_called and 'run' in methods_called
671 692​
672 def test_ssh_bad_usermachine(self, monkeypatch): ​
673 ref_inputdata = {'local': False, 'ssh': 'user#machine', 'ssh_privkey': None} ​
674 693​
675 stub_error = Dingus() 694​@pytest.mark.usefixtures('setup')
676 stub_parser = Dingus() 695​class TestCheckArgs():
677 stub_parser.error = stub_error ​
678 runner.check_args(stub_parser, ref_inputdata) ​
679 696​
680 assert len(stub_error.calls()) == 1 697​ @pytest.fixture
698​ def setup(self):
699​ self.empty_args = {'local': None, 'libvirt': None, 'ssh': None}
700​
701​ def test_runmode_exclusive(self, monkeypatch):
702​ '''allow only single runmode selection'''
703​ ref_inputdata_bad = [{'local': True, 'ssh': 'user@machine'},
704​ {'local': True, 'libvirt': True},
705​ {'libvirt': True, 'ssh': 'user@machine'},
706​ {'local': True, 'ssh': 'user@machine', 'libvirt': True}]
707​ ref_inputdata_good = [{'local': True},
708​ {'ssh': 'user@machine'},
709​ {'libvirt': True},
710​ {}]
711​
712​ for ref_inputdata in ref_inputdata_bad:
713​ stub_error = Dingus()
714​ stub_parser = Dingus()
715​ stub_parser.error = stub_error
716​ args = self.empty_args.copy()
717​ args.update(ref_inputdata)
718​ runner.check_args(stub_parser, args)
719​ assert len(stub_error.calls()) == 1
720​
721​ for ref_inputdata in ref_inputdata_good:
722​ stub_error = Dingus()
723​ stub_parser = Dingus()
724​ stub_parser.error = stub_error
725​ args = self.empty_args.copy()
726​ args.update(ref_inputdata)
727​ runner.check_args(stub_parser, args)
728​ assert len(stub_error.calls()) == 0
681 729​
682 def test_local_ssh_together(self, monkeypatch): 730​ def test_ssh_bad_usermachine(self, monkeypatch):
683 ref_inputdata = {'local': True, 'ssh': 'user@machine', 'ssh_privkey': None} 731​ '''fail on invalid user@machine[:port] specifier'''
732​ ref_inputdata = {'ssh': 'user#machine'}
684 733​
685 stub_error = Dingus() 734​ stub_error = Dingus()
686 stub_parser = Dingus() 735​ stub_parser = Dingus()
687 stub_parser.error = stub_error 736​ stub_parser.error = stub_error
688 runner.check_args(stub_parser, ref_inputdata) 737​ args = self.empty_args
738​ args.update(ref_inputdata)
739​ runner.check_args(stub_parser, args)
689 740​
690 assert len(stub_error.calls()) == 1 741​ assert len(stub_error.calls()) == 1