Differential D821

Allow using a dot (``.``) to access attributes of a variable
ClosedPublic
Actions

Authored by jskladan on Apr 21 2016, 1:52 PM.

Details

Reviewers
lbrabec
tflink
Group Reviewers
libtaskotron
Commits
rLTRN6822b1c5a960: Allow using a dot (``.``) to access attributes of a variable
Summary

The formula yaml file uses modified string.Template_ syntax for variables. As in
the standard, you can use `$variable or ${variable}` format, and you need to
make the dollar sign doubled if you want to include it literally (`$$not_a_variable`).

On top of that, you can use a dot (`.`) to access attributes of a variable.
For the sake of convenience, `${foo.bar}` does the following things on the Python layer:

  • check for an item `bar in foo (foo.__getitem__('bar')`)
  • if there is not, check for an attribute `bar in foo (getattr(foo, 'bar')`)
  • if there is not, raises `TaskotronYamlError`

.. code-block:: yaml

actions:
    - name: first step
      directive_one:
          arg1: value1
      export: firststep_output

    - name: second step
      directive_two:
          arg1: ${firststep_output.bar}
Test Plan

Added unittests

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
jskladan retitled this revision from to Allow using a dot (``.``) to access attributes of a variable.Apr 21 2016, 1:52 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added a reviewer: libtaskotron.
lbrabec accepted this revision.Apr 27 2016, 1:07 PM
lbrabec mentioned this in T788: Remove return type restriction of python directive.
lbrabec added a reviewer: lbrabec.
lbrabec added a subscriber: lbrabec.

LGTM, but see T788.

This revision is now accepted and ready to land.Apr 27 2016, 1:09 PM
tflink accepted this revision.Apr 28 2016, 4:55 AM
tflink added a reviewer: tflink.

minor question otherwise looks good

libtaskotron/taskformula.py
22

does this need to be here if it's commented out?

Closed by commit rLTRN6822b1c5a960: Allow using a dot (``.``) to access attributes of a variable (authored by Josef Skladanka <jskladan@redhat.com>). · Explain WhyMay 3 2016, 12:40 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2136

View Options

docs/source/taskyaml.rst

Show First 20 Lines • Show All 95 Lines • ▼ Show 20 Line(s)95 directivename:
96 arg1: value1 96​ arg1: value1
97 arg2: ${some_variable} 97​ arg2: ${some_variable}
98 export: somestep_output 98​ export: somestep_output
99 99​
100 100​
101Variable Storage and Syntax 101​Variable Storage and Syntax
102--------------------------- 102​---------------------------
103 103​
104The formula yaml file uses standard `string.Template`_ syntax for variables. You 104​The formula yaml file uses modified `string.Template`_ syntax for variables. As in
105can use ``$variable`` or ``${variable}`` format, and you need to make the dollar 105​the standard, you can use ``$variable`` or ``${variable}`` format, and you need to
106sign doubled if you want to include it literally (``$$not_a_variable``). 106​make the dollar sign doubled if you want to include it literally (``$$not_a_variable``).
107​
108​On top of that, you can use a dot (``.``) to access attributes of a variable.
109​For the sake of convenience, ``${foo.bar}`` does the following things on the Python layer:
110​
111​ * check for an item ``bar`` in ``foo`` (``foo.__getitem__('bar')``)
112​ * if there is not, check for an attribute ``bar`` in ``foo`` (``getattr(foo, 'bar')``)
113​ * if there is not, raises ``TaskotronYamlError``
114​
115​.. code-block:: yaml
116​
117​ actions:
118​ - name: first step
119​ directive_one:
120​ arg1: value1
121​ export: firststep_output
122​
123​ - name: second step
124​ directive_two:
125​ arg1: ${firststep_output.bar}
126​
107Variables can be created during task execution or provided by default by the 127​Variables can be created during task execution or provided by default by the
108task runner. 128​task runner.
109 129​
110.. _string.Template: https://docs.python.org/2.7/library/string.html?highlight=string#template-strings 130​.. _string.Template: https://docs.python.org/2.7/library/string.html?highlight=string#template-strings
111 131​
112Provided Variables 132​Provided Variables
113------------------ 133​------------------
114 134​
▲ Show 20 Lines • Show All 118 Lines • Show Last 20 Lines
View Options

libtaskotron/taskformula.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2014, Red Hat, Inc. 2​# Copyright 2009-2014, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6'''Methods for operating with a task formula''' 6​'''Methods for operating with a task formula'''
7 7​
8from __future__ import absolute_import 8​from __future__ import absolute_import
9import collections 9​import collections
10import string 10​import string
11 11​
12from libtaskotron import exceptions as exc 12​from libtaskotron import exceptions as exc
13from libtaskotron.logger import log 13​from libtaskotron.logger import log
14from libtaskotron.ext.fedora import rpm_utils 14​from libtaskotron.ext.fedora import rpm_utils
15 15​
16 16​
17​class DotTemplate(string.Template):
18​ """Redefines the :class:`string.Template`'s ``idpattern`` property to
19​ allow using ``.`` in the variable names. This is then later used for
20​ key/property access in variable replacement.
21​ """
22​ idpattern = '[_a-z](?:\.?[_a-z0-9]+)*'
tflinkUnsubmitted
Not Done

does this need to be here if it's commented out?

23​
24​
25​def _get_key_or_attribute(variables, var_name):
26​ """Allows for the ``.`` selector to work as selector for variables.
27​ When a variable is dict-like (collections.MutableMapping), the selector looks for a key.
28​ When a variable is not dict-like, it looks for the object's attribute.
29​
30​ :param dict variables: names (keys) and values of available variables
31​ :param str var_name: variable name, possibly containing ``.`` selector to be found
32​ :raise KeyError: when a dict-like object (anywhere in the chain) does not contain requested key
33​ :raise AttributeError: when an non-dic object (anywhere in the chain)
34​ does not have the requested attribute
35​ """
36​ if '.' not in var_name:
37​ return variables[var_name]
38​
39​ var_name, selector = var_name.split('.', 1)
40​ outcome = variables[var_name]
41​ for part in selector.split('.'):
42​ if isinstance(outcome, collections.MutableMapping):
43​ outcome = outcome[part]
44​ else:
45​ outcome = outcome.__getattribute__(part)
46​ return outcome
47​
17def _replace_vars(text, variables): 48​def _replace_vars(text, variables):
18 '''Go through ``text`` and replace all variables (in the form of what is 49​ '''Go through ``text`` and replace all variables (in the form of what is
19 supported by :class:`string.Template`) with their values, as provided in 50​ supported by :class:`VariableTemplate`) with their values, as provided in
20 ``variables``. This is used for variable expansion in the task formula. 51​ ``variables``. This is used for variable expansion in the task formula.
21 52​
22 :param str text: input text where to search for variables 53​ :param str text: input text where to search for variables
23 :param dict variables: names (keys) and values of variables to replace 54​ :param dict variables: names (keys) and values of variables to replace
24 :return: if ``text`` contains just a single variable and nothing else, the 55​ :return: if ``text`` contains just a single variable and nothing else, the
25 variable value is directly returned (i.e. with matching type, not 56​ variable value is directly returned (i.e. with matching type, not
26 cast to ``str``). If ``text`` contains something else as well 57​ cast to ``str``). If ``text`` contains something else as well
27 (other variables or text), a string is returned. 58​ (other variables or text), a string is returned.
28 :raise TaskotronYamlError: if ``text`` contains a variable that is not 59​ :raise TaskotronYamlError: if ``text`` contains a variable that is not
29 present in ``variables``, or if the variable 60​ present in ``variables``, or if the variable
30 syntax is incorrect 61​ syntax is incorrect
31 ''' 62​ '''
32 63​
33 try: 64​ try:
34 # try to find the first match 65​ # try to find the first match
35 match = string.Template.pattern.search(text) 66​ match = DotTemplate.pattern.search(text)
36 67​
37 if not match: 68​ if not match:
38 return text 69​ return text
39 70​
40 # There are 4 groups in the pattern: 1 - escaped, 2 - named, 3 - braced, 71​ # There are 4 groups in the pattern: 1 - escaped, 2 - named, 3 - braced,
41 # 4 - invalid. Group 0 returns the whole match. 72​ # 4 - invalid. Group 0 returns the whole match.
42 if match.group(0) == text and (match.group(2) or match.group(3)): 73​ if match.group(0) == text and (match.group(2) or match.group(3)):
43 # We found a single variable and nothing more. We shouldn't return 74​ # We found a single variable and nothing more. We shouldn't return
44 # a string, but the exact value, so that we don't lose value type. 75​ # a string, but the exact value, so that we don't lose value type.
45 # This makes it possible to pass lists, dicts, etc as variables. 76​ # This makes it possible to pass lists, dicts, etc as variables.
46 var_name = match.group(2) or match.group(3) 77​ var_name = match.group(2) or match.group(3)
47 return variables[var_name] 78​ return _get_key_or_attribute(variables, var_name)
48 79​
49 # Now it's clear there's also something else in `text` than just a 80​ # Now it's clear there's also something else in `text` than just a
50 # single variable. We will replace all variables and return a string 81​ # single variable. We will replace all variables and return a string
51 # again. 82​ # again.
52 output = string.Template(text).substitute(variables) 83​
84​ # To make things easy, we'll create expanded_variables dict, that contains values for
85​ # all the replacable variables in the text.
86​ # i.e. if variables = {"n": 1}, and text is "foo bar {$n.imag}" then
87​ # "n.imag" key is added to variables, with the respective value.
88​
89​ expanded_variables = {}
90​ for match in DotTemplate.pattern.findall(text):
91​ var_name = match[1] or match[2]
92​ if var_name:
93​ expanded_variables[var_name] = _get_key_or_attribute(variables, var_name)
94​
95​ output = DotTemplate(text).substitute(expanded_variables)
53 return output 96​ return output
54 97​
55 except KeyError as e: 98​ except (KeyError, AttributeError) as e:
56 raise exc.TaskotronYamlError("The task formula includes a variable, " 99​ raise exc.TaskotronYamlError("The task formula includes a variable, "
57 "but no value has been provided for it: %s" % e) 100​ "but no value has been provided for it: %s" % e)
58 101​
59 except ValueError as e: 102​ except ValueError as e:
60 raise exc.TaskotronYamlError("The task formula includes an incorrect " 103​ raise exc.TaskotronYamlError("The task formula includes an incorrect "
61 "variable definition. Dollar signs must be doubled if they " 104​ "variable definition. Dollar signs must be doubled if they "
62 "shouldn't be considered as a variable denotation.\n" 105​ "shouldn't be considered as a variable denotation.\n"
63 "Error: %s\n" 106​ "Error: %s\n"
▲ Show 20 Lines • Show All 131 Lines • Show Last 20 Lines
View Options

testing/test_taskformula.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2014, Red Hat, Inc. 2​# Copyright 2009-2014, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6'''Unit tests for libtaskotron/taskformula.py''' 6​'''Unit tests for libtaskotron/taskformula.py'''
7 7​
8import copy 8​import copy
9 9​
10import pytest 10​import pytest
11from libtaskotron import taskformula 11​from libtaskotron import taskformula
12import libtaskotron.exceptions as exc 12​import libtaskotron.exceptions as exc
13 13​
14 14​
15​class TestGetKeyOrAttribute(object):
16​ def test_dictionary(self):
17​ variables = {"mydict": {"foo": {"bar": "moo"}}}
18​ res = taskformula._get_key_or_attribute(variables, "mydict")
19​ assert res == variables['mydict']
20​ res = taskformula._get_key_or_attribute(variables, "mydict.foo")
21​ assert res == variables['mydict']['foo']
22​ res = taskformula._get_key_or_attribute(variables, "mydict.foo.bar")
23​ assert res == variables['mydict']['foo']['bar']
24​
25​ def test_object(self):
26​ variables = {"number": 2+1j}
27​ res = taskformula._get_key_or_attribute(variables, "number")
28​ assert res == variables['number']
29​ res = taskformula._get_key_or_attribute(variables, "number.real")
30​ assert res == variables['number'].real
31​ res = taskformula._get_key_or_attribute(variables, "number.real.imag")
32​ assert res == variables['number'].real.imag
33​
34​ def test_mixed(self):
35​ variables = {"mydict": {"number": 2+1j}}
36​ res = taskformula._get_key_or_attribute(variables, "mydict")
37​ assert res == variables['mydict']
38​ res = taskformula._get_key_or_attribute(variables, "mydict.number")
39​ assert res == variables['mydict']['number']
40​ res = taskformula._get_key_or_attribute(variables, "mydict.number.real")
41​ assert res == variables['mydict']['number'].real
42​
43​ def test_exceptions(self):
44​ with pytest.raises(KeyError):
45​ taskformula._get_key_or_attribute({}, "foo")
46​ with pytest.raises(KeyError):
47​ taskformula._get_key_or_attribute({"foo": {}}, "foo.bar")
48​ with pytest.raises(AttributeError):
49​ taskformula._get_key_or_attribute({"foo": None}, "foo.bar")
50​
15class TestReplaceVarsInAction(object): 51​class TestReplaceVarsInAction(object):
16 52​
17 def setup_method(self, method): 53​ def setup_method(self, method):
18 self.action = { 54​ self.action = {
19 'name': 'run foo${foo}', 55​ 'name': 'run foo${foo}',
20 'export': '${export}', 56​ 'export': '${export}',
21 'python': { 57​ 'python': {
22 'file': 'somefile.py', 58​ 'file': 'somefile.py',
23 'args': ['arg1', '${number}', 'arg2', '${list}', '${dict}'], 59​ 'args': ['arg1', '${number}', 'arg2', '${list}', '${dict}'],
24 'cmdline': '-s ${foo} -i ${number}', 60​ 'cmdline': '-s ${foo} -i ${number}',
25 'nest': { 61​ 'nest': {
26 'deepnest': '${dict}', 62​ 'deepnest': '${dict}',
27 'null': '${null}', 63​ 'null': '${null}',
28 } 64​ },
65​ 'from_dict': '${dict.pi}',
66​ 'from_object': '${number.imag}',
67​ 'from_mixed': '${dict.pi.imag}',
29 } 68​ }
30 } 69​ }
31 self.vars = { 70​ self.vars = {
32 'foo': 'bar', 71​ 'foo': 'bar',
33 'export': 'a:b\nc:d\n-e', 72​ 'export': 'a:b\nc:d\n-e',
34 'number': 1, 73​ 'number': 1,
35 'list': [1, 'item'], 74​ 'list': [1, 'item'],
36 'dict': {'key': 'val', 'pi': 3.14}, 75​ 'dict': {'key': 'val', 'pi': 3.14},
37 'null': None, 76​ 'null': None,
38 'extra': 'juicy', 77​ 'extra': 'juicy',
39 } 78​ }
40 79​
41 def test_complex(self): 80​ def test_complex(self):
42 rend_action = copy.deepcopy(self.action) 81​ rend_action = copy.deepcopy(self.action)
43 rend_action['name'] = 'run foo%s' % self.vars['foo'] 82​ rend_action['name'] = 'run foo%s' % self.vars['foo']
44 rend_action['export'] = self.vars['export'] 83​ rend_action['export'] = self.vars['export']
45 rend_action['python']['args'] = ['arg1', self.vars['number'], 'arg2', 84​ rend_action['python']['args'] = ['arg1', self.vars['number'], 'arg2',
46 self.vars['list'], self.vars['dict']] 85​ self.vars['list'], self.vars['dict']]
47 rend_action['python']['cmdline'] = '-s %s -i %s' % (self.vars['foo'], 86​ rend_action['python']['cmdline'] = '-s %s -i %s' % (self.vars['foo'],
48 self.vars['number']) 87​ self.vars['number'])
49 rend_action['python']['nest']['deepnest'] = self.vars['dict'] 88​ rend_action['python']['nest']['deepnest'] = self.vars['dict']
50 rend_action['python']['nest']['null'] = self.vars['null'] 89​ rend_action['python']['nest']['null'] = self.vars['null']
90​ rend_action['python']['from_dict'] = self.vars['dict']['pi']
91​ rend_action['python']['from_object'] = self.vars['number'].imag
92​ rend_action['python']['from_mixed'] = self.vars['dict']['pi'].imag
51 93​
52 taskformula.replace_vars_in_action(self.action, self.vars) 94​ taskformula.replace_vars_in_action(self.action, self.vars)
53 assert self.action == rend_action 95​ assert self.action == rend_action
54 96​
55 def test_raise(self): 97​ def test_raise(self):
56 with pytest.raises(exc.TaskotronYamlError): 98​ with pytest.raises(exc.TaskotronYamlError):
57 taskformula.replace_vars_in_action(self.action, {}) 99​ taskformula.replace_vars_in_action(self.action, {})
58 100​
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Line(s)147 def test_empty(self):
106 res = taskformula._replace_vars('', {'extra': 'var'}) 148​ res = taskformula._replace_vars('', {'extra': 'var'})
107 assert res == '' 149​ assert res == ''
108 150​
109 def test_same_var(self): 151​ def test_same_var(self):
110 '''Only variables must get replaced, not same-sounding text''' 152​ '''Only variables must get replaced, not same-sounding text'''
111 res = taskformula._replace_vars('foo ${foo}', {'foo': 'bar'}) 153​ res = taskformula._replace_vars('foo ${foo}', {'foo': 'bar'})
112 assert res == 'foo bar' 154​ assert res == 'foo bar'
113 155​
156​ def test_dot_selector(self):
157​ res = taskformula._replace_vars('foo ${foo.bar}', {'foo': {'bar': 'moo'}})
158​ assert res == 'foo moo'
159​ res = taskformula._replace_vars('foo ${foo.bar.imag}', {'foo': {'bar': 1}})
160​ assert res == 'foo 0'
161​ res = taskformula._replace_vars('foo $foo.bar', {'foo': {'bar': 'moo'}})
162​ assert res == 'foo moo'
163​ res = taskformula._replace_vars('foo $foo.bar.imag', {'foo': {'bar': 1}})
164​ assert res == 'foo 0'
165​
114 def test_combined_syntax(self): 166​ def test_combined_syntax(self):
115 '''Both $foo and ${foo} syntax must work, together with escaping''' 167​ '''Both $foo and ${foo} syntax must work, together with escaping'''
116 res = taskformula._replace_vars('foo $foo ${foo} $$foo', {'foo': 'bar'}) 168​ res = taskformula._replace_vars('foo $foo ${foo} $$foo', {'foo': 'bar'})
117 assert res == 'foo bar bar $foo' 169​ assert res == 'foo bar bar $foo'
118 170​
119 def test_raise_missing_var(self): 171​ def test_raise_missing_var(self):
120 with pytest.raises(exc.TaskotronYamlError): 172​ with pytest.raises(exc.TaskotronYamlError):
121 taskformula._replace_vars('${foo}', {'bar': 'foo'}) 173​ taskformula._replace_vars('${foo}', {'bar': 'foo'})
122 174​
123 def test_raise_missing_vars(self): 175​ def test_raise_missing_vars(self):
124 with pytest.raises(exc.TaskotronYamlError): 176​ with pytest.raises(exc.TaskotronYamlError):
125 taskformula._replace_vars('${foo} ${bar}', {'bar': 'foo'}) 177​ taskformula._replace_vars('${foo} ${bar}', {'bar': 'foo'})
126 178​
127 def test_raise_bad_escape(self): 179​ def test_raise_bad_escape(self):
128 '''Dollars must be doubled''' 180​ '''Dollars must be doubled'''
129 with pytest.raises(exc.TaskotronYamlError): 181​ with pytest.raises(exc.TaskotronYamlError):
130 taskformula._replace_vars('${foo} $', {'foo': 'bar'}) 182​ taskformula._replace_vars('${foo} $', {'foo': 'bar'})
131 183​
184​ def test_raise_bad_selector(self):
185​ '''Selectors must point to existing properties'''
186​ with pytest.raises(exc.TaskotronYamlError):
187​ taskformula._replace_vars('${foo.bar}', {'foo': None})
188​ with pytest.raises(exc.TaskotronYamlError):
189​ taskformula._replace_vars('${foo.bar}', {'foo': {}})
132 190​
133class TestDeviseEnvironment(object): 191​class TestDeviseEnvironment(object):
134 def test_raise_invalid_env(self): 192​ def test_raise_invalid_env(self):
135 with pytest.raises(exc.TaskotronValueError): 193​ with pytest.raises(exc.TaskotronValueError):
136 taskformula.devise_environment({'environment': ''}, None) 194​ taskformula.devise_environment({'environment': ''}, None)
137 195​
138 def test_empty_env_no_infer_data(self): 196​ def test_empty_env_no_infer_data(self):
139 res = taskformula.devise_environment({}, {}) 197​ res = taskformula.devise_environment({}, {})
▲ Show 20 Lines • Show All 41 Lines • Show Last 20 Lines