Differential D838

Add distro/release inferring from koji_tag
ClosedPublic
Actions

Authored by jskladan on May 3 2016, 12:19 PM.

Details

Reviewers
kparal
Group Reviewers
libtaskotron
Commits
rLTRNca0a1ad94665: Add distro/release inferring from koji_tag
Summary

The same idea of inferring distro/release as in D778 just applied to koji_tag.
This will make sure that all our current tests actually run on the "best" available image.
At the moment, upgradepath/depcheck run on "default fallback", which proved to be counter-intuitive

Test Plan

Added unittests

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
jskladan retitled this revision from to Add distro/release inferring from koji_tag.May 3 2016, 12:19 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added a reviewer: libtaskotron.
tflink added a comment.May 30 2016, 3:37 PM

Overall, looks pretty good to me. Just some questions before approving

libtaskotron/taskformula.py
197

Have these env vars been documented somewhere?

198

If I'm correctly remembering the use case for the distro envvar, wouldn't this log message be better as something like "Environment/distro overriden with ENVVAR from %r:%r to %r"?

mkrizek added a subscriber: mkrizek.Jul 7 2016, 10:33 AM

Other than the questions from @tflink, LGTM.

kparal accepted this revision.Jul 8 2016, 10:39 AM
kparal added a reviewer: kparal.
kparal added a subscriber: kparal.

Apart from the missing docs, looks OK.

libtaskotron/taskformula.py
197

It seems we only have this:
https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/docs/taskyaml.html#dependencies

So, yes, please document this, @jskladan, thanks.

This revision is now accepted and ready to land.Jul 8 2016, 10:39 AM
jskladan updated this revision to Diff 2388.Jul 19 2016, 2:24 PM
  • Added docs
kparal added a comment.Jul 19 2016, 2:38 PM

Docs look reasonable, one typo.

docs/source/taskyaml.rst
92

affects

Closed by commit rLTRNca0a1ad94665: Add distro/release inferring from koji_tag (authored by Josef Skladanka <jskladan@redhat.com>). · Explain WhyJul 19 2016, 2:46 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2389

View Options

docs/source/taskyaml.rst

Show First 20 Lines • Show All 74 Lines • ▼ Show 20 Line(s)74 rpm:
75 - python-solv 75​ - python-solv
76 - python-librepo 76​ - python-librepo
77 77​
78.. note:: 78​.. note::
79 79​
80 Future versions will also support specifying dependencies in other formats 80​ Future versions will also support specifying dependencies in other formats
81 than RPM. 81​ than RPM.
82 82​
83​Environment
84​-----------
85​
86​.. note::
87​
88​ This is only applicable when running in the disposable-minion mode, which
89​ is the default on our deployment.
90​
91​By default, runtime environment (distro, release, arch and flavor) is devised
92​from the item under test for Koji builds and Bodhi updates. This affects the
kparalUnsubmitted
Not Done

affects

93​base-image used to spawn the new virtual machine in the disposable-minion mode.
94​
95​When the environment can not be devised, either because of unsupported
96​``item_type`` or unrecognized value of the ``item``, default environment (defined
97​in configuration by the ``default_disposable_{distro, release, arch, flavor}``
98​options) is used.
99​
100​For example, ``runtask.py -i xchatex-2.8.8-21.fc20 -t koji_build -a x86_64 rpmlint.yml``
101​would use x86_64 (arch) Fedora (distro) 20 (release).
102​
103​On top of that, environment requirements can be specified in the Task Formula,
104​although we do not think that these should be broadly used, there might be
105​a valid reason to, for example, aways run a task on x86_64 disregarding the item
106​under test.
107​
108​All of the distro, release, arch and flavor can be defined in the environment
109​section of the task formula, and take precedence over the auto-devised values.
110​
111​To specify environment in the formula use the ``environment`` section like
112​this:
113​
114​.. code-block:: yaml
115​
116​ environment:
117​ distro: fedora
118​ release: 24
119​ arch: x86_64
120​
121​.. note::
122​
123​ The task execution fails, when there is no base image fitting the
124​ distro/release/arch/flavor requirements.
83 125​
84Task Execution 126​Task Execution
85============== 127​==============
86 128​
87Every task is defined in the ``actions`` block of the formula yaml file. The task 129​Every task is defined in the ``actions`` block of the formula yaml file. The task
88consists of one or more actions which consist of a name, directive and optional 130​consists of one or more actions which consist of a name, directive and optional
89export of data. 131​export of data.
90 132​
▲ Show 20 Lines • Show All 162 Lines • Show Last 20 Lines
View Options

libtaskotron/taskformula.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2014, Red Hat, Inc. 2​# Copyright 2009-2014, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6'''Methods for operating with a task formula''' 6​'''Methods for operating with a task formula'''
7 7​
8from __future__ import absolute_import 8​from __future__ import absolute_import
9import collections 9​import collections
10import string 10​import string
11​import re
11 12​
12from libtaskotron import exceptions as exc 13​from libtaskotron import exceptions as exc
13from libtaskotron.logger import log 14​from libtaskotron.logger import log
14from libtaskotron.ext.fedora import rpm_utils 15​from libtaskotron.ext.fedora import rpm_utils
15 16​
16 17​
17class DotTemplate(string.Template): 18​class DotTemplate(string.Template):
18 """Redefines the :class:`string.Template`'s ``idpattern`` property to 19​ """Redefines the :class:`string.Template`'s ``idpattern`` property to
▲ Show 20 Lines • Show All 165 Lines • ▼ Show 20 Line(s)184 if item_type == 'koji_build':
184 # FIXME: find a way to make this not Fedora-specific 185​ # FIXME: find a way to make this not Fedora-specific
185 # For `xchat-2.8.8-21.fc20` disttag is `fc20` for example 186​ # For `xchat-2.8.8-21.fc20` disttag is `fc20` for example
186 try: 187​ try:
187 distro = rpm_utils.get_dist_tag(item)[:2] 188​ distro = rpm_utils.get_dist_tag(item)[:2]
188 env['distro'] = {'fc': 'fedora'}.get(distro, None) 189​ env['distro'] = {'fc': 'fedora'}.get(distro, None)
189 except exc.TaskotronValueError: 190​ except exc.TaskotronValueError:
190 env['distro'] = None 191​ env['distro'] = None
191 192​
192 if env['distro']: 193​ elif item_type == 'koji_tag':
193 log.debug("Environment/distro inferred from %s:%r to %r" % 194​ if re.match(r'^f[0-9]{2}-.*', item):
194 (item_type, item, env['distro'])) 195​ env['distro'] = 'fedora'
195 else: 196​
196 log.debug("Environment/distro can not be inferred from %s:%r. " 197​ if env['distro']:
tflinkUnsubmitted
Not Done

Have these env vars been documented somewhere?

kparalUnsubmitted
Not Done

It seems we only have this:
https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/docs/taskyaml.html#dependencies

So, yes, please document this, @jskladan, thanks.

197 "Using default." % (item_type, item)) 198​ log.debug("Environment/distro overriden with ENVVAR from %r:%r to %r" %
tflinkUnsubmitted
Not Done

If I'm correctly remembering the use case for the distro envvar, wouldn't this log message be better as something like "Environment/distro overriden with ENVVAR from %r:%r to %r"?

199​ (item_type, item, env['distro']))
198 else: 200​ else:
199 log.debug("Environment/distro can not be inferred for %r. Using default." % 201​ log.debug("Environment/distro can not be inferred from %r:%r. Using default." %
200 item_type) 202​ (item_type, item))
201 203​
202 if not env['release']: 204​ if not env['release']:
203 if item_type == 'koji_build': 205​ if item_type == 'koji_build':
204 # FIXME: find a way to make this not Fedora-specific 206​ # FIXME: find a way to make this not Fedora-specific
205 # Last two characters in rpm's disttag are the Fedora release. 207​ # Last two characters in rpm's disttag are the Fedora release.
206 # For `xchat-2.8.8-21.fc20` disttag is `fc20` for example 208​ # For `xchat-2.8.8-21.fc20` disttag is `fc20` for example
207 try: 209​ try:
208 env['release'] = rpm_utils.get_dist_tag(item)[-2:] 210​ env['release'] = rpm_utils.get_dist_tag(item)[-2:]
209 except exc.TaskotronValueError: 211​ except exc.TaskotronValueError:
210 env['release'] = None 212​ env['release'] = None
211 213​
212 if env['release']: 214​ elif item_type == 'koji_tag':
213 log.debug("Environment/release inferred from %s:%r to %r" % 215​ if re.match(r'^f[0-9]{2}-.*', item):
214 (item_type, item, env['release'])) 216​ env['release'] = item[1:3]
215 else: ​
216 log.debug("Environment/release can not be inferred from %s:%r. " ​
217 "Using default." % (item_type, item)) ​
218 217​
218​ if env['release']:
219​ log.debug("Environment/release inferred from %r:%r to %r" %
220​ (item_type, item, env['release']))
219 else: 221​ else:
220 log.debug("Environment/release can not be inferred for %r. Using default." % 222​ log.debug("Environment/release can not be inferred from %r:%r. Using default." %
221 item_type) 223​ (item_type, item))
222 224​
223 if not env['flavor']: 225​ if not env['flavor']:
224 log.debug("Environment/flavor not specified. Using default") 226​ log.debug("Environment/flavor not specified. Using default")
225 227​
226 if not env['arch']: 228​ if not env['arch']:
227 arches = [i for i in arg_data.get('arch', []) if i != 'noarch'] 229​ arches = [i for i in arg_data.get('arch', []) if i != 'noarch']
228 230​
229 if arches and len(arches) == 1: 231​ if arches and len(arches) == 1:
230 env['arch'] = arches[0] 232​ env['arch'] = arches[0]
231 log.debug("Environment/arch inferred from %r to %r" % 233​ log.debug("Environment/arch inferred from %r to %r" %
232 (arg_data.get('arch', []), env['arch'])) 234​ (arg_data.get('arch', []), env['arch']))
233 else: 235​ else:
234 log.warn("Environment/arch can not be inferred from %r. Using default." % 236​ log.warn("Environment/arch can not be inferred from %r. Using default." %
235 arg_data.get('arch', [])) 237​ arg_data.get('arch', []))
236 238​
237 return env 239​ return env
View Options

testing/test_taskformula.py

Show First 20 Lines • Show All 209 Lines • ▼ Show 20 Line(s)208 def test_infer_from_koji_build(self):
210 ret = taskformula.devise_environment({'environment': {}}, arg_data) 210​ ret = taskformula.devise_environment({'environment': {}}, arg_data)
211 assert ret == {'distro': 'fedora', 'release': '20', 'flavor': None, 'arch': None} 211​ assert ret == {'distro': 'fedora', 'release': '20', 'flavor': None, 'arch': None}
212 212​
213 def test_infer_from_koji_build_unknown_distro(self): 213​ def test_infer_from_koji_build_unknown_distro(self):
214 arg_data = {"type": "koji_build", "item": "xchat-2.8.8-21.XY20"} 214​ arg_data = {"type": "koji_build", "item": "xchat-2.8.8-21.XY20"}
215 ret = taskformula.devise_environment({'environment': {}}, arg_data) 215​ ret = taskformula.devise_environment({'environment': {}}, arg_data)
216 assert ret == {'distro': None, 'release': None, 'flavor': None, 'arch': None} 216​ assert ret == {'distro': None, 'release': None, 'flavor': None, 'arch': None}
217 217​
218​ def test_infer_from_koji_tag(self):
219​ arg_data = {"type": "koji_tag", "item": "f23-updates-pending"}
220​ ret = taskformula.devise_environment({'environment': {}}, arg_data)
221​ assert ret == {'distro': 'fedora', 'release': '23', 'flavor': None, 'arch': None}
222​
223​ def test_infer_from_koji_tag_unknown_distro(self):
224​ arg_data = {"type": "koji_tag", "item": "el6-updates-pending"}
225​ ret = taskformula.devise_environment({'environment': {}}, arg_data)
226​ assert ret == {'distro': None, 'release': None, 'flavor': None, 'arch': None}
227​
218 def test_infer_from_arch(self): 228​ def test_infer_from_arch(self):
219 arg_data = {"arch": []} 229​ arg_data = {"arch": []}
220 ret = taskformula.devise_environment({'environment': {}}, arg_data) 230​ ret = taskformula.devise_environment({'environment': {}}, arg_data)
221 assert ret == {'distro': None, 'release': None, 'flavor': None, 'arch': None} 231​ assert ret == {'distro': None, 'release': None, 'flavor': None, 'arch': None}
222 232​
223 arg_data = {"arch": ['noarch']} 233​ arg_data = {"arch": ['noarch']}
224 ret = taskformula.devise_environment({'environment': {}}, arg_data) 234​ ret = taskformula.devise_environment({'environment': {}}, arg_data)
225 assert ret == {'distro': None, 'release': None, 'flavor': None, 'arch': None} 235​ assert ret == {'distro': None, 'release': None, 'flavor': None, 'arch': None}
Show All 13 Lines