Differential D1093

distgit_directive: bring back NEVR parsing for ease of use
ClosedPublic
Actions

Authored by kparal on Jan 23 2017, 3:05 PM.

Details

Reviewers
ralph
mkrizek
Group Reviewers
libtaskotron
Commits
rLTRN08b68ce92d01: distgit_directive: bring back NVR parsing for ease of use
Summary

This makes it again possible to just provide nevr: ${koji_build}
parameter for koji build tasks, without parsing, as discussed in
D1069#20353. Test suite simplified and extended.

Test Plan

The test suite works but if @ralph could run his check_modulemd (since it's the only task using distgit directive currently, simply arc patch D1093) that would be appreciated :-) I tried to run it, but it requires some setup and I could not find documentation easily, so I gave up.

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to distgit_directive: bring back NEVR parsing for ease of use.Jan 23 2017, 3:05 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added reviewers: libtaskotron, ralph.
kparal added a subscriber: ralph.
kparal edited the test plan for this revision. (Show Details)Jan 24 2017, 8:08 AM
kparal updated this revision to Diff 2789.Jan 26 2017, 12:58 PM
  • change NEVR to NVR, because that's what we provide in ${kojibuild}
mkrizek accepted this revision.Jan 31 2017, 12:09 PM
mkrizek added a reviewer: mkrizek.
This revision is now accepted and ready to land.Jan 31 2017, 12:09 PM
Closed by commit rLTRN08b68ce92d01: distgit_directive: bring back NVR parsing for ease of use (authored by Kamil Páral <kparal@redhat.com>). · Explain WhyJan 31 2017, 12:27 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2803

View Options

libtaskotron/directives/distgit_directive.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2015, Red Hat, Inc. 2​# Copyright 2009-2015, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6from __future__ import absolute_import 6​from __future__ import absolute_import
7 7​
8​import os.path
9​
10​from libtaskotron.directives import BaseDirective
11​from libtaskotron import file_utils, python_utils
12​from libtaskotron.ext.fedora import rpm_utils
13​import libtaskotron.exceptions as exc
14​
15​
8DOCUMENTATION = """ 16​DOCUMENTATION = """
9module: distgit_directive 17​module: distgit_directive
10short_description: download files from distgit 18​short_description: download files from distgit
11description: | 19​description: |
12 Download files from Fedora package repository (usually called 'distgit') hosted at 20​ Download files from Fedora package repository (usually called 'distgit') hosted at
13 http://pkgs.fedoraproject.org/. Any files hosted at that repository can be downloaded for a 21​ http://pkgs.fedoraproject.org/. Any files hosted at that repository can be downloaded for a
14 chosen package. 22​ chosen package.
15parameters: 23​parameters:
16 package: 24​ package:
17 required: true 25​ required: true
18 description: | 26​ description: |
19 Name of a package. 27​ Name of a package. You need to provide either ``package`` or ``nvr`` parameter.
20 Example: ``xchat`` 28​ Example: ``xchat``
21 type: str 29​ type: str
30​ nvr:
31​ required: true
32​ description: |
33​ N(E)VR of the package build. If you input this, then the ``package``, ``gitref`` and
34​ ``namespace`` parameters will be automatically filled out (by parsing package name, dist tag,
35​ and using ``rpms`` namespace, respectively), but you can still override those if needed. You
36​ need to provide either ``package`` or ``nvr`` parameter.
37​ Example: ``xchat-2.8.8-32.fc25``
38​ type: str
22 gitref: 39​ gitref:
23 required: false 40​ required: false
24 description: | 41​ description: |
25 A git ref to check out. May be a branch, tag, or commit hash. 42​ A git ref to check out. May be a branch, tag, or commit hash.
26 Example: ``f24`` or ``04164165a840405e6bb5acc54a51e22346d84e0d`` 43​ Example: ``f24`` or ``04164165a840405e6bb5acc54a51e22346d84e0d``
27 type: str 44​ type: str
28 default: master 45​ default: master
29 namespace: 46​ namespace:
▲ Show 20 Lines • Show All 41 Lines • ▼ Show 20 Line(s)
71""" 88​"""
72 89​
73EXAMPLES = """ 90​EXAMPLES = """
74A task needs to download a spec file and httpd configuration check to run a check 91​A task needs to download a spec file and httpd configuration check to run a check
75on those files:: 92​on those files::
76 93​
77 - name: download spec file and httpd conf 94​ - name: download spec file and httpd conf
78 distgit: 95​ distgit:
96​ nvr: ${koji_build}
97​ path:
98​ - yourls.spec
99​ - yourls-httpd.conf
100​ localpath:
101​ - download/yourls_downloaded.spec
102​ - download/yourls-httpd_downloaded.conf
103​
104​You can make some of that configuration static and not depend on NVR parsing, if you need so::
105​
106​ - name: download spec file and httpd conf
107​ distgit:
79 package: yourls 108​ package: yourls
80 gitref: f22 109​ gitref: f22
81 path: 110​ path:
82 - yourls.spec 111​ - yourls.spec
83 - yourls-httpd.conf 112​ - yourls-httpd.conf
84 localpath: 113​ localpath:
85 - download/yourls_downloaded.spec 114​ - download/yourls_downloaded.spec
86 - download/yourls-httpd_downloaded.conf 115​ - download/yourls-httpd_downloaded.conf
87""" 116​"""
88 117​
89import os.path ​
90 ​
91from libtaskotron.directives import BaseDirective ​
92from libtaskotron import file_utils, python_utils ​
93 ​
94import libtaskotron.exceptions as exc ​
95 ​
96 ​
97 ​
98directive_class = 'DistGitDirective' 118​directive_class = 'DistGitDirective'
99 119​
120​BASEURL = 'http://pkgs.fedoraproject.org'
100URL_FMT = '{baseurl}/cgit/{namespace}/{package}.git/plain/{path}?h={gitref}' 121​URL_FMT = '{baseurl}/cgit/{namespace}/{package}.git/plain/{path}?h={gitref}'
101 122​
102 123​
103class DistGitDirective(BaseDirective): 124​class DistGitDirective(BaseDirective):
104 def __init__(self): 125​ def __init__(self):
105 super(DistGitDirective, self).__init__() 126​ super(DistGitDirective, self).__init__()
106 127​
107 def process(self, params, arg_data): 128​ def process(self, params, arg_data):
108 if 'package' not in params or 'path' not in params: 129​ if ('package' not in params and 'nvr' not in params) or 'path' not in params:
109 detected_args = ', '.join(params.keys()) 130​ detected_args = ', '.join(params.keys())
110 raise exc.TaskotronDirectiveError( 131​ raise exc.TaskotronDirectiveError(
111 "The distgit directive requires 'package' and 'path' arguments." 132​ "The distgit directive requires 'package' (or 'nvr') and 'path' arguments."
112 "Detected arguments: %s" % detected_args) 133​ "Detected arguments: %s" % detected_args)
113 134​
135​ package = None
136​ gitref = None
137​ namespace = None
138​
139​ if 'nvr' in params:
140​ nvr = params['nvr']
141​ package = rpm_utils.rpmformat(nvr, fmt='n')
142​ gitref = rpm_utils.get_dist_tag(nvr).replace('c', '')
143​ namespace = 'rpms'
144​
114 # Assign defaults 145​ # Assign defaults
115 gitref = params.get('gitref', 'master') 146​ package = params.get('package', package)
116 namespace = params.get('namespace', 'rpms') 147​ gitref = params.get('gitref', gitref or 'master')
117 baseurl = params.get('baseurl', 'http://pkgs.fedoraproject.org') 148​ namespace = params.get('namespace', namespace or 'rpms')
149​ baseurl = params.get('baseurl', BASEURL)
118 target_dir = params.get('target_dir', arg_data['workdir']) 150​ target_dir = params.get('target_dir', arg_data['workdir'])
119 151​
120 ​
121 if not python_utils.iterable(params['path']): 152​ if not python_utils.iterable(params['path']):
122 raise exc.TaskotronValueError("Incorrect value type of the 'path' argument: " 153​ raise exc.TaskotronValueError("Incorrect value type of the 'path' argument: "
123 "%s" % type(params['path'])) 154​ "%s" % type(params['path']))
124 155​
125 target_path = params['path'] 156​ target_path = params['path']
126 output_data = {} 157​ output_data = {}
127 158​
128 if 'localpath' in params: 159​ if 'localpath' in params:
129 if not python_utils.iterable(params['localpath']): 160​ if not python_utils.iterable(params['localpath']):
130 raise exc.TaskotronValueError("Incorrect value type of the 'localpath' argument: " 161​ raise exc.TaskotronValueError("Incorrect value type of the 'localpath' argument: "
131 "%s" % type(params['path'])) 162​ "%s" % type(params['path']))
132 163​
133 if not len(params['path']) == len(params['localpath']): 164​ if not len(params['path']) == len(params['localpath']):
134 raise exc.TaskotronValueError('path and localpath lists must be of the same ' 165​ raise exc.TaskotronValueError('path and localpath lists must be of the same '
135 'length.') 166​ 'length.')
136 167​
137 target_path = params['localpath'] 168​ target_path = params['localpath']
138 169​
139 format_fields = { 170​ format_fields = {
140 'package': params['package'], 171​ 'package': package,
141 'gitref': gitref, 172​ 'gitref': gitref,
142 'namespace': namespace, 173​ 'namespace': namespace,
143 'baseurl': baseurl, 174​ 'baseurl': baseurl,
144 } 175​ }
145 output_data['downloaded_files'] = [] 176​ output_data['downloaded_files'] = []
146 for path, localpath in zip(params['path'], target_path): 177​ for path, localpath in zip(params['path'], target_path):
147 localpath = os.path.join(target_dir, localpath) 178​ localpath = os.path.join(target_dir, localpath)
148 file_utils.makedirs(os.path.dirname(localpath)) 179​ file_utils.makedirs(os.path.dirname(localpath))
149 url = URL_FMT.format(path=path, **format_fields) 180​ url = URL_FMT.format(path=path, **format_fields)
150 output_data['downloaded_files'].append( 181​ output_data['downloaded_files'].append(
151 file_utils.download(url, '.', localpath) 182​ file_utils.download(url, '.', localpath)
152 ) 183​ )
153 184​
154 return output_data 185​ return output_data
View Options

testing/test_distgit_directive.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2015, Red Hat, Inc. 2​# Copyright 2009-2015, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6import os.path 6​import os.path
7import pytest 7​import pytest
8from dingus import Dingus 8​from dingus import Dingus
9 9​
10from libtaskotron import file_utils 10​from libtaskotron import file_utils
11from libtaskotron.directives import distgit_directive 11​from libtaskotron.directives import distgit_directive
12import libtaskotron.exceptions as exc 12​import libtaskotron.exceptions as exc
13 13​
14 14​
15​@pytest.mark.usefixtures('setup')
15class TestDistGitDirective(): 16​class TestDistGitDirective():
16 def setup_method(self, method): 17​
18​ @pytest.fixture
19​ def setup(self, monkeypatch):
20​ self.ref_nvr = 'foo-1.2-3.fc99'
17 self.ref_name = 'foo' 21​ self.ref_name = 'foo'
18 self.ref_path = ['foo.spec'] 22​ self.ref_path = ['foo.spec']
19 self.ref_localpath = ['local.foo.spec'] 23​ self.ref_localpath = ['local.foo.spec']
20 self.ref_branch = 'f99' 24​ self.ref_branch = 'f99'
25​ self.ref_namespace = 'rpms'
21 self.ref_arg_data = {'workdir': '/var/tmp/foo'} 26​ self.ref_arg_data = {'workdir': '/var/tmp/foo'}
27​ self.baseurl = distgit_directive.BASEURL
28​
22 self.helper = distgit_directive.DistGitDirective() 29​ self.helper = distgit_directive.DistGitDirective()
30​ self.ref_input = {'nvr': self.ref_nvr,
31​ 'path': self.ref_path}
32​
33​ self.mock_download = Dingus()
34​ monkeypatch.setattr(file_utils, 'download', self.mock_download)
23 35​
24 def _get_url(self, path): 36​ def _get_url(self, path):
25 return distgit_directive.URL_FMT.format( 37​ return distgit_directive.URL_FMT.format(
26 baseurl='http://pkgs.fedoraproject.org', 38​ baseurl=self.baseurl,
27 namespace='rpms', 39​ namespace=self.ref_namespace,
28 package='foo', 40​ package=self.ref_name,
29 path=path, 41​ path=path,
30 gitref=self.ref_branch) 42​ gitref=self.ref_branch)
31 43​
32 def test_parse_download_command(self, monkeypatch): 44​ def test_download(self, monkeypatch):
33 ref_input = {'action': 'download', 45​ self.helper.process(self.ref_input, self.ref_arg_data)
34 'package': self.ref_name, ​
35 'gitref': self.ref_branch, ​
36 'path': self.ref_path} ​
37 ​
38 monkeypatch.setattr(file_utils, 'download', Dingus()) ​
39 ​
40 self.helper.process(ref_input, self.ref_arg_data) ​
41 ​
42 download_calls = file_utils.download.calls ​
43 46​
47​ download_calls = self.mock_download.calls
44 assert len(download_calls) == 1 48​ assert len(download_calls) == 1
45 assert download_calls[0][1][0] == self._get_url(self.ref_path[0]) 49​ assert download_calls[0][1][0] == self._get_url(self.ref_path[0])
46 assert download_calls[0][1][2] == os.path.join(self.ref_arg_data['workdir'], 50​ assert download_calls[0][1][2] == os.path.join(self.ref_arg_data['workdir'],
47 self.ref_path[0]) 51​ self.ref_path[0])
48 52​
49 def test_parse_download_command_localpath(self, monkeypatch): 53​ def test_localpath(self, monkeypatch):
50 ref_input = {'action': 'download', 54​ self.ref_input['localpath'] = self.ref_localpath
51 'package': self.ref_name, ​
52 'gitref': self.ref_branch, ​
53 'path': self.ref_path, ​
54 'localpath': self.ref_localpath} ​
55 55​
56 monkeypatch.setattr(file_utils, 'download', Dingus()) 56​ self.helper.process(self.ref_input, self.ref_arg_data)
57 ​
58 self.helper.process(ref_input, self.ref_arg_data) ​
59 ​
60 download_calls = file_utils.download.calls ​
61 57​
58​ download_calls = self.mock_download.calls
62 assert len(download_calls) == 1 59​ assert len(download_calls) == 1
63 assert download_calls[0][1][0] == self._get_url(self.ref_path[0]) 60​ assert download_calls[0][1][0] == self._get_url(self.ref_path[0])
64 assert download_calls[0][1][2] == os.path.join(self.ref_arg_data['workdir'], 61​ assert download_calls[0][1][2] == os.path.join(self.ref_arg_data['workdir'],
65 self.ref_localpath[0]) 62​ self.ref_localpath[0])
66 63​
67 def test_parse_download_command_multiple_localpath(self, monkeypatch): 64​ def test_multiple_localpath(self, monkeypatch):
68 ref_path = ['file1', 'file2'] 65​ ref_path = ['file1', 'file2']
69 ref_localpath = ['file3', 'file4'] 66​ ref_localpath = ['file3', 'file4']
70 ref_input = {'action': 'download', 67​ self.ref_input['path'] = ref_path
71 'package': self.ref_name, 68​ self.ref_input['localpath'] = ref_localpath
72 'gitref': self.ref_branch, ​
73 'path': ref_path, ​
74 'localpath': ref_localpath} ​
75 ​
76 monkeypatch.setattr(file_utils, 'download', Dingus()) ​
77 69​
78 self.helper.process(ref_input, self.ref_arg_data) 70​ self.helper.process(self.ref_input, self.ref_arg_data)
79 ​
80 download_calls = file_utils.download.calls ​
81 71​
72​ download_calls = self.mock_download.calls
82 assert len(download_calls) == 2 73​ assert len(download_calls) == 2
83 assert download_calls[0][1][0] == self._get_url(ref_path[0]) 74​ assert download_calls[0][1][0] == self._get_url(ref_path[0])
84 assert download_calls[1][1][0] == self._get_url(ref_path[1]) 75​ assert download_calls[1][1][0] == self._get_url(ref_path[1])
85 assert download_calls[0][1][2] == os.path.join(self.ref_arg_data['workdir'], 76​ assert download_calls[0][1][2] == os.path.join(self.ref_arg_data['workdir'],
86 ref_localpath[0]) 77​ ref_localpath[0])
87 assert download_calls[1][1][2] == os.path.join(self.ref_arg_data['workdir'], 78​ assert download_calls[1][1][2] == os.path.join(self.ref_arg_data['workdir'],
88 ref_localpath[1]) 79​ ref_localpath[1])
89 80​
90 def test_parse_download_command_incorrect_localpath(self): 81​ def test_incorrect_localpath(self):
91 ref_path = ['file1', 'file2'] 82​ self.ref_input['path'] = ['file1', 'file2']
92 ref_localpath = ['file3'] 83​ self.ref_input['localpath'] = ['file3']
93 ref_input = {'action': 'download', ​
94 'package': self.ref_name, ​
95 'gitref': self.ref_branch, ​
96 'path': ref_path, ​
97 'localpath': ref_localpath} ​
98 84​
99 with pytest.raises(exc.TaskotronValueError): 85​ with pytest.raises(exc.TaskotronValueError):
100 self.helper.process(ref_input, self.ref_arg_data) 86​ self.helper.process(self.ref_input, self.ref_arg_data)
101 87​
102 def test_parse_download_command_incorrect_path_str(self): 88​ def test_incorrect_path_str(self):
103 ref_path = 'file1' 89​ self.ref_input['path'] = 'file1'
104 ref_input = {'action': 'download', ​
105 'package': self.ref_name, ​
106 'gitref': self.ref_branch, ​
107 'path': ref_path} ​
108 90​
109 with pytest.raises(exc.TaskotronValueError): 91​ with pytest.raises(exc.TaskotronValueError):
110 self.helper.process(ref_input, self.ref_arg_data) 92​ self.helper.process(self.ref_input, self.ref_arg_data)
111 93​
112 def test_parse_download_command_incorrect_localpath_str(self): 94​ def test_incorrect_localpath_str(self):
113 ref_localpath = 'file2' 95​ self.ref_input['localpath'] = 'file2'
114 ref_input = {'action': 'download', ​
115 'package': self.ref_name, ​
116 'gitref': self.ref_branch, ​
117 'path': self.ref_path, ​
118 'localpath': ref_localpath} ​
119 96​
120 with pytest.raises(exc.TaskotronValueError): 97​ with pytest.raises(exc.TaskotronValueError):
121 self.helper.process(ref_input, self.ref_arg_data) 98​ self.helper.process(self.ref_input, self.ref_arg_data)
122 99​
123 def test_parse_download_command_missing_package(self): 100​ def test_missing_package_or_nvr(self):
124 ref_input = {'action': 'download', 101​ self.ref_input.pop('nvr', None)
125 'path': self.ref_path} 102​ self.ref_input.pop('package', None)
126 103​
127 with pytest.raises(exc.TaskotronDirectiveError): 104​ with pytest.raises(exc.TaskotronDirectiveError):
128 self.helper.process(ref_input, self.ref_arg_data) 105​ self.helper.process(self.ref_input, self.ref_arg_data)
129 106​
130 def test_parse_download_command_missing_path(self): 107​ def test_only_package(self):
131 ref_input = {'action': 'download', 108​ self.ref_input.pop('nvr', None)
132 'package': self.ref_name, 109​ self.ref_input['package'] = self.ref_name
133 'gitref': self.ref_branch} 110​ self.ref_input['gitref'] = self.ref_branch
111​
112​ self.helper.process(self.ref_input, self.ref_arg_data)
113​
114​ download_calls = self.mock_download.calls
115​ assert len(download_calls) == 1
116​ assert download_calls[0][1][0] == self._get_url(self.ref_path[0])
117​
118​ def test_missing_path(self):
119​ self.ref_input.pop('path', None)
134 120​
135 with pytest.raises(exc.TaskotronDirectiveError): 121​ with pytest.raises(exc.TaskotronDirectiveError):
136 self.helper.process(ref_input, self.ref_arg_data) 122​ self.helper.process(self.ref_input, self.ref_arg_data)
123​
124​ def test_override_nvr(self):
125​ '''nvr parsing should be overriden when filling in parameters manually'''
126​ self.ref_name = 'bar'
127​ self.ref_branch = 'v17'
128​ self.ref_namespace = 'onions'
129​ self.ref_input.update({'nvr': self.ref_nvr,
130​ 'package': self.ref_name,
131​ 'gitref': self.ref_branch,
132​ 'namespace': self.ref_namespace})
133​
134​ self.helper.process(self.ref_input, self.ref_arg_data)
135​
136​ download_calls = self.mock_download.calls
137​ assert len(download_calls) == 1
138​ # self.ref_nvr was not touched, so if nvr parsing gets priority, the URL will not match
139​ assert download_calls[0][1][0] == self._get_url(self.ref_path[0])