Differential D916

koji_utils/directive: add option to download build.log
ClosedPublic
Actions

Authored by kparal on Jun 30 2016, 2:23 PM.

Details

Reviewers
jskladan
Group Reviewers
libtaskotron
Maniphest Tasks
T812: koji directive: allow to download build logs
Commits
rLTRN89a13bffac56: koji_utils/directive: add option to download build.log
Summary

This allows you to download build.log for all arch for a specific Koji
build.

I considered whether to download all available logs for the build, or
just build.log. Downloading all logs was more difficult to do and also
required sending more Koji queries. For the moment, the code supports
build.log only, which is the only one we have some use for (rpmgrill).

Test Plan

Unit tests added. Tested manually with modified rpmgrill.
Check out feature/buildlog branch from rpmgrill and you can see it live yourself.

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to koji_utils/directive: add option to download build.log.Jun 30 2016, 2:23 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added a reviewer: libtaskotron.
kparal updated this revision to Diff 2334.Jul 1 2016, 1:52 PM
  • add unit tests
kparal updated this object.Jul 1 2016, 1:52 PM
kparal edited the test plan for this revision. (Show Details)
jskladan requested changes to this revision.Jul 7 2016, 1:42 PM
jskladan added a reviewer: jskladan.
jskladan added a subscriber: jskladan.

Looks good other than the build_log_url copy-paste error.

libtaskotron/directives/koji_directive.py
235–240

Maybe log a warning if build_log and action not in (...)? Or is it too defensive?

libtaskotron/ext/fedora/koji_utils.py
32–33

Seems like a typo here - guess it was not supposed to be {nevra.version}/{nevra.version} but {nevra.version}/{nevra.release} based on the example. Maybe add a test? :)

This revision now requires changes to proceed.Jul 7 2016, 1:42 PM
kparal marked 2 inline comments as done.Jul 7 2016, 3:12 PM
kparal added inline comments.
libtaskotron/directives/koji_directive.py
235–240

Added.

libtaskotron/ext/fedora/koji_utils.py
32–33

OOOPS! Thanks! I improved my test coverage.

The reason this slipped is because it "worked" - build.logs were getting created. However, they contained only "404 error" html code, which I didn't check. I don't think our file_utils.download() method should ignore http 404 and similar, it should raise an error instead. I'll address that in a separate diff.

kparal updated this revision to Diff 2352.Jul 7 2016, 3:14 PM
kparal marked 2 inline comments as done.
  • fix typo in build_log_url, add tests to catch it next time
jskladan accepted this revision.Jul 8 2016, 6:58 AM
This revision is now accepted and ready to land.Jul 8 2016, 6:58 AM
Closed by commit rLTRN89a13bffac56: koji_utils/directive: add option to download build.log (authored by kparal). · Explain WhyJul 8 2016, 7:47 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2357

View Options

libtaskotron/directives/koji_directive.py

Show All 36 Lines16parameters:
37 arch_exclude: 37​ arch_exclude:
38 required: false 38​ required: false
39 description: | 39​ description: |
40 a list of architectures to exclude from downloading (overrides ``arch`` 40​ a list of architectures to exclude from downloading (overrides ``arch``
41 value). You can use it to exclude some specific archicture while using 41​ value). You can use it to exclude some specific archicture while using
42 ``arch=['all']``. Example: ``['armhfp', 'noarch']`` 42​ ``arch=['all']``. Example: ``['armhfp', 'noarch']``
43 type: list of str 43​ type: list of str
44 choices: [supported architectures] 44​ choices: [supported architectures]
45​ build_log:
46​ required: false
47​ description: download also ``build.log`` files for each requested architecture (the files will
48​ be saved as ``build.log.<arch>``). Note that some logs can be already deleted in Koji and
49​ might not be available for download. Such missing logs will be skipped. This option will be
50​ considered only for single build downloads (i.e. ``action='download'`` or
51​ ``action='download_latest_stable'``).
52​ type: bool
53​ default: False
54​ debuginfo:
55​ required: false
56​ description: download also ``debuginfo`` RPM files
57​ type: bool
58​ default: False
45 koji_build: 59​ koji_build:
46 required: true 60​ required: true
47 description: | 61​ description: |
48 N(E)VR of a Koji build to download (for ``action="download"``) or to search 62​ N(E)VR of a Koji build to download (for ``action="download"``) or to search
49 the latest stable build for (for ``action="download_latest_stable"``). Not 63​ the latest stable build for (for ``action="download_latest_stable"``). Not
50 required for ``action="download_tag"``. Example: ``xchat-2.8.8-21.fc20`` 64​ required for ``action="download_tag"``. Example: ``xchat-2.8.8-21.fc20``
51 type: str 65​ type: str
52 koji_tag: 66​ koji_tag:
53 required: true 67​ required: true
54 description: | 68​ description: |
55 name of a Koji tag to download all builds from. Only required when 69​ name of a Koji tag to download all builds from. Only required when
56 ``action="download_tag"``. Example: ``f20-updates-pending`` 70​ ``action="download_tag"``. Example: ``f20-updates-pending``
57 type: str 71​ type: str
58 src: 72​ src:
59 required: false 73​ required: false
60 description: download also ``src`` RPM files 74​ description: download also ``src`` RPM files
61 type: bool 75​ type: bool
62 default: False 76​ default: False
63 debuginfo: ​
64 required: false ​
65 description: download also ``debuginfo`` RPM files ​
66 type: bool ​
67 default: False ​
68 target_dir: 77​ target_dir:
69 required: false 78​ required: false
70 description: directory into which to download builds 79​ description: directory into which to download builds
71 type: str 80​ type: str
72 default: ${workdir} 81​ default: ${workdir}
73returns: | 82​returns: |
74 A dictionary containing following items: 83​ A dictionary containing following items:
75 84​
76 * `downloaded_rpms`: (list of str) a list of local filenames of the downloaded 85​ * `downloaded_rpms`: (list of str) a list of absolute paths of the downloaded RPMs
77 RPMs 86​ * `downloaded_logs`: (list of str) a list of absolute paths of the downloaded build logs
87​ * `log_errors`: (list of str) a list of architectures for which the ``build.log`` could not be
88​ downloaded
78raises: | 89​raises: |
79 * :class:`.TaskotronRemoteError`: if downloading failed 90​ * :class:`.TaskotronRemoteError`: if downloading failed
80version_added: 0.4 91​version_added: 0.4
81""" 92​"""
82 93​
83EXAMPLES = """ 94​EXAMPLES = """
84Rpmlint needs to download a specific build from Koji, all architectures 95​Rpmlint needs to download a specific build from Koji, all architectures
85including src.rpm:: 96​including src.rpm::
Show All 37 Lines128 koji:
123 target_dir: ${workdir}/update 134​ target_dir: ${workdir}/update
124 135​
125""" 136​"""
126 137​
127from libtaskotron.directives import BaseDirective 138​from libtaskotron.directives import BaseDirective
128import libtaskotron.exceptions as exc 139​import libtaskotron.exceptions as exc
129from libtaskotron.ext.fedora.koji_utils import KojiClient 140​from libtaskotron.ext.fedora.koji_utils import KojiClient
130from libtaskotron.ext.fedora import rpm_utils 141​from libtaskotron.ext.fedora import rpm_utils
142​from libtaskotron.logger import log
131 143​
132 144​
133directive_class = 'KojiDirective' 145​directive_class = 'KojiDirective'
134 146​
135 147​
136class KojiDirective(BaseDirective): 148​class KojiDirective(BaseDirective):
137 149​
138 def __init__(self, koji_session=None): 150​ def __init__(self, koji_session=None):
Show All 24 Lines157 def process(self, params, arg_data):
163 175​
164 arches = list(params['arch']) 176​ arches = list(params['arch'])
165 if arches and ('all' not in arches) and ('noarch' not in arches): 177​ if arches and ('all' not in arches) and ('noarch' not in arches):
166 arches.append('noarch') 178​ arches.append('noarch')
167 179​
168 arch_exclude = params.get('arch_exclude', []) 180​ arch_exclude = params.get('arch_exclude', [])
169 debuginfo = params.get('debuginfo', False) 181​ debuginfo = params.get('debuginfo', False)
170 src = params.get('src', False) 182​ src = params.get('src', False)
183​ build_log = params.get('build_log', False)
171 184​
172 # download files 185​ # download files
173 output_data = {} 186​ output_data = {}
174 187​
175 if action == 'download': 188​ if action == 'download':
176 if 'koji_build' not in params: 189​ if 'koji_build' not in params:
177 detected_args = ', '.join(params.keys()) 190​ detected_args = ', '.join(params.keys())
178 raise exc.TaskotronDirectiveError( 191​ raise exc.TaskotronDirectiveError(
Show All 29 Lines214 elif action == 'download_latest_stable':
208 name = rpm_utils.rpmformat(params['koji_build'], 'n') 221​ name = rpm_utils.rpmformat(params['koji_build'], 'n')
209 disttag = rpm_utils.get_dist_tag(params['koji_build']) 222​ disttag = rpm_utils.get_dist_tag(params['koji_build'])
210 # we need to do 'fc22' -> 'f22' conversion 223​ # we need to do 'fc22' -> 'f22' conversion
211 tag = disttag.replace('c', '') 224​ tag = disttag.replace('c', '')
212 225​
213 # first we need to check updates tag and if that fails, the latest 226​ # first we need to check updates tag and if that fails, the latest
214 # stable nvr is in the base repo 227​ # stable nvr is in the base repo
215 tags = ['%s-updates' % tag, tag] 228​ tags = ['%s-updates' % tag, tag]
216 latest_stable_nvr = self.koji.latest_by_tag(tags, name) 229​ nvr = self.koji.latest_by_tag(tags, name)
217 230​
218 output_data['downloaded_rpms'] = self.koji.get_nvr_rpms( 231​ output_data['downloaded_rpms'] = self.koji.get_nvr_rpms(
219 latest_stable_nvr, target_dir, arch_exclude=arch_exclude, 232​ nvr, target_dir, arch_exclude=arch_exclude,
220 arches=arches, debuginfo=debuginfo, src=src) 233​ arches=arches, debuginfo=debuginfo, src=src)
221 234​
235​ # download build.log if requested
236​ if build_log:
237​ if action in ('download', 'download_latest_stable'):
238​ ret_log = self.koji.get_build_log(
239​ nvr, target_dir, arches=arches, arch_exclude=arch_exclude)
240​ output_data['downloaded_logs'] = ret_log['ok']
jskladanUnsubmitted
Done

Maybe log a warning if build_log and action not in (...)? Or is it too defensive?

kparalAuthorUnsubmitted
Not Done

Added.

241​ output_data['log_errors'] = ret_log['error']
242​ else:
243​ log.warn("Downloading build logs is not supported for action '%s', ignoring.",
244​ action)
245​
222 return output_data 246​ return output_data
View Options

libtaskotron/ext/fedora/koji_utils.py

Show All 14 Lines
15from libtaskotron import file_utils 15​from libtaskotron import file_utils
16from libtaskotron.logger import log 16​from libtaskotron.logger import log
17from libtaskotron import exceptions as exc 17​from libtaskotron import exceptions as exc
18from libtaskotron import config 18​from libtaskotron import config
19from libtaskotron import arch_utils 19​from libtaskotron import arch_utils
20 20​
21from . import rpm_utils 21​from . import rpm_utils
22 22​
23​
23class KojiClient(object): 24​class KojiClient(object):
24 '''Helper Koji methods. 25​ '''Helper Koji methods.
25 26​
26 :ivar koji.ClientSession session: Koji client session 27​ :ivar koji.ClientSession session: Koji client session
27 ''' 28​ '''
28 29​
30​ # example:
31​ # https://kojipkgs.fedoraproject.org/packages/kcalc/16.04.2/1.fc24/data/logs/x86_64/build.log
32​ build_log_url = \
33​ '{pkg_url}/{nevra.name}/{nevra.version}/{nevra.release}/data/logs/{nevra.arch}/build.log'
jskladanUnsubmitted
Done

Seems like a typo here - guess it was not supposed to be {nevra.version}/{nevra.version} but {nevra.version}/{nevra.release} based on the example. Maybe add a test? :)

kparalAuthorUnsubmitted
Not Done

OOOPS! Thanks! I improved my test coverage.

The reason this slipped is because it "worked" - build.logs were getting created. However, they contained only "404 error" html code, which I didn't check. I don't think our file_utils.download() method should ignore http 404 and similar, it should raise an error instead. I'll address that in a separate diff.

34​
29 def __init__(self, koji_session=None): 35​ def __init__(self, koji_session=None):
30 '''Create a new KojiClient 36​ '''Create a new KojiClient
31 37​
32 :param koji_session: an existing Koji session instance or ``None`` if 38​ :param koji_session: an existing Koji session instance or ``None`` if
33 you want a new default session to be created 39​ you want a new default session to be created
34 :type koji_session: :class:`koji.ClientSession` 40​ :type koji_session: :class:`koji.ClientSession`
35 ''' 41​ '''
36 opts = { 42​ opts = {
▲ Show 20 Lines • Show All 227 Lines • ▼ Show 20 Line(s)251 def get_tagged_rpms(self, tag, dest, arches=['all'], arch_exclude=[], debuginfo=False,
264 log.info("Fetching %s builds for tag: %s", len(nvrs), tag) 270​ log.info("Fetching %s builds for tag: %s", len(nvrs), tag)
265 log.debug('Builds to be downloaded:\n %s', '\n '.join(nvrs)) 271​ log.debug('Builds to be downloaded:\n %s', '\n '.join(nvrs))
266 272​
267 for nvr in nvrs: 273​ for nvr in nvrs:
268 rpms.extend(self.get_nvr_rpms(nvr, dest, arches, arch_exclude, debuginfo, src)) 274​ rpms.extend(self.get_nvr_rpms(nvr, dest, arches, arch_exclude, debuginfo, src))
269 275​
270 return rpms 276​ return rpms
271 277​
278​ def get_build_log(self, nvr, dest, arches=['all'], arch_exclude=[]):
279​ '''Download a build.log file for NVR for each specified architecture and save it in
280​ ``dest`` as ``build.log.<arch>``.
281​
282​ Note: Koji cleans up old log files regularly, i.e. they might not be available for a
283​ certain NVR. This method will not raise an error if that happens (the log file can't be
284​ downloaded). Instead, it will be simply missing in that directory and also marked in the
285​ returned object.
286​
287​ For parameters undocumented here, see :meth:`nvr_to_urls`.
288​
289​ :param str nvr: build NVR
290​ :param str dest: location where to store the log file
291​ :return: a dictionary of this structure:
292​ ::
293​
294​ {'ok': ['/path/build.log.x86_64', '/path/build.log.i686'],
295​ 'error': ['armv7hl']
296​ }
297​
298​ which describes which build logs were correctly downloaded and for which architectures
299​ we failed to download the logs.
300​ :rtype: dict
301​ :raise TaskotronRemoteError: when the requested build doesn't exist or its info can't be
302​ retrieved
303​ '''
304​ log.info('Getting build logs for: %s', nvr)
305​ arch_exclude = list(arch_exclude)
306​
307​ # add binary arches if base arches are specified
308​ archlist = list(arches)
309​ for arch in arches:
310​ if arch in arch_utils.Arches.base:
311​ archlist.extend(arch_utils.Arches.binary[arch])
312​ arches = list(set(archlist))
313​ # there seem to be no logs in Koji for 'src'
314​ arch_exclude.append('src')
315​
316​ # list the RPMs for the koji build, so that we know which arches to download logs for
317​ try:
318​ build_rpms = self.session.listBuildRPMs(nvr)
319​ except koji.GenericError as e:
320​ log.exception("Could not query build %s: %s", nvr, e)
321​ raise exc.TaskotronRemoteError(e)
322​
323​ # download for each arch
324​ build_arches = set([rpm['arch'] for rpm in build_rpms])
325​ ok = []
326​ error = []
327​ for build_arch in build_arches:
328​ if build_arch not in arches and 'all' not in arches:
329​ continue
330​ if build_arch in arch_exclude:
331​ continue
332​ nevra = hawkey.split_nevra(nvr + '.' + build_arch)
333​ url = self.build_log_url.format(pkg_url=config.get_config().pkg_url, nevra=nevra)
334​
335​ log.debug('Getting build.log for: %s %s', nvr, build_arch)
336​ try:
337​ log_file = file_utils.download(url=url, dirname=dest,
338​ filename='build.log.' + build_arch)
339​ except exc.TaskotronRemoteError:
340​ error.append(build_arch)
341​ # file_utils.download() already logged important details, nothing more to do here
342​ else:
343​ ok.append(log_file)
344​
345​ return {'ok': ok, 'error': error}
346​
272 347​
273def getNEVR(build): 348​def getNEVR(build):
274 '''Extract RPM version identifier in NEVR format from Koji build object 349​ '''Extract RPM version identifier in NEVR format from Koji build object
275 350​
276 :param dict build: Koji buildinfo dictionary (as returned e.g. from 351​ :param dict build: Koji buildinfo dictionary (as returned e.g. from
277 :meth:`koji.getBuild`) 352​ :meth:`koji.getBuild`)
278 :return: NEVR string; epoch is included when non-zero, otherwise omitted 353​ :return: NEVR string; epoch is included when non-zero, otherwise omitted
279 :raise TaskotronValueError: if ``build`` is of incorrect type 354​ :raise TaskotronValueError: if ``build`` is of incorrect type
Show All 14 Lines
View Options

testing/test_koji_directive.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2016, Red Hat, Inc. 2​# Copyright 2009-2016, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6from dingus import Dingus 6​from dingus import Dingus
7import pytest 7​import pytest
8​import mock
8 9​
9from libtaskotron.directives import koji_directive 10​from libtaskotron.directives import koji_directive
10 11​
11ACTIONS = ('download', 'download_tag', 'download_latest_stable') 12​ACTIONS = ('download', 'download_tag', 'download_latest_stable')
13​BUILD_LOG_ACTIONS = ('download', 'download_latest_stable')
14​
12 15​
13class TestKojiDirective(): 16​class TestKojiDirective():
14 def setup_method(self, method): 17​ def setup_method(self, method):
15 self.koji_calls = ('get_nvr_rpms', 'get_tagged_rpms') 18​ self.koji_calls = ('get_nvr_rpms', 'get_tagged_rpms')
16 19​
17 self.ref_nvr = 'foo-1.2-3.fc99' 20​ self.ref_nvr = 'foo-1.2-3.fc99'
18 self.ref_previous_nvr = 'foo-1.2-2.fc99' 21​ self.ref_previous_nvr = 'foo-1.2-2.fc99'
19 self.ref_arch = ['noarch'] 22​ self.ref_arch = ['noarch']
▲ Show 20 Lines • Show All 187 Lines • ▼ Show 20 Line(s)203 def test_debuginfo(self, action):
207 210​
208 getrpm_calls = [call for call in stub_koji.calls() if call[0] in self.koji_calls] 211​ getrpm_calls = [call for call in stub_koji.calls() if call[0] in self.koji_calls]
209 assert len(getrpm_calls) >= 1 212​ assert len(getrpm_calls) >= 1
210 for call in getrpm_calls: 213​ for call in getrpm_calls:
211 requested_arches = call[2]['arches'] 214​ requested_arches = call[2]['arches']
212 requested_debuginfo = call[2]['debuginfo'] 215​ requested_debuginfo = call[2]['debuginfo']
213 assert requested_arches == ref_input['arch'] 216​ assert requested_arches == ref_input['arch']
214 assert requested_debuginfo 217​ assert requested_debuginfo
218​
219​ @pytest.mark.parametrize('action', ACTIONS)
220​ def test_build_log_only_for_certain_actions(self, action):
221​ ref_input = {'action': action, 'arch': [], 'build_log': True,
222​ 'koji_build': self.ref_nvr, 'koji_tag': self.ref_tag}
223​ stub_koji = mock.MagicMock()
224​ test_helper = koji_directive.KojiDirective(stub_koji)
225​
226​ test_helper.process(ref_input, self.ref_arg_data)
227​
228​ build_log_calls = [call for call in stub_koji.method_calls if call[0] == 'get_build_log']
229​ if action in BUILD_LOG_ACTIONS:
230​ assert len(build_log_calls) == 1
231​ else:
232​ assert len(build_log_calls) == 0
233​
234​ @pytest.mark.parametrize('action', BUILD_LOG_ACTIONS)
235​ def test_build_log_return_value(self, action):
236​ ref_input = {'action': action, 'arch': [], 'build_log': True,
237​ 'koji_build': self.ref_nvr, 'koji_tag': self.ref_tag}
238​ ref_get_bl_return = {'ok': ['/path/build.log.arch'], 'error': ['some_arch']}
239​ stub_koji = mock.MagicMock()
240​ stub_koji.get_build_log.return_value = ref_get_bl_return
241​ test_helper = koji_directive.KojiDirective(stub_koji)
242​
243​ retval = test_helper.process(ref_input, self.ref_arg_data)
244​
245​ assert retval['downloaded_logs'] == ref_get_bl_return['ok']
246​ assert retval['log_errors'] == ref_get_bl_return['error']
View Options

testing/test_koji_utils.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2016, Red Hat, Inc. 2​# Copyright 2009-2016, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6'''Unit tests for libtaskotron/koji_utils.py''' 6​'''Unit tests for libtaskotron/koji_utils.py'''
7 7​
8import pytest 8​import pytest
9from dingus import Dingus 9​from dingus import Dingus
10import os 10​import os
11import itertools 11​import itertools
12import mock 12​import mock
13​import koji
13 14​
14from libtaskotron.ext.fedora import koji_utils 15​from libtaskotron.ext.fedora import koji_utils
15from libtaskotron import exceptions as exc 16​from libtaskotron import exceptions as exc
16from libtaskotron import config 17​from libtaskotron import config
17from libtaskotron import file_utils 18​from libtaskotron import file_utils
18 19​
19from test_file_utils import mock_download 20​from test_file_utils import mock_download
20 21​
21# http://stackoverflow.com/questions/3190706/nonlocal-keyword-in-python-2-x 22​# http://stackoverflow.com/questions/3190706/nonlocal-keyword-in-python-2-x
22def create_multicall(first, second): 23​def create_multicall(first, second):
23 first_call = {"value": True} 24​ first_call = {"value": True}
24 25​
25 def multicall(): 26​ def multicall():
26 if first_call["value"]: 27​ if first_call["value"]:
27 first_call["value"] = False 28​ first_call["value"] = False
28 return first 29​ return first
29 else: 30​ else:
30 return second 31​ return second
31 32​
32 return multicall 33​ return multicall
33 34​
34 35​
35class TestKojiClient(): 36​class TestKojiClient(object):
36 def setup_method(self, method): 37​ def setup_method(self, method):
37 self.ref_nvr = 'foo-1.2-3.fc99' 38​ self.ref_nvr = 'foo-1.2-3.fc99'
38 self.ref_latest_stable = 'foo-1.2-2.fc99' 39​ self.ref_latest_stable = 'foo-1.2-2.fc99'
39 self.ref_arch = 'x86_64' 40​ self.ref_arch = 'x86_64'
40 self.ref_name = 'foo' 41​ self.ref_name = 'foo'
41 self.ref_version = '1.2' 42​ self.ref_version = '1.2'
42 self.ref_release = '3.fc99' 43​ self.ref_release = '3.fc99'
43 self.ref_buildid = 123456 44​ self.ref_buildid = 123456
44 self.ref_filename = "%s.%s.rpm" % (self.ref_nvr, self.ref_arch) 45​ self.ref_filename = "%s.%s.rpm" % (self.ref_nvr, self.ref_arch)
45 self.ref_tags = ['f99-updates', 'f99'] 46​ self.ref_tags = ['f99-updates', 'f99']
46 47​
47 self.ref_build = {'package_name': self.ref_name, 48​ self.ref_build = {'package_name': self.ref_name,
48 'version': self.ref_version, 49​ 'version': self.ref_version,
49 'release': self.ref_release, 50​ 'release': self.ref_release,
50 'id': self.ref_buildid, 51​ 'id': self.ref_buildid,
51 'nvr': '%s-%s-%s' % (self.ref_name, self.ref_version, 52​ 'nvr': '%s-%s-%s' % (self.ref_name, self.ref_version,
52 self.ref_release)} 53​ self.ref_release)}
53 54​
55​ @property
56​ def ref_rpms(self):
54 # rpms: standard, src, 2x debuginfo, noarch 57​ # rpms: standard, src, 2x debuginfo, noarch
55 self.ref_rpms = [{'name': self.ref_name, 'version': self.ref_version, 58​ return [{'name': self.ref_name, 'version': self.ref_version,
56 'release': self.ref_release, 'nvr': self.ref_nvr, 59​ 'release': self.ref_release, 'nvr': self.ref_nvr,
57 'arch': self.ref_arch, 'build_id': self.ref_buildid}, 60​ 'arch': self.ref_arch, 'build_id': self.ref_buildid},
58 {'name': self.ref_name, 'version': self.ref_version, 61​ {'name': self.ref_name, 'version': self.ref_version,
59 'release': self.ref_release, 'nvr': self.ref_nvr, 62​ 'release': self.ref_release, 'nvr': self.ref_nvr,
60 'arch': 'src', 'build_id': self.ref_buildid}, 63​ 'arch': 'src', 'build_id': self.ref_buildid},
61 {'name': self.ref_name + '-debuginfo', 64​ {'name': self.ref_name + '-debuginfo',
62 'version': self.ref_version, 65​ 'version': self.ref_version,
63 'release': self.ref_release, 'nvr': self.ref_nvr, 66​ 'release': self.ref_release, 'nvr': self.ref_nvr,
64 'arch': self.ref_arch, 'build_id': self.ref_buildid}, 67​ 'arch': self.ref_arch, 'build_id': self.ref_buildid},
65 {'name': self.ref_name + '-debuginfo-common', 68​ {'name': self.ref_name + '-debuginfo-common',
66 'version': self.ref_version, 69​ 'version': self.ref_version,
67 'release': self.ref_release, 'nvr': self.ref_nvr, 70​ 'release': self.ref_release, 'nvr': self.ref_nvr,
68 'arch': self.ref_arch, 'build_id': self.ref_buildid}, 71​ 'arch': self.ref_arch, 'build_id': self.ref_buildid},
69 {'name': self.ref_name + '-data', 72​ {'name': self.ref_name + '-data',
70 'version': self.ref_version, 73​ 'version': self.ref_version,
71 'release': self.ref_release, 'nvr': self.ref_nvr, 74​ 'release': self.ref_release, 'nvr': self.ref_nvr,
72 'arch': 'noarch', 'build_id': self.ref_buildid}] 75​ 'arch': 'noarch', 'build_id': self.ref_buildid}]
76​
77​ # =====================
78​ # latest_by_tag()
79​ # =====================
73 80​
74 def test_latest_by_tag_first_tag_miss(self): 81​ def test_latest_by_tag_first_tag_miss(self):
75 stub_koji = Dingus(listTagged__returns=None, 82​ stub_koji = Dingus(listTagged__returns=None,
76 multiCall__returns=[[[]], [[{'nvr': self.ref_latest_stable}]]]) 83​ multiCall__returns=[[[]], [[{'nvr': self.ref_latest_stable}]]])
77 84​
78 test_koji = koji_utils.KojiClient(stub_koji) 85​ test_koji = koji_utils.KojiClient(stub_koji)
79 outcome = test_koji.latest_by_tag(self.ref_tags, self.ref_name) 86​ outcome = test_koji.latest_by_tag(self.ref_tags, self.ref_name)
80 87​
Show All 10 Lines
91 98​
92 def test_latest_by_tag_build_not_found(self): 99​ def test_latest_by_tag_build_not_found(self):
93 stub_koji = Dingus(listTagged__returns=None, 100​ stub_koji = Dingus(listTagged__returns=None,
94 multiCall__returns=[[[]], [[]]]) 101​ multiCall__returns=[[[]], [[]]])
95 102​
96 test_koji = koji_utils.KojiClient(stub_koji) 103​ test_koji = koji_utils.KojiClient(stub_koji)
97 assert test_koji.latest_by_tag(self.ref_tags, self.ref_name) is None 104​ assert test_koji.latest_by_tag(self.ref_tags, self.ref_name) is None
98 105​
106​ # =====================
107​ # rpms_to_build()
108​ # =====================
109​
99 def test_rpms_to_build(self): 110​ def test_rpms_to_build(self):
100 stub_koji = Dingus(getBuild__returns=None, 111​ stub_koji = Dingus(getBuild__returns=None,
101 getRPM__returns=None, 112​ getRPM__returns=None,
102 multiCall=create_multicall( 113​ multiCall=create_multicall(
103 [[self.ref_rpms[0]], [self.ref_rpms[1]]], 114​ [[self.ref_rpms[0]], [self.ref_rpms[1]]],
104 [[self.ref_build], [self.ref_build]])) 115​ [[self.ref_build], [self.ref_build]]))
105 116​
106 test_koji = koji_utils.KojiClient(stub_koji) 117​ test_koji = koji_utils.KojiClient(stub_koji)
Show All 35 Lines152 stub_koji = Dingus(getBuild__returns=None,
142 getRPM__returns=None, 153​ getRPM__returns=None,
143 multiCall=create_multicall([[self.ref_rpms[0]]], 154​ multiCall=create_multicall([[self.ref_rpms[0]]],
144 [[None]])) 155​ [[None]]))
145 156​
146 test_koji = koji_utils.KojiClient(stub_koji) 157​ test_koji = koji_utils.KojiClient(stub_koji)
147 with pytest.raises(exc.TaskotronRemoteError): 158​ with pytest.raises(exc.TaskotronRemoteError):
148 test_koji.rpms_to_build([self.ref_filename]) 159​ test_koji.rpms_to_build([self.ref_filename])
149 160​
161​ # ===================
162​ # nvr_to_urls()
163​ # ===================
164​
150 def test_get_urls(self): 165​ def test_get_urls(self):
151 stub_koji = Dingus(getBuild__returns=self.ref_build, 166​ stub_koji = Dingus(getBuild__returns=self.ref_build,
152 listRPMs__returns=self.ref_rpms) 167​ listRPMs__returns=self.ref_rpms)
153 test_koji = koji_utils.KojiClient(stub_koji) 168​ test_koji = koji_utils.KojiClient(stub_koji)
154 koji_baseurl = config.get_config().pkg_url 169​ koji_baseurl = config.get_config().pkg_url
155 170​
156 test_urls = test_koji.nvr_to_urls(self.ref_nvr) 171​ test_urls = test_koji.nvr_to_urls(self.ref_nvr)
157 for url in test_urls: 172​ for url in test_urls:
▲ Show 20 Lines • Show All 101 Lines • ▼ Show 20 Line(s)266 def test_arch_exclude_override_explicit(self):
259 # arch_exclude enabled 274​ # arch_exclude enabled
260 urls = test_koji.nvr_to_urls(self.ref_nvr, arches=ref_arch, arch_exclude=ref_exclude, 275​ urls = test_koji.nvr_to_urls(self.ref_nvr, arches=ref_arch, arch_exclude=ref_exclude,
261 src=True, debuginfo=True) 276​ src=True, debuginfo=True)
262 assert len(urls) == len([rpm for rpm in self.ref_rpms 277​ assert len(urls) == len([rpm for rpm in self.ref_rpms
263 if rpm['arch'] not in ref_exclude]) 278​ if rpm['arch'] not in ref_exclude])
264 for arch in ref_exclude: 279​ for arch in ref_exclude:
265 assert all([arch not in url for url in urls]) 280​ assert all([arch not in url for url in urls])
266 281​
267 282​ # ====================
268 # === get_nvr_rpms === 283​ # get_nvr_rpms()
284​ # ====================
269 285​
270 def test_get_nvr_rpms_simple(self, monkeypatch): 286​ def test_get_nvr_rpms_simple(self, monkeypatch):
271 '''NVR contains a few RPMs''' 287​ '''NVR contains a few RPMs'''
272 test_koji = koji_utils.KojiClient(Dingus()) 288​ test_koji = koji_utils.KojiClient(Dingus())
273 289​
274 stub_urls = [ 290​ stub_urls = [
275 'http://localhost/file1.rpm', 291​ 'http://localhost/file1.rpm',
276 'http://localhost/file2.rpm'] 292​ 'http://localhost/file2.rpm']
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Line(s)335 def test_get_nvr_rpms_development_profile(self, monkeypatch):
328 monkeypatch.setattr(config, '_config', config.Config) 344​ monkeypatch.setattr(config, '_config', config.Config)
329 345​
330 rpmdir = '/fake' 346​ rpmdir = '/fake'
331 test_koji.get_nvr_rpms(self.ref_nvr, dest=rpmdir) 347​ test_koji.get_nvr_rpms(self.ref_nvr, dest=rpmdir)
332 348​
333 call = stub_download.calls[0] 349​ call = stub_download.calls[0]
334 assert call[2]['cachedir'] is not None 350​ assert call[2]['cachedir'] is not None
335 351​
336 352​ # =======================
337 # === get_tagged_rpms === 353​ # get_tagged_rpms()
354​ # =======================
338 355​
339 def test_get_tagged_rpms_single(self, monkeypatch): 356​ def test_get_tagged_rpms_single(self, monkeypatch):
340 '''Single NVR in a tag''' 357​ '''Single NVR in a tag'''
341 stub_koji = Dingus(listTagged__returns=[self.ref_build]) 358​ stub_koji = Dingus(listTagged__returns=[self.ref_build])
342 test_koji = koji_utils.KojiClient(stub_koji) 359​ test_koji = koji_utils.KojiClient(stub_koji)
343 360​
344 stub_rpms = [ 361​ stub_rpms = [
345 '/fake/file1.rpm', 362​ '/fake/file1.rpm',
Show All 34 Lines390 def test_get_tagged_rpms_none(self, monkeypatch):
380 monkeypatch.setattr(file_utils, 'makedirs', lambda *args, **kwargs: None) 397​ monkeypatch.setattr(file_utils, 'makedirs', lambda *args, **kwargs: None)
381 398​
382 rpm_files = test_koji.get_tagged_rpms('some tag', dest=None) 399​ rpm_files = test_koji.get_tagged_rpms('some tag', dest=None)
383 400​
384 assert rpm_files == [] 401​ assert rpm_files == []
385 # get_nvr_rpms() should not be called 402​ # get_nvr_rpms() should not be called
386 assert stub_get_nvr_rpms.calls() == [] 403​ assert stub_get_nvr_rpms.calls() == []
387 404​
405​ # =====================
406​ # get_build_log()
407​ # =====================
408​
409​ def test_build_log_no_build(self):
410​ '''raise error if build info can't be retrieved'''
411​ stub_koji = mock.Mock()
412​ stub_koji.listBuildRPMs.side_effect = koji.GenericError()
413​ test_koji = koji_utils.KojiClient(stub_koji)
414​
415​ with pytest.raises(exc.TaskotronRemoteError):
416​ test_koji.get_build_log(self.ref_nvr, dest=None)
417​
418​ def test_build_log_add_binary_archs(self, monkeypatch):
419​ '''specifying basearch should query for concrete binary arches'''
420​ self.ref_arch = 'i686'
421​ ref_base_arch = 'i386'
422​ stub_koji = mock.Mock()
423​ stub_koji.listBuildRPMs.return_value = self.ref_rpms
424​ test_koji = koji_utils.KojiClient(stub_koji)
425​ stub_download = mock.Mock()
426​ monkeypatch.setattr(file_utils, 'download', stub_download)
427​
428​ test_koji.get_build_log(self.ref_nvr, dest=None, arches=[ref_base_arch])
429​
430​ assert stub_download.call_count == 1
431​ assert '/%s/' % self.ref_arch in stub_download.call_args[1]['url']
432​
433​ def test_build_log_do_nothing_different_arch(self, monkeypatch):
434​ '''if only unavailable arches are requested, do nothing and don't crash'''
435​ ref_arch = 'armv7hl' # not available in self.rpms
436​ stub_koji = mock.Mock()
437​ stub_koji.listBuildRPMs.return_value = self.ref_rpms
438​ test_koji = koji_utils.KojiClient(stub_koji)
439​ stub_download = mock.Mock()
440​ monkeypatch.setattr(file_utils, 'download', stub_download)
441​
442​ test_koji.get_build_log(self.ref_nvr, dest=None, arches=[ref_arch])
443​
444​ assert stub_download.call_count == 0
445​
446​ def test_build_log_dont_download_src(self, monkeypatch):
447​ '''src logs should not be attempted to be downloaded'''
448​ ref_archs = ('all', 'src')
449​ stub_koji = mock.Mock()
450​ stub_koji.listBuildRPMs.return_value = self.ref_rpms
451​ test_koji = koji_utils.KojiClient(stub_koji)
452​ stub_download = mock.Mock()
453​ monkeypatch.setattr(file_utils, 'download', stub_download)
454​
455​ test_koji.get_build_log(self.ref_nvr, dest=None, arches=ref_archs)
456​
457​ assert stub_download.call_count > 0
458​ for call in stub_download.call_args_list:
459​ assert '/src/' not in call[1]['url']
460​
461​ def test_build_log_return_value(self, monkeypatch):
462​ '''populate return values for working and failed downloads'''
463​ ref_dest = '/path/'
464​
465​ def fake_download(url, dirname, filename, *args, **kwargs):
466​ if '/noarch/' in url:
467​ raise exc.TaskotronRemoteError()
468​ else:
469​ return os.path.join(ref_dest, filename)
470​
471​ stub_koji = mock.Mock()
472​ stub_koji.listBuildRPMs.return_value = self.ref_rpms
473​ test_koji = koji_utils.KojiClient(stub_koji)
474​ stub_download = mock.Mock()
475​ stub_download.side_effect = fake_download
476​ monkeypatch.setattr(file_utils, 'download', stub_download)
477​
478​ retval = test_koji.get_build_log(self.ref_nvr, dest=ref_dest, arches=['all'])
479​
480​ assert stub_download.call_count == 2 # x86_64 and noarch
481​ assert retval['ok'] == [os.path.join(ref_dest, 'build.log.x86_64')]
482​ assert retval['error'] == ['noarch']
483​
484​ def test_build_log_correct_url(self, monkeypatch):
485​ '''the url to be downloaded must be created correctly'''
486​ stub_koji = mock.Mock()
487​ stub_koji.listBuildRPMs.return_value = self.ref_rpms
488​ test_koji = koji_utils.KojiClient(stub_koji)
489​ stub_download = mock.Mock()
490​ monkeypatch.setattr(file_utils, 'download', stub_download)
491​
492​ test_koji.get_build_log(self.ref_nvr, dest=None, arches=[self.ref_arch])
493​
494​ assert stub_download.call_count == 1
495​ call = stub_download.call_args
496​ for part in (self.ref_name, self.ref_version, self.ref_release, self.ref_arch):
497​ assert '/%s/' % part in call[1]['url']
498​ assert call[1]['url'].startswith(config.get_config().pkg_url)
499​ assert call[1]['url'].endswith('/build.log')
500​
388 501​
389class TestGetENVR(object): 502​class TestGetENVR(object):
390 503​
391 def test_epoch(self): 504​ def test_epoch(self):
392 '''Epoch included in build''' 505​ '''Epoch included in build'''
393 build = {'nvr': 'foo-1.2-3.fc20', 506​ build = {'nvr': 'foo-1.2-3.fc20',
394 'epoch': 1, 507​ 'epoch': 1,
395 } 508​ }
Show All 13 Lines