Differential D506

adjust for Bodhi 2.0
ClosedPublic
Actions

Authored by kparal on Aug 18 2015, 3:28 PM.

Details

Reviewers
lbrabec
tflink
jskladan
Maniphest Tasks
T558: Test libtaskotron against bodhi 2.0
Commits
rLTRNbbf9ec8e1515: adjust for Bodhi 2.0
Summary

A lot of changes, some temporary:

  • bodhi_utils.build2update() has been adjusted to the new API
  • bodhi_utils.query_update() is blocked on https://github.com/fedora-infra/bodhi/issues/164 , so it has been disabled. Nothing seems to use it ATM except for bodhi directive, which in turn nothing uses, so we have some time to fix this. We might remove the whole method completely.
  • fas_password can't be non-empty anymore, auth fails. We will need to deal with fake_fedorainfra issues (doesn't accept empty password) in a different way.
Test Plan

test suite works, and upgradepath and depcheck when run manually also seem to work

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to adjust for Bodhi 2.0.Aug 18 2015, 3:28 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added reviewers: tflink, jskladan, lbrabec.
kparal added a task: T558: Test libtaskotron against bodhi 2.0.
tflink accepted this revision.Aug 18 2015, 8:28 PM

Other than the question about == vs >= for the python-fedora version in requirements.txt, looks good to me

requirements.txt
19–20

I know why all this was changed but I suspect that there's going to be more churn in python-fedora soon, maybe a >= 0.5.2 would be better here for the near future?

This revision is now accepted and ready to land.Aug 18 2015, 8:28 PM
Closed by commit rLTRNbbf9ec8e1515: adjust for Bodhi 2.0 (authored by kparal). · Explain WhyAug 18 2015, 8:40 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 1334

View Options

conf/taskotron.yaml.example

Show First 20 Lines • Show All 132 Lines • ▼ Show 20 Line(s)
133 133​
134## ==== SECRETS section ==== 134​## ==== SECRETS section ====
135## All login credentials and other secrets are here. If you add some secret 135​## All login credentials and other secrets are here. If you add some secret
136## here, make sure you make this file readable just for the right user accounts. 136​## here, make sure you make this file readable just for the right user accounts.
137 137​
138## FAS (Fedora Accounts System) credentials 138​## FAS (Fedora Accounts System) credentials
139## These credentials are used when reporting results into Bodhi. 139​## These credentials are used when reporting results into Bodhi.
140#fas_username: taskotron 140​#fas_username: taskotron
141#fas_password: password 141​#fas_password: ''
142​
View Options

libtaskotron.spec

Show All 15 Lines
16Requires: createrepo 16​Requires: createrepo
17Requires: koji 17​Requires: koji
18Requires: libtaskotron-config 18​Requires: libtaskotron-config
19Requires: mash 19​Requires: mash
20Requires: pyOpenSSL 20​Requires: pyOpenSSL
21Requires: pytap13 >= 0.3.0 21​Requires: pytap13 >= 0.3.0
22Requires: python-bayeux >= 0.9 22​Requires: python-bayeux >= 0.9
23Requires: python-bunch >= 1.0.1 23​Requires: python-bunch >= 1.0.1
24Requires: python-fedora >= 0.3.36 24​Requires: python-fedora >= 0.5.2
25Requires: python-hawkey >= 0.4.13-1 25​Requires: python-hawkey >= 0.4.13-1
26Requires: python-pycurl 26​Requires: python-pycurl
27Requires: python-setuptools 27​Requires: python-setuptools
28Requires: python-urlgrabber 28​Requires: python-urlgrabber
29Requires: python-yamlish >= 0.18 29​Requires: python-yamlish >= 0.18
30Requires: PyYAML >= 3.11 30​Requires: PyYAML >= 3.11
31Requires: resultsdb_api >= 1.2.1 31​Requires: resultsdb_api >= 1.2.1
32Requires: rpm-python 32​Requires: rpm-python
33BuildRequires: koji 33​BuildRequires: koji
34BuildRequires: mash 34​BuildRequires: mash
35BuildRequires: pytap13 >= 0.3.0 35​BuildRequires: pytap13 >= 0.3.0
36BuildRequires: pytest >= 2.6.4 36​BuildRequires: pytest >= 2.6.4
37BuildRequires: python-bayeux >= 0.9 37​BuildRequires: python-bayeux >= 0.9
38BuildRequires: python-bunch >= 1.0.1 38​BuildRequires: python-bunch >= 1.0.1
39BuildRequires: python-devel 39​BuildRequires: python-devel
40BuildRequires: python-dingus >= 0.3.4 40​BuildRequires: python-dingus >= 0.3.4
41BuildRequires: python-fedora >= 0.3.36 41​BuildRequires: python-fedora >= 0.5.2
42BuildRequires: python-hawkey >= 0.4.13-1 42​BuildRequires: python-hawkey >= 0.4.13-1
43BuildRequires: python-setuptools 43​BuildRequires: python-setuptools
44BuildRequires: python-urlgrabber 44​BuildRequires: python-urlgrabber
45BuildRequires: PyYAML >= 3.11 45​BuildRequires: PyYAML >= 3.11
46BuildRequires: resultsdb_api >= 1.2.1 46​BuildRequires: resultsdb_api >= 1.2.1
47 47​
48 48​
49%description 49​%description
▲ Show 20 Lines • Show All 171 Lines • Show Last 20 Lines
View Options

libtaskotron/bodhi_utils.py

Show All 15 Lines
16 16​
17 17​
18class BodhiUtils(object): 18​class BodhiUtils(object):
19 '''Helper Bodhi methods. 19​ '''Helper Bodhi methods.
20 20​
21 :ivar fedora.client.BodhiClient client: Bodhi client instance 21​ :ivar fedora.client.BodhiClient client: Bodhi client instance
22 ''' 22​ '''
23 23​
24 #: How many requests to make in a single time when using "multicall"-like 24​ #: How many requests to make in a single call. The maximum page limit is
25 #: Bodhi methods. '50' is a good size, there is a 2 minute timeout and it 25​ #: 100, so it can't be above that. With high numbers, we increase the
26 #: usually takes about 20 seconds for Bodhi to evaluate 50 requests, so 26​ #: chance to receive a timeout.
27 #: there's still a lot of buffer (increasing the size further doesn't ​
28 #: seem to speed it up noticeably) ​
29 _MULTICALL_REQUEST_SIZE = 50 27​ _MULTICALL_REQUEST_SIZE = 50
30 28​
31 def __init__(self, client=None): 29​ def __init__(self, client=None):
32 '''Create a new BodhiUtils instance. 30​ '''Create a new BodhiUtils instance.
33 31​
34 :param client: custom :class:`BodhiClient` instance. If ``None``, a 32​ :param client: custom :class:`BodhiClient` instance. If ``None``, a
35 default BodhiClient instance is used. 33​ default BodhiClient instance is used.
36 ''' 34​ '''
Show All 19 Lines51 :param str package: package NVR or package name or update title or
56 .. note:: 54​ .. note::
57 Only NVR allowed, not NEVR. See 55​ Only NVR allowed, not NEVR. See
58 https://fedorahosted.org/bodhi/ticket/592. 56​ https://fedorahosted.org/bodhi/ticket/592.
59 :return: most recent Bodhi update object matching criteria, or ``None`` 57​ :return: most recent Bodhi update object matching criteria, or ``None``
60 when no update is found 58​ when no update is found
61 :rtype: :class:`bunch.Bunch` 59​ :rtype: :class:`bunch.Bunch`
62 ''' 60​ '''
63 log.debug('Searching Bodhi updates for: %s', package) 61​ log.debug('Searching Bodhi updates for: %s', package)
62​ # FIXME: https://github.com/fedora-infra/bodhi/issues/164
63​ raise NotImplementedError('This has been broken when upgrading to Bodhi 2. Fix pending.')
64 res = self.client.query(package=package, limit=1) 64​ res = self.client.query(package=package, limit=1)
65 if res['updates']: 65​ if res['updates']:
66 return res['updates'][0] 66​ return res['updates'][0]
67 else: 67​ else:
68 return None 68​ return None
69 69​
70 def build2update(self, builds, strict=False): 70​ def build2update(self, builds, strict=False):
71 '''Find matching Bodhi updates for provided builds. 71​ '''Find matching Bodhi updates for provided builds.
▲ Show 20 Lines • Show All 44 Lines • ▼ Show 20 Line(s)
116 builds_queue = list(builds_nvr) 116​ builds_queue = list(builds_nvr)
117 117​
118 log.info('Querying Bodhi to map %d builds to their updates...', len(builds)) 118​ log.info('Querying Bodhi to map %d builds to their updates...', len(builds))
119 119​
120 # retrieve all update data 120​ # retrieve all update data
121 while builds_queue: 121​ while builds_queue:
122 builds_chunk = builds_queue[:self._MULTICALL_REQUEST_SIZE] 122​ builds_chunk = builds_queue[:self._MULTICALL_REQUEST_SIZE]
123 builds_chunk = ' '.join(builds_chunk) 123​ builds_chunk = ' '.join(builds_chunk)
124 # this method doesn't seem to be exposed or documented anywhere, it's 124​ res = self.client.query(builds=builds_chunk)
125 # just available in the Bodhi source code: 125​
126 # https://github.com/fedora-infra/bodhi/blob/master/bodhi/controllers.py#L202-L220 126​ updates.extend(res['updates'])
127 # For a space-delimited list of builds in format of NVR, it returns a ​
128 # dictionary in format of {build: update_data}. ​
129 res = self.client.send_request(method='get_updates_from_builds', ​
130 req_params={'builds': builds_chunk}) ​
131 # there is a bug - it contains an extra item "tg_flash=None" ​
132 # https://github.com/fedora-infra/bodhi/issues/152 ​
133 res.pop('tg_flash', None) ​
134 ​
135 updates.extend(res.values()) ​
136 builds_queue = builds_queue[self._MULTICALL_REQUEST_SIZE:] 127​ builds_queue = builds_queue[self._MULTICALL_REQUEST_SIZE:]
137 128​
138 # don't query for builds which were already found 129​ # don't query for builds which were already found
139 for update in res.values(): 130​ for update in res['updates']:
140 for build in update['builds']: 131​ for build in update['builds']:
141 if build['nvr'] in builds_queue: 132​ if build['nvr'] in builds_queue:
142 builds_queue.remove(build['nvr']) 133​ builds_queue.remove(build['nvr'])
143 134​
144 log.info('Bodhi queries done: %d/%d', len(builds_nvr)-len(builds_queue), 135​ log.info('Bodhi queries done: %d/%d', len(builds_nvr)-len(builds_queue),
145 len(builds_nvr)) 136​ len(builds_nvr))
146 137​
147 # separate builds into OK and failures 138​ # separate builds into OK and failures
Show All 29 Lines
View Options

libtaskotron/config_defaults.py

Show First 20 Lines • Show All 71 Lines • ▼ Show 20 Line(s)23class Config(object):
72 cachedir = '/var/cache/taskotron' #: 72​ cachedir = '/var/cache/taskotron' #:
73 73​
74 log_level_stream = 'INFO' #: 74​ log_level_stream = 'INFO' #:
75 log_level_file = 'DEBUG' #: 75​ log_level_file = 'DEBUG' #:
76 76​
77 log_file_enabled = False #: 77​ log_file_enabled = False #:
78 78​
79 fas_username = 'taskotron' #: 79​ fas_username = 'taskotron' #:
80 fas_password = 'password' #: 80​ fas_password = None #:
81 81​
82 def __str__(self): 82​ def __str__(self):
83 ''' Make this object more readable when printing ''' 83​ ''' Make this object more readable when printing '''
84 return '<%s: %s>' % (self.__class__.__name__, 84​ return '<%s: %s>' % (self.__class__.__name__,
85 pprint.pformat(vars(self))) 85​ pprint.pformat(vars(self)))
86 86​
87 87​
88class ProductionConfig(Config): 88​class ProductionConfig(Config):
Show All 28 Lines
View Options

requirements.txt

Show All 10 Lines
11# you can define a dependency range here instead of an exact version. 11​# you can define a dependency range here instead of an exact version.
12 12​
13bayeux == 0.9 13​bayeux == 0.9
14bunch == 1.0.1 14​bunch == 1.0.1
15dingus == 0.3.4 15​dingus == 0.3.4
16pytap13 == 0.3.0 16​pytap13 == 0.3.0
17pytest == 2.6.4 17​pytest == 2.6.4
18pytest-cov == 1.8.1 18​pytest-cov == 1.8.1
19python-fedora == 0.3.36 19​# heavy changes in python-fedora these days, temporarily use >= instead ==
20​python-fedora >= 0.5.2
tflinkUnsubmitted
Not Done

I know why all this was changed but I suspect that there's going to be more churn in python-fedora soon, maybe a >= 0.5.2 would be better here for the near future?

20PyYAML == 3.11 21​PyYAML == 3.11
21resultsdb_api == 1.2.1 22​resultsdb_api == 1.2.1
22Sphinx == 1.2.3 23​Sphinx == 1.2.3
23yamlish == 0.18 24​yamlish == 0.18
View Options

testing/test_bodhi_utils.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2014, Red Hat, Inc. 2​# Copyright 2009-2014, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6'''Unit tests for libtaskotron/bodhi_utils.py''' 6​'''Unit tests for libtaskotron/bodhi_utils.py'''
7 7​
8import pytest 8​import pytest
9from dingus import Dingus 9​from dingus import Dingus
10from bunch import Bunch 10​from munch import Munch
11 11​
12from libtaskotron import bodhi_utils 12​from libtaskotron import bodhi_utils
13from libtaskotron import exceptions as exc 13​from libtaskotron import exceptions as exc
14 14​
15 15​
16class TestQueryUpdate(): 16​# FIXME: once query_update() does not throw NotImplementedError
17 '''Test query_update()''' 17​# class TestQueryUpdate():
18 18​# '''Test query_update()'''
19 def setup_method(self, method): 19​#
20 '''Run this before every test invocation''' 20​# def setup_method(self, method):
21 self.ref_bodhi_id = 'FEDORA-1234-56789' 21​# '''Run this before every test invocation'''
22 self.ref_update = {'title': 'Random update', 22​# self.ref_bodhi_id = 'FEDORA-1234-56789'
23 'builds': [{'nvr': 'foo-1.2-3.fc99', 23​# self.ref_update = {'title': 'Random update',
24 'package': {} 24​# 'builds': [{'nvr': 'foo-1.2-3.fc99',
25 }], 25​# 'package': {}
26 'other_keys': {}, 26​# }],
27 } 27​# 'other_keys': {},
28 28​# }
29 def test_query_existing_update(self): 29​#
30 '''Test query for existing update''' 30​# def test_query_existing_update(self):
31 ref_query_answer = {'tg_flash': None, 31​# '''Test query for existing update'''
32 'num_items': 1, 32​# ref_query_answer = {'tg_flash': None,
33 'title': '1 update found', 33​# 'num_items': 1,
34 'updates': [self.ref_update], 34​# 'title': '1 update found',
35 } 35​# 'updates': [self.ref_update],
36 stub_bodhi = Dingus(query__returns=ref_query_answer) 36​# }
37 bodhi = bodhi_utils.BodhiUtils(client=stub_bodhi) 37​# stub_bodhi = Dingus(query__returns=ref_query_answer)
38 update = bodhi.query_update(self.ref_bodhi_id) 38​# bodhi = bodhi_utils.BodhiUtils(client=stub_bodhi)
39 39​# update = bodhi.query_update(self.ref_bodhi_id)
40 assert update == self.ref_update 40​#
41 41​# assert update == self.ref_update
42 def test_query_non_existing_update(self, monkeypatch): 42​#
43 '''Test query for non-existing update''' 43​# def test_query_non_existing_update(self, monkeypatch):
44 ref_query_answer = {'tg_flash': None, 44​# '''Test query for non-existing update'''
45 'num_items': 0, 45​# ref_query_answer = {'tg_flash': None,
46 'title': '0 updates found', 46​# 'num_items': 0,
47 'updates': [], 47​# 'title': '0 updates found',
48 } 48​# 'updates': [],
49 stub_bodhi = Dingus(query__returns=ref_query_answer) 49​# }
50 bodhi = bodhi_utils.BodhiUtils(client=stub_bodhi) 50​# stub_bodhi = Dingus(query__returns=ref_query_answer)
51 update = bodhi.query_update(self.ref_bodhi_id) 51​# bodhi = bodhi_utils.BodhiUtils(client=stub_bodhi)
52 52​# update = bodhi.query_update(self.ref_bodhi_id)
53 assert update is None 53​#
54​# assert update is None
54 55​
55 56​
56@pytest.mark.usefixtures('setup') 57​@pytest.mark.usefixtures('setup')
57class TestBuild2Update(): 58​class TestBuild2Update():
58 '''Test build2update()''' 59​ '''Test build2update()'''
59 60​
60 # fake requests and responses for query_update() 61​ # fake requests and responses for query_update()
62​ multi1 = Munch(builds=[Munch(nvr=u'multi1-a-1.0-1.fc20'),
63​ Munch(nvr=u'multi1-b-2.0-2.fc20')])
64​ multi2 = Munch(builds=[Munch(nvr=u'multi2-a-1.0-1.fc20'),
65​ Munch(nvr=u'multi2-b-2.0-2.fc20')])
61 requests = { 66​ requests = {
62 'foo-ok-1.2-3.fc20': Bunch(builds=[Bunch(nvr=u'foo-ok-1.2-3.fc20')]), 67​ 'foo-ok-1.2-3.fc20': Munch(builds=[Munch(nvr=u'foo-ok-1.2-3.fc20')]),
63 'bar-ok-1.8.0-4.fc20': 68​ 'bar-ok-1.8.0-4.fc20': Munch(builds=[Munch(nvr=u'bar-ok-1.8.0-4.fc20')]),
64 Bunch(builds=[Bunch(nvr=u'bar-ok-1.8.0-4.fc20')]), 69​ 'multi1-a-1.0-1.fc20': multi1,
65 'multi1-a-1.0-1.fc20': 70​ 'multi1-b-2.0-2.fc20': multi1,
66 Bunch(builds=[Bunch(nvr=u'multi1-a-1.0-1.fc20'), 71​ 'multi2-a-1.0-1.fc20': multi2,
67 Bunch(nvr=u'multi1-b-2.0-2.fc20')]), 72​ 'multi2-b-2.0-2.fc20': multi2,
68 'multi1-b-2.0-2.fc20': ​
69 Bunch(builds=[Bunch(nvr=u'multi1-a-1.0-1.fc20'), ​
70 Bunch(nvr=u'multi1-b-2.0-2.fc20')]), ​
71 'multi2-a-1.0-1.fc20': ​
72 Bunch(builds=[Bunch(nvr=u'multi2-a-1.0-1.fc20'), ​
73 Bunch(nvr=u'multi2-b-2.0-2.fc20')]), ​
74 'multi2-b-2.0-2.fc20': ​
75 Bunch(builds=[Bunch(nvr=u'multi2-a-1.0-1.fc20'), ​
76 Bunch(nvr=u'multi2-b-2.0-2.fc20')]), ​
77 } 73​ }
78 74​
79 def mock_get_updates_from_builds(self, method, req_params): 75​ def mock_query(self, builds):
80 '''Mock bodhi.client.get_updates_from_builds()''' 76​ '''Mock bodhi.client.query()'''
81 # this will allow us to track calls 77​ # this will allow us to track calls
82 self.send_request_dingus.mock_get_updates_from_builds(method, req_params) 78​ self.query_dingus.mock_query(builds)
83 assert method == 'get_updates_from_builds' 79​ builds = builds.split()
84 builds = req_params['builds'].split() 80​ updates = []
85 response = {} ​
86 for build in builds: 81​ for build in builds:
87 if build in self.requests: 82​ if build in self.requests:
88 response[build] = self.requests[build] 83​ if self.requests[build] not in updates:
89 return response 84​ updates.append(self.requests[build])
85​ return Munch(updates=updates)
90 86​
91 @pytest.fixture 87​ @pytest.fixture
92 def setup(self, monkeypatch): 88​ def setup(self, monkeypatch):
93 '''Run this before every test invocation''' 89​ '''Run this before every test invocation'''
94 self.client = Dingus() 90​ self.client = Dingus()
95 self.bodhi = bodhi_utils.BodhiUtils(self.client) 91​ self.bodhi = bodhi_utils.BodhiUtils(self.client)
96 # replace send_request() with a fake function 92​ # replace send_request() with a fake function
97 self.client.send_request = self.mock_get_updates_from_builds 93​ self.client.query = self.mock_query
98 # this will allow us to track calls 94​ # this will allow us to track calls
99 self.send_request_dingus = Dingus() 95​ self.query_dingus = Dingus()
100 96​
101 def test_basic(self): 97​ def test_basic(self):
102 '''One update per build, mixed results''' 98​ '''One update per build, mixed results'''
103 updates, failures = self.bodhi.build2update(['foo-ok-1.2-3.fc20', 99​ updates, failures = self.bodhi.build2update(['foo-ok-1.2-3.fc20',
104 'bar-fail-1.8.0-4.fc20']) 100​ 'bar-fail-1.8.0-4.fc20'])
105 101​
106 assert len(updates) == 1 102​ assert len(updates) == 1
107 assert 'foo-ok-1.2-3.fc20' in updates 103​ assert 'foo-ok-1.2-3.fc20' in updates
▲ Show 20 Lines • Show All 94 Lines • ▼ Show 20 Line(s)195 def test_more_requests_efficient(self):
202 # decrease the request size to make sure we make several calls 198​ # decrease the request size to make sure we make several calls
203 self.bodhi._MULTICALL_REQUEST_SIZE = 1 199​ self.bodhi._MULTICALL_REQUEST_SIZE = 1
204 200​
205 updates, failures = self.bodhi.build2update(['multi1-a-1.0-1.fc20', 201​ updates, failures = self.bodhi.build2update(['multi1-a-1.0-1.fc20',
206 'multi1-b-2.0-2.fc20', 202​ 'multi1-b-2.0-2.fc20',
207 'multi2-a-1.0-1.fc20', 203​ 'multi2-a-1.0-1.fc20',
208 'foo-fail-1.2-3.fc20']) 204​ 'foo-fail-1.2-3.fc20'])
209 205​
210 calls = self.send_request_dingus.calls() 206​ calls = self.query_dingus.calls()
211 # 3 calls, before one of multi1 packages should be skipped 207​ # 3 calls, before one of multi1 packages should be skipped
212 assert len(calls) == 3 208​ assert len(calls) == 3
213 209​
214 assert len(updates) == 3 210​ assert len(updates) == 3
215 assert 'multi1-a-1.0-1.fc20' in updates 211​ assert 'multi1-a-1.0-1.fc20' in updates
216 assert 'multi1-b-2.0-2.fc20' in updates 212​ assert 'multi1-b-2.0-2.fc20' in updates
217 assert (updates['multi1-a-1.0-1.fc20'] == updates['multi1-b-2.0-2.fc20'] == 213​ assert (updates['multi1-a-1.0-1.fc20'] == updates['multi1-b-2.0-2.fc20'] ==
218 self.requests['multi1-a-1.0-1.fc20']) 214​ self.requests['multi1-a-1.0-1.fc20'])
219 # incomplete updates are allowed without strict=True 215​ # incomplete updates are allowed without strict=True
220 assert 'multi2-a-1.0-1.fc20' in updates 216​ assert 'multi2-a-1.0-1.fc20' in updates
221 217​
222 assert len(failures) == 1 218​ assert len(failures) == 1
223 assert 'foo-fail-1.2-3.fc20' in failures 219​ assert 'foo-fail-1.2-3.fc20' in failures
224 assert failures['foo-fail-1.2-3.fc20'] is None 220​ assert failures['foo-fail-1.2-3.fc20'] is None