Differential D127

enhance taskotron-trigger to convert from bodhi_id to koji_tag where needed
ClosedPublic
Actions

Authored by mkrizek on Jun 5 2014, 2:30 PM.

Details

Reviewers
kparal
ralph
tflink
Maniphest Tasks
T200: enhance taskotron-trigger to convert from bodhi_id to koji_tag where needed
Commits
rTRGRb8e933db1c0b: enhance taskotron-trigger to convert from bodhi_id to koji_tag where needed
Summary

Fixes T200

Test Plan

Tests altered

Diff Detail

Repository
rTRGR taskotron-trigger
Lint
Lint Skipped
Unit
Unit Tests Skipped
mkrizek retitled this revision from to enhance taskotron-trigger to convert from bodhi_id to koji_tag where needed.Jun 5 2014, 2:30 PM
mkrizek updated this object.
mkrizek edited the test plan for this revision. (Show Details)
mkrizek added reviewers: tflink, kparal, ralph.
mkrizek added a task: T200: enhance taskotron-trigger to convert from bodhi_id to koji_tag where needed.
tflink requested changes to this revision.Jun 9 2014, 4:44 AM

Other than the concern about updates-testing, looks pretty good to me

jobtriggers/bodhi_msg.py
34

Is it safe to make this assumption? Updates that are still in updates-testing or pending push to updates-testing would be '%s-updates-testing', no?

This revision now requires changes to proceed.Jun 9 2014, 4:44 AM
mkrizek updated this revision to Diff 369.Jun 9 2014, 11:24 AM

Work with updates-testing as well

tflink added a comment.Jun 9 2014, 2:10 PM

I don't see any unit tests for the fXX-updates-testing case - am I missing something? If not, would you add a test or two to cover that case?

mkrizek updated this revision to Diff 370.Jun 9 2014, 2:23 PM

kojitag -testing test case

tflink accepted this revision.Jun 10 2014, 2:05 PM

Looks good to me

This revision is now accepted and ready to land.Jun 10 2014, 2:05 PM
tflink requested changes to this revision.Jun 12 2014, 2:33 PM

sorry for changing my mind on this but I was reading through it again and having "koji_tag_tasks" doesn't make sense to me since we aren't directly triggering on koji tag change, just triggering a job using the koji tag of an update when we receive a message about a changed update.

conf/trigger.cfg.example
8

I'd like to see this renamed to indicate that it's triggering with koji_tag but is triggered by a bodhi message. something like bodhi_to_koji_tag_tasks

This revision now requires changes to proceed.Jun 12 2014, 2:33 PM
tflink accepted this revision.Jun 13 2014, 2:18 PM

Looks good to me

This revision is now accepted and ready to land.Jun 13 2014, 2:18 PM
mkrizek closed this revision.Jun 13 2014, 3:00 PM
mkrizek updated this revision to Diff 408.

Closed by commit rTRGRb8e933db1c0b (authored by @mkrizek).

Revision Contents

Diff 408

View Options

conf/trigger.cfg.example

1[buildbot] 1​[buildbot]
2username = username 2​username = username
3password = password 3​password = password
4url = http://localhost/buildbot 4​url = http://localhost/buildbot
5 5​
6[trigger] 6​[trigger]
7koji_tasks = rpmlint 7​koji_tasks = rpmlint
8​bodhi_to_koji_tag_tasks = upgradepath
tflinkUnsubmitted
Not Done

I'd like to see this renamed to indicate that it's triggering with koji_tag but is triggered by a bodhi message. something like bodhi_to_koji_tag_tasks

8bodhi_tasks = examplebodhi 9​bodhi_tasks = examplebodhi
9valid_arches = i386,i686,x86_64,noarch 10​valid_arches = i386,i686,x86_64,noarch
10koji_url = http://koji.fedoraproject.org/kojihub 11​koji_url = http://koji.fedoraproject.org/kojihub
11type = StreamTrigger 12​type = StreamTrigger
12blacklist_releases = el5,el6 13​blacklist_releases = el5,el6
13job_logging = False 14​job_logging = False
14joblog_file = /var/log/taskotron-trigger/jobs.csv 15​joblog_file = /var/log/taskotron-trigger/jobs.csv
View Options

jobtriggers/bodhi_msg.py

1import fedmsg 1​import fedmsg
2import fedmsg.encoding 2​import fedmsg.encoding
3import fedmsg.consumers 3​import fedmsg.consumers
4 4​
5from . import triggers, utils, config 5​from . import triggers, utils, config
6 6​
7 7​
8ITEM_TYPE = 'bodhi_id' ​
9 ​
10class BodhiJobTrigger(fedmsg.consumers.FedmsgConsumer): 8​class BodhiJobTrigger(fedmsg.consumers.FedmsgConsumer):
11 topic = "org.fedoraproject.prod.bodhi.update.request.*" 9​ topic = "org.fedoraproject.prod.bodhi.update.request.*"
12 jsonify = False 10​ jsonify = False
13 config_key = 'taskotron.bodhijobtrigger.enabled' 11​ config_key = 'taskotron.bodhijobtrigger.enabled'
14 12​
15 def __init__(self, *args, **kw): 13​ def __init__(self, *args, **kw):
16 super(BodhiJobTrigger, self).__init__(*args, **kw) 14​ super(BodhiJobTrigger, self).__init__(*args, **kw)
17 self.trigger = triggers.get_trigger(config.trigger_type) 15​ self.trigger = triggers.get_trigger(config.trigger_type)
Show All 9 Lines17 def consume(self, message):
27 if not topic.endswith('stable') and not topic.endswith('testing'): 25​ if not topic.endswith('stable') and not topic.endswith('testing'):
28 return 26​ return
29 27​
30 # do not schledule jobs for blacklisted releases 28​ # do not schledule jobs for blacklisted releases
31 if nvrs[0].endswith(tuple(config.blacklist_releases)): 29​ if nvrs[0].endswith(tuple(config.blacklist_releases)):
32 return 30​ return
33 31​
34 updatetitle = usable_message['msg']['update']['title'] 32​ updatetitle = usable_message['msg']['update']['title']
33​ dist_tag = usable_message['msg']['update']['release']['dist_tag']
34​
tflinkUnsubmitted
Not Done

Is it safe to make this assumption? Updates that are still in updates-testing or pending push to updates-testing would be '%s-updates-testing', no?

35​ koji_tag = '%s-updates' % dist_tag
36​ if topic.endswith('testing'):
37​ koji_tag += '-testing'
35 38​
36 arches = utils.get_arches(nvr=nvrs[0]) 39​ arches = utils.get_arches(nvr=nvrs[0])
37 40​
38 for task in config.bodhi_tasks: 41​ tasks = []
42​ tasks += [(updatetitle, 'bodhi_id', task) for task in config.bodhi_tasks]
43​ tasks += [(koji_tag, 'koji_tag', task) for task in config.bodhi_to_koji_tag_tasks]
44​
45​ for (item, item_type, task) in tasks:
39 for arch in arches: 46​ for arch in arches:
40 self.log.info('triggering %s for %s on %s', task, updatetitle, arch) 47​ self.log.info('triggering %s for %s on %s', task, item, arch)
41 48​
42 if config.job_logging: 49​ if config.job_logging:
43 try: 50​ try:
44 utils.log_job(config.joblog_file, updatetitle, ITEM_TYPE, task, arch) 51​ utils.log_job(config.joblog_file, item, item_type, task, arch)
45 except IOError, e: 52​ except IOError, e:
46 self.log.warning('Cannot log a job into %s', config.joblog_file) 53​ self.log.warning('Cannot log a job into %s', config.joblog_file)
47 54​
48 output = self.trigger.trigger_job(updatetitle, ITEM_TYPE, task, arch) 55​ output = self.trigger.trigger_job(item, item_type, task, arch)
49 for line in output: 56​ for line in output:
50 self.log.info(line) 57​ self.log.info(line)
View Options

jobtriggers/config.py

1import ConfigParser 1​import ConfigParser
2import io 2​import io
3 3​
4defaults = """ 4​defaults = """
5[buildbot] 5​[buildbot]
6username = username 6​username = username
7password = password 7​password = password
8url = http://localhost/buildbot 8​url = http://localhost/buildbot
9 9​
10[trigger] 10​[trigger]
11koji_tasks = rpmlint 11​koji_tasks = rpmlint
12​bodhi_to_koji_tag_tasks = upgradepath
12bodhi_tasks = examplebodhi 13​bodhi_tasks = examplebodhi
13valid_arches = i386,i686,x86_64,noarch 14​valid_arches = i386,i686,x86_64,noarch
14koji_url = http://koji.fedoraproject.org/kojihub 15​koji_url = http://koji.fedoraproject.org/kojihub
15type = StreamTrigger 16​type = StreamTrigger
16blacklist_releases = el5,el6 17​blacklist_releases = el5,el6
17job_logging = False 18​job_logging = False
18joblog_file = /var/log/taskotron-trigger/jobs.cvs 19​joblog_file = /var/log/taskotron-trigger/jobs.cvs
19""" 20​"""
20 21​
21config = ConfigParser.ConfigParser() 22​config = ConfigParser.ConfigParser()
22config.readfp(io.BytesIO(defaults)) 23​config.readfp(io.BytesIO(defaults))
23config.read(['/etc/taskotron/trigger.cfg', './conf/trigger.cfg']) 24​config.read(['/etc/taskotron/trigger.cfg', './conf/trigger.cfg'])
24 25​
25 26​
26buildbot_username = config.get('buildbot', 'username') 27​buildbot_username = config.get('buildbot', 'username')
27buildbot_password = config.get('buildbot', 'password') 28​buildbot_password = config.get('buildbot', 'password')
28buildbot_url = config.get('buildbot', 'url') 29​buildbot_url = config.get('buildbot', 'url')
29 30​
30koji_tasks = [e.strip() for e in config.get('trigger', 'koji_tasks').split(',')] 31​koji_tasks = [e.strip() for e in config.get('trigger', 'koji_tasks').split(',')]
32​bodhi_to_koji_tag_tasks = [e.strip() for e in config.get('trigger', 'bodhi_to_koji_tag_tasks').split(',')]
31bodhi_tasks = [e.strip() for e in config.get('trigger', 'bodhi_tasks').split(',')] 33​bodhi_tasks = [e.strip() for e in config.get('trigger', 'bodhi_tasks').split(',')]
32valid_arches = [e.strip() for e in config.get('trigger', 'valid_arches').split(',')] 34​valid_arches = [e.strip() for e in config.get('trigger', 'valid_arches').split(',')]
33koji_url = config.get('trigger', 'koji_url') 35​koji_url = config.get('trigger', 'koji_url')
34trigger_type = config.get('trigger', 'type') 36​trigger_type = config.get('trigger', 'type')
35blacklist_releases = [e.strip() for e in config.get('trigger', 'blacklist_releases').split(',')] 37​blacklist_releases = [e.strip() for e in config.get('trigger', 'blacklist_releases').split(',')]
36job_logging = True if config.get('trigger', 'job_logging') == 'True' else False 38​job_logging = True if config.get('trigger', 'job_logging') == 'True' else False
37joblog_file = config.get('trigger', 'joblog_file') 39​joblog_file = config.get('trigger', 'joblog_file')
View Options

testing/test_bodhi_trigger.py

1import pytest 1​import pytest
2from dingus import Dingus 2​from dingus import Dingus
3from bunch import Bunch 3​from bunch import Bunch
4 4​
5from jobtriggers import bodhi_msg 5​from jobtriggers import bodhi_msg
6 6​
7 7​
8@pytest.mark.usefixtures('prepare') 8​@pytest.mark.usefixtures('prepare')
9class TestBodhiJobTrigger(): 9​class TestBodhiJobTrigger():
10 @pytest.fixture 10​ @pytest.fixture
11 def prepare(self, monkeypatch): 11​ def prepare(self, monkeypatch):
12 self.ref_updatetitle = 'foo' 12​ self.ref_updatetitle = 'foo'
13​ self.ref_dist_tag = 'f18'
14​ self.ref_koji_tag = '%s-updates' % self.ref_dist_tag
13 self.ref_nvr = "gnome-settings-daemon-3.6.1-1.fc18" 15​ self.ref_nvr = "gnome-settings-daemon-3.6.1-1.fc18"
14 self.ref_topic = 'org.fedoraproject.prod.bodhi.update.request.stable' 16​ self.ref_topic = 'org.fedoraproject.prod.bodhi.update.request.stable'
15 17​
16 self.ref_tasks = ['bodhi_task1', 'bodhi_task2'] 18​ self.ref_tasks = ['bodhi_task1', 'bodhi_task2']
17 self.ref_validarches = ['i386', 'x86_64'] 19​ self.ref_validarches = ['i386', 'x86_64']
18 self.ref_blacklist_releases= ['el5', 'el6'] 20​ self.ref_blacklist_releases= ['el5', 'el6']
19 21​
20 self._create_msg(self.ref_updatetitle, self.ref_nvr, self.ref_topic) 22​ self._create_msg(self.ref_updatetitle, self.ref_dist_tag, self.ref_nvr, self.ref_topic)
21 23​
22 stub_hub = Bunch(config=Bunch(get=0)) 24​ stub_hub = Bunch(config=Bunch(get=0))
23 self.helper = bodhi_msg.BodhiJobTrigger(stub_hub) 25​ self.helper = bodhi_msg.BodhiJobTrigger(stub_hub)
24 26​
25 self.helper.trigger = Dingus() 27​ self.helper.trigger = Dingus()
26 28​
27 monkeypatch.setattr(bodhi_msg.utils, 'get_arches', lambda *a, **k: self.ref_validarches) 29​ monkeypatch.setattr(bodhi_msg.utils, 'get_arches', lambda *a, **k: self.ref_validarches)
28 30​
29 bodhi_msg.config.bodhi_tasks = self.ref_tasks 31​ bodhi_msg.config.bodhi_tasks = self.ref_tasks
32​ bodhi_msg.config.bodhi_to_koji_tag_tasks = self.ref_tasks
30 bodhi_msg.config.valid_arches = self.ref_validarches 33​ bodhi_msg.config.valid_arches = self.ref_validarches
31 bodhi_msg.config.blacklist_releases = self.ref_blacklist_releases 34​ bodhi_msg.config.blacklist_releases = self.ref_blacklist_releases
32 bodhi_msg.config.job_logging = False 35​ bodhi_msg.config.job_logging = False
33 36​
34 def _create_msg(self, ref_updatetitle, ref_nvr, ref_topic): 37​ def _create_msg(self, ref_updatetitle, ref_dist_tag, ref_nvr, ref_topic):
35 self.ref_message = Bunch(body='{"msg": {"agent": "lmacken",\ 38​ self.ref_message = Bunch(body='{"msg": {"agent": "lmacken",\
36 "update": {"submitter": "lmacken",\ 39​ "update": {"submitter": "lmacken",\
37 "title": "%s",\ 40​ "title": "%s",\
41​ "release": {"dist_tag": "%s"},\
38 "builds": [{"nvr": "%s"}]}},\ 42​ "builds": [{"nvr": "%s"}]}},\
39 "topic": "%s"}' % 43​ "topic": "%s"}' %
40 (ref_updatetitle, ref_nvr, ref_topic)) 44​ (ref_updatetitle, ref_dist_tag, ref_nvr, ref_topic))
41 45​
42 def test_consume(self): 46​ def test_consume(self):
43 self.helper.consume(self.ref_message) 47​ self.helper.consume(self.ref_message)
44 48​
45 trigger_calls = self.helper.trigger.trigger_job.calls() 49​ trigger_calls = self.helper.trigger.trigger_job.calls()
46 50​
47 assert len(trigger_calls) == 4 51​ assert len(trigger_calls) == 8
48 52​
49 assert trigger_calls[0][1] == (self.ref_updatetitle, bodhi_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[0]) 53​ assert trigger_calls[0][1] == (self.ref_updatetitle, 'bodhi_id', self.ref_tasks[0], self.ref_validarches[0])
50 assert trigger_calls[1][1] == (self.ref_updatetitle, bodhi_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[1]) 54​ assert trigger_calls[1][1] == (self.ref_updatetitle, 'bodhi_id', self.ref_tasks[0], self.ref_validarches[1])
51 assert trigger_calls[2][1] == (self.ref_updatetitle, bodhi_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_validarches[0]) 55​ assert trigger_calls[2][1] == (self.ref_updatetitle, 'bodhi_id', self.ref_tasks[1], self.ref_validarches[0])
52 assert trigger_calls[3][1] == (self.ref_updatetitle, bodhi_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_validarches[1]) 56​ assert trigger_calls[3][1] == (self.ref_updatetitle, 'bodhi_id', self.ref_tasks[1], self.ref_validarches[1])
57​ assert trigger_calls[4][1] == (self.ref_koji_tag, 'koji_tag', self.ref_tasks[0], self.ref_validarches[0])
58​ assert trigger_calls[5][1] == (self.ref_koji_tag, 'koji_tag', self.ref_tasks[0], self.ref_validarches[1])
59​ assert trigger_calls[6][1] == (self.ref_koji_tag, 'koji_tag', self.ref_tasks[1], self.ref_validarches[0])
60​ assert trigger_calls[7][1] == (self.ref_koji_tag, 'koji_tag', self.ref_tasks[1], self.ref_validarches[1])
53 61​
54 def test_consume_notwatched_topic(self): 62​ def test_consume_notwatched_topic(self):
55 ref_topic = 'org.fedoraproject.pro.bodhi.update.request.randomtopic' 63​ ref_topic = 'org.fedoraproject.pro.bodhi.update.request.randomtopic'
56 self._create_msg(self.ref_updatetitle, self.ref_nvr, ref_topic) 64​ self._create_msg(self.ref_updatetitle, self.ref_dist_tag, self.ref_nvr, ref_topic)
57 65​
58 self.helper.consume(self.ref_message) 66​ self.helper.consume(self.ref_message)
59 67​
60 trigger_calls = self.helper.trigger.trigger_job.calls() 68​ trigger_calls = self.helper.trigger.trigger_job.calls()
61 69​
62 assert trigger_calls == [] 70​ assert trigger_calls == []
63 71​
64 def test_consume_blacklisted_release(self): 72​ def test_consume_blacklisted_release(self):
65 ref_nvr = 'gnome-settings-daemon-3.6.1-1.el5' 73​ ref_nvr = 'gnome-settings-daemon-3.6.1-1.el5'
66 self._create_msg(self.ref_updatetitle, ref_nvr, self.ref_topic) 74​ self._create_msg(self.ref_updatetitle, self.ref_dist_tag, ref_nvr, self.ref_topic)
67 75​
68 self.helper.consume(self.ref_message) 76​ self.helper.consume(self.ref_message)
69 77​
70 trigger_calls = self.helper.trigger.trigger_job.calls() 78​ trigger_calls = self.helper.trigger.trigger_job.calls()
71 79​
72 assert trigger_calls == [] 80​ assert trigger_calls == []
81​
82​ def test_consume_kojitag_testing(self):
83​ ref_topic = 'org.fedoraproject.pro.bodhi.update.request.testing'
84​ self._create_msg(self.ref_updatetitle, self.ref_dist_tag, self.ref_nvr, ref_topic)
85​
86​ self.helper.consume(self.ref_message)
87​
88​ trigger_calls = self.helper.trigger.trigger_job.calls()
89​
90​ assert trigger_calls[4][1][0] == self.ref_koji_tag + '-testing'