Differential D96

Use FedmsgConsumer.log instead of custom one
ClosedPublic
Actions

Authored by mkrizek on May 13 2014, 2:37 PM.

Details

Reviewers
ralph
tflink
Maniphest Tasks
T103: create service/daemon for taskotron trigger
Commits
rTRGRed9ad67182fb: Use FedmsgConsumer.log instead of custom one
Summary

Works on my system with fedmsg-hub.service. The logs go to /var/log/messages, we'd need to change /etc/fedmsg.d/logging.py to put them in a file but that's a change outside this repo.

Fixes T103

Test Plan

None

Diff Detail

Lint
Lint Skipped
Unit
Unit Tests Skipped
tflink added a comment.May 13 2014, 3:27 PM

This means we lose the stdout functionality but I suppose that tailing a file is functionally equivalent. Is there a sane way to use a fedmsg log file like @ralph was talking about instead of letting stuff go to /var/log/messages by default?

jobtriggers/config.py
27–29

I hadn't gotten around to filing a bug about that yet - thanks for fixing it

ralph added a comment.May 13 2014, 4:58 PM

Here's a thought: You won't get your log output until the entire trigger_job call is done. It builds a list of output strings and returns them at the end.

You could instead use yield statements in trigger_job to yield output lines and then modify KojiJobTrigger.consume to do:

output = self.trigger.trigger_job(envr, UNIT_TYPE, task, arch)
for line in output:
    self.log.info(line)
mkrizek added a comment.May 14 2014, 11:47 AM

Is there a sane way to use a fedmsg log file like @ralph was talking about instead of letting stuff go to /var/log/messages by default?

I guess we'd need to do this (/etc/fedmg.d/logging.py): {F2117}

tflink added a comment.May 14 2014, 2:18 PM

I guess we'd need to do this (/etc/fedmg.d/logging.py

That's not part of this diff, though. I wonder if we would be wise to include a logging.py in the source tree even if it wasn't installed with the package.

mkrizek updated this revision.May 15 2014, 2:13 PM
  • Use yield instead of returning all output at once
tflink requested changes to this revision.May 15 2014, 2:23 PM

Overall, the code looks good to me. One nit about the number of logging statements for buildbot but that's a minor detail.

I'd like to see the readme updated with suggested production changes to /etc/fedmsg.d/logging.py, though.

jobtriggers/triggers.py
63

I think that some of these logging statements are excessive. Status from login, force and logout in addition to url for the force request should be sufficient

mkrizek updated this revision.May 15 2014, 3:00 PM
  • readme: Add instructions
mkrizek updated this revision.May 15 2014, 3:03 PM
  • Fix typo
tflink accepted this revision.May 15 2014, 3:05 PM

Small wording change suggestion but other than that, looks good to me

readme.rst
32

one small wording suggestion: "Then replace the console handler with the filelog handler for the fedmsg logger so that log messages are sent to a separate file instead of to the console"

mkrizek closed this revision.May 15 2014, 3:17 PM

Closed by commit rTRGRed9ad67182fb (authored by @mkrizek).

Revision Contents

Diff 277

View Options

jobtriggers/bodhi_msg.py

1import logging ​
2import fedmsg 1​import fedmsg
3import fedmsg.encoding 2​import fedmsg.encoding
4import fedmsg.consumers 3​import fedmsg.consumers
5 4​
6from . import triggers, utils, config 5​from . import triggers, utils, config
7 6​
8 7​
9UNIT_TYPE = 'bodhi_id' 8​UNIT_TYPE = 'bodhi_id'
10 9​
11logging.basicConfig() ​
12log = logging.getLogger(__name__) ​
13 ​
14 ​
15class BodhiJobTrigger(fedmsg.consumers.FedmsgConsumer): 10​class BodhiJobTrigger(fedmsg.consumers.FedmsgConsumer):
16 topic = "org.fedoraproject.prod.bodhi.update.request.*" 11​ topic = "org.fedoraproject.prod.bodhi.update.request.*"
17 jsonify = False 12​ jsonify = False
18 config_key = 'taskotron.bodhijobtrigger.enabled' 13​ config_key = 'taskotron.bodhijobtrigger.enabled'
19 14​
20 def __init__(self, *args, **kw): 15​ def __init__(self, *args, **kw):
21 super(BodhiJobTrigger, self).__init__(*args, **kw) 16​ super(BodhiJobTrigger, self).__init__(*args, **kw)
22 self.trigger = triggers.get_trigger(config.trigger_type) 17​ self.trigger = triggers.get_trigger(config.trigger_type)
Show All 9 Lines19 def consume(self, message):
32 updatetitle = usable_message['msg']['update']['title'] 27​ updatetitle = usable_message['msg']['update']['title']
33 builds = usable_message['msg']['update']['builds'] 28​ builds = usable_message['msg']['update']['builds']
34 nvrs = [build['nvr'] for build in builds] 29​ nvrs = [build['nvr'] for build in builds]
35 30​
36 arches = utils.get_arches(nvr=nvrs[0]) 31​ arches = utils.get_arches(nvr=nvrs[0])
37 32​
38 for task in config.bodhi_tasks: 33​ for task in config.bodhi_tasks:
39 for arch in arches: 34​ for arch in arches:
40 log.info('triggering %s for %s on %s', task, updatetitle, arch) 35​ self.log.info('triggering %s for %s on %s', task, updatetitle, arch)
41 self.trigger.trigger_job(updatetitle, UNIT_TYPE, task, arch) 36​ output = self.trigger.trigger_job(updatetitle, UNIT_TYPE, task, arch)
37​ for line in output:
38​ self.log.info(line)
View Options

jobtriggers/config.py

Show All 18 Lines
19config.readfp(io.BytesIO(defaults)) 19​config.readfp(io.BytesIO(defaults))
20config.read(['/etc/taskotron/trigger.cfg', './conf/trigger.cfg']) 20​config.read(['/etc/taskotron/trigger.cfg', './conf/trigger.cfg'])
21 21​
22 22​
23buildbot_username = config.get('buildbot', 'username') 23​buildbot_username = config.get('buildbot', 'username')
24buildbot_password = config.get('buildbot', 'password') 24​buildbot_password = config.get('buildbot', 'password')
25buildbot_url = config.get('buildbot', 'url') 25​buildbot_url = config.get('buildbot', 'url')
26 26​
27koji_tasks = config.get('trigger', 'koji_tasks').split(',') 27​koji_tasks = [e.strip() for e in config.get('trigger', 'koji_tasks').split(',')]
28bodhi_tasks = config.get('trigger', 'bodhi_tasks').split(',') 28​bodhi_tasks = [e.strip() for e in config.get('trigger', 'bodhi_tasks').split(',')]
29valid_arches = config.get('trigger', 'valid_arches').split(',') 29​valid_arches = [e.strip() for e in config.get('trigger', 'valid_arches').split(',')]
tflinkUnsubmitted
Not Done

I hadn't gotten around to filing a bug about that yet - thanks for fixing it

30koji_url = config.get('trigger', 'koji_url') 30​koji_url = config.get('trigger', 'koji_url')
31trigger_type = config.get('trigger', 'type') 31​trigger_type = config.get('trigger', 'type')
View Options

jobtriggers/koji_msg.py

1import logging ​
2 ​
3import fedmsg 1​import fedmsg
4import fedmsg.encoding 2​import fedmsg.encoding
5import fedmsg.consumers 3​import fedmsg.consumers
6 4​
7from . import triggers, utils, config 5​from . import triggers, utils, config
8 6​
9 7​
10UNIT_TYPE = 'koji_build' 8​UNIT_TYPE = 'koji_build'
11 9​
12logging.basicConfig() ​
13log = logging.getLogger(__name__) ​
14 ​
15 ​
16class KojiJobTrigger(fedmsg.consumers.FedmsgConsumer): 10​class KojiJobTrigger(fedmsg.consumers.FedmsgConsumer):
17 topic = "org.fedoraproject.prod.buildsys.tag" 11​ topic = "org.fedoraproject.prod.buildsys.tag"
18 jsonify = False 12​ jsonify = False
19 config_key = 'taskotron.kojijobtrigger.enabled' 13​ config_key = 'taskotron.kojijobtrigger.enabled'
20 14​
21 def __init__(self, *args, **kw): 15​ def __init__(self, *args, **kw):
22 super(KojiJobTrigger, self).__init__(*args, **kw) 16​ super(KojiJobTrigger, self).__init__(*args, **kw)
23 self.trigger = triggers.get_trigger(config.trigger_type) 17​ self.trigger = triggers.get_trigger(config.trigger_type)
Show All 12 Lines19 def consume(self, message):
36 buildid = usable_message['msg']['build_id'] 30​ buildid = usable_message['msg']['build_id']
37 31​
38 envr = "%s-%s-%s" % (buildname, buildversion, buildrelease) 32​ envr = "%s-%s-%s" % (buildname, buildversion, buildrelease)
39 33​
40 arches = utils.get_arches(buildid=buildid) 34​ arches = utils.get_arches(buildid=buildid)
41 35​
42 for task in config.koji_tasks: 36​ for task in config.koji_tasks:
43 for arch in arches: 37​ for arch in arches:
44 log.info('triggering %s for %s on %s', task, envr, arch) 38​ self.log.info('triggering %s for %s on %s', task, envr, arch)
45 self.trigger.trigger_job(envr, UNIT_TYPE, task, arch) 39​ output = self.trigger.trigger_job(envr, UNIT_TYPE, task, arch)
40​ for line in output:
41​ self.log.info(line)
View Options

jobtriggers/triggers.py

1import pprint ​
2import requests 1​import requests
3 2​
4from . import config 3​from . import config
5 4​
6class BaseTrigger(object): 5​class BaseTrigger(object):
7 """ 6​ """
8 Format: 7​ Format:
9 { 8​ {
10 'item': <name of item - envr, update id, koji tag, etc.>, 9​ 'item': <name of item - envr, update id, koji tag, etc.>,
11 'item_type': <koji_build, koji_tag, bodhi_id>, 10​ 'item_type': <koji_build, koji_tag, bodhi_id>,
12 'taskname': <task to be scheduled>, 11​ 'taskname': <task to be scheduled>,
13 'arch': <arch needed for execution, x86_64, i386 etc.> 12​ 'arch': <arch needed for execution, x86_64, i386 etc.>
14 } 13​ }
15 """ 14​ """
16 def trigger_job(self, item, item_type, taskname, arch): 15​ def trigger_job(self, item, item_type, taskname, arch):
17 raise NotImplementedError 16​ raise NotImplementedError
18 17​
19 18​
20class StreamTrigger(BaseTrigger): 19​class StreamTrigger(BaseTrigger):
21 def trigger_job(self, item, item_type, taskname, arch): 20​ def trigger_job(self, item, item_type, taskname, arch):
22 job = {'item': item, 'item_type': item_type, 'taskname': taskname, 'arch': arch} 21​ output = ["StreamTrigger received:",
23 pprint.pprint(job) 22​ "item: %s" % item,
24 print 23​ "item_type: %s" % item_type,
24​ "taskname: %s" % taskname,
25​ "arch: %s" % arch]
26​
27​ yield '\n'.join(output)
25 28​
26 29​
27class BuildbotTrigger(BaseTrigger): 30​class BuildbotTrigger(BaseTrigger):
28 def trigger_job(self, item, item_type, taskname, arch): 31​ def trigger_job(self, item, item_type, taskname, arch):
29 job = {'item': item, 'item_type': item_type, 'taskname': taskname, 'arch': arch} 32​ output = ["Buildbot received:",
30 pprint.pprint(job) 33​ "item: %s" % item,
31 print 34​ "item_type: %s" % item_type,
35​ "taskname: %s" % taskname,
36​ "arch: %s" % arch]
32 37​
33 s = requests.session() 38​ yield '\n'.join(output)
34 39​
35 print "Sending Login request to: %s/login" % config.buildbot_url 40​ s = requests.session()
36 print ​
37 41​
38 r = s.post('%s/login' % config.buildbot_url, 42​ r = s.post('%s/login' % config.buildbot_url,
39 data={'username': config.buildbot_username, 43​ data={'username': config.buildbot_username,
40 'passwd': config.buildbot_password}) 44​ 'passwd': config.buildbot_password})
41 print "login request status: %s" % str(r.status_code) 45​
42# print "login request text: %s" % str(r.text) 46​ yield "Login request status: %s" % str(r.status_code)
43 print "" ​
44 47​
45 if arch not in ['i386', 'x86_64']: 48​ if arch not in ['i386', 'x86_64']:
46 builder = 'all' 49​ builder = 'all'
47 else: 50​ else:
48 builder = arch 51​ builder = arch
49 52​
50 r = s.post('%s/builders/%s/force' % (config.buildbot_url, builder), 53​ r = s.post('%s/builders/%s/force' % (config.buildbot_url, builder),
51 data={'forcescheduler': 'rpmcheck', 54​ data={'forcescheduler': 'rpmcheck',
52 'reason': 'triggered by fedmsg', 55​ 'reason': 'triggered by fedmsg',
53 'item': item, 56​ 'item': item,
54 'item_type': item_type, 57​ 'item_type': item_type,
55 'taskname': taskname, 58​ 'taskname': taskname,
56 'arch': arch}) 59​ 'arch': arch})
57 60​
58 print "force request status: %s" % str(r.status_code) 61​ yield "Force request ('%s/builders/%s/force') status: %s" % (config.buildbot_url, builder, str(r.status_code))
59# print "force request text: %s" % str(r.text) 62​
60 print "" 63​ r = s.get('%s/logout' % config.buildbot_url)
tflinkUnsubmitted
Not Done

I think that some of these logging statements are excessive. Status from login, force and logout in addition to url for the force request should be sufficient

61 64​
62 s.get('%s/logout' % config.buildbot_url) 65​ yield "Logout request status: %s" % str(r.status_code)
63 66​
64 print "force request status: %s" % str(r.status_code) ​
65# print "force request text: %s" % str(r.text) ​
66 print "" ​
67 67​
68_triggers = {'StreamTrigger': StreamTrigger, 'BuildbotTrigger': BuildbotTrigger} 68​_triggers = {'StreamTrigger': StreamTrigger, 'BuildbotTrigger': BuildbotTrigger}
69 69​
70def get_trigger(trigger_type): 70​def get_trigger(trigger_type):
71 return _triggers[trigger_type]() 71​ return _triggers[trigger_type]()
View Options

readme.rst

Show All 9 Lines
10Install the rpm dependencies on the system (only works on Fedora for now):: 10​Install the rpm dependencies on the system (only works on Fedora for now)::
11 11​
12 yum install fedmsg fedmsg-hub 12​ yum install fedmsg fedmsg-hub
13 13​
14Parts of the trigger do need to be installed on the local system:: 14​Parts of the trigger do need to be installed on the local system::
15 15​
16 python setup.py install 16​ python setup.py install
17 17​
18​Updating Production Logging Configuration
19​-----------------------------------------
20​
21​In /etc/fedmsg.d/logging, add the following handler::
22​
23​ filelog={
24​ "class": "logging.handlers.RotatingFileHandler",
25​ "formatter": "bare",
26​ "level": "INFO",
27​ "filename": "/var/log/fedmsg/taskotron-trigger.log",
28​ "mode": "a",
29​ }
30​
31​Then replace the console handler with the filelog handler for the fedmsg
32​logger so that log messages are sent to a separate file instead of to the
tflinkUnsubmitted
Not Done

one small wording suggestion: "Then replace the console handler with the filelog handler for the fedmsg logger so that log messages are sent to a separate file instead of to the console"

33​console::
34​
35​ fedmsg={
36​ "level": "INFO",
37​ "propagate": False,
38​ - "handlers": ["console"],
39​ + "handlers": ["filelog"],
40​ },
41​
18Running the Trigger 42​Running the Trigger
19------------------- 43​-------------------
20 44​
21Once the fedmsg component has been installed, run the following command in the 45​Once the fedmsg component has been installed, run the following command in the
22working directory with triggerjob.py:: 46​working directory with triggerjob.py::
23 47​
24 fedmsg-hub 48​ fedmsg-hub
25 49​
Show All 27 Lines