Differential D1180

Add MBS (Module Build Service) consumer
ClosedPublic
Actions

Authored by mkrizek on Mar 27 2017, 6:10 PM.

Details

Reviewers
rasibley
ralph
tflink
Group Reviewers
taskotron-trigger
Maniphest Tasks
T915: Create new taskotron trigger ability for responding to new module builds
Commits
rTRGRa9d20b0d8657: Add MBS (Module Build Service) consumer
Test Plan

Unit tests included. More of a "is this what we want" review, I guess more testing will be done when more fedmsgs will be triggered.

Diff Detail

Repository
rTRGR taskotron-trigger
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
mkrizek created this revision.Mar 27 2017, 6:10 PM
tflink added a comment.Mar 29 2017, 11:17 AM

It looks reasonable to me.

@ralph - does this seem to match what you were looking for?

jscotka added a comment.EditedMar 29 2017, 1:15 PM

Hi,
I'm waiting for these messages to be able to use them in taskotron.
I'm doing modularity testing framework https://pagure.io/modularity-testing-framework
what should do testing of build modules and for now I would like to use these messages to trigger that.
I have own parsing lib (it creates for me REPO URL ans some other args, also stores modulemd file to file) https://pagure.io/modularity-testing-framework/blob/master/f/tools/taskotron-msg-reader.py
and then I'll trigger "avocado tests" for module (like examples for memcached, haproxy, etc...)

Thanks
Honza
ralph added a comment.Apr 3 2017, 3:11 PM

Requesting one change to the way item is passed to the check.

Otherwise, this looks good.

(Technically, we can use that scmurl here to figure out the name-stream-version, but passing the name-stream-version in as the item is more direct.)

jobtriggers/module_build_msg.py
14

I think item should be "{name}-{stream}-{version}".format(**msg['msg'])

We can probably *derive* that from the scmurl in a check, but the name-stream-version value is the most straightforward identifier for the built module.

jscotka added a comment.Apr 3 2017, 3:23 PM

Is it possible to pass also raw fedora MBS message via some variable like FEDMSG_RAW what will contains whole message? It could help if there will be missing some data.

tflink added a comment.Apr 3 2017, 3:47 PM
In D1180#21961, @jscotka wrote:

Is it possible to pass also raw fedora MBS message via some variable like FEDMSG_RAW what will contains whole message? It could help if there will be missing some data.

The thing that I'm worried about here is having a huge amount of data as a CLI arg. At the moment, anything passed into libtaskotron needs to be passed as an argument in bash and there are length limits on what you can have in a single command.

What do you need or anticipate needing that isn't covered by name-stream-version?

jscotka added a comment.Apr 4 2017, 6:08 AM

Hi,
I'm not sure now, what could be missing there. I think that everything could be covered by this "name-stream-version" at least there will be way how to get this data
At least this is unique identifier, so that in theory I can find whole message from datagrepper and then I have every fields (I'm not sure how long datagrepper stores these messages) but I can do that at least for one day I think. Or I'll hardcode there some specific fedora paths + this identifier shows me location of everything and in case something will change in infra but I'll need to adapt all changes In every script what will translate it to final sources.

Is it possible to show me how to construct these data like: eg. provide me location of used modulemd file derived from these "name-stream-version"

Thanks&Regards
 Honza
mkrizek updated this revision to Diff 3009.Apr 4 2017, 6:13 AM

Update item format as ralph suggested

tflink accepted this revision.Apr 4 2017, 1:12 PM
This revision is now accepted and ready to land.Apr 4 2017, 1:12 PM
Closed by commit rTRGRa9d20b0d8657: Add MBS (Module Build Service) consumer (authored by Martin Krizek <mkrizek@redhat.com>). · Explain WhyApr 4 2017, 2:21 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 3012

View Options

fedmsg.d/taskotron-trigger.py

1config = { 1​config = {
2 'taskotron.kojitagchangedjobconsumer.enabled': True, 2​ 'taskotron.kojitagchangedjobconsumer.enabled': True,
3 'taskotron.kojibuildcompletedjobconsumer.enabled': True, 3​ 'taskotron.kojibuildcompletedjobconsumer.enabled': True,
4 'taskotron.distgitcommitjobconsumer.enabled': True, 4​ 'taskotron.distgitcommitjobconsumer.enabled': True,
5 'taskotron.paguregitreceivejobconsumer.enabled': True, 5​ 'taskotron.paguregitreceivejobconsumer.enabled': True,
6 'taskotron.cloudcomposecompletedconsumer.enabled': True, 6​ 'taskotron.cloudcomposecompletedconsumer.enabled': True,
7​ 'taskotron.modulebuilddonejobconsumer.enabled': True,
7} 8​}
View Options

jobtriggers/jobrunner.py

1import argparse 1​import argparse
2import logging 2​import logging
3 3​
4import requests 4​import requests
5from . import config 5​from . import config
6from . import exceptions as exc 6​from . import exceptions as exc
7 7​
8from .koji_build_msg import KojiBuildCompletedJobTrigger 8​from .koji_build_msg import KojiBuildCompletedJobTrigger
9from .dist_git_commit_msg import DistGitCommitJobTrigger 9​from .dist_git_commit_msg import DistGitCommitJobTrigger
10from .pagure_git_received import PagureGitReceiveJobTrigger 10​from .pagure_git_received import PagureGitReceiveJobTrigger
11from .cloud_compose_complete_msg import CloudComposeJobTrigger 11​from .cloud_compose_complete_msg import CloudComposeJobTrigger
12​from .module_build_msg import ModuleBuildDoneJobTrigger
13​
12 14​
13# set up logging 15​# set up logging
14logger = logging.getLogger('Jobrunner') 16​logger = logging.getLogger('Jobrunner')
15sh = logging.StreamHandler() 17​sh = logging.StreamHandler()
16formatter = logging.Formatter(fmt='%(asctime)s - %(name)s - %(levelname)s: %(message)s', 18​formatter = logging.Formatter(fmt='%(asctime)s - %(name)s - %(levelname)s: %(message)s',
17 datefmt='%Y-%m-%d %H:%M:%S') 19​ datefmt='%Y-%m-%d %H:%M:%S')
18sh.setFormatter(formatter) 20​sh.setFormatter(formatter)
19logger.addHandler(sh) 21​logger.addHandler(sh)
Show All 19 Lines37 }, {
39 'topic': 'io.pagure.%s.pagure.git.receive' % deployment_type, 41​ 'topic': 'io.pagure.%s.pagure.git.receive' % deployment_type,
40 'page': 1, 42​ 'page': 1,
41 }, { 43​ }, {
42 'start': start, 44​ 'start': start,
43 'end': end, 45​ 'end': end,
44 'category': 'pungi', 46​ 'category': 'pungi',
45 'topic': 'org.fedoraproject.%s.pungi.compose.status.change' % deployment_type, 47​ 'topic': 'org.fedoraproject.%s.pungi.compose.status.change' % deployment_type,
46 'page': 1, 48​ 'page': 1,
49​ }, {
50​ 'start': start,
51​ 'end': end,
52​ 'category': 'mbs',
53​ 'topic': 'org.fedoraproject.%s.mbs.module.state.change' % deployment_type,
54​ 'page': 1,
47 }] 55​ }]
48 56​
49 for params in topics: 57​ for params in topics:
50 while True: 58​ while True:
51 r = requests.get(config.datagrepper_url, params=params) 59​ r = requests.get(config.datagrepper_url, params=params)
52 pages = r.json()['pages'] 60​ pages = r.json()['pages']
53 page = r.json()['arguments']['page'] 61​ page = r.json()['arguments']['page']
54 62​
55 if not pages: 63​ if not pages:
56 break 64​ break
57 65​
58 logger.info('Processing jobs from datagrepper for topic ' 66​ logger.info('Processing jobs from datagrepper for topic '
59 '%s: page %d of %d', params['topic'], page, pages) 67​ '%s: page %d of %d', params['topic'], page, pages)
60 68​
61 for msg in r.json()['raw_messages']: 69​ for msg in r.json()['raw_messages']:
62 if msg['topic'].endswith('build.state.change'): 70​ if msg['topic'].endswith('build.state.change'):
63 trigger = KojiBuildCompletedJobTrigger(logger) 71​ trigger = KojiBuildCompletedJobTrigger(logger)
64 elif 'pungi' in msg['topic']: 72​ elif 'pungi' in msg['topic']:
65 trigger = CloudComposeJobTrigger(logger) 73​ trigger = CloudComposeJobTrigger(logger)
74​ elif 'mbs' in msg['topic']:
75​ trigger = ModuleBuildDoneJobTrigger(logger)
66 elif msg['topic'].endswith('git.receive'): 76​ elif msg['topic'].endswith('git.receive'):
67 if msg['topic'].startswith('io.pagure'): 77​ if msg['topic'].startswith('io.pagure'):
68 trigger = PagureGitReceiveJobTrigger(logger) 78​ trigger = PagureGitReceiveJobTrigger(logger)
69 else: 79​ else:
70 trigger = DistGitCommitJobTrigger(logger) 80​ trigger = DistGitCommitJobTrigger(logger)
71 else: 81​ else:
72 logger.warning('Got an unexpected topic %s', msg['topic']) 82​ logger.warning('Got an unexpected topic %s', msg['topic'])
73 continue 83​ continue
Show All 29 Lines
View Options

jobtriggers/module_build_msg.py

  • This file was added.
1​import fedmsg.consumers
2​
3​from . import config
4​from .jobtrigger import JobTrigger
5​from . import exceptions as exc
6​
7​MESSAGE_TYPE = 'ModuleBuildComplete'
8​ITEM_TYPE = 'module_build'
9​
10​
11​class ModuleBuildDoneJobTrigger(JobTrigger):
12​ def process(self, msg):
13​ item = '{name}-{stream}-{version}'.format(**msg['msg'])
14​ if msg['msg']['state_name'] != 'done':
ralphUnsubmitted
Not Done

I think item should be "{name}-{stream}-{version}".format(**msg['msg'])

We can probably *derive* that from the scmurl in a check, but the name-stream-version value is the most straightforward identifier for the built module.

15​ raise exc.TriggerMsgError('rejecting message for not being module complete build'
16​ ' - %s' % item)
17​
18​ data = {
19​ "_msg": msg,
20​ "message_type": MESSAGE_TYPE,
21​ "item": item,
22​ "item_type": ITEM_TYPE,
23​ }
24​ return data
25​
26​
27​class ModuleBuildDoneJobConsumer(fedmsg.consumers.FedmsgConsumer):
28​ topic = "org.fedoraproject.%s.mbs.module.state.change" % config.deployment_type
29​ config_key = 'taskotron.modulebuilddonejobconsumer.enabled'
30​
31​ def __init__(self, *args, **kw):
32​ super(ModuleBuildDoneJobConsumer, self).__init__(*args, **kw)
33​ self.trigger = ModuleBuildDoneJobTrigger(self.log)
34​
35​ def consume(self, message):
36​ msg = message['body']
37​ try:
38​ data = self.trigger.process(msg)
39​ self.trigger.do_trigger(data)
40​ except exc.TriggerMsgError, e:
41​ self.log.debug(e)
42​ return
View Options

setup.py

Show All 12 Lines
13 13​
14 14​
15moksha_consumer = ( 15​moksha_consumer = (
16 'kojibuildcompletedjobconsumer = jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer', 16​ 'kojibuildcompletedjobconsumer = jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer',
17 'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer', 17​ 'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer',
18 'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer', 18​ 'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer',
19 'paguregitreceivejobconsumer = jobtriggers.pagure_git_received:PagureGitReceiveJobConsumer', 19​ 'paguregitreceivejobconsumer = jobtriggers.pagure_git_received:PagureGitReceiveJobConsumer',
20 'cloudcomposejobconsumer = jobtriggers.cloud_compose_complete_msg:CloudComposeCompletedConsumer', 20​ 'cloudcomposejobconsumer = jobtriggers.cloud_compose_complete_msg:CloudComposeCompletedConsumer',
21​ 'modulebuilddonejobconsumer = jobtriggers.module_build_msg:ModuleBuildDoneJobConsumer',
21) 22​)
22 23​
23setup( 24​setup(
24 name='jobtriggers', 25​ name='jobtriggers',
25 version='0.4.7', 26​ version='0.4.7',
26 description='triggering jobs via fedmsg', 27​ description='triggering jobs via fedmsg',
27 author='Tim Flink', 28​ author='Tim Flink',
28 author_email='tflink@fedoraproject.org', 29​ author_email='tflink@fedoraproject.org',
Show All 12 Lines
View Options

testing/test_module_build_trigger.py

  • This file was added.
1​import pytest
2​from dingus import Dingus
3​from munch import Munch
4​
5​from jobtriggers import module_build_msg
6​
7​
8​@pytest.mark.usefixtures('prepare')
9​class TestModuleBuildDoneJobConsumer():
10​
11​ @pytest.fixture
12​ def prepare(self):
13​ self.ref_task = 'module_task1'
14​ self.ref_name = 'testmodule'
15​ self.ref_stream = 'master'
16​ self.ref_version = '20170325004958'
17​ self.ref_item = '%s-%s-%s' % (self.ref_name, self.ref_stream, self.ref_version)
18​
19​ self.ref_message = self._create_msg(self.ref_name, self.ref_stream, self.ref_version)
20​
21​ stub_hub = Munch(config=Munch(get=0))
22​
23​ self.helper = module_build_msg.ModuleBuildDoneJobConsumer(stub_hub)
24​ self.helper.trigger.runner = Dingus()
25​
26​ module_build_msg.config.trigger_rules_template = """---
27​- when: {message_type: ModuleBuildComplete}
28​ do:
29​ - {tasks: [%s]}
30​""" % self.ref_task
31​
32​ module_build_msg.config.job_logging = False
33​ module_build_msg.config.valid_arches = ['x86_64']
34​
35​ def _create_msg(self, name, stream, version, state_name='done'):
36​ return Munch(body={
37​ "source_name": "datanommer",
38​ "i": 20,
39​ "timestamp": 1490404041.0,
40​ "msg_id": "2017-683a79ca-b143-49df-a659-0d76a65c4a91",
41​ "topic": "org.fedoraproject.prod.mbs.module.state.change",
42​ "source_version": "0.6.5",
43​ "msg": {
44​ "state_reason": "null",
45​ "component_builds": [
46​ 2653,
47​ 2654,
48​ 2655
49​ ],
50​ "name": name,
51​ "stream": stream,
52​ "time_submitted": 1490403045.0,
53​ "tasks": {
54​ "rpms": {
55​ "ed": {
56​ "state": 1,
57​ "nvr": "ed-1.14.1-2.module_b98abea8",
58​ "task_id": 18582394,
59​ "state_reason": ""
60​ },
61​ "module-build-macros": {
62​ "state": 1,
63​ "nvr": "module-build-macros-0.1-1.module_b98abea8",
64​ "task_id": 18582338,
65​ "state_reason": ""
66​ },
67​ "mksh": {
68​ "state": 1,
69​ "nvr": "mksh-54-2.module_b98abea8",
70​ "task_id": 18582475,
71​ "state_reason": ""
72​ }
73​ }
74​ },
75​ "state_url": "http://localhost/module-build-service/1/module-builds/75",
76​ "time_modified": 1490404041.0,
77​ "scmurl": "git://pkgs.fedoraproject.org/modules/testmodule.git?#69d45ad",
78​ "state": 3,
79​ "modulemd": "random data\n",
80​ "time_completed": 1490404041.0,
81​ "version": version,
82​ "owner": "psabata",
83​ "state_trace": [
84​ {
85​ "reason": "null",
86​ "time": 1490403047.0,
87​ "state": 1,
88​ "state_name": "wait"
89​ },
90​ {
91​ "reason": "null",
92​ "time": 1490403073.0,
93​ "state": 2,
94​ "state_name": "build"
95​ },
96​ {
97​ "reason": "null",
98​ "time": 1490403073.0,
99​ "state": 2,
100​ "state_name": "wait"
101​ }
102​ ],
103​ "id": 75,
104​ "state_name": state_name
105​ }
106​ })
107​
108​ def test_consume(self):
109​ self.helper.consume(self.ref_message)
110​
111​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
112​
113​ assert len(runner_calls) == 1
114​
115​ assert runner_calls[0][1] == (
116​ self.ref_item,
117​ module_build_msg.ITEM_TYPE,
118​ self.ref_task,
119​ 'x86_64',
120​ )
121​
122​ def test_consume_not_done(self):
123​ ref_message = self._create_msg(self.ref_name, self.ref_stream,
124​ self.ref_version, state_name='failed')
125​ self.helper.consume(ref_message)
126​
127​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
128​
129​ assert len(runner_calls) == 0