Differential D1110

Rought out the consumer
ClosedPublic
Actions

Authored by jskladan on Feb 2 2017, 3:08 PM.

Details

Reviewers
kparal
tflink
mkrizek
Commits
rTRGRdab986695b11: Add pagure git commit consumer
Summary

This adds a rough idea of what the pagure commit consumer could look
like.
We are once again misusing the item to carry additional information.
I'd really like to be able to have libtaskotron take the arbitrary input
(and I think we will have to do this sooner than later), but it is out
of the scope for this diff

This also fixes a heisenbug in testing, where depending on the order in
which the tests ran, the dist_git_commit/pagure_git_received were getting
either [x86_64] or [i386, x86_64] as valid arches, because some of the
testsuites change the default config value and do not clean up.

Test Plan

Added an unittest, but I'll try this out later on from the CLI anyway
Feel free to test on my behalf ;)

Diff Detail

Repository
rTRGR taskotron-trigger
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
jskladan retitled this revision from to Rought out the consumer.Feb 2 2017, 3:08 PM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added reviewers: mkrizek, tflink, kparal.
tflink added inline comments.Feb 7 2017, 2:16 PM
jobtriggers/pagure_git_received.py
9

I'm being picky, but can you fix this? I don't see a reason why that has to be an "end of line" commit

16

Am I missing a reason why this and the .config line above are here?

testing/test_pagure_commit_trigger.py
36

Can't this be split up onto multiple lines?

tflink added a comment.Feb 7 2017, 2:17 PM

Bah, hit the "submit" button too quickly. Overall it looks good to me, I just had some picky bits and questions before hitting approve

jskladan added inline comments.Feb 7 2017, 2:41 PM
jobtriggers/pagure_git_received.py
9

What do you mean by "end of line commit"? I sure can change it, just used this in order to not make any changes to libtaskotron at the moment. Sure would change it before merging into the code.

16

I'll most probably add the start/end commit to the data structure.
The .config import is important for unit tests, which then change some of the configuration during the test-run. We do this in the other consumers too, for the same exact reason.

testing/test_pagure_commit_trigger.py
36

Sure can. I'll split it before the merge.

jskladan updated this revision to Diff 2856.Feb 8 2017, 8:34 AM
  • make it more git source agnostic
  • store server
mkrizek accepted this revision.Feb 8 2017, 9:43 AM

I didn't test it, but seems like a straightforward change, the code looks good to me.

This revision is now accepted and ready to land.Feb 8 2017, 9:43 AM
Closed by commit rTRGRdab986695b11: Add pagure git commit consumer (authored by Josef Skladanka <jskladan@redhat.com>). · Explain WhyFeb 9 2017, 2:26 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2867

View Options

conf/trigger_rules.yml.example

Show All 35 Lines35- when: {message_type: KojiTagChanged}
36 do: 36​ do:
37 - {tasks: [depcheck]} 37​ - {tasks: [depcheck]}
38 38​
39- when: 39​- when:
40 message_type: KojiTagChanged 40​ message_type: KojiTagChanged
41 tag: {$regex: '/.*(?<!testing-pending)$$/'} 41​ tag: {$regex: '/.*(?<!testing-pending)$$/'}
42 do: 42​ do:
43 - {tasks: [upgradepath]} 43​ - {tasks: [upgradepath]}
44​
45​- when:
46​ message_type: PagureGitReceive
47​ repo:
48​ $in: [taskotron/execdb, taskotron/libtaskotron, taskotron/resultsdb, taskotron/taskotron-trigger]
49​ do:
50​ - {discover: {repo: 'https://pagure.io/taskotron/task-taskotron-ci.git' branch="master"}}
View Options

fedmsg.d/taskotron-trigger.py

1config = { 1​config = {
2 'taskotron.kojitagchangedjobconsumer.enabled': True, 2​ 'taskotron.kojitagchangedjobconsumer.enabled': True,
3 'taskotron.kojibuildcompletedjobconsumer.enabled': True, 3​ 'taskotron.kojibuildcompletedjobconsumer.enabled': True,
4 'taskotron.composecompletedjobconsumer.enabled': True, 4​ 'taskotron.composecompletedjobconsumer.enabled': True,
5 'taskotron.distgitcommitjobconsumer.enabled': True, 5​ 'taskotron.distgitcommitjobconsumer.enabled': True,
6​ 'taskotron.paguregitreceivejobconsumer.enabled': True,
6} 7​}
View Options

jobtriggers/jobrunner.py

1import argparse 1​import argparse
2import logging 2​import logging
3 3​
4import requests 4​import requests
5from . import config 5​from . import config
6from . import exceptions as exc 6​from . import exceptions as exc
7 7​
8from .koji_build_msg import KojiBuildCompletedJobTrigger 8​from .koji_build_msg import KojiBuildCompletedJobTrigger
9from .dist_git_commit_msg import DistGitCommitJobTrigger 9​from .dist_git_commit_msg import DistGitCommitJobTrigger
10from .compose_complete_msg import ComposeCompletedJobTrigger 10​from .compose_complete_msg import ComposeCompletedJobTrigger
11​from .pagure_git_received import PagureGitReceiveJobTrigger
11 12​
12 13​
13# set up logging 14​# set up logging
14logger = logging.getLogger('Jobrunner') 15​logger = logging.getLogger('Jobrunner')
15sh = logging.StreamHandler() 16​sh = logging.StreamHandler()
16formatter = logging.Formatter(fmt='%(asctime)s - %(name)s - %(levelname)s: %(message)s', 17​formatter = logging.Formatter(fmt='%(asctime)s - %(name)s - %(levelname)s: %(message)s',
17 datefmt='%Y-%m-%d %H:%M:%S') 18​ datefmt='%Y-%m-%d %H:%M:%S')
18sh.setFormatter(formatter) 19​sh.setFormatter(formatter)
Show All 14 Lines30 }, {
33 'topic': 'org.fedoraproject.prod.compose.*', 34​ 'topic': 'org.fedoraproject.prod.compose.*',
34 'page': 1, 35​ 'page': 1,
35 }, { 36​ }, {
36 'start': start, 37​ 'start': start,
37 'end': end, 38​ 'end': end,
38 'category': 'git', 39​ 'category': 'git',
39 'topic': 'org.fedoraproject.prod.git.receive', 40​ 'topic': 'org.fedoraproject.prod.git.receive',
40 'page': 1, 41​ 'page': 1,
42​ }, {
43​ 'start': start,
44​ 'end': end,
45​ 'category': 'git',
46​ 'topic': 'io.pagure.prod.pagure.git.receive',
47​ 'page': 1,
41 }] 48​ }]
42 49​
43 for params in topics: 50​ for params in topics:
44 while True: 51​ while True:
45 r = requests.get(config.datagrepper_url, params=params) 52​ r = requests.get(config.datagrepper_url, params=params)
46 pages = r.json()['pages'] 53​ pages = r.json()['pages']
47 page = r.json()['arguments']['page'] 54​ page = r.json()['arguments']['page']
48 55​
49 if not pages: 56​ if not pages:
50 break 57​ break
51 58​
52 logger.info('Processing jobs from datagrepper for topic ' 59​ logger.info('Processing jobs from datagrepper for topic '
53 '%s: page %d of %d', params['topic'], page, pages) 60​ '%s: page %d of %d', params['topic'], page, pages)
54 61​
55 for msg in r.json()['raw_messages']: 62​ for msg in r.json()['raw_messages']:
56 if msg['topic'].endswith('build.state.change'): 63​ if msg['topic'].endswith('build.state.change'):
57 trigger = KojiBuildCompletedJobTrigger(logger) 64​ trigger = KojiBuildCompletedJobTrigger(logger)
58 elif 'compose' in msg['topic']: 65​ elif 'compose' in msg['topic']:
59 trigger = ComposeCompletedJobTrigger(logger) 66​ trigger = ComposeCompletedJobTrigger(logger)
60 elif msg['topic'].endswith('git.receive'): 67​ elif msg['topic'].endswith('git.receive'):
61 trigger = DistGitCommitJobTrigger(logger) 68​ if msg['topic'].startswith('io.pagure'):
69​ trigger = PagureGitReceiveJobTrigger(logger)
70​ else:
71​ trigger = DistGitCommitJobTrigger(logger)
62 else: 72​ else:
63 logger.warning('Got an unexpected topic %s', msg['topic']) 73​ logger.warning('Got an unexpected topic %s', msg['topic'])
64 continue 74​ continue
65 75​
66 try: 76​ try:
67 data = trigger.process(msg) 77​ data = trigger.process(msg)
68 trigger.do_trigger(data) 78​ trigger.do_trigger(data)
69 except exc.TriggerMsgError, e: 79​ except exc.TriggerMsgError, e:
Show All 24 Lines
View Options

jobtriggers/pagure_git_received.py

  • This file was added.
1​import fedmsg.consumers
2​
3​from . import config # NOQA # This gets mocked by the test suite.
4​from .jobtrigger import JobTrigger
5​from . import exceptions as exc
6​
7​MESSAGE_TYPE = 'PagureGitReceive'
8​ITEM_TYPE = 'git_commit'
9​
tflinkUnsubmitted
Not Done

I'm being picky, but can you fix this? I don't see a reason why that has to be an "end of line" commit

jskladanAuthorUnsubmitted
Not Done

What do you mean by "end of line commit"? I sure can change it, just used this in order to not make any changes to libtaskotron at the moment. Sure would change it before merging into the code.

10​
11​class PagureGitReceiveJobTrigger(JobTrigger):
12​ def process(self, msg):
13​ server = 'https://pagure.io'
14​ repo = msg['msg']['repo']['name']
15​ branch = msg['msg']['branch']
16​ start_commit = msg['msg']['start_commit']
tflinkUnsubmitted
Not Done

Am I missing a reason why this and the .config line above are here?

jskladanAuthorUnsubmitted
Not Done

I'll most probably add the start/end commit to the data structure.
The .config import is important for unit tests, which then change some of the configuration during the test-run. We do this in the other consumers too, for the same exact reason.

17​ end_commit = msg['msg']['end_commit']
18​
19​ item = "%s/%s#%s#%s" % (server, repo, branch, end_commit)
20​
21​ data = {
22​ "_msg": msg,
23​ "message_type": MESSAGE_TYPE,
24​ "item": item,
25​ "item_type": ITEM_TYPE,
26​ "server": server,
27​ "repo": repo,
28​ "branch": branch,
29​ "start_commit": start_commit,
30​ "end_commit": end_commit,
31​ }
32​ return data
33​
34​
35​class PagureGitReceiveJobConsumer(fedmsg.consumers.FedmsgConsumer):
36​ topic = "io.pagure.prod.pagure.git.receive"
37​ config_key = 'taskotron.paguregitreceivejobconsumer.enabled'
38​
39​ def __init__(self, *args, **kw):
40​ super(PagureGitReceiveJobConsumer, self).__init__(*args, **kw)
41​ self.trigger = PagureGitReceiveJobTrigger(self.log)
42​
43​ def consume(self, message):
44​ msg = message['body']
45​ try:
46​ data = self.trigger.process(msg)
47​ self.trigger.do_trigger(data)
48​ except exc.TriggerMsgError, e:
49​ self.log.debug(e)
50​ return
View Options

setup.py

Show All 11 Lines9 def run(self):
12 raise SystemExit(errno) 12​ raise SystemExit(errno)
13 13​
14 14​
15moksha_consumer = ( 15​moksha_consumer = (
16 'kojibuildcompletedjobconsumer = jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer', 16​ 'kojibuildcompletedjobconsumer = jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer',
17 'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer', 17​ 'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer',
18 'composecompletedjobconsumer = jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer', 18​ 'composecompletedjobconsumer = jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer',
19 'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer', 19​ 'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer',
20​ 'paguregitreceivejobconsumer = jobtriggers.pagure_git_received:PagureGitReceiveJobConsumer',
20) 21​)
21 22​
22setup( 23​setup(
23 name='jobtriggers', 24​ name='jobtriggers',
24 version='0.4.4', 25​ version='0.4.4',
25 description='triggering jobs via fedmsg', 26​ description='triggering jobs via fedmsg',
26 author='Tim Flink', 27​ author='Tim Flink',
27 author_email='tflink@fedoraproject.org', 28​ author_email='tflink@fedoraproject.org',
Show All 14 Lines
View Options

testing/test_dist_git_trigger.py

Show All 39 Lines36"""
40 mock_discover_tasks = mock.Mock(return_value=[]) 40​ mock_discover_tasks = mock.Mock(return_value=[])
41 monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks) 41​ monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks)
42 monkeypatch.setattr(utils, 'clone_repo', mock_clone_repo) 42​ monkeypatch.setattr(utils, 'clone_repo', mock_clone_repo)
43 43​
44 mock_critpath = {"pkgs": {"f18": [], "master": []}} 44​ mock_critpath = {"pkgs": {"f18": [], "master": []}}
45 mock_get_critpath = mock.Mock(return_value=mock_critpath) 45​ mock_get_critpath = mock.Mock(return_value=mock_critpath)
46 monkeypatch.setattr(utils, 'parse_yaml_from_file', mock_get_critpath) 46​ monkeypatch.setattr(utils, 'parse_yaml_from_file', mock_get_critpath)
47 47​
48​ dist_git_commit_msg.config.valid_arches = ['x86_64']
49​
48 def _create_msg(self): 50​ def _create_msg(self):
49 return Munch(body={ 51​ return Munch(body={
50 "i": 1, 52​ "i": 1,
51 "msg": { 53​ "msg": {
52 "commit": { 54​ "commit": {
53 "agent": "martinkg", 55​ "agent": "martinkg",
54 "branch": "f25", 56​ "branch": "f25",
55 "email": "mgansser@alice.de", 57​ "email": "mgansser@alice.de",
▲ Show 20 Lines • Show All 59 Lines • Show Last 20 Lines
View Options

testing/test_pagure_commit_trigger.py

  • This file was added.
1​import pytest
2​import mock
3​from dingus import Dingus
4​from munch import Munch
5​
6​from jobtriggers import pagure_git_received
7​from jobtriggers import utils
8​
9​
10​def mock_clone_repo(repo, branch='master', fallback=None):
11​ return '/tmp/path_to_repo', branch
12​
13​
14​@pytest.mark.usefixtures('prepare')
15​class TestPagureGitReceivedJobConsumer():
16​
17​ @pytest.fixture
18​ def prepare(self, monkeypatch):
19​ self.item_type = 'dist_git_commit'
20​ self.ref_message = self._create_msg()
21​ self.ref_tasks = ['dist_git_task1', 'dist_git_task2']
22​ self.message_type = 'PagureGitReceive'
23​
24​ stub_hub = Munch(config=Munch(get=0))
25​
26​ self.helper = pagure_git_received.PagureGitReceiveJobConsumer(stub_hub)
27​ self.helper.trigger.runner = Dingus()
28​
29​ pagure_git_received.config.trigger_rules_template = """---
30​- when:
31​ message_type: PagureGitReceive
32​ repo:
33​ $in: [fm-orchestrator]
34​ do:
35​ - {discover: {repo: 'https://pagure.io/${repo}.git'}}
36​ - {tasks: [speclint]}
tflinkUnsubmitted
Not Done

Can't this be split up onto multiple lines?

jskladanAuthorUnsubmitted
Not Done

Sure can. I'll split it before the merge.

37​"""
38​
39​ pagure_git_received.config.job_logging = False
40​
41​ mock_discover_tasks = mock.Mock(return_value=[])
42​ monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks)
43​ monkeypatch.setattr(utils, 'clone_repo', mock_clone_repo)
44​
45​ pagure_git_received.config.valid_arches = ['x86_64']
46​
47​ def _create_msg(self):
48​ return Munch(body={
49​ "i": 1,
50​ "timestamp": 1486024230.0,
51​ "msg_id": "2017-c1117063-673e-4cce-803a-29a57e5a1ae1",
52​ "topic": "io.pagure.prod.pagure.git.receive",
53​ "msg": {
54​ "forced": False,
55​ "agent": "ralph",
56​ "repo": {
57​ "custom_keys": [],
58​ "description": "The module build orchestrator for Modularity",
59​ "parent": None,
60​ "settings": {
61​ "issues_default_to_private": False,
62​ "Minimum_score_to_merge_pull-request": 0,
63​ "Web-hooks": None,
64​ "fedmsg_notifications": True,
65​ "always_merge": False,
66​ "project_documentation": False,
67​ "Enforce_signed-off_commits_in_pull-request": False,
68​ "pull_requests": True,
69​ "Only_assignee_can_merge_pull-request": False,
70​ "issue_tracker": True
71​ },
72​ "tags": [
73​ "modularity",
74​ "fedora-modularity"
75​ ],
76​ "namespace": None,
77​ "priorities": {},
78​ "close_status": [
79​ "Invalid",
80​ "Insufficient data",
81​ "Fixed",
82​ "Duplicate"
83​ ],
84​ "milestones": {},
85​ "user": {
86​ "fullname": "Petr \u0160abata",
87​ "name": "psabata"
88​ },
89​ "date_created": "1466078637",
90​ "id": 774,
91​ "name": "fm-orchestrator"
92​ },
93​ "end_commit": "d093c5eef3f7bc593abce48f9ac6bd79bb279a41",
94​ "branch": "refs/heads/master",
95​ "authors": [
96​ {
97​ "fullname": "Ralph Bean",
98​ "name": "ralph"
99​ }
100​ ],
101​ "total_commits": 1,
102​ "start_commit": "FAKE_d093c5eef3f7bc593abce48f9ac6bd79bb279a41"
103​ }
104​ })
105​
106​ def test_consume(self):
107​ self.helper.consume(self.ref_message)
108​
109​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
110​
111​ assert len(runner_calls) == 1
112​
113​ assert runner_calls[0][1] == (
114​ 'https://pagure.io/fm-orchestrator#refs/heads/master#d093c5eef3f7bc593abce48f9ac6bd79bb279a41', # NOQA
115​ 'git_commit',
116​ 'speclint',
117​ 'x86_64',
118​ )