Differential D1064

Add a trigger for dist-git commit events.
ClosedPublic
Actions

Authored by • ralph2 on Dec 5 2016, 8:48 PM.

Details

Reviewers
tflink
mkrizek
jskladan
Commits
rTRGR55a87db1365d: Add a trigger for dist-git commit events.
Summary

This lets us add tests that trigger in response to dist-git commits.

I originally wrote this with a single message_type for all namespaces,
but then found that I didn't have a way in the trigger rules to specify
that a test (like modlint) should be run on all module dist-git
commits but not on all rpm dist-git commits. To fix this, I created
three separate message types, one for each of the namespaces we
currently have.

This of course means that we can write taskotron checks that check our
taskotron checks.

Test Plan

Run the test suite.

Diff Detail

Repository
rTRGR taskotron-trigger
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
• ralph2 retitled this revision from to Add a trigger for dist-git commit events..Dec 5 2016, 8:48 PM
• ralph2 updated this object.
• ralph2 edited the test plan for this revision. (Show Details)
• ralph2 added reviewers: tflink, mkrizek.
jskladan added a comment.Dec 6 2016, 4:25 PM

... then found that I didn't have a way in the trigger rules to specify that a test (like modlint) should be run on all module dist-git commits but not on all rpm dist-git commits

Looking at the code (and I hope I understood what you mean), this sure is a solution, that is readable, but you could do it without the different message types, if you wanted to, since you effectively just map namespace 1:1 to mesage_type.

I'll try to suggest the change in code. Although I'm not saying it is the better solution, this is exactly what I designed the trigger config for - in the FedMsg Consumer, you parse and provide data (here the 'relevant' bits are message_type and namespace), and the config then specifies rules based on the combination of values of said data.
It might well be, that this is not what I think you wanted (even thought I think my suggestions end up doing the same), and then I'd like to discuss how to make trigger better.
It might also be just a case of missing documentation... /me sucks with docs...

conf/trigger_rules.yml.example
9
- when:
    message_type: DistGitCommit, 
    namespace: rpms
12
- when:
    message_type: DistGitCommit, 
    namespace: modules
15
- when:
    message_type: DistGitCommit, 
    namespace: docker
jobtriggers/dist_git_commit_msg.py
33

"message_type": "DistGitCommit",

• ralph2 updated this revision to Diff 2707.Dec 7 2016, 5:01 PM

Simplify dist-git trigger message type(s), as per review feedback from @jskladan.

jskladan accepted this revision.Dec 8 2016, 10:54 AM

Looks great, thanks!

This revision is now accepted and ready to land.Dec 8 2016, 10:54 AM
Closed by commit rTRGR55a87db1365d: Add a trigger for dist-git commit events. (authored by Ralph Bean <rbean@redhat.com>). · Explain WhyDec 8 2016, 8:02 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2716

View Options

.gitignore

Show All 20 Lines
21# virtualenv 21​# virtualenv
22/env_trigger/ 22​/env_trigger/
23 23​
24# local configuration files 24​# local configuration files
25/conf/trigger.cfg 25​/conf/trigger.cfg
26/conf/trigger_rules.yml 26​/conf/trigger_rules.yml
27/docker_data/trigger.cfg 27​/docker_data/trigger.cfg
28/docker_data/trigger_rules.yml 28​/docker_data/trigger_rules.yml
29​
30​test_env
29.tox 31​.tox
View Options

conf/trigger_rules.yml.example

1--- 1​---
2- when: {message_type: KojiBuildPackageCompleted} 2​- when: {message_type: KojiBuildPackageCompleted}
3 do: 3​ do:
4 - {tasks: [rpmlint, rpmgrill]} 4​ - {tasks: [rpmlint, rpmgrill]}
5 - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-rpms/${name}.git', branch: "${distgit_branch}"}} 5​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-rpms/${name}.git', branch: "${distgit_branch}"}}
6- when: {message_type: KojiBuildDockerCompleted} 6​- when: {message_type: KojiBuildDockerCompleted}
7 do: 7​ do:
8 - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-docker/${name}.git', branch: "${distgit_branch}"}} 8​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-docker/${name}.git', branch: "${distgit_branch}"}}
9​- when: {message_type: DistGitCommit, namespace: rpms}
jskladanUnsubmitted
Not Done
- when:
    message_type: DistGitCommit, 
    namespace: rpms
10​ do:
11​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-rpms/${name}.git', branch: "${distgit_branch}"}}
12​- when: {message_type: DistGitCommit, namespace: modules}
jskladanUnsubmitted
Not Done
- when:
    message_type: DistGitCommit, 
    namespace: modules
13​ do:
14​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-modules/${name}.git', branch: "${distgit_branch}"}}
15​- when: {message_type: DistGitCommit, namespace: docker}
jskladanUnsubmitted
Not Done
- when:
    message_type: DistGitCommit, 
    namespace: docker
16​ do:
17​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-docker/${name}.git', branch: "${distgit_branch}"}}
9- when: {message_type: KojiBuildPackageCompleted, name: docker} 18​- when: {message_type: KojiBuildPackageCompleted, name: docker}
10 do: 19​ do:
11 - {tasks: [dockerautotest]} 20​ - {tasks: [dockerautotest]}
12- when: 21​- when:
13 message_type: KojiBuildPackageCompleted 22​ message_type: KojiBuildPackageCompleted
14 name: 23​ name:
15 $in: ${critpath_pkgs} 24​ $in: ${critpath_pkgs}
16 $nin: [] 25​ $nin: []
Show All 10 Lines
View Options

jobtriggers/dist_git_commit_msg.py

  • This file was added.
1​from . import config # This gets mocked by the test suite.
2​from .jobtrigger import JobTrigger
3​from . import exceptions as exc
4​
5​MESSAGE_TYPE = 'DistGitCommit'
6​ITEM_TYPE = 'dist_git_commit'
7​
8​
9​class DistGitCommitJobTrigger(JobTrigger):
10​ topic = "org.fedoraproject.prod.git.receive"
11​ config_key = 'taskotron.distgitcommitjobtrigger.enabled'
12​
13​ def process(self, msg):
14​ namespace = msg['msg']['commit']['namespace']
15​ repo = msg['msg']['commit']['repo']
16​ branch = msg['msg']['commit']['branch']
17​ rev = msg['msg']['commit']['rev']
18​ item = "%s/%s#%s" % (namespace, repo, rev)
19​
20​ data = {
21​ "_msg": msg,
22​ "message_type": MESSAGE_TYPE,
23​ "item": item,
24​ "item_type": ITEM_TYPE,
25​ "namespace": namespace,
26​ "repo": repo,
27​ "branch": branch,
28​ "rev": rev,
29​ }
30​ self.do_trigger(data)
31​
32​ def consume(self, message):
33​ msg = message.body
jskladanUnsubmitted
Not Done

"message_type": "DistGitCommit",

34​ try:
35​ self.process(msg)
36​ except exc.TriggerMsgError, e:
37​ self.log.debug(e)
38​ return
View Options

testing/test_dist_git_trigger.py

  • This file was added.
1​import pytest
2​import mock
3​from dingus import Dingus
4​from munch import Munch
5​
6​from jobtriggers import dist_git_commit_msg
7​from jobtriggers import utils
8​
9​
10​@pytest.mark.usefixtures('prepare')
11​class TestDistGitCommitJobTrigger():
12​
13​ @pytest.fixture
14​ def prepare(self, monkeypatch):
15​ self.item_type = 'dist_git_commit'
16​ self.ref_message = self._create_msg()
17​ self.ref_tasks = ['dist_git_task1', 'dist_git_task2']
18​ self.message_type = 'DistGitRPMCommit'
19​
20​ stub_hub = Munch(config=Munch(get=0))
21​
22​ self.helper = dist_git_commit_msg.DistGitCommitJobTrigger(stub_hub)
23​ self.helper.runner = Dingus()
24​
25​ dist_git_commit_msg.config.trigger_rules_template = """---
26​- when: {message_type: DistGitCommit, namespace: rpms}
27​ do:
28​ - {discover: {repo: 'http://foo/test-rpms/${name}.git'}}
29​ - {tasks: [speclint]} # Whatever speclint is..
30​- when: {message_type: DistGitCommit, namespace: modules}
31​ do:
32​ - {discover: {repo: 'http://foo/test-modules/${name}.git'}}
33​ - {tasks: [modlint]} # modlint is a real thing :)
34​"""
35​
36​ dist_git_commit_msg.config.job_logging = False
37​
38​ mock_discover_tasks = mock.Mock(return_value=[])
39​ monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks)
40​
41​ mock_critpath = {"pkgs": {"f18": [], "master": []}}
42​ mock_get_critpath = mock.Mock(return_value=mock_critpath)
43​ monkeypatch.setattr(utils, 'parse_yaml_from_file', mock_get_critpath)
44​
45​ def _create_msg(self):
46​ return Munch(body={
47​ "i": 1,
48​ "msg": {
49​ "commit": {
50​ "agent": "martinkg",
51​ "branch": "f25",
52​ "email": "mgansser@alice.de",
53​ "message": "Update to 1.1.61\n",
54​ "name": "Martin Gansser",
55​ "namespace": "rpms",
56​ "path": "/srv/git/repositories/rpms/vdr-epg-daemon.git",
57​ "repo": "vdr-epg-daemon",
58​ "rev": "6e5218a6e0b18c5a739e2164f85436f2ada5e475",
59​ "seen": False,
60​ "stats": {
61​ "files": {
62​ ".gitignore": {
63​ "additions": 1,
64​ "deletions": 0,
65​ "lines": 1
66​ },
67​ "clog": {
68​ "additions": 1,
69​ "deletions": 1,
70​ "lines": 2
71​ },
72​ "sources": {
73​ "additions": 1,
74​ "deletions": 1,
75​ "lines": 2
76​ },
77​ "vdr-epg-daemon.spec": {
78​ "additions": 4,
79​ "deletions": 1,
80​ "lines": 5
81​ }
82​ },
83​ "total": {
84​ "additions": 7,
85​ "deletions": 3,
86​ "files": 4,
87​ "lines": 10
88​ }
89​ },
90​ "summary": "Update to 1.1.61",
91​ "username": "martinkg"
92​ }
93​ },
94​ "msg_id": "2016-c0f8f391-7936-4c4e-9401-13c0c2372102",
95​ "timestamp": 1480707154.0,
96​ "topic": "org.fedoraproject.prod.git.receive"
97​ })
98​
99​ def test_consume(self):
100​ self.helper.consume(self.ref_message)
101​
102​ runner_calls = self.helper.runner.trigger_job.calls()
103​
104​ assert len(runner_calls) == 1
105​
106​ assert runner_calls[0][1] == (
107​ 'rpms/vdr-epg-daemon#6e5218a6e0b18c5a739e2164f85436f2ada5e475',
108​ 'dist_git_commit',
109​ 'speclint',
110​ 'x86_64',
111​ )