Differential D1081

Fix CLI to work with the trigger rules
ClosedPublic
Actions

Authored by mkrizek on Jan 13 2017, 3:08 PM.

Details

Reviewers
ralph
tflink
jskladan
Maniphest Tasks
T853: Fix taskotron-trigger CLI
Commits
rTRGRb77ce99d5ef2: Fix CLI to work with the trigger rules
Summary

This fixes the CLI to work with the new trigger rules, a few notes:

  1. It is not possible to replay jobs from a file (jobs.csv) since the file no longer provides all the values that are neccesary to trigger the jobs according to the trigger rules. Some code is still there, will be removed if we agree not to support it anymore, reworked otherwise.
  1. Koji tag jobs are not replayed from the CLI. It does not make much sense to run depcheck since the state of the repo was changed since the fedmsg was emitted.
  1. Topics and triggers are hardcoded in the CLI for now. I guess I can write some code that would discover what triggers should be run on what topics but not sure it's worth the effort now.

The patch also fixes a couple of other small issues that I run into, sorry about that.

Test Plan

Run both modes of the trigger:

$ fedmsg-hub

and

$ python cli.py -d --start `date --date="2017-01-09 07:00" +%s` --end `date --date="2017-01-09 10:00" +%s`

Diff Detail

Repository
rTRGR taskotron-trigger
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
mkrizek retitled this revision from to WIP: Fix CLI to work with the trigger rules.Jan 13 2017, 3:08 PM
mkrizek updated this object.
mkrizek edited the test plan for this revision. (Show Details)
mkrizek added reviewers: tflink, jskladan, ralph.
mkrizek updated this revision to Diff 2749.Jan 13 2017, 3:16 PM

Fix some lint errors

mkrizek edited the test plan for this revision. (Show Details)Jan 13 2017, 3:17 PM
mkrizek added a task: T853: Fix taskotron-trigger CLI.
jskladan added a comment.Jan 16 2017, 12:53 PM

I'll read the code a bit later, but regarding:

It is not possible to replay jobs from a file (jobs.csv) since the file no longer provides all the values that are neccesary to trigger the jobs according to the trigger rules. Some code is still there, will be removed if we agree not to support it anymore, reworked otherwise.

I'd rather have this supported, than removed. It honestly was the 99% usecase for me, so I'm not sure what are the other things to be done via the CLI, but this one I'd love to keep.

tflink added a comment.Jan 18 2017, 8:32 AM

I don't see any conceptual issues with the code. If you've tested it locally and it works, I'd say go for it once the lint issues and test failures are taken care of

ralph added a comment.Jan 18 2017, 4:47 PM

No blockers from me.

mkrizek updated this revision to Diff 2770.Jan 18 2017, 7:48 PM

Fixing unittests and most of the lint errors

mkrizek added a comment.Jan 18 2017, 7:50 PM
In D1081#20148, @jskladan wrote:

I'll read the code a bit later, but regarding:

It is not possible to replay jobs from a file (jobs.csv) since the file no longer provides all the values that are neccesary to trigger the jobs according to the trigger rules. Some code is still there, will be removed if we agree not to support it anymore, reworked otherwise.

I'd rather have this supported, than removed. It honestly was the 99% usecase for me, so I'm not sure what are the other things to be done via the CLI, but this one I'd love to keep.

That makes sense to me. Can we agree on creating a phab task for reworking the file support and deal with that in other review?

mkrizek retitled this revision from WIP: Fix CLI to work with the trigger rules to Fix CLI to work with the trigger rules.Jan 18 2017, 7:51 PM
mkrizek edited the test plan for this revision. (Show Details)
mkrizek updated this revision to Diff 2771.Jan 18 2017, 7:56 PM

Rebase to develop

ralph accepted this revision.Jan 18 2017, 8:07 PM

Looks good to me (pending whether or not @jskladan approves create an issue to restore the old functionality).

This revision is now accepted and ready to land.Jan 18 2017, 8:07 PM
kparal added a subscriber: kparal.Jan 19 2017, 12:39 PM

Excuse: Any suggestions what to do with those?

Sure, here comes the lint guru :)

jobtriggers/dist_git_commit_msg.py
3

First try whether this line can be removed and the test suite still runs. config is not used anywhere, so I'm not sure why the test suite would need it (I haven't looked). If it is needed, add an exception since it's necessary in this case:

from . import config  # This gets mocked by the test suite.   # noqa: F401
setup.py
32–33

This seems to be the correct solution:

entry_points={
    'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'],
    'moksha.consumer': (
        'kojibuildcompletedjobconsumer = '
        'jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer',
        'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer',
        'composecompletedjobconsumer = '
        'jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer',
        'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer',
    ),
},

but it's hard to read. What about this?

moksha_consumer = (
    'kojibuildcompletedjobconsumer = jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer',
    'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer',
    'composecompletedjobconsumer = jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer',
    'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer',
)

setup(
    name='jobtriggers',
    version='0.4.3',
    description='triggering jobs via fedmsg',
    author='Tim Flink',
    author_email='tflink@fedoraproject.org',
    license='GPLv2+',
    url='https://fedorahosted.org/fedora-qa',
    packages=['jobtriggers'],
    entry_points={
        'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'],
        'moksha.consumer': moksha_consumer,
    },
    include_package_data=True,
    install_requires=[
        'fedmsg',
        'requests',
    ],
    cmdclass={'test': PyTest}
)
testing/test_jobtrigger.py
108

Remove tasks = .

mkrizek updated this revision to Diff 2783.Jan 23 2017, 1:38 PM

minor fix

jskladan accepted this revision.Jan 23 2017, 2:18 PM

Looks good, I'd still want to have the jobsfile functionality, but we can add that later on. /me will spin up a ticket for that.

tflink accepted this revision.Jan 23 2017, 2:21 PM

Lint issues aside, looks good to me. Works when I test locally

Closed by commit rTRGRb77ce99d5ef2: Fix CLI to work with the trigger rules (authored by Martin Krizek <mkrizek@redhat.com>). · Explain WhyJan 23 2017, 2:58 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2784

View Options

conf/trigger_rules.yml.example

1--- 1​---
2- when: {message_type: KojiBuildPackageCompleted} 2​- when: {message_type: KojiBuildPackageCompleted}
3 do: 3​ do:
4 - {tasks: [rpmlint, rpmgrill]} 4​ - {tasks: [rpmlint, rpmgrill]}
5 - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-rpms/${name}.git', branch: "${distgit_branch}"}} 5​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-rpms/${name}.git', branch: "${distgit_branch}", fallback_branch: 'master'}}
6​
6- when: {message_type: KojiBuildDockerCompleted} 7​- when: {message_type: KojiBuildDockerCompleted}
7 do: 8​ do:
8 - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-docker/${name}.git', branch: "${distgit_branch}"}} 9​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-docker/${name}.git', branch: "${distgit_branch}", fallback_branch: 'master'}}
10​
9- when: {message_type: DistGitCommit, namespace: rpms} 11​- when: {message_type: DistGitCommit, namespace: rpms}
10 do: 12​ do:
11 - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-rpms/${name}.git', branch: "${distgit_branch}"}} 13​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-rpms/${repo}.git', branch: "${distgit_branch}"}}
14​
12- when: {message_type: DistGitCommit, namespace: modules} 15​- when: {message_type: DistGitCommit, namespace: modules}
13 do: 16​ do:
14 - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-modules/${name}.git', branch: "${distgit_branch}"}} 17​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-modules/${repo}.git', branch: "${distgit_branch}"}}
18​
15- when: {message_type: DistGitCommit, namespace: docker} 19​- when: {message_type: DistGitCommit, namespace: docker}
16 do: 20​ do:
17 - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-docker/${name}.git', branch: "${distgit_branch}"}} 21​ - {discover: {repo: 'http://pkgs.fedoraproject.org/git/test-docker/${repo}.git', branch: "${distgit_branch}"}}
22​
18- when: {message_type: KojiBuildPackageCompleted, name: docker} 23​- when: {message_type: KojiBuildPackageCompleted, name: docker}
19 do: 24​ do:
20 - {tasks: [dockerautotest]} 25​ - {tasks: [dockerautotest]}
26​
21- when: 27​- when:
22 message_type: KojiBuildPackageCompleted 28​ message_type: KojiBuildPackageCompleted
23 name: 29​ name:
24 $in: ${critpath_pkgs} 30​ $in: ${critpath_pkgs}
25 $nin: [] 31​ $nin: []
26 do: 32​ do:
27 - {tasks: [abicheck]} 33​ - {tasks: [abicheck]}
34​
28- when: {message_type: KojiTagChanged} 35​- when: {message_type: KojiTagChanged}
29 do: 36​ do:
30 - {tasks: [depcheck]} 37​ - {tasks: [depcheck]}
38​
31- when: 39​- when:
32 message_type: KojiTagChanged 40​ message_type: KojiTagChanged
33 tag: {$regex: '/.*(?<!testing-pending)$$/'} 41​ tag: {$regex: '/.*(?<!testing-pending)$$/'}
34 do: 42​ do:
35 - {tasks: [upgradepath]} 43​ - {tasks: [upgradepath]}
View Options

fedmsg.d/taskotron-trigger.py

1config = { 1​config = {
2 'taskotron.kojitagchangedjobtrigger.enabled': True, 2​ 'taskotron.kojitagchangedjobconsumer.enabled': True,
3 'taskotron.kojibuildcompletedjobtrigger.enabled': True, 3​ 'taskotron.kojibuildcompletedjobconsumer.enabled': True,
4 'taskotron.composecompletedjobtrigger.enabled': True, 4​ 'taskotron.composecompletedjobconsumer.enabled': True,
5​ 'taskotron.distgitcommitjobconsumer.enabled': True,
5} 6​}
View Options

jobtriggers/compose_complete_msg.py

1import fedmsg 1​import fedmsg
2import fedmsg.encoding 2​import fedmsg.encoding
3​import fedmsg.consumers
3 4​
4from .jobtrigger import JobTrigger 5​from .jobtrigger import JobTrigger
6​from . import exceptions as exc
5 7​
6ITEM_TYPE = 'compose' 8​ITEM_TYPE = 'compose'
7 9​
8 10​
9class ComposeCompletedJobTrigger(JobTrigger): 11​class ComposeCompletedJobTrigger(JobTrigger):
10 topic = "org.fedoraproject.prod.compose.*" 12​ def process(self, msg):
11 jsonify = False ​
12 config_key = 'taskotron.composecompletedjobtrigger.enabled' ​
13 ​
14 def __init__(self, *args, **kw): ​
15 super(ComposeCompletedJobTrigger, self).__init__(*args, **kw) ​
16 ​
17 def consume(self, message): ​
18 msg = fedmsg.encoding.loads(message.body) ​
19 ​
20 topic = msg['topic'] 13​ topic = msg['topic']
21 14​
22 if not 'branched.complete' in topic and not 'rawhide.complete' in topic: 15​ if not 'branched.complete' in topic and not 'rawhide.complete' in topic:
23 return 16​ raise exc.TriggerMsgError('rejecting message not being branched/rawhide '
17​ 'complete compose')
24 18​
25 branch = msg['msg']['branch'] 19​ branch = msg['msg']['branch']
26 arch = msg['msg']['arch'] 20​ arch = msg['msg']['arch']
27 21​
28 # non empty arch means secondary arch 22​ # non empty arch means secondary arch
29 if arch: 23​ if arch:
30 return 24​ raise exc.TriggerMsgError('rejecting message not being primary arch: %s' % arch)
31 25​
32 self.log.debug('should be scheduling job for completed compose: %s' % branch) 26​ self.log.debug('should be scheduling job for completed compose: %s' % branch)
33 27​
34 data = { 28​ data = {
35 "_msg": msg, 29​ "_msg": msg,
36 "message_type": "ComposeCompleted", 30​ "message_type": "ComposeCompleted",
37 "item": branch, 31​ "item": branch,
38 "item_type": ITEM_TYPE, 32​ "item_type": ITEM_TYPE,
39 "branch": branch, 33​ "branch": branch,
40 } 34​ }
41 self.do_trigger(data) 35​ return data
36​
37​
38​class ComposeCompletedJobConsumer(fedmsg.consumers.FedmsgConsumer):
39​ topic = "org.fedoraproject.prod.compose.*"
40​ jsonify = False
41​ config_key = 'taskotron.composecompletedjobconsumer.enabled'
42​
43​ def __init__(self, *args, **kw):
44​ super(ComposeCompletedJobConsumer, self).__init__(*args, **kw)
45​ self.trigger = ComposeCompletedJobTrigger(self.log)
46​
47​ def consume(self, message):
48​ msg = fedmsg.encoding.loads(message.body)
49​ try:
50​ data = self.trigger.process(msg)
51​ self.trigger.do_trigger(data)
52​ except exc.TriggerMsgError, e:
53​ self.log.debug(e)
54​ return
View Options

jobtriggers/dist_git_commit_msg.py

1​import fedmsg.consumers
2​
1from . import config # This gets mocked by the test suite. 3​from . import config # This gets mocked by the test suite.
kparalUnsubmitted
Not Done

First try whether this line can be removed and the test suite still runs. config is not used anywhere, so I'm not sure why the test suite would need it (I haven't looked). If it is needed, add an exception since it's necessary in this case:

from . import config  # This gets mocked by the test suite.   # noqa: F401
2from .jobtrigger import JobTrigger 4​from .jobtrigger import JobTrigger
3from . import exceptions as exc 5​from . import exceptions as exc
4 6​
5MESSAGE_TYPE = 'DistGitCommit' 7​MESSAGE_TYPE = 'DistGitCommit'
6ITEM_TYPE = 'dist_git_commit' 8​ITEM_TYPE = 'dist_git_commit'
7 9​
8 10​
9class DistGitCommitJobTrigger(JobTrigger): 11​class DistGitCommitJobTrigger(JobTrigger):
10 topic = "org.fedoraproject.prod.git.receive" ​
11 config_key = 'taskotron.distgitcommitjobtrigger.enabled' ​
12 ​
13 def process(self, msg): 12​ def process(self, msg):
14 namespace = msg['msg']['commit']['namespace'] 13​ namespace = msg['msg']['commit']['namespace']
15 repo = msg['msg']['commit']['repo'] 14​ repo = msg['msg']['commit']['repo']
16 branch = msg['msg']['commit']['branch'] 15​ branch = msg['msg']['commit']['branch']
17 rev = msg['msg']['commit']['rev'] 16​ rev = msg['msg']['commit']['rev']
18 item = "%s/%s#%s" % (namespace, repo, rev) 17​ item = "%s/%s#%s" % (namespace, repo, rev)
19 18​
20 data = { 19​ data = {
21 "_msg": msg, 20​ "_msg": msg,
22 "message_type": MESSAGE_TYPE, 21​ "message_type": MESSAGE_TYPE,
23 "item": item, 22​ "item": item,
24 "item_type": ITEM_TYPE, 23​ "item_type": ITEM_TYPE,
25 "namespace": namespace, 24​ "namespace": namespace,
26 "repo": repo, 25​ "repo": repo,
27 "branch": branch, 26​ "distgit_branch": branch,
28 "rev": rev, 27​ "rev": rev,
29 } 28​ }
30 self.do_trigger(data) 29​ return data
30​
31​
32​class DistGitCommitJobConsumer(fedmsg.consumers.FedmsgConsumer):
33​ topic = "org.fedoraproject.prod.git.receive"
34​ config_key = 'taskotron.distgitcommitjobconsumer.enabled'
35​
36​ def __init__(self, *args, **kw):
37​ super(DistGitCommitJobConsumer, self).__init__(*args, **kw)
38​ self.trigger = DistGitCommitJobTrigger(self.log)
31 39​
32 def consume(self, message): 40​ def consume(self, message):
33 msg = message.body 41​ msg = message['body']
34 try: 42​ try:
35 self.process(msg) 43​ data = self.trigger.process(msg)
44​ self.trigger.do_trigger(data)
36 except exc.TriggerMsgError, e: 45​ except exc.TriggerMsgError, e:
37 self.log.debug(e) 46​ self.log.debug(e)
38 return 47​ return
View Options

jobtriggers/jobrunner.py

1import csv ​
2import argparse 1​import argparse
3import itertools ​
4import logging 2​import logging
5from copy import deepcopy ​
6 3​
7import requests 4​import requests
8from . import config 5​from . import config
9from . import koji_build_msg ​
10from . import koji_tag_msg ​
11from . import exceptions as exc 6​from . import exceptions as exc
12from .runners import get_runner ​
13 7​
8​from .koji_build_msg import KojiBuildCompletedJobTrigger
9​from .dist_git_commit_msg import DistGitCommitJobTrigger
10​from .compose_complete_msg import ComposeCompletedJobTrigger
14 11​
15runner = get_runner(config.runner_type) ​
16 12​
17# set up logging 13​# set up logging
18logger = logging.getLogger('Jobrunner') 14​logger = logging.getLogger('Jobrunner')
19sh = logging.StreamHandler() 15​sh = logging.StreamHandler()
20formatter = logging.Formatter(fmt='%(asctime)s - %(name)s - %(levelname)s: %(message)s', 16​formatter = logging.Formatter(fmt='%(asctime)s - %(name)s - %(levelname)s: %(message)s',
21 datefmt='%Y-%m-%d %H:%M:%S') 17​ datefmt='%Y-%m-%d %H:%M:%S')
22sh.setFormatter(formatter) 18​sh.setFormatter(formatter)
23logger.addHandler(sh) 19​logger.addHandler(sh)
24 20​
25 21​
26def trigger_task(item, item_type, task, arch, **kwargs): ​
27 logger.debug('triggering %s for %s on %s' % (task, item, arch)) ​
28 ​
29 output = runner.trigger_job(item, item_type, task, arch, **kwargs) ​
30 for line in output: ​
31 logger.info(line) ​
32 ​
33 ​
34def trigger_tasks(item, item_type, tasks, arches, **kwargs): ​
35 for task, arch in itertools.product(tasks, arches): ​
36 trigger_task(item, item_type, task, arch, **kwargs) ​
37 ​
38 ​
39def file_jobs(jobsfile): ​
40 with open(jobsfile, 'r') as csvfile: ​
41 jobs = csv.reader(csvfile, delimiter=';') ​
42 for job in jobs: ​
43 trigger_task(job[1], job[2], job[3], job[4]) ​
44 ​
45 ​
46def datagrepper_jobs(start, end): 22​def datagrepper_jobs(start, end):
47 params = { 23​ topics = [{
48 'start': start, 24​ 'start': start,
49 'end': end, 25​ 'end': end,
50 'category': 'buildsys', 26​ 'category': 'buildsys',
51 'topic': ['org.fedoraproject.prod.buildsys.build.state.change', 27​ 'topic': 'org.fedoraproject.prod.buildsys.build.state.change',
52 'org.fedoraproject.prod.buildsys.tag'], ​
53 'page': 1, 28​ 'page': 1,
54 } 29​ }, {
55 30​ 'start': start,
56 while True: 31​ 'end': end,
57 r = requests.get(config.datagrepper_url, params=params) 32​ 'category': 'compose',
58 pages = r.json()['pages'] 33​ 'topic': 'org.fedoraproject.prod.compose.*',
59 page = r.json()['arguments']['page'] 34​ 'page': 1,
60 35​ }, {
61 logger.info('Processing jobs from datagrepper: page %d of %d', page, pages) 36​ 'start': start,
37​ 'end': end,
38​ 'category': 'git',
39​ 'topic': 'org.fedoraproject.prod.git.receive',
40​ 'page': 1,
41​ }]
42​
43​ for params in topics:
44​ while True:
45​ r = requests.get(config.datagrepper_url, params=params)
46​ pages = r.json()['pages']
47​ page = r.json()['arguments']['page']
48​
49​ if not pages:
50​ break
51​
52​ logger.info('Processing jobs from datagrepper for topic '
53​ '%s: page %d of %d', params['topic'], page, pages)
54​
55​ for msg in r.json()['raw_messages']:
56​ if msg['topic'].endswith('build.state.change'):
57​ trigger = KojiBuildCompletedJobTrigger(logger)
58​ elif 'compose' in msg['topic']:
59​ trigger = ComposeCompletedJobTrigger(logger)
60​ elif msg['topic'].endswith('git.receive'):
61​ trigger = DistGitCommitJobTrigger(logger)
62​ else:
63​ logger.warning('Got an unexpected topic %s', msg['topic'])
64​ continue
62 65​
63 koji_tag_items = set() ​
64 for msg in r.json()['raw_messages']: ​
65 if msg['topic'].endswith('build.state.change'): ​
66 try: 66​ try:
67 koji_build_msg.trigger_tasks(msg, trigger_tasks) 67​ data = trigger.process(msg)
68​ trigger.do_trigger(data)
68 except exc.TriggerMsgError, e: 69​ except exc.TriggerMsgError, e:
69 logger.debug(e) 70​ logger.debug(e)
70 except exc.TriggerError, e: 71​ except KeyError, e:
71 logger.error(e) 72​ logger.exception(e)
72 elif msg['topic'].endswith('tag'): 73​ logger.info('Missing a key in the fedmsg. Maybe consumer needs an update or '
73 try: 74​ 'you\'re replaying fedmsgs that have old format?')
74 koji_tag_items.add(koji_tag_msg.get_item(msg)) ​
75 except exc.TriggerMsgError, e: ​
76 logger.debug(e) ​
77 else: ​
78 logger.warning('Got an unexpected topic %s', msg['topic']) ​
79 ​
80 item_type = koji_tag_msg.ITEM_TYPE ​
81 for item in koji_tag_items: ​
82 tasks = deepcopy(config.koji_tag_changed_tasks) ​
83 if item.endswith('testing-pending'): ​
84 tasks.remove('upgradepath') ​
85 trigger_tasks(item, item_type, tasks, config.valid_arches) ​
86 75​
87 next_page = page + 1 76​ next_page = page + 1
88 if next_page > pages: 77​ if next_page > pages:
89 break 78​ break
90 params['page'] = next_page 79​ params['page'] = next_page
91 80​
92 81​
93def main(): 82​def main():
94 parser = argparse.ArgumentParser(prog='Jobrunner', 83​ parser = argparse.ArgumentParser(prog='Jobrunner',
95 usage='Gets jobs from datagrepper from a given time ' 84​ usage='Gets jobs from datagrepper from a given time '
96 'period and runs them.') 85​ 'period and runs them.')
97 parser.add_argument('-s', '--start', help='Start time (Unix time format)') 86​ parser.add_argument('-s', '--start', help='Start time (Unix time format)')
98 parser.add_argument('-e', '--end', help='End time (Unix time format)') 87​ parser.add_argument('-e', '--end', help='End time (Unix time format)')
99 parser.add_argument('-d', '--debug', action='store_true', help='Print debug message') 88​ parser.add_argument('-d', '--debug', action='store_true', help='Print debug message')
100 parser.add_argument('-f', '--jobsfile', help='A file containing jobs in csv format') ​
101 args = parser.parse_args() 89​ args = parser.parse_args()
102 90​
103 logger.setLevel(logging.DEBUG if args.debug else logging.INFO) 91​ logger.setLevel(logging.DEBUG if args.debug else logging.INFO)
104 92​
105 # let's make running jobs from the file a priority 93​ datagrepper_jobs(args.start, args.end)
106 if args.jobsfile: ​
107 file_jobs(args.jobsfile) ​
108 else: ​
109 datagrepper_jobs(args.start, args.end) ​
View Options

jobtriggers/jobtrigger.py

1import os 1​import os
2import copy 2​import copy
3import yaml 3​import yaml
4import itertools 4​import itertools
5import fedmsg.consumers ​
6from mongoquery import Query, QueryError 5​from mongoquery import Query, QueryError
7 6​
8from . import config, runners, utils 7​from . import config, runners, utils
9from . import exceptions as exc 8​from . import exceptions as exc
10from .mongoquery_string_template import MongoTemplate 9​from .mongoquery_string_template import MongoTemplate
11 10​
12 11​
13class DefaultDict(dict): 12​class DefaultDict(dict):
14 13​
15 """Since the rules will contain unreplacable variables (for example KojiBuildCompleted 14​ """Since the rules will contain unreplacable variables (for example KojiBuildCompleted
16 provides ${critpath_pkgs} and KojiTagChanged does not), we need a default value. 15​ provides ${critpath_pkgs} and KojiTagChanged does not), we need a default value.
17 None/null would be probably safer, but to make debugging easier, let's replace the missing 16​ None/null would be probably safer, but to make debugging easier, let's replace the missing
18 keys with stringified placeholder instead. 17​ keys with stringified placeholder instead.
19 """ 18​ """
20 19​
21 def __missing__(self, key): 20​ def __missing__(self, key):
22 return str("MISSING_VAR_%s" % key) 21​ return str("MISSING_VAR_%s" % key)
23 22​
24 23​
25class JobTrigger(fedmsg.consumers.FedmsgConsumer): 24​class JobTrigger(object):
26 25​ def __init__(self, log):
27 def __init__(self, *args, **kw): 26​ self.log = log
28 super(JobTrigger, self).__init__(*args, **kw) ​
29 self.runner = runners.get_runner(config.runner_type) 27​ self.runner = runners.get_runner(config.runner_type)
30 28​
31 def trigger_tasks(self, item, item_type, tasks, arches, **kwargs): 29​ def trigger_tasks(self, item, item_type, tasks, arches, **kwargs):
32 for task, arch in itertools.product(tasks, arches): 30​ for task, arch in itertools.product(tasks, arches):
33 self.log.info('triggering %s for %s on %s', task, item, arch) 31​ self.log.info('triggering %s for %s on %s', task, item, arch)
34 32​
35 if config.job_logging: 33​ if config.job_logging:
36 try: 34​ try:
▲ Show 20 Lines • Show All 95 Lines • Show Last 20 Lines
View Options

jobtriggers/koji_build_msg.py

1import koji 1​import koji
2import fedmsg 2​import fedmsg
3import fedmsg.encoding 3​import fedmsg.encoding
4​import fedmsg.consumers
4 5​
5from . import config, utils 6​from . import config, utils
6from .jobtrigger import JobTrigger 7​from .jobtrigger import JobTrigger
7from . import exceptions as exc 8​from . import exceptions as exc
8 9​
9ITEM_TYPE_PACKAGE = 'koji_build' 10​ITEM_TYPE_PACKAGE = 'koji_build'
10ITEM_TYPE_DOCKER = 'docker_image' 11​ITEM_TYPE_DOCKER = 'docker_image'
11 12​
12 13​
13class KojiBuildCompletedJobTrigger(JobTrigger): 14​class KojiBuildCompletedJobTrigger(JobTrigger):
14 topic = "org.fedoraproject.prod.buildsys.build.state.change" ​
15 jsonify = False ​
16 config_key = 'taskotron.kojibuildcompletedjobtrigger.enabled' ​
17 15​
18 def get_nvr(self, msg): 16​ def _get_nvr(self, msg):
19 release = msg['msg']['release'] 17​ release = msg['msg']['release']
20 name = msg['msg']['name'] 18​ name = msg['msg']['name']
21 version = msg['msg']['version'] 19​ version = msg['msg']['version']
22 20​
23 return name, version, release 21​ return name, version, release
24 22​
25 def process(self, msg): ​
26 state = msg['msg']['new'] ​
27 instance = msg['msg']['instance'] ​
28 owner = msg['msg']['owner'] ​
29 name, version, release = self.get_nvr(msg) ​
30 nvr = '-'.join((name, version, release)) ​
31 ​
32 if not instance == 'primary': ​
33 raise exc.TriggerMsgError('rejecting message not from primary koji instance ' ​
34 '(%s from %s)' % (nvr, instance)) ​
35 ​
36 if state != koji.BUILD_STATES['COMPLETE']: ​
37 raise exc.TriggerMsgError('rejecting message for not being a build complete message ' ​
38 '(%s, status %s)' % (nvr, state)) ​
39 ​
40 # FIXME This is probably a temporary approach, could change in future, see D1002#19239 ​
41 if owner == 'containerbuild': ​
42 return self._process_docker(msg) ​
43 else: ​
44 if 'fc' not in release: ​
45 raise exc.TriggerMsgError('rejecting message for not containing "fc" in release ' ​
46 '(%s)' % nvr) ​
47 return self._process_package(msg) ​
48 ​
49 def _process_package(self, msg): 23​ def _process_package(self, msg):
50 name, version, release = self.get_nvr(msg) 24​ name, version, release = self._get_nvr(msg)
51 item = '-'.join([name, version, release]) 25​ item = '-'.join([name, version, release])
52 critpath = utils.parse_yaml_from_file(config.critpath_filepath) 26​ critpath = utils.parse_yaml_from_file(config.critpath_filepath)
53 distgit_branch = utils.get_distgit_branch(release) 27​ distgit_branch = utils.get_distgit_branch(release)
54 critpath_pkgs = critpath['pkgs'].get(distgit_branch, critpath['pkgs']['master']) 28​ critpath_pkgs = critpath['pkgs'].get(distgit_branch, critpath['pkgs']['master'])
55 data = { 29​ data = {
56 "_msg": msg, 30​ "_msg": msg,
57 "message_type": "KojiBuildPackageCompleted", 31​ "message_type": "KojiBuildPackageCompleted",
58 "item": item, 32​ "item": item,
59 "item_type": ITEM_TYPE_PACKAGE, 33​ "item_type": ITEM_TYPE_PACKAGE,
60 "name": name, 34​ "name": name,
61 "version": version, 35​ "version": version,
62 "release": release, 36​ "release": release,
63 "critpath_pkgs": critpath_pkgs, 37​ "critpath_pkgs": critpath_pkgs,
64 "distgit_branch": distgit_branch, 38​ "distgit_branch": distgit_branch,
65 } 39​ }
66 self.do_trigger(data) 40​ return data
67 41​
68 def _process_docker(self, msg): 42​ def _process_docker(self, msg):
69 name, version, release = self.get_nvr(msg) 43​ name, version, release = self._get_nvr(msg)
70 build_id = msg['msg']['build_id'] 44​ build_id = msg['msg']['build_id']
71 item = utils.get_docker_registry_url(build_id) 45​ item = utils.get_docker_registry_url(build_id)
72 distgit_branch = utils.get_distgit_branch_docker(build_id) 46​ distgit_branch = utils.get_distgit_branch_docker(build_id)
73 critpath_pkgs = [] # FIXME do we need this for docker images? 47​ critpath_pkgs = [] # FIXME do we need this for docker images?
74 data = { 48​ data = {
75 "_msg": msg, 49​ "_msg": msg,
76 "message_type": "KojiBuildDockerCompleted", 50​ "message_type": "KojiBuildDockerCompleted",
77 "item": item, 51​ "item": item,
78 "item_type": ITEM_TYPE_DOCKER, 52​ "item_type": ITEM_TYPE_DOCKER,
79 "name": name, 53​ "name": name,
80 "version": version, 54​ "version": version,
81 "release": release, 55​ "release": release,
82 "critpath_pkgs": critpath_pkgs, 56​ "critpath_pkgs": critpath_pkgs,
83 "distgit_branch": distgit_branch, 57​ "distgit_branch": distgit_branch,
84 } 58​ }
85 self.do_trigger(data) 59​ return data
60​
61​ def process(self, msg):
62​ state = msg['msg']['new']
63​ instance = msg['msg']['instance']
64​ owner = msg['msg']['owner']
65​ name, version, release = self._get_nvr(msg)
66​ nvr = '-'.join((name, version, release))
67​
68​ if not instance == 'primary':
69​ raise exc.TriggerMsgError('rejecting message not from primary koji instance '
70​ '(%s from %s)' % (nvr, instance))
71​
72​ if state != koji.BUILD_STATES['COMPLETE']:
73​ raise exc.TriggerMsgError('rejecting message for not being a build complete message '
74​ '(%s, status %s)' % (nvr, state))
75​
76​ # FIXME This is probably a temporary approach, could change in future, see D1002#19239
77​ if owner == 'containerbuild':
78​ return self._process_docker(msg)
79​ else:
80​ if 'fc' not in release:
81​ raise exc.TriggerMsgError('rejecting message for not containing "fc" in release '
82​ '(%s)' % nvr)
83​ return self._process_package(msg)
84​
85​
86​class KojiBuildCompletedJobConsumer(fedmsg.consumers.FedmsgConsumer):
87​ topic = "org.fedoraproject.prod.buildsys.build.state.change"
88​ jsonify = False
89​ config_key = 'taskotron.kojibuildcompletedjobconsumer.enabled'
90​
91​ def __init__(self, *args, **kw):
92​ super(KojiBuildCompletedJobConsumer, self).__init__(*args, **kw)
93​ self.trigger = KojiBuildCompletedJobTrigger(self.log)
86 94​
87 def consume(self, message): 95​ def consume(self, message):
88 msg = fedmsg.encoding.loads(message.body) 96​ msg = fedmsg.encoding.loads(message.body)
89 try: 97​ try:
90 self.process(msg) 98​ data = self.trigger.process(msg)
99​ self.trigger.do_trigger(data)
91 except exc.TriggerMsgError, e: 100​ except exc.TriggerMsgError, e:
92 self.log.debug(e) 101​ self.log.debug(e)
93 return 102​ return
View Options

jobtriggers/koji_tag_msg.py

1import re 1​import re
2 2​
3import fedmsg 3​import fedmsg
4import fedmsg.encoding 4​import fedmsg.encoding
5 5​
6from twisted.internet.task import LoopingCall 6​from twisted.internet.task import LoopingCall
7 7​
8from . import config 8​from . import config
9from .jobtrigger import JobTrigger 9​from .jobtrigger import JobTrigger
10from . import exceptions as exc 10​from . import exceptions as exc
11 11​
12ITEM_TYPE = 'koji_tag' 12​ITEM_TYPE = 'koji_tag'
13 13​
14 14​
15def get_item(msg): 15​class KojiTagChangedJobTrigger(JobTrigger):
16 koji_tag = msg['msg']['tag'] 16​ def get_item(self, msg):
17 instance = msg['msg']['instance'] 17​ koji_tag = msg['msg']['tag']
18​ instance = msg['msg']['instance']
18 19​
19 if not instance == 'primary': 20​ if not instance == 'primary':
20 raise exc.TriggerMsgError('rejecting message not from primary koji instance (%s from %s)' 21​ raise exc.TriggerMsgError('rejecting message not from primary koji instance '
21 % (koji_tag, instance)) 22​ '(%s from %s)' % (koji_tag, instance))
22 23​
23 if not re.match('.*-updates(-testing)?-pending', koji_tag): 24​ if not re.match('.*-updates(-testing)?-pending', koji_tag):
24 raise exc.TriggerMsgError('rejecting message for not being an updates-(testing?)-pending', 25​ raise exc.TriggerMsgError('rejecting message for not being an '
25 'tag (%s)' % koji_tag) 26​ 'updates-(testing?)-pending tag (%s)' % koji_tag)
26 27​
27 m = re.match(r'^f\d\d-.*', koji_tag) 28​ m = re.match(r'^f\d\d-.*', koji_tag)
28 if m is None: 29​ if m is None:
29 raise exc.TriggerMsgError('rejecting message for not starting with fedora version ' 30​ raise exc.TriggerMsgError('rejecting message for not starting with fedora version '
30 '(e.g. f22) (%s)' % koji_tag) 31​ '(e.g. f22) (%s)' % koji_tag)
31 32​
32 return koji_tag 33​ return koji_tag
33 34​
35​ def process(self, msg):
36​ item = self.get_item(msg)
37​ data = {
38​ "_msg": msg,
39​ "message_type": "KojiTagChanged",
40​ "item": item,
41​ "item_type": ITEM_TYPE,
42​ "tag": item,
43​ }
34 44​
35class KojiTagChangedJobTrigger(JobTrigger): 45​ return data
46​
47​
48​class KojiTagChangedJobConsumer(fedmsg.consumers.FedmsgConsumer):
36 topic = "org.fedoraproject.prod.buildsys.tag" 49​ topic = "org.fedoraproject.prod.buildsys.tag"
37 jsonify = False 50​ jsonify = False
38 config_key = 'taskotron.kojitagchangedjobtrigger.enabled' 51​ config_key = 'taskotron.kojitagchangedjobconsumer.enabled'
39 52​
40 def __init__(self, *args, **kw): 53​ def __init__(self, *args, **kw):
41 super(KojiTagChangedJobTrigger, self).__init__(*args, **kw) 54​ super(KojiTagChangedJobConsumer, self).__init__(*args, **kw)
55​ self.trigger = KojiTagChangedJobTrigger(self.log)
42 56​
43 self.queued_tags = set() 57​ self.queued_tags = set()
44 self.queued_data = dict() 58​ self.queued_data = dict()
45 # check for bodhi_to_koji_tags tasks queue every config.fuse_delay seconds 59​ # check for bodhi_to_koji_tags tasks queue every config.fuse_delay seconds
46 lc = LoopingCall(self.delayed_consume) 60​ lc = LoopingCall(self.delayed_consume)
47 lc.start(config.fuse_delay) 61​ lc.start(config.fuse_delay)
48 62​
49 def delayed_consume(self): 63​ def delayed_consume(self):
50 if self.queued_tags: 64​ if self.queued_tags:
51 try: 65​ try:
52 for tag, data in self.queued_data.items(): 66​ for tag, data in self.queued_data.items():
53 self.do_trigger(data) 67​ self.trigger.do_trigger(data)
54 finally: 68​ finally:
55 self.queued_data = dict() 69​ self.queued_data = dict()
56 self.queued_tags = set() 70​ self.queued_tags = set()
57 else: 71​ else:
58 self.log.debug("Woke up, but there were no messages.") 72​ self.log.debug("Woke up, but there were no messages.")
59 73​
60 def consume(self, message): 74​ def consume(self, message):
61 msg = fedmsg.encoding.loads(message.body) 75​ msg = fedmsg.encoding.loads(message.body)
62 try: 76​ try:
63 item = get_item(msg) 77​ item = self.trigger.get_item(msg)
64 if item in self.queued_tags: 78​ if item in self.queued_tags:
65 self.log.info("koji_tag %s already in the queue, skipping", item) 79​ self.log.info("koji_tag %s already in the queue, skipping", item)
66 return 80​ return
67 81​
68 self.log.info('Adding koji_tag %s to the queue', item) 82​ self.log.info('Adding koji_tag %s to the queue', item)
69 83​
70 data = { 84​ data = self.trigger.process(msg)
71 "_msg": msg, ​
72 "message_type": "KojiTagChanged", ​
73 "item": item, ​
74 "item_type": ITEM_TYPE, ​
75 "tag": item, ​
76 } ​
77 85​
78 self.queued_tags.add(item) 86​ self.queued_tags.add(item)
79 self.queued_data[item] = data 87​ self.queued_data[item] = data
80 except exc.TriggerMsgError, e: 88​ except exc.TriggerMsgError, e:
81 self.log.debug(e) 89​ self.log.debug(e)
View Options

setup.py

1from setuptools import setup, Command 1​from setuptools import setup, Command
2 2​
3class PyTest(Command): 3​class PyTest(Command):
4 user_options = [] 4​ user_options = []
5 def initialize_options(self): 5​ def initialize_options(self):
6 pass 6​ pass
7 def finalize_options(self): 7​ def finalize_options(self):
8 pass 8​ pass
9 def run(self): 9​ def run(self):
10 import subprocess 10​ import subprocess
11 errno = subprocess.call(['py.test', 'testing']) 11​ errno = subprocess.call(['py.test', 'testing'])
12 raise SystemExit(errno) 12​ raise SystemExit(errno)
13 13​
14 14​
15​moksha_consumer = (
16​ 'kojibuildcompletedjobconsumer = jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer',
17​ 'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer',
18​ 'composecompletedjobconsumer = jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer',
19​ 'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer',
20​)
21​
15setup( 22​setup(
16 name='jobtriggers', 23​ name='jobtriggers',
17 version='0.4.3', 24​ version='0.4.3',
18 description='triggering jobs via fedmsg', 25​ description='triggering jobs via fedmsg',
19 author='Tim Flink', 26​ author='Tim Flink',
20 author_email='tflink@fedoraproject.org', 27​ author_email='tflink@fedoraproject.org',
21 license='GPLv2+', 28​ license='GPLv2+',
22 url='https://fedorahosted.org/fedora-qa', 29​ url='https://fedorahosted.org/fedora-qa',
23 packages=['jobtriggers'], 30​ packages=['jobtriggers'],
24 entry_points={ 31​ entry_points={
25 'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'], 32​ 'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'],
26 'moksha.consumer': ( 33​ 'moksha.consumer': moksha_consumer,
kparalUnsubmitted
Not Done

This seems to be the correct solution:

entry_points={
    'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'],
    'moksha.consumer': (
        'kojibuildcompletedjobconsumer = '
        'jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer',
        'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer',
        'composecompletedjobconsumer = '
        'jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer',
        'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer',
    ),
},

but it's hard to read. What about this?

moksha_consumer = (
    'kojibuildcompletedjobconsumer = jobtriggers.koji_build_msg:KojiBuildCompletedJobConsumer',
    'kojitagchangedjobconsumer = jobtriggers.koji_tag_msg:KojiTagChangedJobConsumer',
    'composecompletedjobconsumer = jobtriggers.compose_complete_msg:ComposeCompletedJobConsumer',
    'distgitcommitjobconsumer = jobtriggers.dist_git_commit_msg:DistGitCommitJobConsumer',
)

setup(
    name='jobtriggers',
    version='0.4.3',
    description='triggering jobs via fedmsg',
    author='Tim Flink',
    author_email='tflink@fedoraproject.org',
    license='GPLv2+',
    url='https://fedorahosted.org/fedora-qa',
    packages=['jobtriggers'],
    entry_points={
        'console_scripts': ['jobrunner=jobtriggers.jobrunner:main'],
        'moksha.consumer': moksha_consumer,
    },
    include_package_data=True,
    install_requires=[
        'fedmsg',
        'requests',
    ],
    cmdclass={'test': PyTest}
)
27 'kojibuildcompletedjobtrigger = jobtriggers.koji_build_msg:KojiBuildCompletedJobTrigger', ​
28 'kojitagchangedjobtrigger = jobtriggers.koji_tag_msg:KojiTagChangedJobTrigger', ​
29 'composecompletedjobtrigger = jobtriggers.compose_complete_msg:ComposeCompletedJobTrigger', ​
30 ), ​
31 }, 34​ },
32 include_package_data=True, 35​ include_package_data=True,
33 install_requires=[ 36​ install_requires=[
34 'fedmsg', 37​ 'fedmsg',
35 'requests', 38​ 'requests',
36 ], 39​ ],
37 cmdclass={'test': PyTest} 40​ cmdclass={'test': PyTest}
38) 41​)
View Options

testing/functest_cli.py

  • This file was deleted.
This file was completely deleted. Show File Contents
View Options

testing/test_compose_trigger.py

1import pytest 1​import pytest
2from dingus import Dingus 2​from dingus import Dingus
3from munch import Munch 3​from munch import Munch
4 4​
5from jobtriggers import jobtrigger 5​from jobtriggers import jobtrigger
6from jobtriggers import compose_complete_msg 6​from jobtriggers import compose_complete_msg
7 7​
8 8​
9@pytest.mark.usefixtures('prepare') 9​@pytest.mark.usefixtures('prepare')
10class TestComposeCompletedJobTrigger(): 10​class TestComposeCompletedJobConsumer():
11 11​
12 @pytest.fixture 12​ @pytest.fixture
13 def prepare(self, monkeypatch): 13​ def prepare(self, monkeypatch):
14 self.ref_arch = '' 14​ self.ref_arch = ''
15 self.ref_branch = '21' 15​ self.ref_branch = '21'
16 self.ref_topic = 'org.fedoraproject.prod.compose.branched.complete' 16​ self.ref_topic = 'org.fedoraproject.prod.compose.branched.complete'
17 17​
18 self._create_msg(self.ref_arch, self.ref_branch, self.ref_topic) 18​ self._create_msg(self.ref_arch, self.ref_branch, self.ref_topic)
19 19​
20 self.ref_tasks = ['compose_task1', 'compose_task2'] 20​ self.ref_tasks = ['compose_task1', 'compose_task2']
21 self.ref_arches = 'x86_64' 21​ self.ref_arches = 'x86_64'
22 22​
23 stub_hub = Munch(config=Munch(get=0)) 23​ stub_hub = Munch(config=Munch(get=0))
24 24​
25 self.helper = compose_complete_msg.ComposeCompletedJobTrigger(stub_hub) 25​ self.helper = compose_complete_msg.ComposeCompletedJobConsumer(stub_hub)
26 self.helper.runner = Dingus() 26​ self.helper.trigger.runner = Dingus()
27 27​
28 jobtrigger.config.compose_completed_tasks = self.ref_tasks 28​ jobtrigger.config.compose_completed_tasks = self.ref_tasks
29 jobtrigger.config.valid_arches = [self.ref_arches] 29​ jobtrigger.config.valid_arches = [self.ref_arches]
30 jobtrigger.config.job_logging = False 30​ jobtrigger.config.job_logging = False
31 31​
32 jobtrigger.config.trigger_rules_template = """--- 32​ jobtrigger.config.trigger_rules_template = """---
33- do: 33​- do:
34 - {tasks: %r} 34​ - {tasks: %r}
35 when: {message_type: ComposeCompleted} 35​ when: {message_type: ComposeCompleted}
36""" % (self.ref_tasks) 36​""" % (self.ref_tasks)
37 37​
38 def _create_msg(self, ref_arch, ref_branch, ref_topic): 38​ def _create_msg(self, ref_arch, ref_branch, ref_topic):
39 self.ref_message = Munch(body='{"i": 1,\ 39​ self.ref_message = Munch(body='{"i": 1,\
40 "msg": {"arch": "%s",\ 40​ "msg": {"arch": "%s",\
41 "branch": "%s",\ 41​ "branch": "%s",\
42 "log": "done"},\ 42​ "log": "done"},\
43 "topic": "%s"}' % 43​ "topic": "%s"}' %
44 (ref_arch, ref_branch, ref_topic)) 44​ (ref_arch, ref_branch, ref_topic))
45 45​
46 def test_consume(self): 46​ def test_consume(self):
47 self.helper.consume(self.ref_message) 47​ self.helper.consume(self.ref_message)
48 48​
49 runner_calls = self.helper.runner.trigger_job.calls() 49​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
50 50​
51 assert len(runner_calls) == 2 51​ assert len(runner_calls) == 2
52 52​
53 assert runner_calls[0][1] == ( 53​ assert runner_calls[0][1] == (
54 self.ref_branch, compose_complete_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_arches) 54​ self.ref_branch, compose_complete_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_arches)
55 assert runner_calls[1][1] == ( 55​ assert runner_calls[1][1] == (
56 self.ref_branch, compose_complete_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_arches) 56​ self.ref_branch, compose_complete_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_arches)
57 57​
58 def test_consume_secondary_arch(self): 58​ def test_consume_secondary_arch(self):
59 ref_arch = 's390' 59​ ref_arch = 's390'
60 self._create_msg(ref_arch, self.ref_branch, self.ref_topic) 60​ self._create_msg(ref_arch, self.ref_branch, self.ref_topic)
61 61​
62 self.helper.consume(self.ref_message) 62​ self.helper.consume(self.ref_message)
63 63​
64 runner_calls = self.helper.runner.trigger_job.calls() 64​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
65 65​
66 assert len(runner_calls) == 0 66​ assert len(runner_calls) == 0
67 67​
68 def test_consume_incorrect_topic(self): 68​ def test_consume_incorrect_topic(self):
69 ref_topic = 'fake_topic' 69​ ref_topic = 'fake_topic'
70 self._create_msg(self.ref_arch, self.ref_branch, ref_topic) 70​ self._create_msg(self.ref_arch, self.ref_branch, ref_topic)
71 71​
72 self.helper.consume(self.ref_message) 72​ self.helper.consume(self.ref_message)
73 73​
74 runner_calls = self.helper.runner.trigger_job.calls() 74​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
75 75​
76 assert len(runner_calls) == 0 76​ assert len(runner_calls) == 0
77 77​
78 def test_consume_allowed_topics(self): 78​ def test_consume_allowed_topics(self):
79 ref_topics = ['org.fedoraproject.prod.compose.branched.complete', 79​ ref_topics = ['org.fedoraproject.prod.compose.branched.complete',
80 'org.fedoraproject.prod.compose.rawhide.complete'] 80​ 'org.fedoraproject.prod.compose.rawhide.complete']
81 81​
82 for ref_topic in ref_topics: 82​ for ref_topic in ref_topics:
83 self._create_msg(self.ref_arch, self.ref_branch, ref_topic) 83​ self._create_msg(self.ref_arch, self.ref_branch, ref_topic)
84 84​
85 self.helper.consume(self.ref_message) 85​ self.helper.consume(self.ref_message)
86 86​
87 runner_calls = self.helper.runner.trigger_job.calls() 87​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
88 88​
89 assert len(runner_calls) == 4 89​ assert len(runner_calls) == 4
View Options

testing/test_dist_git_trigger.py

1import pytest 1​import pytest
2import mock 2​import mock
3from dingus import Dingus 3​from dingus import Dingus
4from munch import Munch 4​from munch import Munch
5 5​
6from jobtriggers import dist_git_commit_msg 6​from jobtriggers import dist_git_commit_msg
7from jobtriggers import utils 7​from jobtriggers import utils
8 8​
9def mock_clone_repo(repo, branch='master', fallback=None): 9​def mock_clone_repo(repo, branch='master', fallback=None):
10 return '/tmp/path_to_repo', branch 10​ return '/tmp/path_to_repo', branch
11 11​
12@pytest.mark.usefixtures('prepare') 12​@pytest.mark.usefixtures('prepare')
13class TestDistGitCommitJobTrigger(): 13​class TestDistGitCommitJobConsumer():
14 14​
15 @pytest.fixture 15​ @pytest.fixture
16 def prepare(self, monkeypatch): 16​ def prepare(self, monkeypatch):
17 self.item_type = 'dist_git_commit' 17​ self.item_type = 'dist_git_commit'
18 self.ref_message = self._create_msg() 18​ self.ref_message = self._create_msg()
19 self.ref_tasks = ['dist_git_task1', 'dist_git_task2'] 19​ self.ref_tasks = ['dist_git_task1', 'dist_git_task2']
20 self.message_type = 'DistGitRPMCommit' 20​ self.message_type = 'DistGitRPMCommit'
21 21​
22 stub_hub = Munch(config=Munch(get=0)) 22​ stub_hub = Munch(config=Munch(get=0))
23 23​
24 self.helper = dist_git_commit_msg.DistGitCommitJobTrigger(stub_hub) 24​ self.helper = dist_git_commit_msg.DistGitCommitJobConsumer(stub_hub)
25 self.helper.runner = Dingus() 25​ self.helper.trigger.runner = Dingus()
26 26​
27 dist_git_commit_msg.config.trigger_rules_template = """--- 27​ dist_git_commit_msg.config.trigger_rules_template = """---
28- when: {message_type: DistGitCommit, namespace: rpms} 28​- when: {message_type: DistGitCommit, namespace: rpms}
29 do: 29​ do:
30 - {discover: {repo: 'http://foo/test-rpms/${name}.git'}} 30​ - {discover: {repo: 'http://foo/test-rpms/${name}.git'}}
31 - {tasks: [speclint]} # Whatever speclint is.. 31​ - {tasks: [speclint]} # Whatever speclint is..
32- when: {message_type: DistGitCommit, namespace: modules} 32​- when: {message_type: DistGitCommit, namespace: modules}
33 do: 33​ do:
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Line(s)49 return Munch(body={
97 "msg_id": "2016-c0f8f391-7936-4c4e-9401-13c0c2372102", 97​ "msg_id": "2016-c0f8f391-7936-4c4e-9401-13c0c2372102",
98 "timestamp": 1480707154.0, 98​ "timestamp": 1480707154.0,
99 "topic": "org.fedoraproject.prod.git.receive" 99​ "topic": "org.fedoraproject.prod.git.receive"
100 }) 100​ })
101 101​
102 def test_consume(self): 102​ def test_consume(self):
103 self.helper.consume(self.ref_message) 103​ self.helper.consume(self.ref_message)
104 104​
105 runner_calls = self.helper.runner.trigger_job.calls() 105​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
106 106​
107 assert len(runner_calls) == 1 107​ assert len(runner_calls) == 1
108 108​
109 assert runner_calls[0][1] == ( 109​ assert runner_calls[0][1] == (
110 'rpms/vdr-epg-daemon#6e5218a6e0b18c5a739e2164f85436f2ada5e475', 110​ 'rpms/vdr-epg-daemon#6e5218a6e0b18c5a739e2164f85436f2ada5e475',
111 'dist_git_commit', 111​ 'dist_git_commit',
112 'speclint', 112​ 'speclint',
113 'x86_64', 113​ 'x86_64',
114 ) 114​ )
View Options

testing/test_jobtrigger.py

Show All 15 Lines15def mock_clone_repo(repo, branch='master', fallback=None):
16 return '/tmp/path_to_repo', branch 16​ return '/tmp/path_to_repo', branch
17 17​
18 18​
19@pytest.mark.usefixtures('prepare') 19​@pytest.mark.usefixtures('prepare')
20class TestJobtrigger(): 20​class TestJobtrigger():
21 21​
22 @pytest.fixture 22​ @pytest.fixture
23 def prepare(self, monkeypatch): 23​ def prepare(self, monkeypatch):
24 stub_hub = Munch(config=Munch(get=0)) 24​ stub_logging = Munch(info=lambda *args: True)
25 self.helper = BogusJobTrigger(stub_hub) 25​ self.helper = BogusJobTrigger(stub_logging)
26 self.helper.runner = Dingus() 26​ self.helper.runner = Dingus()
27 27​
28 self.ref_discovered_tasks = ['discovered_task_1'] 28​ self.ref_discovered_tasks = ['discovered_task_1']
29 mock_discover_tasks = mock.Mock(return_value=self.ref_discovered_tasks) 29​ mock_discover_tasks = mock.Mock(return_value=self.ref_discovered_tasks)
30 monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks) 30​ monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks)
31 31​
32 monkeypatch.setattr(utils, 'clone_repo', mock_clone_repo) 32​ monkeypatch.setattr(utils, 'clone_repo', mock_clone_repo)
33 33​
▲ Show 20 Lines • Show All 66 Lines • ▼ Show 20 Line(s)91 def test__tasks_from_rule_discover_set_branch(self):
100 assert tasks[0]['branch'] == 'mybranch' 100​ assert tasks[0]['branch'] == 'mybranch'
101 101​
102 def test__tasks_from_rule_discover_recursive(self, monkeypatch): 102​ def test__tasks_from_rule_discover_recursive(self, monkeypatch):
103 utils_dingus = Dingus(discover_tasks__returns=self.ref_discovered_tasks) 103​ utils_dingus = Dingus(discover_tasks__returns=self.ref_discovered_tasks)
104 monkeypatch.setattr(utils, 'discover_tasks', utils_dingus.discover_tasks) 104​ monkeypatch.setattr(utils, 'discover_tasks', utils_dingus.discover_tasks)
105 105​
106 ref_repo = 'http://bogus.repo/firefox.git' 106​ ref_repo = 'http://bogus.repo/firefox.git'
107 rule = {'do': [{'discover': {'repo': ref_repo, 'recursive': True}}]} 107​ rule = {'do': [{'discover': {'repo': ref_repo, 'recursive': True}}]}
108 tasks = self.helper._tasks_from_rule(rule, self.ref_message_data) 108​ self.helper._tasks_from_rule(rule, self.ref_message_data)
kparalUnsubmitted
Not Done

Remove tasks = .

109 109​
110 discover_calls = utils_dingus.calls() 110​ discover_calls = utils_dingus.calls()
111 assert len(discover_calls) == 1 111​ assert len(discover_calls) == 1
112 assert discover_calls[0][0] == 'discover_tasks' 112​ assert discover_calls[0][0] == 'discover_tasks'
113 assert discover_calls[0][1][0] == '/tmp/path_to_repo/' 113​ assert discover_calls[0][1][0] == '/tmp/path_to_repo/'
114 assert discover_calls[0][1][1] == self.ref_message_data['item_type'] 114​ assert discover_calls[0][1][1] == self.ref_message_data['item_type']
115 assert discover_calls[0][1][2] == True 115​ assert discover_calls[0][1][2] == True
116 116​
▲ Show 20 Lines • Show All 54 Lines • Show Last 20 Lines
View Options

testing/test_koji_build_trigger.py

Show All 9 Lines
10params = [('someowner', koji_build_msg.ITEM_TYPE_PACKAGE, "KojiBuildPackageCompleted"), 10​params = [('someowner', koji_build_msg.ITEM_TYPE_PACKAGE, "KojiBuildPackageCompleted"),
11 ('containerbuild', koji_build_msg.ITEM_TYPE_DOCKER, "KojiBuildDockerCompleted")] 11​ ('containerbuild', koji_build_msg.ITEM_TYPE_DOCKER, "KojiBuildDockerCompleted")]
12 12​
13def mock_clone_repo(repo, branch='master', fallback=None): 13​def mock_clone_repo(repo, branch='master', fallback=None):
14 return '/tmp/path_to_repo', branch 14​ return '/tmp/path_to_repo', branch
15 15​
16@pytest.mark.usefixtures('prepare') 16​@pytest.mark.usefixtures('prepare')
17@pytest.mark.parametrize('buildtype', params) 17​@pytest.mark.parametrize('buildtype', params)
18class TestKojiBuildCompletedJobTrigger(): 18​class TestKojiBuildCompletedJobConsumer():
19 19​
20 @pytest.fixture 20​ @pytest.fixture
21 def prepare(self, monkeypatch, buildtype): 21​ def prepare(self, monkeypatch, buildtype):
22 self.ref_instance = 'primary' 22​ self.ref_instance = 'primary'
23 self.ref_new = koji.BUILD_STATES['COMPLETE'] 23​ self.ref_new = koji.BUILD_STATES['COMPLETE']
24 self.ref_name = 'stage' 24​ self.ref_name = 'stage'
25 self.ref_version = '4.1.1' 25​ self.ref_version = '4.1.1'
26 self.ref_release = '3.fc18' 26​ self.ref_release = '3.fc18'
Show All 9 Lines35 self.ref_message = self._create_msg(self.ref_instance, self.ref_new, self.ref_name,
36 self.ref_release, self.ref_version, self.ref_owner) 36​ self.ref_release, self.ref_version, self.ref_owner)
37 37​
38 self.ref_tasks = ['koji_build_task1', 'koji_build_task2'] 38​ self.ref_tasks = ['koji_build_task1', 'koji_build_task2']
39 self.ref_tasks_docker = ['docker_build_task'] 39​ self.ref_tasks_docker = ['docker_build_task']
40 self.ref_validarches = ['i386', 'x86_64'] 40​ self.ref_validarches = ['i386', 'x86_64']
41 41​
42 stub_hub = Munch(config=Munch(get=0)) 42​ stub_hub = Munch(config=Munch(get=0))
43 43​
44 self.helper = koji_build_msg.KojiBuildCompletedJobTrigger(stub_hub) 44​ self.helper = koji_build_msg.KojiBuildCompletedJobConsumer(stub_hub)
45 self.helper.runner = Dingus() 45​ self.helper.trigger.runner = Dingus()
46 46​
47 koji_build_msg.config.trigger_rules_template = """--- 47​ koji_build_msg.config.trigger_rules_template = """---
48- do: 48​- do:
49 - {discover: {repo: 'http://bogus/firefox.git'}} 49​ - {discover: {repo: 'http://bogus/firefox.git'}}
50 - {tasks: %s} 50​ - {tasks: %s}
51 when: {message_type: %s} 51​ when: {message_type: %s}
52- do: 52​- do:
53 - {tasks: %s} 53​ - {tasks: %s}
Show All 33 Lines78 msg = Munch(body='{"i": 1,\
87 "topic": "org.fedoraproject.prod.buildsys.build.state.change",\ 87​ "topic": "org.fedoraproject.prod.buildsys.build.state.change",\
88 "username": "apache"}' % 88​ "username": "apache"}' %
89 (ref_owner, ref_instance, ref_new, ref_name, ref_release, ref_version)) 89​ (ref_owner, ref_instance, ref_new, ref_name, ref_release, ref_version))
90 return msg 90​ return msg
91 91​
92 def test_consume(self): 92​ def test_consume(self):
93 self.helper.consume(self.ref_message) 93​ self.helper.consume(self.ref_message)
94 94​
95 runner_calls = self.helper.runner.trigger_job.calls() 95​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
96 96​
97 assert len(runner_calls) == 4 97​ assert len(runner_calls) == 4
98 98​
99 assert runner_calls[0][1] == (self.ref_nvr, self.item_type, 99​ assert runner_calls[0][1] == (self.ref_nvr, self.item_type,
100 self.ref_tasks[0], self.ref_validarches[0]) 100​ self.ref_tasks[0], self.ref_validarches[0])
101 assert runner_calls[1][1] == (self.ref_nvr, self.item_type, 101​ assert runner_calls[1][1] == (self.ref_nvr, self.item_type,
102 self.ref_tasks[0], self.ref_validarches[1]) 102​ self.ref_tasks[0], self.ref_validarches[1])
103 assert runner_calls[2][1] == (self.ref_nvr, self.item_type, 103​ assert runner_calls[2][1] == (self.ref_nvr, self.item_type,
104 self.ref_tasks[1], self.ref_validarches[0]) 104​ self.ref_tasks[1], self.ref_validarches[0])
105 assert runner_calls[3][1] == (self.ref_nvr, self.item_type, 105​ assert runner_calls[3][1] == (self.ref_nvr, self.item_type,
106 self.ref_tasks[1], self.ref_validarches[1]) 106​ self.ref_tasks[1], self.ref_validarches[1])
107 107​
108 def test_consume_docker(self): 108​ def test_consume_docker(self):
109 # skip test when item is docker image 109​ # skip test when item is docker image
110 if self.item_type == koji_build_msg.ITEM_TYPE_DOCKER: 110​ if self.item_type == koji_build_msg.ITEM_TYPE_DOCKER:
111 return 111​ return
112 ref_name = 'docker' 112​ ref_name = 'docker'
113 self.ref_message = self._create_msg(self.ref_instance, self.ref_new, ref_name, 113​ self.ref_message = self._create_msg(self.ref_instance, self.ref_new, ref_name,
114 self.ref_release, self.ref_version, self.ref_owner) 114​ self.ref_release, self.ref_version, self.ref_owner)
115 115​
116 self.helper.consume(self.ref_message) 116​ self.helper.consume(self.ref_message)
117 117​
118 runner_calls = self.helper.runner.trigger_job.calls() 118​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
119 119​
120 assert len(runner_calls) == 6 120​ assert len(runner_calls) == 6
121 121​
122 assert runner_calls[0][1] == (self.ref_nvr_docker, self.item_type, 122​ assert runner_calls[0][1] == (self.ref_nvr_docker, self.item_type,
123 self.ref_tasks[0], self.ref_validarches[0]) 123​ self.ref_tasks[0], self.ref_validarches[0])
124 assert runner_calls[1][1] == (self.ref_nvr_docker, self.item_type, 124​ assert runner_calls[1][1] == (self.ref_nvr_docker, self.item_type,
125 self.ref_tasks[0], self.ref_validarches[1]) 125​ self.ref_tasks[0], self.ref_validarches[1])
126 assert runner_calls[2][1] == (self.ref_nvr_docker, self.item_type, 126​ assert runner_calls[2][1] == (self.ref_nvr_docker, self.item_type,
127 self.ref_tasks[1], self.ref_validarches[0]) 127​ self.ref_tasks[1], self.ref_validarches[0])
128 assert runner_calls[3][1] == (self.ref_nvr_docker, self.item_type, 128​ assert runner_calls[3][1] == (self.ref_nvr_docker, self.item_type,
129 self.ref_tasks[1], self.ref_validarches[1]) 129​ self.ref_tasks[1], self.ref_validarches[1])
130 assert runner_calls[4][1] == (self.ref_nvr_docker, self.item_type, 130​ assert runner_calls[4][1] == (self.ref_nvr_docker, self.item_type,
131 self.ref_tasks_docker[0], self.ref_validarches[0]) 131​ self.ref_tasks_docker[0], self.ref_validarches[0])
132 assert runner_calls[5][1] == (self.ref_nvr_docker, self.item_type, 132​ assert runner_calls[5][1] == (self.ref_nvr_docker, self.item_type,
133 self.ref_tasks_docker[0], self.ref_validarches[1]) 133​ self.ref_tasks_docker[0], self.ref_validarches[1])
134 134​
135 def test_consume_distgit(self, monkeypatch): 135​ def test_consume_distgit(self, monkeypatch):
136 ref_distgit_task = 'abicheck' 136​ ref_distgit_task = 'abicheck'
137 mock_discover_tasks = mock.Mock(return_value=[ref_distgit_task]) 137​ mock_discover_tasks = mock.Mock(return_value=[ref_distgit_task])
138 monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks) 138​ monkeypatch.setattr(utils, 'discover_tasks', mock_discover_tasks)
139 139​
140 self.helper.consume(self.ref_message) 140​ self.helper.consume(self.ref_message)
141 141​
142 runner_calls = self.helper.runner.trigger_job.calls() 142​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
143 143​
144 assert len(runner_calls) == 6 144​ assert len(runner_calls) == 6
145 145​
146 assert runner_calls[0][1] == (self.ref_nvr, self.item_type, 146​ assert runner_calls[0][1] == (self.ref_nvr, self.item_type,
147 ref_distgit_task, self.ref_validarches[0]) 147​ ref_distgit_task, self.ref_validarches[0])
148 assert runner_calls[1][1] == (self.ref_nvr, self.item_type, 148​ assert runner_calls[1][1] == (self.ref_nvr, self.item_type,
149 ref_distgit_task, self.ref_validarches[1]) 149​ ref_distgit_task, self.ref_validarches[1])
150 assert runner_calls[2][1] == (self.ref_nvr, self.item_type, 150​ assert runner_calls[2][1] == (self.ref_nvr, self.item_type,
151 self.ref_tasks[0], self.ref_validarches[0]) 151​ self.ref_tasks[0], self.ref_validarches[0])
152 assert runner_calls[3][1] == (self.ref_nvr, self.item_type, 152​ assert runner_calls[3][1] == (self.ref_nvr, self.item_type,
153 self.ref_tasks[0], self.ref_validarches[1]) 153​ self.ref_tasks[0], self.ref_validarches[1])
154 assert runner_calls[4][1] == (self.ref_nvr, self.item_type, 154​ assert runner_calls[4][1] == (self.ref_nvr, self.item_type,
155 self.ref_tasks[1], self.ref_validarches[0]) 155​ self.ref_tasks[1], self.ref_validarches[0])
156 assert runner_calls[5][1] == (self.ref_nvr, self.item_type, 156​ assert runner_calls[5][1] == (self.ref_nvr, self.item_type,
157 self.ref_tasks[1], self.ref_validarches[1]) 157​ self.ref_tasks[1], self.ref_validarches[1])
158 158​
159 def test_consume_not_primary_instance(self): 159​ def test_consume_not_primary_instance(self):
160 ref_instance = 'ppc' 160​ ref_instance = 'ppc'
161 self.ref_message = self._create_msg(ref_instance, self.ref_new, self.ref_name, 161​ self.ref_message = self._create_msg(ref_instance, self.ref_new, self.ref_name,
162 self.ref_release, self.ref_version, self.ref_owner) 162​ self.ref_release, self.ref_version, self.ref_owner)
163 163​
164 self.helper.consume(self.ref_message) 164​ self.helper.consume(self.ref_message)
165 165​
166 runner_calls = self.helper.runner.trigger_job.calls() 166​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
167 167​
168 assert runner_calls == [] 168​ assert runner_calls == []
169 169​
170 def test_consume_notcomplete_state(self): 170​ def test_consume_notcomplete_state(self):
171 ref_new = koji.BUILD_STATES['BUILDING'] 171​ ref_new = koji.BUILD_STATES['BUILDING']
172 self.ref_message = self._create_msg(self.ref_instance, ref_new, self.ref_name, 172​ self.ref_message = self._create_msg(self.ref_instance, ref_new, self.ref_name,
173 self.ref_release, self.ref_version, self.ref_owner) 173​ self.ref_release, self.ref_version, self.ref_owner)
174 174​
175 self.helper.consume(self.ref_message) 175​ self.helper.consume(self.ref_message)
176 176​
177 runner_calls = self.helper.runner.trigger_job.calls() 177​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
178 178​
179 assert runner_calls == [] 179​ assert runner_calls == []
180 180​
181 def test_consume_blacklisted_release(self): 181​ def test_consume_blacklisted_release(self):
182 # skip test when item is docker image 182​ # skip test when item is docker image
183 if self.item_type == koji_build_msg.ITEM_TYPE_DOCKER: 183​ if self.item_type == koji_build_msg.ITEM_TYPE_DOCKER:
184 return 184​ return
185 185​
186 ref_release = '3.el5.1' 186​ ref_release = '3.el5.1'
187 self.ref_message = self._create_msg(self.ref_instance, self.ref_new, self.ref_name, 187​ self.ref_message = self._create_msg(self.ref_instance, self.ref_new, self.ref_name,
188 ref_release, self.ref_version, self.ref_owner) 188​ ref_release, self.ref_version, self.ref_owner)
189 189​
190 self.helper.consume(self.ref_message) 190​ self.helper.consume(self.ref_message)
191 191​
192 runner_calls = self.helper.runner.trigger_job.calls() 192​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
193 193​
194 assert runner_calls == [] 194​ assert runner_calls == []
View Options

testing/test_koji_tag_trigger.py

1import pytest 1​import pytest
2from dingus import Dingus 2​from dingus import Dingus
3from munch import Munch 3​from munch import Munch
4from copy import deepcopy 4​from copy import deepcopy
5import fedmsg.encoding 5​import fedmsg.encoding
6​import fedmsg.consumers
6 7​
7from jobtriggers import koji_tag_msg 8​from jobtriggers import koji_tag_msg
8 9​
9 10​
10@pytest.mark.usefixtures('prepare') 11​@pytest.mark.usefixtures('prepare')
11class TestKojiTagChangedJobTrigger(): 12​class TestKojiTagChangedJobConsumer():
12 13​
13 @pytest.fixture 14​ @pytest.fixture
14 def prepare(self, monkeypatch): 15​ def prepare(self, monkeypatch):
15 self.ref_instance = 'primary' 16​ self.ref_instance = 'primary'
16 self.ref_name = 'stage' 17​ self.ref_name = 'stage'
17 self.ref_version = '4.1.1' 18​ self.ref_version = '4.1.1'
18 self.ref_release = '3.fc18' 19​ self.ref_release = '3.fc18'
19 self.ref_tag = 'f18-updates-pending' 20​ self.ref_tag = 'f18-updates-pending'
Show All 10 Lines30 self.ref_data = {
30 "_msg": {}, 31​ "_msg": {},
31 "message_type": "KojiTagChanged", 32​ "message_type": "KojiTagChanged",
32 "item": self.ref_tag, 33​ "item": self.ref_tag,
33 "item_type": "koji_tag", 34​ "item_type": "koji_tag",
34 "tag": self.ref_tag, 35​ "tag": self.ref_tag,
35 } 36​ }
36 37​
37 stub_hub = Munch(config=Munch(get=0)) 38​ stub_hub = Munch(config=Munch(get=0))
38 self.helper = koji_tag_msg.KojiTagChangedJobTrigger(stub_hub) 39​ self.helper = koji_tag_msg.KojiTagChangedJobConsumer(stub_hub)
39 40​
40 self.helper.runner = Dingus() 41​ self.helper.trigger.runner = Dingus()
41 koji_tag_msg.config.trigger_rules_template = """--- 42​ koji_tag_msg.config.trigger_rules_template = """---
42- do: 43​- do:
43 - {tasks: [depcheck]} 44​ - {tasks: [depcheck]}
44 when: {message_type: KojiTagChanged} 45​ when: {message_type: KojiTagChanged}
45- do: 46​- do:
46 - {tasks: [upgradepath]} 47​ - {tasks: [upgradepath]}
47 when: {message_type: KojiTagChanged, tag: {$regex: '/.*(?<!testing-pending)$$/'}} 48​ when: {message_type: KojiTagChanged, tag: {$regex: '/.*(?<!testing-pending)$$/'}}
48""" 49​"""
Show All 30 Lines70 def test_delayed_queue(self):
79 } 80​ }
80 81​
81 def test_delayed_consume(self): 82​ def test_delayed_consume(self):
82 self.helper.queued_tags = [self.ref_tag] 83​ self.helper.queued_tags = [self.ref_tag]
83 self.helper.queued_data = {self.ref_tag: self.ref_data} 84​ self.helper.queued_data = {self.ref_tag: self.ref_data}
84 85​
85 self.helper.delayed_consume() 86​ self.helper.delayed_consume()
86 87​
87 runner_calls = self.helper.runner.trigger_job.calls() 88​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
88 89​
89 assert len(runner_calls) == 4 90​ assert len(runner_calls) == 4
90 91​
91 assert runner_calls[0][1] == ( 92​ assert runner_calls[0][1] == (
92 self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[0]) 93​ self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[0])
93 assert runner_calls[1][1] == ( 94​ assert runner_calls[1][1] == (
94 self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[1]) 95​ self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[1])
95 assert runner_calls[2][1] == ( 96​ assert runner_calls[2][1] == (
96 self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_validarches[0]) 97​ self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_validarches[0])
97 assert runner_calls[3][1] == ( 98​ assert runner_calls[3][1] == (
98 self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_validarches[1]) 99​ self.ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[1], self.ref_validarches[1])
99 100​
100 assert self.helper.queued_tags == set() 101​ assert self.helper.queued_tags == set()
101 assert self.helper.queued_data == dict() 102​ assert self.helper.queued_data == dict()
102 103​
103 def test_consume_not_primary_instance(self): 104​ def test_consume_not_primary_instance(self):
104 ref_instance = 'ppc' 105​ ref_instance = 'ppc'
105 self._create_msg( 106​ self._create_msg(
106 ref_instance, self.ref_name, self.ref_release, self.ref_tag, self.ref_version) 107​ ref_instance, self.ref_name, self.ref_release, self.ref_tag, self.ref_version)
107 108​
108 self.helper.consume(self.ref_message) 109​ self.helper.consume(self.ref_message)
109 110​
110 runner_calls = self.helper.runner.trigger_job.calls() 111​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
111 112​
112 assert runner_calls == [] 113​ assert runner_calls == []
113 114​
114 def test_consume_kojitag_not_pending(self): 115​ def test_consume_kojitag_not_pending(self):
115 ref_tag = 'f18-updates-testing' 116​ ref_tag = 'f18-updates-testing'
116 self._create_msg( 117​ self._create_msg(
117 self.ref_instance, self.ref_name, self.ref_release, ref_tag, self.ref_version) 118​ self.ref_instance, self.ref_name, self.ref_release, ref_tag, self.ref_version)
118 self.helper.consume(self.ref_message) 119​ self.helper.consume(self.ref_message)
119 120​
120 assert self.helper.queued_tags == set() 121​ assert self.helper.queued_tags == set()
121 122​
122 def test_delayed_consume_upgradepath(self): 123​ def test_delayed_consume_upgradepath(self):
123 ref_tag = 'f18-updates-testing-pending' 124​ ref_tag = 'f18-updates-testing-pending'
124 ref_data = deepcopy(self.ref_data) 125​ ref_data = deepcopy(self.ref_data)
125 ref_data['item'] = ref_data['tag'] = ref_tag 126​ ref_data['item'] = ref_data['tag'] = ref_tag
126 127​
127 self.helper.queued_tags = [ref_tag] 128​ self.helper.queued_tags = [ref_tag]
128 self.helper.queued_data = {ref_tag: ref_data} 129​ self.helper.queued_data = {ref_tag: ref_data}
129 130​
130 self.helper.delayed_consume() 131​ self.helper.delayed_consume()
131 132​
132 runner_calls = self.helper.runner.trigger_job.calls() 133​ runner_calls = self.helper.trigger.runner.trigger_job.calls()
133 134​
134 assert len(runner_calls) == 2 135​ assert len(runner_calls) == 2
135 136​
136 assert runner_calls[0][1] == ( 137​ assert runner_calls[0][1] == (
137 ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[0]) 138​ ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[0])
138 assert runner_calls[1][1] == ( 139​ assert runner_calls[1][1] == (
139 ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[1]) 140​ ref_tag, koji_tag_msg.ITEM_TYPE, self.ref_tasks[0], self.ref_validarches[1])
140 141​
Show All 35 Lines