Differential D827

cli: Get all results from datagrepper
ClosedPublic
Actions

Authored by mkrizek on Apr 26 2016, 12:39 PM.

Details

Reviewers
tflink
Maniphest Tasks
T764: taskotron-trigger's jobrunner doesn't seem to replay all jobs from datagrepper
Commits
rTRGR41771a5e9171: cli: Get all results from datagrepper
Summary

Until now we (me to blame) just requested the first page of results :/

Test Plan

One way to try this is the following (debug output enabled):

python cli.py -d --start `date --date="2016-04-26 07:00" +%s` --end `date --date="2016-04-26 08:00" +%s`

For me (my local time), this returned 14 pages of results. While most of the jobs were ignored (epel builds, not primary koji instance, ...), running this with bigger time period as an input might take a while.

Diff Detail

Repository
rTRGR taskotron-trigger
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
mkrizek retitled this revision from to cli: Get all results from datagrepper.Apr 26 2016, 12:39 PM
mkrizek updated this object.
mkrizek edited the test plan for this revision. (Show Details)
mkrizek added a reviewer: tflink.
tflink accepted this revision.Apr 28 2016, 4:56 AM
This revision is now accepted and ready to land.Apr 28 2016, 4:56 AM
Closed by commit rTRGR41771a5e9171: cli: Get all results from datagrepper (authored by mkrizek). · Explain WhyApr 28 2016, 7:52 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mjobtriggers/jobrunner.py (61 lines)

Diff 2117

View Options

jobtriggers/jobrunner.py

Show First 20 Lines • Show All 44 Lines • ▼ Show 20 Line(s)
45 45​
46def datagrepper_jobs(start, end): 46​def datagrepper_jobs(start, end):
47 params = { 47​ params = {
48 'start': start, 48​ 'start': start,
49 'end': end, 49​ 'end': end,
50 'category': 'buildsys', 50​ 'category': 'buildsys',
51 'topic': ['org.fedoraproject.prod.buildsys.build.state.change', 51​ 'topic': ['org.fedoraproject.prod.buildsys.build.state.change',
52 'org.fedoraproject.prod.buildsys.tag'], 52​ 'org.fedoraproject.prod.buildsys.tag'],
53​ 'page': 1,
53 } 54​ }
54 55​
55 r = requests.get(config.datagrepper_url, params=params) 56​ while True:
56 57​ r = requests.get(config.datagrepper_url, params=params)
57 koji_tag_items = set() 58​ pages = r.json()['pages']
58 for msg in r.json()['raw_messages']: 59​ page = r.json()['arguments']['page']
59 if msg['topic'].endswith('build.state.change'): 60​
60 try: 61​ logger.info('Processing jobs from datagrepper: page %d of %d', page, pages)
61 koji_build_msg.trigger_tasks(msg, trigger_tasks) 62​
62 except exc.TriggerMsgError, e: 63​ koji_tag_items = set()
63 logger.debug(e) 64​ for msg in r.json()['raw_messages']:
64 except exc.TriggerError, e: 65​ if msg['topic'].endswith('build.state.change'):
65 logger.error(e) 66​ try:
66 elif msg['topic'].endswith('tag'): 67​ koji_build_msg.trigger_tasks(msg, trigger_tasks)
67 try: 68​ except exc.TriggerMsgError, e:
68 koji_tag_items.add(koji_tag_msg.get_item(msg)) 69​ logger.debug(e)
69 except exc.TriggerMsgError, e: 70​ except exc.TriggerError, e:
70 logger.debug(e) 71​ logger.error(e)
71 else: 72​ elif msg['topic'].endswith('tag'):
72 logger.warning('Got an unexpected topic %s', msg['topic']) 73​ try:
73 74​ koji_tag_items.add(koji_tag_msg.get_item(msg))
74 item_type = koji_tag_msg.ITEM_TYPE 75​ except exc.TriggerMsgError, e:
75 for item in koji_tag_items: 76​ logger.debug(e)
76 tasks = deepcopy(config.koji_tag_changed_tasks) 77​ else:
77 if item.endswith('testing-pending'): 78​ logger.warning('Got an unexpected topic %s', msg['topic'])
78 tasks.remove('upgradepath') 79​
79 trigger_tasks(item, item_type, tasks, config.valid_arches) 80​ item_type = koji_tag_msg.ITEM_TYPE
81​ for item in koji_tag_items:
82​ tasks = deepcopy(config.koji_tag_changed_tasks)
83​ if item.endswith('testing-pending'):
84​ tasks.remove('upgradepath')
85​ trigger_tasks(item, item_type, tasks, config.valid_arches)
86​
87​ next_page = page + 1
88​ if next_page > pages:
89​ break
90​ params['page'] = next_page
80 91​
81 92​
82def main(): 93​def main():
83 parser = argparse.ArgumentParser(prog='Jobrunner', 94​ parser = argparse.ArgumentParser(prog='Jobrunner',
84 usage='Gets jobs from datagrepper from a given time ' 95​ usage='Gets jobs from datagrepper from a given time '
85 'period and runs them.') 96​ 'period and runs them.')
86 parser.add_argument('-s', '--start', help='Start time (Unix time format)') 97​ parser.add_argument('-s', '--start', help='Start time (Unix time format)')
87 parser.add_argument('-e', '--end', help='End time (Unix time format)') 98​ parser.add_argument('-e', '--end', help='End time (Unix time format)')
Show All 11 Lines