Differential D287

Initial support for ExecDB
ClosedPublic
Actions

Authored by jskladan on Feb 23 2015, 8:44 AM.

Details

Reviewers
kparal
tflink
mkrizek
Maniphest Tasks
T307: Support global UUIDs throughout taskotron
Commits
rLTRN6d8d0a33c66e: Initial support for ExecDB
Summary

Added execdb_url to the config file and changed the ResultsDB calls, so
the uuid gets stored with the job.

Note, that this patch pre-requires updated versions of ResultsDB and
resultsdb_api.

Test Plan

unittests pass; Tested with the whole taskotron-local stuff, and is currently running at taskotron-demo

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
jskladan retitled this revision from to Initial support for ExecDB.Feb 23 2015, 8:44 AM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added reviewers: tflink, kparal, mkrizek.
kparal added inline comments.Feb 23 2015, 10:02 AM
conf/taskotron.yaml.example
62

Let's be a bit more maintainer friendly and add a few words describing it:

## URL of ExecDB server API interface, which tracks task execution status
63

Just a few lines above we have resultsdb_server key. Could we keep the naming consistent and call this execdb_server? (I know, we have some other keys which have _url in their name).

libtaskotron/directives/resultsdb_directive.py
122–130

Can you please update the docstring as well?

jskladan updated this revision to Diff 794.Feb 23 2015, 5:12 PM
  • Changed execdb_url to execdb_server
tflink accepted this revision.Feb 24 2015, 5:10 AM
This revision is now accepted and ready to land.Feb 24 2015, 5:10 AM
kparal accepted this revision.Feb 24 2015, 12:28 PM
Closed by commit rLTRN6d8d0a33c66e: Initial support for ExecDB (authored by Josef Skladanka <jskladan@redhat.com>). · Explain WhyFeb 26 2015, 10:45 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 812

View Options

conf/taskotron.yaml.example

Show First 20 Lines • Show All 53 Lines • ▼ Show 20 Line(s)
54 54​
55## Number of maximum retries to Bodhi server if the query request 55​## Number of maximum retries to Bodhi server if the query request
56## fails with 5xx HTTP error 56​## fails with 5xx HTTP error
57#bodhi_request_max_retries: 3 57​#bodhi_request_max_retries: 3
58 58​
59## URL of ResultsDB server API interface, which can store all test results 59​## URL of ResultsDB server API interface, which can store all test results
60#resultsdb_server: http://127.0.0.1/resultsdb/api/v1.0/ 60​#resultsdb_server: http://127.0.0.1/resultsdb/api/v1.0/
61 61​
62​## URL of ExecDB server API interface, which tracks task execution status
kparalUnsubmitted
Not Done

Let's be a bit more maintainer friendly and add a few words describing it:

## URL of ExecDB server API interface, which tracks task execution status
63​#execdb_server: http://127.0.0.1/execdb/
kparalUnsubmitted
Not Done

Just a few lines above we have resultsdb_server key. Could we keep the naming consistent and call this execdb_server? (I know, we have some other keys which have _url in their name).

64​
62## URL of taskotron buildmaster 65​## URL of taskotron buildmaster
63#taskotron_master: http://127.0.0.1/taskmaster/ 66​#taskotron_master: http://127.0.0.1/taskmaster/
64 67​
65## name of step in buildbot that executes the task 68​## name of step in buildbot that executes the task
66#buildbot_task_step = 'runtask' 69​#buildbot_task_step = 'runtask'
67 70​
68## ==== BODHI EMAIL section ==== 71​## ==== BODHI EMAIL section ====
69## These configuration options affect how Taskotron decideds to send emails 72​## These configuration options affect how Taskotron decideds to send emails
▲ Show 20 Lines • Show All 44 Lines • Show Last 20 Lines
View Options

libtaskotron/config_defaults.py

Show All 40 Lines23class Config(object):
41 41​
42 reporting_enabled = False #: 42​ reporting_enabled = False #:
43 report_to_bodhi = True #: 43​ report_to_bodhi = True #:
44 report_to_resultsdb = True #: 44​ report_to_resultsdb = True #:
45 45​
46 koji_url = 'http://koji.fedoraproject.org/kojihub' #: 46​ koji_url = 'http://koji.fedoraproject.org/kojihub' #:
47 pkg_url = 'http://kojipkgs.fedoraproject.org/packages' #: 47​ pkg_url = 'http://kojipkgs.fedoraproject.org/packages' #:
48 bodhi_server = 'https://admin.fedoraproject.org/updates/' #: 48​ bodhi_server = 'https://admin.fedoraproject.org/updates/' #:
49​ execdb_server = 'http://127.0.0.1/execdb/' #:
49 resultsdb_server = 'http://127.0.0.1/resultsdb/api/v1.0/' #: 50​ resultsdb_server = 'http://127.0.0.1/resultsdb/api/v1.0/' #:
50 taskotron_master = 'http://127.0.0.1/taskmaster/' #: 51​ taskotron_master = 'http://127.0.0.1/taskmaster/' #:
51 buildbot_task_step = 'runtask' #: 52​ buildbot_task_step = 'runtask' #:
52 53​
53 bodhi_posting_comments_span = 4320 #: 54​ bodhi_posting_comments_span = 4320 #:
54 # 3 days (3*24*60 = 4320) 55​ # 3 days (3*24*60 = 4320)
55 bodhi_request_max_retries = 3 #: 56​ bodhi_request_max_retries = 3 #:
56 57​
▲ Show 20 Lines • Show All 46 Lines • Show Last 20 Lines
View Options

libtaskotron/directives/resultsdb_directive.py

Show First 20 Lines • Show All 89 Lines • ▼ Show 20 Line(s)
90class ResultsdbDirective(BaseDirective): 90​class ResultsdbDirective(BaseDirective):
91 91​
92 def __init__(self, resultsdb = None): 92​ def __init__(self, resultsdb = None):
93 self.resultsdb = resultsdb 93​ self.resultsdb = resultsdb
94 94​
95 conf = config.get_config() 95​ conf = config.get_config()
96 self.masterurl = conf.taskotron_master 96​ self.masterurl = conf.taskotron_master
97 self.task_stepname = conf.buildbot_task_step 97​ self.task_stepname = conf.buildbot_task_step
98​ self.execdb_server = "%s/jobs" % conf.execdb_server
98 99​
99 if self.resultsdb is None: 100​ if self.resultsdb is None:
100 self.resultsdb = resultsdb_api.ResultsDBapi(conf.resultsdb_server) 101​ self.resultsdb = resultsdb_api.ResultsDBapi(conf.resultsdb_server)
101 102​
102 def ensure_testcase_exists(self, name): 103​ def ensure_testcase_exists(self, name):
103 """ Make sure that the testcase exists in resultsdb, otherwise create 104​ """ Make sure that the testcase exists in resultsdb, otherwise create
104 the testcase using a dummy url as a reference 105​ the testcase using a dummy url as a reference
105 106​
106 :param str name: name of the testcase to check for 107​ :param str name: name of the testcase to check for
107 """ 108​ """
108 109​
109 try: 110​ try:
110 self.resultsdb.get_testcase(name) 111​ self.resultsdb.get_testcase(name)
111 return 112​ return
112 except resultsdb_api.ResultsDBapiException, e: 113​ except resultsdb_api.ResultsDBapiException, e:
113 if e.message != 'Testcase not found': 114​ if e.message != 'Testcase not found':
114 raise e 115​ raise e
115 116​
116 # since the testcase doesn't exist, create it with a dummy value for url 117​ # since the testcase doesn't exist, create it with a dummy value for url
117 # it can be updated later when it's not blocking results reporting 118​ # it can be updated later when it's not blocking results reporting
118 dummyurl = 'http://faketestcasesRus.com/%s' % name 119​ dummyurl = 'http://faketestcasesRus.com/%s' % name
119 self.resultsdb.create_testcase(name, dummyurl) 120​ self.resultsdb.create_testcase(name, dummyurl)
120 121​
121 def create_resultsdb_job(self, name, refurl=None): 122​ def create_resultsdb_job(self, name, uuid=None, refurl=None):
122 """ Create a job in resultsdb for reporting results against 123​ """ Create a job in resultsdb for reporting results against
123 124​
124 :param str name: name of job to report against 125​ :param str name: name of job to report against
126​ :param str uuid: UUID of the job (most probably provided by ExecDB)
127​ :param str refurl: url pointing to the execution overview.
128​ If set to None, ExecDB url is created from UUID
125 :returns: dict of rendered json results 129​ :returns: dict of rendered json results
126 """ 130​ """
kparalUnsubmitted
Not Done

Can you please update the docstring as well?

127 131​
128 # this is completly hacky for now since the runner doesn't have a 132​ url = "%s/%s" % (self.execdb_server, uuid)
129 # good way to get the actual refurl. This will be addressed in T122 and ​
130 # T145 ​
131 ​
132 url = 'http://not-a-real-domain.com/noworky' ​
133 if refurl is not None: 133​ if refurl is not None:
134 url = refurl 134​ url = refurl
135 135​
136 jobdata = self.resultsdb.create_job(url, status='SCHEDULED', name=name) 136​ jobdata = self.resultsdb.create_job(url, status='SCHEDULED', name=name,
137​ uuid=uuid)
137 self.resultsdb.update_job(id=jobdata['id'], status='RUNNING') 138​ self.resultsdb.update_job(id=jobdata['id'], status='RUNNING')
138 139​
139 return jobdata 140​ return jobdata
140 141​
141 def complete_resultsdb_job(self, jobid): 142​ def complete_resultsdb_job(self, jobid):
142 """ Change the resultsdb job to a status of ``COMPLETED``, indicating 143​ """ Change the resultsdb job to a status of ``COMPLETED``, indicating
143 that the reporting is complete. 144​ that the reporting is complete.
144 145​
Show All 28 Lines168 if not (conf.reporting_enabled and conf.report_to_resultsdb):
173 return 174​ return
174 175​
175 # for now, we're creating the resultsdb job at reporting time 176​ # for now, we're creating the resultsdb job at reporting time
176 # the job needs to be 'RUNNING' in order to append any results 177​ # the job needs to be 'RUNNING' in order to append any results
177 job_url, log_url = buildbot_utils.get_urls(env_data['jobid'], 178​ job_url, log_url = buildbot_utils.get_urls(env_data['jobid'],
178 self.masterurl, 179​ self.masterurl,
179 self.task_stepname) 180​ self.task_stepname)
180 job_data = self.create_resultsdb_job(env_data['checkname'], 181​ job_data = self.create_resultsdb_job(env_data['checkname'],
181 refurl=job_url) 182​ uuid=env_data['uuid'],
183​ )
182 184​
183 self.ensure_testcase_exists(env_data['checkname']) 185​ self.ensure_testcase_exists(env_data['checkname'])
184 186​
185 output = [] 187​ output = []
186 188​
187 log.info('Posting %s results to ResultsDB...' % len(check_details)) 189​ log.info('Posting %s results to ResultsDB...' % len(check_details))
188 for detail in check_details: 190​ for detail in check_details:
189 try: 191​ try:
▲ Show 20 Lines • Show All 50 Lines • Show Last 20 Lines
View Options

testing/test_resultsdb_directive.py

Show All 36 Lines25 def setup(self, monkeypatch):
37 37​
38 tap = check.export_TAP(self.cd) 38​ tap = check.export_TAP(self.cd)
39 39​
40 self.ref_input = {'results': tap} 40​ self.ref_input = {'results': tap}
41 self.ref_envdata = { 41​ self.ref_envdata = {
42 'resultsdb_job_id': 1, 42​ 'resultsdb_job_id': 1,
43 'checkname': 'test_resultsdb_report', 43​ 'checkname': 'test_resultsdb_report',
44 'jobid': 'all/123', 44​ 'jobid': 'all/123',
45​ 'uuid': 'c25237a4-b6b3-11e4-b98a-3c970e018701',
45 } 46​ }
46 47​
47 self.ref_jobid = 1234 48​ self.ref_jobid = 1234
48 self.ref_refurl = u'http://example.com/%s' % self.ref_envdata['checkname'] 49​ self.ref_refurl = u'http://example.com/%s' % self.ref_envdata['checkname']
49 self.ref_jobdata = { u'end_time': None, 50​ self.ref_jobdata = { u'end_time': None,
50 u'href': u'http://127.0.0.1/api/v1.0/jobs/%d' % self.ref_jobid, 51​ u'href': u'http://127.0.0.1/api/v1.0/jobs/%d' % self.ref_jobid,
51 u'id': self.ref_jobid, 52​ u'id': self.ref_jobid,
52 u'name': self.ref_envdata['checkname'], 53​ u'name': self.ref_envdata['checkname'],
▲ Show 20 Lines • Show All 107 Lines • ▼ Show 20 Line(s)155 def test_complete_job(self):
160 assert len(rdb_calls) == 1 161​ assert len(rdb_calls) == 1
161 assert rdb_calls[0][0] == 'update_job' 162​ assert rdb_calls[0][0] == 'update_job'
162 assert rdb_calls[0][2]['status'] == 'COMPLETED' 163​ assert rdb_calls[0][2]['status'] == 'COMPLETED'
163 164​
164 def test_ensure_testcase_creation_notexist(self): 165​ def test_ensure_testcase_creation_notexist(self):
165 # check to see if the testcase is created in the case that it doesn't 166​ # check to see if the testcase is created in the case that it doesn't
166 # already exist 167​ # already exist
167 168​
168 self.stub_rdb.get_testcase =exception_raiser(ResultsDBapiException('Testcase not found')) 169​ self.stub_rdb.get_testcase = exception_raiser(ResultsDBapiException('Testcase not found'))
169 170​
170 self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname']) 171​ self.test_rdb.ensure_testcase_exists(self.ref_envdata['checkname'])
171 172​
172 rdb_calls = self.stub_rdb.calls() 173​ rdb_calls = self.stub_rdb.calls()
173 174​
174 # dingus doesn't record calls to exception_raiser 175​ # dingus doesn't record calls to exception_raiser
175 assert len(rdb_calls) == 1 176​ assert len(rdb_calls) == 1
176 assert rdb_calls[0][0] == 'create_testcase' 177​ assert rdb_calls[0][0] == 'create_testcase'
Show All 12 Lines