Differential D1208

master.cfg for Buildbot and new ExecDB
AcceptedPublic
Actions

Authored by jskladan on Jun 12 2017, 12:46 PM.

Details

Reviewers
tflink
mkrizek
Summary

This changes the steps we use so they report their start/stop to Buildbot. There almost certainly is a nicer/better way of doing this like using factory-like code to produce the modified classes, or reimplementing the Bbots push notification code, but given the limited docs, this is the most straight-forward solution

Test Plan

Tested localy with docker setup (which has slightly different master.cfg), so there might be some copy-paste errors. But the overall idea is sound and working.

Diff Detail

Lint
Lint Skipped
Unit
Unit Tests Skipped
jskladan created this revision.Jun 12 2017, 12:46 PM
tflink accepted this revision.Jun 12 2017, 2:05 PM

I'd like to see some more comments to describe what's going on and why we're overriding some pretty base buildsteps but other than that, it looks good to me.

This revision is now accepted and ready to land.Jun 12 2017, 2:05 PM
jskladan added a comment.Jun 12 2017, 2:40 PM

The patch overrides all the buildsteps we use - since we want every step to report on its progress, we ought to change how they behave, base or not. (I'm not sure I understand what you mean, though).

Overall, the changes are the same for all buildsteps - we override the start and finished methods, to inject the report call into them. The report method just plain HTTP POSTs to an url given to it, and sends the provided data.

All the steps have gained the execdb_url parameter in constructor, in order to store that url (which is constructed partialy in the jinja, and during runtime).

The patch then sets the said execdb_url parameter in every addStep call, so the constructor can be called without exceptions raising.

Does this answer the questions? If not, I'd probably need you to be a bit more specific :)

tflink added a comment.Jun 12 2017, 3:06 PM

Yeah, that answers most of the questions - I just wanted them in the master.cfg file for future reference or if someone is looking at it for the first time

mkrizek resigned from this revision.Thu, Jul 13, 9:03 AM

Revision Contents

Diff 3066

View Options

inventory/group_vars/taskotron-dev

Context not available.
44############################################################ 44​############################################################
45 45​
46execdb_endpoint: execdb 46​execdb_endpoint: execdb
47execdb_statuspush: http://resultsdb-dev01.qa.fedoraproject.org/execdb/buildbottest 47​execdb_statuspush: http://resultsdb-dev01.qa.fedoraproject.org/execdb/api/v1/jobs
48execdb_server: http://resultsdb-dev01.qa.fedoraproject.org/execdb 48​execdb_server: http://resultsdb-dev01.qa.fedoraproject.org/execdb
49resultsdb_api_endpoint: resultsdb_api 49​resultsdb_api_endpoint: resultsdb_api
50resultsdb_host: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb_api/ 50​resultsdb_host: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb_api/
Context not available.
View Options

inventory/group_vars/taskotron-prod

Context not available.
37############################################################ 37​############################################################
38 38​
39execdb_server: http://resultsdb01.qa.fedoraproject.org/execdb 39​execdb_server: http://resultsdb01.qa.fedoraproject.org/execdb
40execdb_statuspush: http://resultsdb01.qa.fedoraproject.org/execdb/buildbottest 40​execdb_statuspush: http://resultsdb01.qa.fedoraproject.org/execdb/api/v1/jobs
41resultsdb_external_url: https://taskotron.fedoraproject.org/resultsdb/ 41​resultsdb_external_url: https://taskotron.fedoraproject.org/resultsdb/
42resultsdb_fe_endpoint: resultsdb 42​resultsdb_fe_endpoint: resultsdb
43resultsdb_frontend_url: http://resultsdb01.qa.fedoraproject.org/resultsdb 43​resultsdb_frontend_url: http://resultsdb01.qa.fedoraproject.org/resultsdb
Context not available.
View Options

inventory/group_vars/taskotron-stg

Context not available.
42############################################################ 42​############################################################
43 43​
44execdb_server: http://resultsdb-stg01.qa.fedoraproject.org/execdb 44​execdb_server: http://resultsdb-stg01.qa.fedoraproject.org/execdb
45execdb_statuspush: http://resultsdb-stg01.qa.fedoraproject.org/execdb/buildbottest 45​execdb_statuspush: http://resultsdb-stg01.qa.fedoraproject.org/execdb/api/v1/jobs
46 46​
47resultsdb_url: http://resultsdb-stg01.qa.fedoraproject.org/resultsdb_api/api/v2.0 47​resultsdb_url: http://resultsdb-stg01.qa.fedoraproject.org/resultsdb_api/api/v2.0
48resultsdb_frontend_url: http://resultsdb-stg01.qa.fedoraproject.org/ 48​resultsdb_frontend_url: http://resultsdb-stg01.qa.fedoraproject.org/
Context not available.
View Options

roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2

Context not available.
35# the 'change_source' setting tells the buildmaster how it should find out 35​# the 'change_source' setting tells the buildmaster how it should find out
36# about source code changes. Here we point to the buildbot clone of pyflakes. 36​# about source code changes. Here we point to the buildbot clone of pyflakes.
37 37​
38​from buildbot.steps.slave import RemoveDirectory #as RemoveDirectory_orig
38#from buildbot.changes.gitpoller import GitPoller 39​#from buildbot.changes.gitpoller import GitPoller
39#c['change_source'] = [] 40​#c['change_source'] = []
40#c['change_source'].append(GitPoller( 41​#c['change_source'].append(GitPoller(
Context not available.
171# only take place on one slave. 172​# only take place on one slave.
172 173​
173from buildbot.process.factory import BuildFactory 174​from buildbot.process.factory import BuildFactory
174from buildbot.steps.source.git import Git ​
175from buildbot.steps.shell import ShellCommand ​
176from buildbot.process.properties import Property, Interpolate 175​from buildbot.process.properties import Property, Interpolate
177from buildbot.steps.slave import RemoveDirectory 176​from build.process import results as buildbot_results
178from buildbot.steps.transfer import DirectoryUpload, FileUpload 177​
179from buildbot.steps.master import MasterShellCommand 178​from buildbot.steps.source.git import Git as Git_orig
179​from buildbot.steps.shell import ShellCommand as ShellCommand_orig
180​from buildbot.steps.slave import RemoveDirectory as RemoveDirectory_orig
181​from buildbot.steps.transfer import DirectoryUpload as DirectoryUpload_orig
182​from buildbot.steps.transfer import FileUpload as FileUpload_orig
183​from buildbot.steps.master import MasterShellCommand as MasterShellCommand_orig
184​
185​import json
186​import urllib2
187​
188​def map_result(result):
189​ if result in [buildbot_results.SUCCESS, buildbot_results.WARNINGS]:
190​ return 'COMPLETED'
191​ if result in [buildbot_results.SKIPPED, buildbot_results.CANCELLED]:
192​ return 'ABORTED'
193​ if result == buildbot_results.EXCEPTION:
194​ return 'CRASHED'
195​ return 'FAILED'
196​
197​def report(url, data):
198​ data = json.dumps(data)
199​ req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
200​ response = urllib2.urlopen(req)
201​ data = json.load(response)
202​ response.close()
203​ return data
204​
205​class JobStartStopStep(ShellCommand_orig):
206​ def __init__(self, execdb_url, execdb_args, **kwargs):
207​ self.execdb_url = execdb_url
208​ self.execdb_args = execdb_args
209​ super(self.__class__, self).__init__(**kwargs)
210​
211​ def start(self, **kwargs):
212​ report(self.execdb_url, execdb_args)
213​ return super(self.__class__, self).start(**kwargs)
214​
215​class Git(Git_orig):
216​ def __init__(self, execdb_url, **kwargs):
217​ self.execdb_url = execdb_url
218​ self.execdb_step_id = None
219​ super(self.__class__, self).__init__(**kwargs)
220​
221​ def start(self, **kwargs):
222​ data = report(self.execdb_url, {"name": self.name, "description": self.name})
223​ self.execdb_step_id = data['id']
224​ return super(self.__class__, self).start(**kwargs)
225​
226​ def finished(self, results):
227​ report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)})
228​ return super(self.__class__, self).finished(results)
229​
230​class ShellCommand(ShellCommand_orig):
231​ def __init__(self, execdb_url, **kwargs):
232​ self.execdb_url = execdb_url
233​ self.execdb_step_id = None
234​ super(self.__class__, self).__init__(**kwargs)
235​
236​ def start(self, **kwargs):
237​ data = report(self.execdb_url, {"name": self.name, "description": self.name})
238​ self.execdb_step_id = data['id']
239​ return super(self.__class__, self).start(**kwargs)
240​
241​ def finished(self, results):
242​ report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)})
243​ return super(self.__class__, self).finished(results)
244​
245​class RemoveDirectory(RemoveDirectory_orig):
246​ def __init__(self, execdb_url, **kwargs):
247​ self.execdb_url = execdb_url
248​ self.execdb_step_id = None
249​ super(self.__class__, self).__init__(**kwargs)
250​
251​ def start(self, **kwargs):
252​ data = report(self.execdb_url, {"name": self.name, "description": self.name})
253​ self.execdb_step_id = data['id']
254​ return super(self.__class__, self).start(**kwargs)
255​
256​ def finished(self, results):
257​ report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)})
258​ return super(self.__class__, self).finished(results)
259​
260​class DirectoryUpload(DirectoryUpload_orig):
261​ def __init__(self, execdb_url, **kwargs):
262​ self.execdb_url = execdb_url
263​ self.execdb_step_id = None
264​ super(self.__class__, self).__init__(**kwargs)
265​
266​ def start(self, **kwargs):
267​ data = report(self.execdb_url, {"name": self.name, "description": self.name})
268​ self.execdb_step_id = data['id']
269​ return super(self.__class__, self).start(**kwargs)
270​
271​ def finished(self, results):
272​ report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)})
273​ return super(self.__class__, self).finished(results)
274​
275​class FileUpload(FileUpload_orig):
276​ def __init__(self, execdb_url, **kwargs):
277​ self.execdb_url = execdb_url
278​ self.execdb_step_id = None
279​ super(self.__class__, self).__init__(**kwargs)
280​
281​ def start(self, **kwargs):
282​ data = report(self.execdb_url, {"name": self.name, "description": self.name})
283​ self.execdb_step_id = data['id']
284​ return super(self.__class__, self).start(**kwargs)
285​
286​ def finished(self, results):
287​ report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)})
288​ return super(self.__class__, self).finished(results)
289​
290​class MasterShellCommand(MasterShellCommand_orig):
291​ def __init__(self, execdb_url, **kwargs):
292​ self.execdb_url = execdb_url
293​ self.execdb_step_id = None
294​ super(self.__class__, self).__init__(**kwargs)
295​
296​ def start(self, **kwargs):
297​ data = report(self.execdb_url, {"name": self.name, "description": self.name})
298​ self.execdb_step_id = data['id']
299​ return super(self.__class__, self).start(**kwargs)
300​
301​ def finished(self, results):
302​ report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)})
303​ return super(self.__class__, self).finished(results)
304​
180 305​
181factory = BuildFactory() 306​factory = BuildFactory()
182 307​
308​factory.addStep(JobStartStopStep(
309​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"),
310​ execdb_args={"start": True},
311​ command=["true"], name="start_job", descriptionDone=['Start ExecDB job']))
312​
183{% if deployment_type in ['local'] %} 313​{% if deployment_type in ['local'] %}
184# clean out /var/tmp/taskotron (see T253) 314​# clean out /var/tmp/taskotron (see T253)
185factory.addStep(ShellCommand(command="rm -rf /var/tmp/taskotron/*", name="rm_tmp", descriptionDone=['Clean tmp'])) 315​factory.addStep(ShellCommand(
316​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
317​ command="rm -rf /var/tmp/taskotron/*", name="rm_tmp", descriptionDone=['Clean tmp']))
186 318​
187# clean the log (see T230) 319​# clean the log (see T230)
188factory.addStep(ShellCommand(command=["rm", "-f", "/var/log/taskotron/taskotron.log"], name="rm_log", descriptionDone=['Clean log'])) 320​factory.addStep(ShellCommand(
321​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
322​ command=["rm", "-f", "/var/log/taskotron/taskotron.log"], name="rm_log", descriptionDone=['Clean log']))
189{% endif %} 323​{% endif %}
190 324​
191{% if deployment_type in ['dev', 'stg', 'prod'] %} 325​{% if deployment_type in ['dev', 'stg', 'prod'] %}
192# clean out /var/tmp/taskotron (see T253) 326​# clean out /var/tmp/taskotron (see T253)
193factory.addStep(ShellCommand(command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp'])) 327​factory.addStep(ShellCommand(
328​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
329​ command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp']))
194{% endif %} 330​{% endif %}
195 331​
196# check out the source 332​# check out the source
197factory.addStep(Git(repourl=Property('repo', default=Interpolate('{{ grokmirror_user }}@{{ buildmaster }}:/var/lib/git/mirror/fedoraqa/%(prop:taskname)s/')), 333​factory.addStep(Git(
334​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
335​ repourl=Property('repo', default=Interpolate('{{ grokmirror_user }}@{{ buildmaster }}:/var/lib/git/mirror/fedoraqa/%(prop:taskname)s/')),
198 branch=Property('branch', default='{{ grokmirror_default_branch }}'), 336​ branch=Property('branch', default='{{ grokmirror_default_branch }}'),
199 mode='full', 337​ mode='full',
200 method='clobber', 338​ method='clobber',
201 shallow=True)) 339​ shallow=True))
202 340​
203# run the runner 341​# run the runner
204factory.addStep(ShellCommand(command=["runtask", 342​factory.addStep(ShellCommand(
343​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
344​ command=["runtask",
205 '-i', Interpolate('%(prop:item)s'), 345​ '-i', Interpolate('%(prop:item)s'),
206 '-t', Interpolate('%(prop:item_type)s'), 346​ '-t', Interpolate('%(prop:item_type)s'),
207 '-a', Interpolate('%(prop:arch)s'), 347​ '-a', Interpolate('%(prop:arch)s'),
Context not available.
224{% endif %} 364​{% endif %}
225 365​
226 366​
227factory.addStep(ShellCommand(command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'), 367​factory.addStep(ShellCommand(
228 descriptionDone=['Make sure the minion is removed'])) 368​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
369​ command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'),
370​ descriptionDone=['Make sure the minion is removed']))
229 371​
230# create artifacts dir on master 372​# create artifacts dir on master
231factory.addStep(MasterShellCommand(command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')], 373​factory.addStep(MasterShellCommand(
232 descriptionDone=['Create artifacs dir'])) 374​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
375​ command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')],
376​ descriptionDone=['Create artifacs dir']))
233 377​
234# copy artifacts to master 378​# copy artifacts to master
235factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), 379​factory.addStep(DirectoryUpload(
236 masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output'))) 380​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
381​ slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'),
382​ masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output')))
237 383​
238# gzip artifacts 384​# gzip artifacts
239factory.addStep(MasterShellCommand(command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'), 385​factory.addStep(MasterShellCommand(
240 descriptionDone=['gzip artifacs dir content'])) 386​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
387​ command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'),
388​ descriptionDone=['gzip artifacs dir content']))
241 389​
242{% if deployment_type in ['local'] %} 390​{% if deployment_type in ['local'] %}
243# copy taskotron log to master 391​# copy taskotron log to master
244factory.addStep(FileUpload(slavesrc='/var/log/taskotron/taskotron.log', 392​factory.addStep(FileUpload(
245 masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/taskotron.log'), 393​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
246 mode=0644)) 394​ slavesrc='/var/log/taskotron/taskotron.log',
395​ masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/taskotron.log'),
396​ mode=0644))
247{% endif %} 397​{% endif %}
248 398​
249# move the artifacts to the correct dir on the master 399​# move the artifacts to the correct dir on the master
Context not available.
255 return datetime.datetime.now().strftime("%Y%m%d") 405​ return datetime.datetime.now().strftime("%Y%m%d")
256 406​
257# move artifacts dir 407​# move artifacts dir
258factory.addStep(MasterShellCommand(command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today), 408​factory.addStep(MasterShellCommand(
259 descriptionDone=['Move artifacs dir'])) 409​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
410​ command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today),
411​ descriptionDone=['Move artifacs dir']))
412​
260 413​
414​factory.addStep(JobStartStopStep(
415​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"),
416​ execdb_args={"outcome": '_COMPUTED_'},
417​ command=["true"], name="end_job", descriptionDone=['End ExecDB job']))
261 418​
262 419​
263{% if deployment_type in ['dev', 'stg', 'prod'] %} 420​{% if deployment_type in ['dev', 'stg', 'prod'] %}
264# this is for package-specific tasks 421​# this is for package-specific tasks
265distgit_factory = BuildFactory() 422​distgit_factory = BuildFactory()
266 423​
424​distgit_factory.addStep(JobStartStopStep(
425​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"),
426​ execdb_args={"start": True},
427​ command=["true"], name="start_job", descriptionDone=['Start ExecDB job']))
428​
429​
267# clean out /var/tmp/taskotron (see T253) 430​# clean out /var/tmp/taskotron (see T253)
268distgit_factory.addStep(ShellCommand(command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp'])) 431​distgit_factory.addStep(ShellCommand(
432​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
433​ command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp']))
269 434​
270{% if deployment_type in ['dev', 'prod'] %} 435​{% if deployment_type in ['dev', 'prod'] %}
271# check out the source from prod dist-git 436​# check out the source from prod dist-git
272distgit_factory.addStep(Git(repourl=Property('git_repo', default=Interpolate('http://pkgs02.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))), 437​distgit_factory.addStep(Git(
438​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
439​ repourl=Property('git_repo', default=Interpolate('http://pkgs02.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))),
273 branch=Property('git_branch', default='master'), 440​ branch=Property('git_branch', default='master'),
274 mode='full', 441​ mode='full',
275 method='clobber', 442​ method='clobber',
276 shallow=True)) 443​ shallow=True))
277{% else %} 444​{% else %}
278# check out the source from stg dist-git 445​# check out the source from stg dist-git
279distgit_factory.addStep(Git(repourl=Property('git_repo', default=Interpolate('http://pkgs01.stg.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))), 446​distgit_factory.addStep(Git(
447​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
448​ repourl=Property('git_repo', default=Interpolate('http://pkgs01.stg.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))),
280 branch=Property('git_branch', default='master'), 449​ branch=Property('git_branch', default='master'),
281 mode='full', 450​ mode='full',
282 method='clobber', 451​ method='clobber',
283 shallow=True)) 452​ shallow=True))
284{% endif %} 453​{% endif %}
285# run the runner 454​# run the runner
286distgit_factory.addStep(ShellCommand(command=["runtask", 455​distgit_factory.addStep(ShellCommand(
456​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
457​ command=["runtask",
287 '-i', Interpolate('%(prop:item)s'), 458​ '-i', Interpolate('%(prop:item)s'),
288 '-t', Interpolate('%(prop:item_type)s'), 459​ '-t', Interpolate('%(prop:item_type)s'),
289 '-a', Interpolate('%(prop:arch)s'), 460​ '-a', Interpolate('%(prop:arch)s'),
Context not available.
301)) 472​))
302 473​
303 474​
304distgit_factory.addStep(ShellCommand(command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'), 475​distgit_factory.addStep(ShellCommand(
305 descriptionDone=['Make sure the minion is removed'])) 476​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
477​ command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'),
478​ descriptionDone=['Make sure the minion is removed']))
306 479​
307# create artifacts dir on master 480​# create artifacts dir on master
308distgit_factory.addStep(MasterShellCommand(command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')], 481​distgit_factory.addStep(MasterShellCommand(
309 descriptionDone=['Create artifacs dir'])) 482​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
483​ command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')],
484​ descriptionDone=['Create artifacs dir']))
310 485​
311# copy artifacts to master 486​# copy artifacts to master
312distgit_factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), 487​distgit_factory.addStep(DirectoryUpload(
313 masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output'))) 488​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
489​ slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'),
490​ masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output')))
314 491​
315# gzip artifacts 492​# gzip artifacts
316distgit_factory.addStep(MasterShellCommand(command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'), 493​distgit_factory.addStep(MasterShellCommand(
317 descriptionDone=['gzip artifacs dir content'])) 494​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
495​ command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'),
496​ descriptionDone=['gzip artifacs dir content']))
318 497​
319{% endif %} 498​{% endif %}
320 499​
321# move artifacts dir 500​# move artifacts dir
322distgit_factory.addStep(MasterShellCommand(command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today), 501​distgit_factory.addStep(MasterShellCommand(
323 descriptionDone=['Move artifacs dir'])) 502​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"),
503​ command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today),
504​ descriptionDone=['Move artifacs dir']))
505​
506​distgit_factory.addStep(JobStartStopStep(
507​ execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"),
508​ execdb_args={"outcome": '_COMPUTED_'},
509​ command=["true"], name="end_job", descriptionDone=['End ExecDB job']))
510​
511​
512​
324####### RESOURCE LOCKS 513​####### RESOURCE LOCKS
325# 514​#
326# This is a set of resource locks to make sure that we don't have too many things 515​# This is a set of resource locks to make sure that we don't have too many things
Context not available.