Differential D631

Upgrade buildslave and buildslave-configure ansible roles to handle multiple slaves per host
ClosedPublic
Actions

Authored by mkrizek on Oct 29 2015, 2:35 PM.

Details

Reviewers
tflink
Maniphest Tasks
T633: Upgrade buildslave and buildslave-configure ansible roles to handle multiple slaves per host
Summary

Contains also changes to support libtaskotron 0.4.0 in non-disposable mode.

Test Plan

Did some testing with allinone deployment on my laptop but not everything was covered. I guess deploying to dev is a test plan :)

Diff Detail

Lint
Lint Skipped
Unit
Unit Tests Skipped
mkrizek retitled this revision from to Upgrade buildslave and buildslave-configure ansible roles to handle multiple slaves per host.Oct 29 2015, 2:35 PM
mkrizek updated this object.
mkrizek edited the test plan for this revision. (Show Details)
mkrizek added a reviewer: tflink.
mkrizek added a project: infrastructure.
mkrizek added a task: T633: Upgrade buildslave and buildslave-configure ansible roles to handle multiple slaves per host.
tflink requested changes to this revision.Oct 29 2015, 3:22 PM

A couple of minor concerns but looks pretty good to me, overall.

roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2
155

Don't those %(prop:slavenam)s need to be wrapped in Interpolate instead of used as part of a raw string?

199

If we have multiple buildslaves on the same host, i think this step will cause problems. I wonder if we need to separate out the tmpdirs per-user like we did for the log files

roles/taskotron/buildslave/tasks/main.yml
40

I think we should add the homedir here as well. It's not needed at the moment but enough stuff has been moved around on the master to deal with disk space issues, I'd rather be prepared since we're specifying the homedir in inventory, anyways

This revision now requires changes to proceed.Oct 29 2015, 3:22 PM
mkrizek updated this revision to Diff 1643.Oct 29 2015, 8:54 PM
tflink requested changes to this revision.Oct 29 2015, 9:07 PM

changes look good to me but I caught something new this time around

roles/taskotron/buildslave-configure/tasks/main.yml
49

i didn't catch this before but this is not a wise thing to do - that /etc/ssh/ssh_known_hosts is controlled elsewhere by ansible - changing it would end up with a war between the two tasks.

why not continue to use the user's .ssh/known_hosts file?

This revision now requires changes to proceed.Oct 29 2015, 9:07 PM
mkrizek updated this revision to Diff 1645.Oct 30 2015, 5:30 PM
mkrizek added a comment.EditedOct 30 2015, 5:33 PM

I just realized that we also write directly into taskotron.log in artifactsdir so I changed master.cfg to reflect that.

mkrizek updated this revision to Diff 1646.Nov 2 2015, 2:21 PM
mkrizek updated this object.
mkrizek added a comment.Nov 2 2015, 2:24 PM

Added conditions to run the change only on dev.

tflink accepted this revision.Nov 2 2015, 4:05 PM

I agree that the conditionals around "is dev" or "is not dev" is kinda ugly but I can't think of other reasonable options. We'll just have to remove them once this is all deployed to prod

This revision is now accepted and ready to land.Nov 2 2015, 4:05 PM
mkrizek closed this revision.Nov 2 2015, 8:39 PM

Revision Contents

Diff 1646

View Options

inventory/group_vars/taskotron-dev

Context not available.
68buildmaster_template: taskotron.master.cfg.j2 68​buildmaster_template: taskotron.master.cfg.j2
69buildmaster_user: buildmaster 69​buildmaster_user: buildmaster
70 70​
71buildslave_dir: /home/buildslave/slave ​
72buildslave_port: 9989 71​buildslave_port: 9989
73buildslave_ssh_pubkey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqNpJlS14KKQQ8gAaXo0untJRccLJSrdUt/hWlpZ9Oad5KKYoRNznsGK3j9WCO4YDFXKGGxBrpqdyegPLO4J9zHmtlUPNfM4kbXVlVtIMGGKmBi9rQPNRpeclUpsjlo3uF9IqIIUSS1IMcx1X1oz60kmrAWO9+Z7mYs/AuCCeomlET3OU7paQYxL5Dgs2n6/IjhvoCitN9T9bBf91nLuRh+EtEbBL5ztZ1kVJxj5oTQN7iGKw2V0tW0MJ1WtnK2/j4v9MivnZPSPPKDysveNze2Vy9kioQACvrvjyHMIQrc+2YS/38D/fv7e4WhhuoOGT5Tip20PPspiCQRiSUBDN1 buildslave@taskotron-stg.fedoraproject.org' 72​buildslave_ssh_pubkey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqNpJlS14KKQQ8gAaXo0untJRccLJSrdUt/hWlpZ9Oad5KKYoRNznsGK3j9WCO4YDFXKGGxBrpqdyegPLO4J9zHmtlUPNfM4kbXVlVtIMGGKmBi9rQPNRpeclUpsjlo3uF9IqIIUSS1IMcx1X1oz60kmrAWO9+Z7mYs/AuCCeomlET3OU7paQYxL5Dgs2n6/IjhvoCitN9T9bBf91nLuRh+EtEbBL5ztZ1kVJxj5oTQN7iGKw2V0tW0MJ1WtnK2/j4v9MivnZPSPPKDysveNze2Vy9kioQACvrvjyHMIQrc+2YS/38D/fv7e4WhhuoOGT5Tip20PPspiCQRiSUBDN1 buildslave@taskotron-stg.fedoraproject.org'
Context not available.
View Options

inventory/group_vars/taskotron-dev-clients

Context not available.
21# buildslave config 21​# buildslave config
22############################################################ 22​############################################################
23 23​
24slave_user: buildslave 24​slaves_group: buildslaves
25slave_home: /home/buildslave/ ​
26slave_dir: /home/buildslave/slave ​
27slave_user: buildslave ​
28buildmaster: 10.5.124.181 25​buildmaster: 10.5.124.181
29buildslave_port: 9989 26​buildslave_port: 9989
30taskotron_admin_email: taskotron-admin-members@fedoraproject.org 27​taskotron_admin_email: taskotron-admin-members@fedoraproject.org
Context not available.
View Options

inventory/host_vars/taskotron-client26.qa.fedoraproject.org

Context not available.
10datacenter: phx2 10​datacenter: phx2
11 11​
12short_hostname: taskotron-client26.qa 12​short_hostname: taskotron-client26.qa
13buildslave_name: taskotron-client26 13​slaves:
14​ - { user: taskotron-client26-1, home: /home/taskotron-client26-1, dir: /home/taskotron-client26-1/slave }
15​ - { user: taskotron-client26-2, home: /home/taskotron-client26-2, dir: /home/taskotron-client26-2/slave }
14fas_client_groups: sysadmin-qa,sysadmin-main 16​fas_client_groups: sysadmin-qa,sysadmin-main
Context not available.
View Options

inventory/host_vars/taskotron-client27.qa.fedoraproject.org

Context not available.
12#libdir: /usr/lib 12​#libdir: /usr/lib
13 13​
14short_hostname: taskotron-client27.qa 14​short_hostname: taskotron-client27.qa
15buildslave_name: taskotron-client27 15​slaves:
16​ - { user: taskotron-client27-1, home: /home/taskotron-client27-1, dir: /home/taskotron-client27-1/slave }
17​ - { user: taskotron-client27-2, home: /home/taskotron-client27-2, dir: /home/taskotron-client27-2/slave }
16fas_client_groups: sysadmin-qa,sysadmin-main 18​fas_client_groups: sysadmin-qa,sysadmin-main
Context not available.
View Options

inventory/host_vars/taskotron-client28.qa.fedoraproject.org

Context not available.
10datacenter: phx2 10​datacenter: phx2
11 11​
12short_hostname: taskotron-client28.qa 12​short_hostname: taskotron-client28.qa
13buildslave_name: taskotron-client28 13​slaves:
14​ - { user: taskotron-client28-1, home: /home/taskotron-client28-1, dir: /home/taskotron-client28-1/slave }
15​ - { user: taskotron-client28-2, home: /home/taskotron-client28-2, dir: /home/taskotron-client28-2/slave }
14fas_client_groups: sysadmin-qa,sysadmin-main 16​fas_client_groups: sysadmin-qa,sysadmin-main
Context not available.
View Options

inventory/host_vars/taskotron-client29.qa.fedoraproject.org

Context not available.
11libdir: /usr/lib 11​libdir: /usr/lib
12 12​
13short_hostname: taskotron-client29.qa 13​short_hostname: taskotron-client29.qa
14buildslave_name: taskotron-client29 14​slaves:
15​ - { user: taskotron-client29-1, home: /home/taskotron-client29-1, dir: /home/taskotron-client29-1/slave }
16​ - { user: taskotron-client29-2, home: /home/taskotron-client29-2, dir: /home/taskotron-client29-2/slave }
15fas_client_groups: sysadmin-qa,sysadmin-main 17​fas_client_groups: sysadmin-qa,sysadmin-main
Context not available.
View Options

inventory/host_vars/taskotron-dev01.qa.fedoraproject.org

Context not available.
47buildmaster: 10.5.124.181 47​buildmaster: 10.5.124.181
48 48​
49buildslaves: 49​buildslaves:
50 - taskotron-client26 50​ - taskotron-client26-1
51 - taskotron-client27 51​ - taskotron-client26-2
52 - taskotron-client28 52​ - taskotron-client27-1
53 - taskotron-client29 53​ - taskotron-client27-2
54​ - taskotron-client28-1
55​ - taskotron-client28-2
56​ - taskotron-client29-1
57​ - taskotron-client29-2
54i386_buildslaves: 58​i386_buildslaves:
55 - taskotron-client29 59​ - taskotron-client29-1
60​ - taskotron-client29-2
56x86_64_buildslaves: 61​x86_64_buildslaves:
57 - taskotron-client27 62​ - taskotron-client26-1
58 - taskotron-client26 63​ - taskotron-client26-2
59 - taskotron-client28 64​ - taskotron-client27-1
65​ - taskotron-client27-2
66​ - taskotron-client28-1
67​ - taskotron-client28-2
Context not available.
View Options

roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2

Context not available.
151 151​
152factory = BuildFactory() 152​factory = BuildFactory()
153 153​
154​{% if deployment_type in ['prod', 'stg', 'local'] %}
154# clean out /var/tmp/taskotron (see T253) 155​# clean out /var/tmp/taskotron (see T253)
tflinkUnsubmitted
Not Done

Don't those %(prop:slavenam)s need to be wrapped in Interpolate instead of used as part of a raw string?

155factory.addStep(ShellCommand(command="rm -rf /var/tmp/taskotron/*", name="rm_tmp", descriptionDone=['Clean tmp'])) 156​factory.addStep(ShellCommand(command="rm -rf /var/tmp/taskotron/*", name="rm_tmp", descriptionDone=['Clean tmp']))
156 157​
157# clean the log (see T230) 158​# clean the log (see T230)
158factory.addStep(ShellCommand(command=["rm", "-f", "/var/log/taskotron/taskotron.log"], name="rm_log", descriptionDone=['Clean log'])) 159​factory.addStep(ShellCommand(command=["rm", "-f", "/var/log/taskotron/taskotron.log"], name="rm_log", descriptionDone=['Clean log']))
160​{% endif %}
161​
162​{% if deployment_type == 'dev' %}
163​# clean out /var/tmp/taskotron (see T253)
164​factory.addStep(ShellCommand(command=["rm", "-rf", Interpolate("/var/tmp/taskotron/%(prop:slavename)s/*")], name="rm_tmp", descriptionDone=['Clean tmp']))
165​{% endif %}
159 166​
160# check out the source 167​# check out the source
161factory.addStep(Git(repourl=Interpolate('{{ grokmirror_user }}@{{ buildmaster }}:/var/lib/git/mirror/fedoraqa/%(prop:taskname)s/'), 168​factory.addStep(Git(repourl=Interpolate('{{ grokmirror_user }}@{{ buildmaster }}:/var/lib/git/mirror/fedoraqa/%(prop:taskname)s/'),
Context not available.
172 descriptionDone=[Interpolate('%(prop:taskname)s on %(prop:item)s')], 179​ descriptionDone=[Interpolate('%(prop:taskname)s on %(prop:item)s')],
173 name='runtask', 180​ name='runtask',
174 timeout=2400, 181​ timeout=2400,
182​{% if deployment_type == 'dev' %}
183​ logfiles={'taskotron.log': {'filename': Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/taskotron.log'), }}))
184​{% endif %}
185​{% if deployment_type in ['prod', 'stg', 'local'] %}
175 logfiles={'taskotron.log': {'filename': '/var/log/taskotron/taskotron.log', }})) 186​ logfiles={'taskotron.log': {'filename': '/var/log/taskotron/taskotron.log', }}))
176 187​{% endif %}
177 188​
178# create artifacts dir on master 189​# create artifacts dir on master
179factory.addStep(MasterShellCommand(command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')], 190​factory.addStep(MasterShellCommand(command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')],
Context not available.
183factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), 194​factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'),
184 masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output'))) 195​ masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output')))
185 196​
197​
198​{% if deployment_type in ['prod', 'stg', 'local'] %}
186# copy taskotron log to master 199​# copy taskotron log to master
tflinkUnsubmitted
Not Done

If we have multiple buildslaves on the same host, i think this step will cause problems. I wonder if we need to separate out the tmpdirs per-user like we did for the log files

187factory.addStep(FileUpload(slavesrc='/var/log/taskotron/taskotron.log', 200​factory.addStep(FileUpload(slavesrc='/var/log/taskotron/taskotron.log',
188 masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/taskotron.log'), 201​ masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/taskotron.log'),
189 mode=0644)) 202​ mode=0644))
203​{% endif %}
204​
190import datetime 205​import datetime
191from buildbot.process.properties import renderer 206​from buildbot.process.properties import renderer
192 207​
Context not available.
View Options

roles/taskotron/buildslave-configure/tasks/main.yml

Context not available.
2 sudo: true 2​ sudo: true
3 sudo_user: "{{ slave_user }}" 3​ sudo_user: "{{ slave_user }}"
4 command: creates={{ slave_dir }} buildslave create-slave {{ slave_dir }} {{ buildmaster }} buildslave passwd 4​ command: creates={{ slave_dir }} buildslave create-slave {{ slave_dir }} {{ buildmaster }} buildslave passwd
5​ when: deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local'
5 6​
6- name: generate slave config 7​- name: generate slave config
7 sudo: true 8​ sudo: true
8 sudo_user: "{{ slave_user }}" 9​ sudo_user: "{{ slave_user }}"
9 template: src=buildbot.tac.j2 dest={{ slave_dir }}/buildbot.tac mode=0600 owner={{ slave_user }} group={{ slave_user }} 10​ template: src=buildbot.tac.j2 dest={{ slave_dir }}/buildbot.tac mode=0600 owner={{ slave_user }} group={{ slave_user }}
11​ when: deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local'
10 12​
11- name: generate slave info 13​- name: generate slave info
12 sudo: true 14​ sudo: true
Context not available.
15 with_items: 17​ with_items:
16 - admin 18​ - admin
17 - host 19​ - host
20​ when: deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local'
18 21​
19- name: create slave ssh directory 22​- name: create slave ssh directory
20 when: buildslave_public_sshkey_file is defined 23​ when: buildslave_public_sshkey_file is defined and (deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local')
21 file: path={{ slave_home }}/.ssh owner={{ slave_user }} group={{ slave_user }} mode=0700 state=directory 24​ file: path={{ slave_home }}/.ssh owner={{ slave_user }} group={{ slave_user }} mode=0700 state=directory
22 25​
23- name: install slave ssh private key 26​- name: install slave ssh private key
24 when: buildslave_private_sshkey_file is defined 27​ when: buildslave_private_sshkey_file is defined and (deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local')
25 copy: src={{ private }}/files/taskotron/{{ buildslave_private_sshkey_file }} dest={{ slave_home }}/.ssh/id_rsa owner={{ slave_user }} group={{ slave_user }} mode=0600 28​ copy: src={{ private }}/files/taskotron/{{ buildslave_private_sshkey_file }} dest={{ slave_home }}/.ssh/id_rsa owner={{ slave_user }} group={{ slave_user }} mode=0600
26 29​
27- name: install slave ssh public key 30​- name: install slave ssh public key
28 when: buildslave_public_sshkey_file is defined 31​ when: buildslave_public_sshkey_file is defined and (deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local')
29 copy: src={{ private }}/files/taskotron/{{ buildslave_public_sshkey_file }} dest={{ slave_home }}/.ssh/id_rsa.pub owner={{ slave_user }} group={{ slave_user }} mode=0644 32​ copy: src={{ private }}/files/taskotron/{{ buildslave_public_sshkey_file }} dest={{ slave_home }}/.ssh/id_rsa.pub owner={{ slave_user }} group={{ slave_user }} mode=0644
30 33​
31- name: make sure master is in known_hosts 34​- name: make sure master is in known_hosts
32 when: buildslave_public_sshkey_file is defined 35​ when: buildslave_public_sshkey_file is defined and (deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local')
33 lineinfile: dest=/home/{{ slave_user }}/.ssh/known_hosts regexp='{{ buildmaster }}' line='{{ buildmaster }} {{ buildmaster_pubkey }}' create=yes owner={{ slave_user }} group={{ slave_user }} 36​ lineinfile: dest=/home/{{ slave_user }}/.ssh/known_hosts regexp='{{ buildmaster }}' line='{{ buildmaster }} {{ buildmaster_pubkey }}' create=yes owner={{ slave_user }} group={{ slave_user }}
34 37​
35- name: generate buildslave service file 38​- name: generate buildslave service file
36 template: src=buildslave.service.j2 dest=/lib/systemd/system/buildslave.service owner=root group=root mode=0744 39​ template: src=buildslave.service.j2 dest=/lib/systemd/system/buildslave.service owner=root group=root mode=0744
40​ when: deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local'
37 41​
38- name: start and enable buildslave service 42​- name: start and enable buildslave service
39 service: name=buildslave enabled=yes state=started 43​ service: name=buildslave enabled=yes state=started
44​ when: deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local'
40 45​
46​
47​- name: create slave
48​ sudo: true
49​ sudo_user: '{{ item.user }}'
tflinkUnsubmitted
Not Done

i didn't catch this before but this is not a wise thing to do - that /etc/ssh/ssh_known_hosts is controlled elsewhere by ansible - changing it would end up with a war between the two tasks.

why not continue to use the user's .ssh/known_hosts file?

50​ command: creates={{ item.dir }} buildslave create-slave {{ item.dir }} {{ buildmaster }} buildslave passwd
51​ with_items:
52​ - '{{ slaves }}'
53​ when: deployment_type == 'dev'
54​
55​- name: generate slave config
56​ sudo: true
57​ sudo_user: '{{ item.user }}'
58​ template: src=buildbot.tac.j2 dest={{ item.dir }}/buildbot.tac mode=0600 owner={{ item.user }} group={{ slaves_group }}
59​ with_items:
60​ - '{{ slaves }}'
61​ when: deployment_type == 'dev'
62​
63​- name: generate slave admin info
64​ sudo: true
65​ sudo_user: '{{ item.user }}'
66​ template: src=admin.j2 dest={{ item.dir }}/info/admin mode=0644 owner={{ item.user }} group={{ slaves_group }}
67​ with_items:
68​ - '{{ slaves }}'
69​ when: deployment_type == 'dev'
70​
71​- name: generate slave host info
72​ sudo: true
73​ sudo_user: '{{ item.user }}'
74​ template: src=host.j2 dest={{ item.dir }}/info/host mode=0644 owner={{ item.user }} group={{ slaves_group }}
75​ with_items:
76​ - '{{ slaves }}'
77​ when: deployment_type == 'dev'
78​
79​- name: create slave ssh directory
80​ when: buildslave_public_sshkey_file is defined and deployment_type == 'dev'
81​ file: path={{ item.home }}/.ssh owner={{ item.user }} group={{ slaves_group }} mode=0700 state=directory
82​ with_items:
83​ - '{{ slaves }}'
84​
85​- name: install slave ssh private key
86​ when: buildslave_private_sshkey_file is defined and deployment_type == 'dev'
87​ copy: src={{ private }}/files/taskotron/{{ buildslave_private_sshkey_file }} dest={{ item.home }}/.ssh/id_rsa owner={{ item.user }} group={{ slaves_group }} mode=0600
88​ with_items:
89​ - '{{ slaves }}'
90​
91​- name: install slave ssh public key
92​ when: buildslave_public_sshkey_file is defined and deployment_type == 'dev'
93​ copy: src={{ private }}/files/taskotron/{{ buildslave_public_sshkey_file }} dest={{ item.home }}/.ssh/id_rsa.pub owner={{ item.user }} group={{ slaves_group }} mode=0644
94​ with_items:
95​ - '{{ slaves }}'
96​
97​- name: make sure master is in known_hosts
98​ when: buildslave_public_sshkey_file is defined and deployment_type == 'dev'
99​ lineinfile: dest={{ item.home }}/.ssh/known_hosts regexp='{{ buildmaster }}' line='{{ buildmaster }} {{ buildmaster_pubkey }}' create=yes owner={{ item.user }} group={{ slaves_group }}
100​ with_items:
101​ - '{{ slaves }}'
102​
103​- name: generate buildslave service file
104​ template: src=buildslave@.service.j2 dest=/lib/systemd/system/buildslave@.service owner=root group=root mode=0744
105​ when: deployment_type == 'dev'
106​
107​- name: start and enable buildslave services
108​ service: name=buildslave@{{ item.user }} enabled=yes state=started
109​ with_items:
110​ - '{{ slaves }}'
111​ when: deployment_type == 'dev'
Context not available.
View Options

roles/taskotron/buildslave-configure/templates/buildbot.tac.j2

Context not available.
4from twisted.application import service 4​from twisted.application import service
5from buildslave.bot import BuildSlave 5​from buildslave.bot import BuildSlave
6 6​
7​{% if deployment_type in ['prod', 'stg', 'local'] %}
7basedir = r'/home/buildslave/slave' 8​basedir = r'/home/buildslave/slave'
9​{% endif %}
10​{% if deployment_type == 'dev' %}
11​basedir = r'{{ item.dir }}'
12​{% endif %}
8rotateLength = 10000000 13​rotateLength = 10000000
9maxRotatedFiles = 10 14​maxRotatedFiles = 10
10 15​
Context not available.
29 34​
30buildmaster_host = '{{ buildmaster }}' 35​buildmaster_host = '{{ buildmaster }}'
31port = 9989 36​port = 9989
37​{% if deployment_type in ['prod', 'stg', 'local'] %}
32slavename = '{{ buildslave_name }}' 38​slavename = '{{ buildslave_name }}'
39​{% endif %}
40​{% if deployment_type == 'dev' %}
41​slavename = '{{ item.user }}'
42​{% endif %}
33{% if deployment_type == 'dev' %} 43​{% if deployment_type == 'dev' %}
34passwd = '{{ dev_buildslave_password }}' 44​passwd = '{{ dev_buildslave_password }}'
35{% endif %} 45​{% endif %}
Context not available.
View Options

roles/taskotron/buildslave-configure/templates/buildslave@.service.j2

  • This file was added.
1​[Unit]
2​Description=Buildslave for taskotron
3​After=network.target
4​
5​[Service]
6​Type=forking
7​PIDFile=/home/%i/slave/twistd.pid
8​ExecStart=/bin/buildslave start /home/%i/slave/
9​ExecStop=/bin/buildslave stop /home/%i/slave/
10​User=%i
11​Group={{ slaves_group }}
12​
13​[Install]
14​WantedBy=multi-user.target
View Options

roles/taskotron/buildslave/tasks/main.yml

Context not available.
35 35​
36- name: add the buildslave user 36​- name: add the buildslave user
37 user: name=buildslave 37​ user: name=buildslave
38​ when: deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local'
39​
40​- name: add buildslaves group
tflinkUnsubmitted
Not Done

I think we should add the homedir here as well. It's not needed at the moment but enough stuff has been moved around on the master to deal with disk space issues, I'd rather be prepared since we're specifying the homedir in inventory, anyways

41​ group: name={{ slaves_group }}
42​ when: deployment_type == 'dev'
43​
44​- name: add buildslave users
45​ user: name={{ item.user }} group={{ slaves_group }} home={{ item.home }}
46​ with_items: slaves
47​ when: deployment_type == 'dev'
Context not available.
View Options

roles/taskotron/taskotron-client/tasks/main.yml

Context not available.
44 44​
45- name: create /var/log/taskotron for task logfiles 45​- name: create /var/log/taskotron for task logfiles
46 file: path=/var/log/taskotron state=directory owner={{ slave_user }} group={{ slave_user }} mode=1755 46​ file: path=/var/log/taskotron state=directory owner={{ slave_user }} group={{ slave_user }} mode=1755
47​ when: deployment_type == 'prod' or deployment_type == 'stg' or deployment_type == 'local'
48​
49​- name: create /var/log/taskotron for task logfiles
50​ file: path=/var/log/taskotron state=directory owner=root group={{ slaves_group }} mode=1775
51​ when: deployment_type == 'dev'
Context not available.
View Options

roles/taskotron/taskotron-client/templates/taskotron.yaml.j2

1## Main configuration file for Taskotron 1​## Main configuration file for Taskotron
2## The file is in YAML syntax, read more about it at: 2​## The file is in YAML syntax, read more about it at:
3## http://en.wikipedia.org/wiki/Yaml 3​## http://en.wikipedia.org/wiki/Yaml
4## FIXME: add a link to Taskotron docs 4​## libtaskotron docs live at:
5​## https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/
6​
5 7​
6## ==== GENERAL section ==== 8​## ==== GENERAL section ====
9​## Config profile selection.
7## There are two major config profiles in Taskotron - development and 10​## There are two major config profiles in Taskotron - development and
8## production. 11​## production.
9## /Development/ profile is used for developing libtaskotron, developing checks 12​## Development profile is used for developing libtaskotron, developing checks
10## based on libtaskotron and local execution of these checks. 13​## based on libtaskotron and local execution of these checks.
11## /Production/ profile is used for deploying Taskotron as a service on a 14​## Production profile is used for deploying Taskotron as a service on a
12## server, periodically executing the checks and reporting results to relevant 15​## server, periodically executing the checks and reporting results to relevant
13## result consumers. 16​## result consumers.
14## *The default profile is /development/*. If you want to switch to the 17​## The default profile is 'development'. If you want to switch to the
15## /production/ profile, uncomment the following line. 18​## 'production' profile, uncomment the following line.
16## You can also switch a profile temporarily by using TASKOTRON_PROFILE=name 19​## You can also switch a profile temporarily by using TASKOTRON_PROFILE=name
17## environment variable, it has a higher priority. All other options set in this 20​## environment variable, it has a higher priority. All other options set in this
18## file still apply of course. 21​## file still apply of course.
19## [choices: production, development; default: development] 22​## [choices: production, development; default: development]
20profile: production 23​profile: production
21 24​
25​## Task execution mode. The tasks can be executed locally (on the current
26​## machine) or remotely in a disposable virtual machine. Local execution is
27​## dangerous with destructive or untrusted tasks, remote execution requires
28​## some additional setup (see Taskotron documentation).
29​## Remote execution is done through libvirt, it creates a throwaway virtual
30​## machine from a specified disk image, executes the task and deletes the
31​## machine.
32​## Local execution is the default mode for development profile and remote
33​## execution for production profile.
34​## [choices: local, libvirt; default: local for development, libvirt for production]
35​runtask_mode: local
36​
37​
38​## ==== SCHEDULING section ====
39​## This section holds options related to the scheduling and execution system,
40​## currently we use Buildbot
41​
42​## name of step in buildbot that executes the task
43​buildbot_task_step = 'runtask'
44​
22 45​
23## ==== REPORTING section ==== 46​## ==== REPORTING section ====
24## This section controls which result reports you want to send after the test 47​## This section controls which result reports you want to send after the test
25## execution is complete 48​## execution is complete.
26 49​
27## Overall setting whether to send any reports at all. If this is False, no 50​## Overall setting whether to send any reports at all. If this is False, no
28## reports will be sent anywhere, regardless of your configuration for 51​## reports will be sent anywhere, regardless of your configuration for
Context not available.
32 55​
33## If True, test results (for selected tests) will be sent as comments to 56​## If True, test results (for selected tests) will be sent as comments to
34## Fedora Update System (Bodhi). This requires that you have Bodhi 57​## Fedora Update System (Bodhi). This requires that you have Bodhi
35## credentials filled in secrets.conf. 58​## credentials filled in secrets.conf. See also 'reporting_enabled' option.
36report_to_bodhi: {{ client_report_to_bodhi }} 59​report_to_bodhi: {{ client_report_to_bodhi }}
37 60​
38## Whether to send test results to the configured ResultsDB server. 61​## Whether to send test results to the configured ResultsDB server. See also
62​## 'reporting_enabled' option.
39report_to_resultsdb: True 63​report_to_resultsdb: True
40 64​
41 65​
42## ==== RESOURCES section ==== 66​## ==== RESOURCES section ====
43## This section specifies access details to various external services 67​## This section specifies access details to various external services.
68​##
69​## Note: Try to keep custom URL addresses without a trailing slash. Otherwise
70​## the rendered URLs might end up containing double slashes, which some
71​## application servers don't handle gracefully (e.g. Flask's internal app
72​## server werkzeug).
44 73​
45## URL of Koji instance used for querying about new builds 74​## URL of Koji instance used for querying about new builds
46#koji_url: http://koji.fedoraproject.org/kojihub ​
47koji_url: {{ kojihub_url }} 75​koji_url: {{ kojihub_url }}
48 76​
49## URL of repository of all the RPM packages built in Koji 77​## URL of repository of all the RPM packages built in Koji
Context not available.
52## URL of Bodhi instance used for communication about package updates 80​## URL of Bodhi instance used for communication about package updates
53bodhi_server: {{ bodhi_server }} 81​bodhi_server: {{ bodhi_server }}
54 82​
55## URL of ResultsDB server API interface, which can store all test results 83​## Whether to use staging Bodhi instance instead of production (the
84​## default one).
85​#bodhi_staging: False
86​
87​## URL of ResultsDB server API interface, which can store all test results.
88​## Please make sure the URL doesn't have a trailing slash.
56resultsdb_server: {{ resultsdb_server }} 89​resultsdb_server: {{ resultsdb_server }}
57 90​
58## URL of ResultsDB frontend, which displays results from ResultsDB. 91​## URL of ResultsDB frontend, which displays results from ResultsDB.
59## Please make sure the URL doesn't have a trailing slash. 92​## Please make sure the URL doesn't have a trailing slash.
60resultsdb_frontend: {{ resultsdb_external_url }} 93​resultsdb_frontend: {{ resultsdb_external_url }}
61 94​
62## URL of ExecDB server API interface, which tracks task execution status 95​## URL of ExecDB server API interface, which tracks task execution status.
96​## Please make sure the URL doesn't have a trailing slash.
63execdb_server: {{ execdb_external_url }} 97​execdb_server: {{ execdb_external_url }}
64 98​
65## URL of taskotron buildmaster 99​## URL of taskotron buildmaster, to construct log URLs from.
100​## Please make sure the URL doesn't have a trailing slash.
66taskotron_master: {{ taskotron_master }} 101​taskotron_master: {{ taskotron_master }}
67 102​
68## URL of artifacts base directory, to construct artifacts URLs from. 103​## URL of artifacts base directory, to construct artifacts URLs from.
69## Please make sure the URL doesn't have a trailing slash. 104​## Please make sure the URL doesn't have a trailing slash.
70artifacts_baseurl: {{ artifacts_base_url }} 105​artifacts_baseurl: {{ artifacts_base_url }}
71 106​
72## name of step in buildbot that executes the task 107​## Whether to cache downloaded files to speed up subsequent downloads. If True,
73buildbot_task_step: 'runtask' 108​## files will be downloaded to a common directory specified by "cachedir". At
109​## the moment, Taskotron only supports Koji RPM downloads to be cached.
110​## [default: False for production, True for development]
111​#download_cache_enabled: False
112​
74 113​
75## ==== BODHI EMAIL section ==== 114​## ==== BODHI EMAIL section ====
76## These configuration options affect how Taskotron decideds to send emails 115​## These configuration options affect how Taskotron decideds to send emails
Context not available.
79## How long (in minutes) should we wait before allowing consequent test to 118​## How long (in minutes) should we wait before allowing consequent test to
80## re-post a 'FAILED' comment into Bodhi once again. 119​## re-post a 'FAILED' comment into Bodhi once again.
81## By default 3 days (3*24*60 = 4320). 120​## By default 3 days (3*24*60 = 4320).
82#bodhi_email_failed_span: 4320 121​#bodhi_posting_comments_span: 4320
83 122​
84 123​
85## ==== PATHS section ==== 124​## ==== PATHS section ====
86## Location of various pieces of the project. 125​## Location of various pieces of the project.
87 126​
88## The main log file location for Taskotron 127​## The location of log files for Taskotron
89#logfile: /var/log/taskotron/taskotron.log 128​#logdir: /var/log/taskotron
129​
130​## The location of task files (git checkout) when running in disposable clients mode
131​#client_taskdir: /var/tmp/taskotron/taskdir
132​
133​## The location of temporary files for Taskotron
134​#tmpdir: /var/tmp/taskotron
135​
136​## The location of artifacts produced by checks
137​#artifactsdir: /var/lib/taskotron/artifacts
138​
139​## The location of cached files downloaded by Taskotron
140​#cachedir: /var/cache/taskotron
141​
142​
143​## ==== LOGGING section ====
144​## This section contains configuration of logging.
145​
146​## Configuration of logging level. Here can be configured which messages
147​## will be logged. You can specify different level for logging to standard
148​## output (option log_level_stream) and logging to file (log_level_file).
149​## Possible values can be found here:
150​## https://docs.python.org/2.7/library/logging.html#logging-levels
151​#log_level_stream: INFO
152​#log_level_file: DEBUG
153​
154​## If True, logging to file will be enabled.
155​## [default: True for production, False for development]
156​#log_file_enabled: True
90 157​
91 158​
92## ==== SECRETS section ==== 159​## ==== SECRETS section ====
Context not available.
98fas_username: '{{ taskotron_fas_user }}' 165​fas_username: '{{ taskotron_fas_user }}'
99fas_password: '{{ taskotron_fas_password}}' 166​fas_password: '{{ taskotron_fas_password}}'
100 167​
168​## SSH private key location. Used for remote task execution, when connecting to
169​## VMs and remote machines. If your systems are not configured for automatic
170​## connection (private keys located in SSH's standard search path), specify a
171​## path to the private key here. An empty value (the default) means to rely on
172​## the standard search path only.
173​#ssh_privkey: /path/to/private.key
Context not available.