Differential D307

Ansible-ize ExecDB
ClosedPublic
Actions

Authored by jskladan on Mar 17 2015, 10:40 AM.

Details

Reviewers
tflink
Maniphest Tasks
T442: ansible-ize execdb
Summary

Initial attempt of the subject, created mostly by using ResultsDB as a guide.

Test Plan

Untested, I'm not really sure how could I test it, though

Diff Detail

Lint
Lint Skipped
Unit
Unit Tests Skipped
jskladan retitled this revision from to Ansible-ize ExecDB.Mar 17 2015, 10:40 AM
jskladan updated this object.
jskladan edited the test plan for this revision. (Show Details)
jskladan added a reviewer: tflink.
jskladan added a project: infrastructure.
tflink added a comment.Mar 17 2015, 4:46 PM

overall, it looks pretty good to me but I'm wondering if we really want/need execdb to be its own host or if it'd be better to co-habitate the resultsdb VM. thoughts?

jskladan added a comment.Mar 18 2015, 11:26 AM
In D307#5537, @tflink wrote:

overall, it looks pretty good to me but I'm wondering if we really want/need execdb to be its own host or if it'd be better to co-habitate the resultsdb VM. thoughts?

Yep, I think it should not be a problem - not sure what would be better - whether resultsdb VM or the VM with Trigger, though. I do not think that it needs its separate VM.

tflink added a comment.Mar 19 2015, 6:21 AM
In D307#5564, @jskladan wrote:
In D307#5537, @tflink wrote:

overall, it looks pretty good to me but I'm wondering if we really want/need execdb to be its own host or if it'd be better to co-habitate the resultsdb VM. thoughts?

Yep, I think it should not be a problem - not sure what would be better - whether resultsdb VM or the VM with Trigger, though. I do not think that it needs its separate VM.

Trigger is on the buildmaster VM, so I think that the resultsdb VM would be a better choice since there are fewer things running on it. Let's just put in on there for now - if it causes problems, we can extract to a separate VM later.

jskladan updated this revision to Diff 856.Mar 19 2015, 2:38 PM

in the spirit of @tflink's comment, I (hopefully) changed the configs, so execdb shares the machine with resultsdb

tflink requested changes to this revision.Mar 19 2015, 2:50 PM

overall, looks pretty good to me. I think it'd be better to add the execdb role to the existing resultsdb playbook/group instead of trying to control the same VM with multiple playbooks, though.

playbooks/groups/execdb-dev.yml
2 ↗(On Diff #856)

If execdb is residing on the resultsdb machine, it doesn't need a separate playbook - that can cause oddness when things are repeated. I think it'd be better to add the role on to the resultsdb playbook instead of creating another group

This revision now requires changes to proceed.Mar 19 2015, 2:50 PM
jskladan updated this revision to Diff 861.Mar 23 2015, 3:17 PM

Remove ExecDB playbook and make it a role in ResultsDB

jskladan updated this revision to Diff 862.Mar 23 2015, 3:31 PM
jskladan updated this revision to Diff 863.Mar 23 2015, 3:49 PM
tflink accepted this revision.Mar 23 2015, 3:52 PM
This revision is now accepted and ready to land.Mar 23 2015, 3:52 PM
jskladan closed this revision.Mar 24 2015, 8:32 AM

Revision Contents

Diff 863

View Options

inventory/group_vars/resultsdb-dev

Context not available.
6# for systems that do not match the above - specify the same parameter in 6​# for systems that do not match the above - specify the same parameter in
7# the host_vars/$hostname file 7​# the host_vars/$hostname file
8 8​
9tcp_ports: [ 80, 443, "{{ resultsdb_db_port }}" ] 9​tcp_ports: [ 80, 443, "{{ resultsdb_db_port }}", "{{ execdb_db_port }}" ]
10fas_client_groups: sysadmin-qa,sysadmin-main 10​fas_client_groups: sysadmin-qa,sysadmin-main
11nrpe_procs_warn: 250 11​nrpe_procs_warn: 250
12nrpe_procs_crit: 300 12​nrpe_procs_crit: 300
Context not available.
24resultsdb_endpoint: 'resultsdb_api' 24​resultsdb_endpoint: 'resultsdb_api'
25resultsdb_fe_endpoint: 'resultsdb' 25​resultsdb_fe_endpoint: 'resultsdb'
26resultsdb_db_name: resultsdb_dev 26​resultsdb_db_name: resultsdb_dev
27​
28​execdb_db_host: db-qa01.qa.fedoraproject.org
29​execdb_db_port: 5432
30​execdb_endpoint: 'execdb'
31​execdb_db_name: execdb_dev
32​
27allowed_hosts: 33​allowed_hosts:
28 - 10.5.124 34​ - 10.5.124
29 35​
Context not available.
View Options

inventory/group_vars/taskotron-dev

Context not available.
21master_dir: /home/buildmaster/master 21​master_dir: /home/buildmaster/master
22master_user: buildmaster 22​master_user: buildmaster
23external_hostname: taskotron-dev.fedoraproject.org 23​external_hostname: taskotron-dev.fedoraproject.org
24​execdb_statuspush: http://resultsdb-dev01.qa.fedoraproject.org/execdb/buildbottest
24resultsdb_url: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb_api/api/v1.0 25​resultsdb_url: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb_api/api/v1.0
25resultsdb_frontend_url: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb/ 26​resultsdb_frontend_url: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb/
26resultsdb_external_url: https://taskotron-dev.fedoraproject.org/resultsdb/ 27​resultsdb_external_url: https://taskotron-dev.fedoraproject.org/resultsdb/
Context not available.
View Options

inventory/group_vars/taskotron-dev-clients

Context not available.
5 5​
6slave_user: buildslave 6​slave_user: buildslave
7taskotron_fas_user: taskotron 7​taskotron_fas_user: taskotron
8​execdb_server: http://resultsdb-dev01.qa.fedoraproject.org/execdb
8resultsdb_server: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb_api/api/v1.0/ 9​resultsdb_server: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb_api/api/v1.0/
9bodhi_server: http://10.5.124.181/fakefedorainfra/bodhi/ 10​bodhi_server: http://10.5.124.181/fakefedorainfra/bodhi/
10kojihub_url: http://koji.fedoraproject.org/kojihub 11​kojihub_url: http://koji.fedoraproject.org/kojihub
Context not available.
View Options

inventory/host_vars/db-qa01.qa.fedoraproject.org

Context not available.
22- fakefedorainfra 22​- fakefedorainfra
23- fakefedorainfra_stg 23​- fakefedorainfra_stg
24- dev_fakefedorainfra 24​- dev_fakefedorainfra
25​- execdb_dev
25- resultsdb 26​- resultsdb
26- resultsdb_stg 27​- resultsdb_stg
27- resultsdb_dev 28​- resultsdb_dev
Context not available.
View Options

playbooks/groups/resultsdb-dev.yml

Context not available.
49 handlers: 49​ handlers:
50 - include: "{{ handlers }}/restart_services.yml" 50​ - include: "{{ handlers }}/restart_services.yml"
51 51​
52- name: configure resultsdb 52​- name: configure resultsdb and execdb
53 hosts: resultsdb-dev 53​ hosts: resultsdb-dev
54 user: root 54​ user: root
55 gather_facts: True 55​ gather_facts: True
Context not available.
62 roles: 62​ roles:
63 - { role: taskotron/resultsdb-backend, tags: ['resultsdb-be'] } 63​ - { role: taskotron/resultsdb-backend, tags: ['resultsdb-be'] }
64 - { role: taskotron/resultsdb-frontend, tags: ['resultsdb-fe'] } 64​ - { role: taskotron/resultsdb-frontend, tags: ['resultsdb-fe'] }
65​ - { role: taskotron/execdb, tags: ['execdb'] }
65 66​
66 handlers: 67​ handlers:
67 - include: "{{ handlers }}/restart_services.yml" 68​ - include: "{{ handlers }}/restart_services.yml"
Context not available.
View Options

roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2

Context not available.
126 label='type of object to use', 126​ label='type of object to use',
127 default='', 127​ default='',
128 size=256), 128​ size=256),
129​{% if deployment_type == 'dev' %}
130​ StringParameter(name='uuid',
131​ label='UUID of the build in progress',
132​ default='',
133​ size=256),
134​{% endif %}
129 StringParameter(name='arch', 135​ StringParameter(name='arch',
130 label='arch of rpm to test', 136​ label='arch of rpm to test',
131 default='x86_64', 137​ default='x86_64',
Context not available.
158 mode='full', method='clobber')) 164​ mode='full', method='clobber'))
159 165​
160# run the runner 166​# run the runner
161factory.addStep(ShellCommand(command=["runtask", '-i', 167​factory.addStep(ShellCommand(command=["runtask",
162 Interpolate('%(prop:item)s'), '-t', 168​ '-i', Interpolate('%(prop:item)s'),
163 Interpolate('%(prop:item_type)s'), 169​ '-t', Interpolate('%(prop:item_type)s'),
164 '-a', Interpolate('%(prop:arch)s'), 170​ '-a', Interpolate('%(prop:arch)s'),
165 '-j', Interpolate('%(prop:buildername)s/%(prop:buildnumber)s'), 171​ '-j', Interpolate('%(prop:buildername)s/%(prop:buildnumber)s'),
166{% if deployment_type == 'dev' %} 172​{% if deployment_type == 'dev' %}
Context not available.
356 362​
357c['status'].append(mn) 363​c['status'].append(mn)
358 364​
365​{% if deployment_type == 'dev' %}
366​# ExecDB's push-notifications:
367​import buildbot.status.status_push
368​sp = buildbot.status.status_push.HttpStatusPush(
369​ serverUrl="{{ execdb_statuspush }}",
370​ debug=True)
371​c['status'].append(sp)
372​{% endif %}
359 373​
360####### PROJECT IDENTITY 374​####### PROJECT IDENTITY
361 375​
Context not available.
View Options

roles/taskotron/execdb/tasks/main.yml

  • This file was added.
1​- name: ensure packages required for execdb are installed
2​ action: yum name={{ item }} state=latest
3​ with_items:
4​ - execdb
5​ - mod_wsgi
6​ - python-psycopg2
7​ - libsemanage-python
8​
9​- name: ensure database is created
10​ delegate_to: "{{ execdb_db_host }}"
11​ sudo_user: postgres
12​ sudo: true
13​ action: postgresql_db db={{ execdb_db_name }}
14​
15​- name: ensure dev execdb db user has access to dev database
16​ when: deployment_type == 'dev'
17​ delegate_to: "{{ execdb_db_host }}"
18​ sudo_user: postgres
19​ sudo: true
20​ action: postgresql_user db={{ execdb_db_name }} user={{ dev_execdb_db_user }} password={{ dev_execdb_db_password }} role_attr_flags=NOSUPERUSER
21​
22​- name: ensure stg execdb db user has access to stg database
23​ when: deployment_type == 'stg'
24​ delegate_to: "{{ execdb_db_host }}"
25​ sudo_user: postgres
26​ sudo: true
27​ action: postgresql_user db={{ execdb_db_name }} user={{ stg_execdb_db_user }} password={{ stg_execdb_db_password }} role_attr_flags=NOSUPERUSER
28​
29​- name: ensure prod execdb db user has access to prod database
30​ when: deployment_type == 'prod'
31​ delegate_to: "{{ execdb_db_host }}"
32​ sudo_user: postgres
33​ sudo: true
34​ action: postgresql_user db={{ execdb_db_name }} user={{ prod_execdb_db_user }} password={{ prod_execdb_db_password }} role_attr_flags=NOSUPERUSER
35​
36​- name: ensure local execdb db user has access to prod database
37​ when: deployment_type == 'local'
38​ delegate_to: "{{ execdb_db_host }}"
39​ sudo_user: postgres
40​ sudo: true
41​ action: postgresql_user db={{ execdb_db_name }} user={{ local_execdb_db_user }} password={{ local_execdb_db_password }} role_attr_flags=NOSUPERUSER
42​
43​- name: ensure selinux lets httpd talk to postgres
44​ seboolean: name=httpd_can_network_connect_db persistent=yes state=yes
45​
46​- name: generate execdb config
47​ template: src=settings.py.j2 dest=/etc/execdb/settings.py owner=root group=root mode=0644
48​ notify:
49​ - restart httpd
50​
51​- name: generate execdb apache config
52​ template: src=execdb.conf.j2 dest=/etc/httpd/conf.d/execdb.conf owner=root group=root mode=0644
53​ notify:
54​ - restart httpd
55​
56​- name: initialize execdb database
57​ shell: PROD='true' execdb init_db
View Options

roles/taskotron/execdb/templates/execdb.conf.j2

  • This file was added.
1​WSGIDaemonProcess execdb user=apache group=apache threads=5
2​WSGIScriptAlias /{{ execdb_endpoint }} /usr/share/execdb/execdb.wsgi
3​WSGISocketPrefix run/wsgi
4​
5​# this isn't the best way to force SSL but it works for now
6​#RewriteEngine On
7​#RewriteCond %{HTTPS} !=on
8​#RewriteRule ^/execdb/admin/?(.*) https://%{SERVER_NAME}/$1 [R,L]
9​
10​<Directory /usr/share/execdb>
11​ WSGIProcessGroup execdb
12​ WSGIApplicationGroup %{GLOBAL}
13​ WSGIScriptReloading On
14​ <IfModule mod_authz_core.c>
15​ # Apache 2.4
16​ <RequireAny>
17​ Require method GET
18​ Require ip 127.0.0.1 ::1{% for host in allowed_hosts %} {{ host }}{% endfor %}
19​
20​ </RequireAny>
21​ </IfModule>
22​ <IfModule !mod_auth_core.c>
23​ Order allow,deny
24​ Allow from all
25​ </IfModule>
26​
27​</Directory>
28​
29​#Alias /execdb/static /var/www/execdb/execdb/static
30​
31​#<Directory /var/www/execdb/execdb/static>
32​#Order allow,deny
33​#Allow from all
34​#</Directory>
View Options

roles/taskotron/execdb/templates/settings.py.j2

  • This file was added.
1​{%- if deployment_type == 'prod' %}
2​SECRET_KEY = '{{ prod_execdb_secret_key }}'
3​SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ prod_execdb_db_user }}:{{ prod_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
4​{% endif %}
5​{%- if deployment_type == 'stg' %}
6​SECRET_KEY = '{{ stg_execdb_secret_key }}'
7​SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ stg_execdb_db_user }}:{{ stg_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
8​{% endif %}
9​{%- if deployment_type == 'dev' %}
10​SECRET_KEY = '{{ dev_execdb_secret_key }}'
11​SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ dev_execdb_db_user }}:{{ dev_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
12​{% endif %}
13​{%- if deployment_type == 'local' %}
14​SECRET_KEY = '{{ local_execdb_secret_key }}'
15​SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ local_execdb_db_user }}:{{ local_execdb_db_password }}@127.0.0.1:{{ execdb_db_port }}/{{ execdb_db_name }}'
16​{% endif %}
17​FILE_LOGGING = False
18​LOGFILR = '/var/log/execdb/execdb.log'
19​SYSLOG_LOGGING = False
20​STREAM_LOGGING = True
View Options

roles/taskotron/taskotron-client/templates/taskotron.yaml.j2

Context not available.
55## URL of ResultsDB server API interface, which can store all test results 55​## URL of ResultsDB server API interface, which can store all test results
56resultsdb_server: {{ resultsdb_server }} 56​resultsdb_server: {{ resultsdb_server }}
57 57​
58​## URL of ExecDB server API interface, which tracks task execution status
59​execdb_server: {{ execdb_server }}
60​
58## URL of taskotron buildmaster 61​## URL of taskotron buildmaster
59taskotron_master: {{ taskotron_master }} 62​taskotron_master: {{ taskotron_master }}
60 63​
Context not available.