Infra Ansible - fix ExecDB's local deployment inconsistency
ClosedPublic
Actions

Authored by jskladan on Oct 27 2015, 12:38 PM.

Details

Reviewers

Summary

roles/taskotron/execdb/tasks/main.yml and roles/taskotron/execdb/templates/alembic.ini.j2 use local_ prefix for the execdb variables, but roles/taskotron/execdb/templates/settings.py.j2 was not. This removes the need for both local_execdb_db_{user, pasword} and execdb_db_{user, pasword} in the private vars for the local deployment.

Test Plan

The ansible allinone works with just local_ execdb variables.

Diff Detail

Lint

Lint Skipped

Unit

Unit Tests Skipped

jskladan retitled this revision from to Infra Ansible - fix ExecDB's local deployment inconsistency.Oct 27 2015, 12:38 PM

jskladan updated this object.

jskladan edited the test plan for this revision. (Show Details)

jskladan added a project: infrastructure.

Herald added a subscriber: tflink. · View Herald TranscriptOct 27 2015, 12:38 PM

jskladan added a reviewer: tflink.Oct 27 2015, 12:39 PM

I'd rather solve this in a similar way to how I've been refactoring the non-local playbooks: keep setting the local_* vars as they are but set the non-prefixed variables in a group_vars file. For example, set execdb_db_password: {{ local_execdb_password }} in the inventory/group_vars/resultsdb file. resultsdb for dev and stg should mostly be set up that way. I won't be doing production until we upgrade to F22, though.

I really want to get rid of those conditional blocks in the templates and tasks - they're confusing and encourage silly repetition. Apparently I missed some of it in the execdb role :-/

This revision now requires changes to proceed.Oct 27 2015, 4:20 PM

In D629#12105, @tflink wrote:

...

roger, will have a look at that

How about this? I "finished" the resultsdb/execdb refactoring, so there are (almost) no conditionals in the tasks/templates, using the aforementioned method of setting stuff in the inventory/group_vars. I hope I grasped the idea right, though.

LGTM - can you wait until tomorrow before pushing this, though - infra freeze is still in place until tomorrow morning and this affects some production systems.

This revision is now accepted and ready to land.Nov 3 2015, 2:15 PM

jskladan mentioned this in 20151109-fedoraqadevel.Nov 9 2015, 2:15 PM

jskladan mentioned this in D630: Taskotron-Ansible - script to automate deployment.Nov 11 2015, 8:13 AM

It doesn't look like this was ever applied to the infra ansible repo

jskladan updated this revision to Diff 1894.Feb 3 2016, 11:22 AM

New changes look good to me

roles/taskotron/resultsdb-backend/templates/settings.py.j2
10	When we have more than one of these, I'd prefer to see `{% when deployment_type in ['local', 'dev'] %}`

jskladan closed this revision.Feb 3 2016, 1:12 PM

Revision Contents

			Path
M			inventory/group_vars/resultsdb-dev (7 lines)
M			inventory/group_vars/resultsdb-prod (42 lines)
M			inventory/group_vars/resultsdb-stg (8 lines)
M			inventory/host_vars/testdays.fedorainfracloud.org (2 lines)
M			roles/taskotron/execdb/tasks/main.yml (24 lines)
M			roles/taskotron/execdb/templates/alembic.ini.j2 (13 lines)
M			roles/taskotron/execdb/templates/settings.py.j2 (11 lines)
M			roles/taskotron/resultsdb-backend/tasks/main.yml (15 lines)
M			roles/taskotron/resultsdb-backend/templates/settings.py.j2 (19 lines)
M			roles/taskotron/resultsdb-frontend/templates/settings.py.j2 (13 lines)

Diff 1894

View Options

inventory/group_vars/resultsdb-dev

Context not available.
23	resultsdb_db_host: "{{ resultsdb_db_host_machine }}"	23	resultsdb_db_host: "{{ resultsdb_db_host_machine }}"
24	resultsdb_db_port: 5432	24	resultsdb_db_port: 5432
25	resultsdb_endpoint: 'resultsdb_api'	25	resultsdb_endpoint: 'resultsdb_api'
26	resultsdb_fe_endpoint: 'resultsdb'
27	resultsdb_db_name: resultsdb_dev	26	resultsdb_db_name: resultsdb_dev
28	resultsdb_db_user: "{{ dev_resultsdb_db_user }}"	27	resultsdb_db_user: "{{ dev_resultsdb_db_user }}"
29	resultsdb_db_password: "{{ dev_resultsdb_db_password }}"	28	resultsdb_db_password: "{{ dev_resultsdb_db_password }}"
Context not available.
32	allowed_hosts:	31	allowed_hosts:
33	- 10.5.124	32	- 10.5.124
34		33
		34	############################################################
		35	# resultsdb frontend details
		36	############################################################
		37	resultsdb_fe_endpoint: "resultsdb"
		38	resultsdb_frontend_secret_key: "{{ dev_resultsdb_frontend_secretkey }}"
		39
35		40
36	############################################################	41	############################################################
37	# execdb details	42	# execdb details
Context not available.

View Options

inventory/group_vars/resultsdb-prod

1	---	1	---
		2	############################################################
		3	# general information
		4	############################################################
2	# common items for the releng-* boxes	5	# common items for the releng-* boxes
3	lvm_size: 50000	6	lvm_size: 50000
4	mem_size: 4096	7	mem_size: 4096
Context not available.
11	nrpe_procs_warn: 250	14	nrpe_procs_warn: 250
12	nrpe_procs_crit: 300	15	nrpe_procs_crit: 300
13		16
		17	external_hostname: taskotron.fedoraproject.org
14	deployment_type: prod	18	deployment_type: prod
15		19
		20
		21	############################################################
		22	# resultsdb details
		23	############################################################
		24
		25	# the db_host_machine bits are so that delegation continues to work, even if
		26	# that db is localhost relative to resultsdb
		27
16	resultsdb_db_host_machine: db-qa01.qa.fedoraproject.org	28	resultsdb_db_host_machine: db-qa01.qa.fedoraproject.org
17	resultsdb_db_host: "{{ resultsdb_db_host_machine }}"	29	resultsdb_db_host: "{{ resultsdb_db_host_machine }}"
18	resultsdb_db_port: 5432	30	resultsdb_db_port: 5432
19	resultsdb_endpoint: 'resultsdb_api'	31	resultsdb_endpoint: 'resultsdb_api'
20	resultsdb_fe_endpoint: 'resultsdb'
21	resultsdb_db_name: resultsdb	32	resultsdb_db_name: resultsdb
		33	resultsdb_db_user: "{{ prod_resultsdb_db_user }}"
		34	resultsdb_db_password: "{{ prod_resultsdb_db_password }}"
		35	resultsdb_secret_key: "{{ prod_resultsdb_secret_key }}"
		36
		37	allowed_hosts:
		38	- 10.5.124
		39	- 10.5.131
22		40
		41
		42	############################################################
		43	# resultsdb-frontend details
		44	############################################################
		45	resultsdb_fe_endpoint: "resultsdb"
		46	resultsdb_frontend_secret_key: "{{ prod_resultsdb_frontend_secretkey }}"
		47
		48
		49	###########################################################
		50	# execdb details
		51	############################################################
23	execdb_db_host_machine: db-qa01.qa.fedoraproject.org	52	execdb_db_host_machine: db-qa01.qa.fedoraproject.org
24	execdb_db_host: "{{ execdb_db_host_machine }}"	53	execdb_db_host: "{{ execdb_db_host_machine }}"
25	execdb_db_port: 5432	54	execdb_db_port: 5432
26	execdb_endpoint: 'execdb'	55	execdb_endpoint: 'execdb'
27	execdb_db_name: execdb	56	execdb_db_name: execdb
		57	execdb_db_user: "{{ prod_execdb_db_user }}"
		58	execdb_db_password: "{{ prod_execdb_db_password }}"
		59	execdb_secret_key: "{{ prod_execdb_secret_key }}"
28		60
29	external_hostname: taskotron.fedoraproject.org
30
31	allowed_hosts:
32	- 10.5.124
33	- 10.5.131
34		61
		62	############################################################
		63	# fedmsg details
		64	############################################################
35	fedmsg_active: True	65	fedmsg_active: True
36	fedmsg_cert_prefix: resultsdb	66	fedmsg_cert_prefix: resultsdb
37		67
Context not available.

View Options

inventory/group_vars/resultsdb-stg

Context not available.
25	resultsdb_db_host: "{{ resultsdb_db_host_machine }}"	25	resultsdb_db_host: "{{ resultsdb_db_host_machine }}"
26	resultsdb_db_port: 5432	26	resultsdb_db_port: 5432
27	resultsdb_endpoint: 'resultsdb_api'	27	resultsdb_endpoint: 'resultsdb_api'
28	resultsdb_fe_endpoint: 'resultsdb'
29	resultsdb_db_name: resultsdb_stg	28	resultsdb_db_name: resultsdb_stg
30	resultsdb_db_user: "{{ stg_resultsdb_db_user }}"	29	resultsdb_db_user: "{{ stg_resultsdb_db_user }}"
31	resultsdb_db_password: "{{ stg_resultsdb_db_password }}"	30	resultsdb_db_password: "{{ stg_resultsdb_db_password }}"
Context not available.
36		35
37		36
38	############################################################	37	############################################################
		38	# resultsdb-frontend details
		39	############################################################
		40	resultsdb_fe_endpoint: "resultsdb"
		41	resultsdb_frontend_secret_key: "{{ stg_resultsdb_frontend_secretkey }}"
		42
		43
		44	############################################################
39	# execdb details	45	# execdb details
40	############################################################	46	############################################################
41		47
Context not available.

View Options

inventory/host_vars/testdays.fedorainfracloud.org

Context not available.
59	# resultsdb_frontend	59	# resultsdb_frontend
60	############################################################	60	############################################################
61		61
62	local_resultsdb_frontend_secret_key: "{{ testdays_resultsdb_frontend_secretkey }}"	62	resultsdb_frontend_secret_key: "{{ testdays_resultsdb_frontend_secretkey }}"
63	resultsdb_fe_endpoint: "resultsdb"	63	resultsdb_fe_endpoint: "resultsdb"
64		64
65	############################################################	65	############################################################
Context not available.

View Options

roles/taskotron/execdb/tasks/main.yml

Context not available.
23	become: true	23	become: true
24	postgresql_db: db={{ execdb_db_name }}	24	postgresql_db: db={{ execdb_db_name }}
25		25
26	- name: ensure dev execdb db user has access to dev database	26	- name: ensure execdb db user has access to database
27	when: deployment_type == 'dev'
28	delegate_to: "{{ execdb_db_host_machine }}"	27	delegate_to: "{{ execdb_db_host_machine }}"
29	become_user: postgres	28	become_user: postgres
30	become: true	29	become: true
31	postgresql_user: db={{ execdb_db_name }} user={{ execdb_db_user }} password={{ execdb_db_password }} role_attr_flags=NOSUPERUSER	30	postgresql_user: db={{ execdb_db_name }} user={{ execdb_db_user }} password={{ execdb_db_password }} role_attr_flags=NOSUPERUSER
32		31
33	- name: ensure stg execdb db user has access to stg database
34	when: deployment_type == 'stg'
35	delegate_to: "{{ execdb_db_host }}"
36	become_user: postgres
37	become: true
38	postgresql_user: db={{ execdb_db_name }} user={{ stg_execdb_db_user }} password={{ stg_execdb_db_password }} role_attr_flags=NOSUPERUSER
39
40	- name: ensure prod execdb db user has access to prod database
41	when: deployment_type == 'prod'
42	delegate_to: "{{ execdb_db_host }}"
43	become_user: postgres
44	become: true
45	postgresql_user: db={{ execdb_db_name }} user={{ prod_execdb_db_user }} password={{ prod_execdb_db_password }} role_attr_flags=NOSUPERUSER
46
47	- name: ensure local execdb db user has access to prod database
48	when: deployment_type == 'local'
49	delegate_to: "{{ execdb_db_host }}"
50	become_user: postgres
51	become: true
52	postgresql_user: db={{ execdb_db_name }} user={{ local_execdb_db_user }} password={{ local_execdb_db_password }} role_attr_flags=NOSUPERUSER
53
54	- name: ensure selinux lets httpd talk to postgres	32	- name: ensure selinux lets httpd talk to postgres
55	seboolean: name=httpd_can_network_connect_db persistent=yes state=yes	33	seboolean: name=httpd_can_network_connect_db persistent=yes state=yes
56		34
Context not available.

View Options

roles/taskotron/execdb/templates/alembic.ini.j2

Context not available.
35	# path to migration scripts on a packaged install	35	# path to migration scripts on a packaged install
36	script_location = /usr/share/execdb/alembic	36	script_location = /usr/share/execdb/alembic
37		37
38	{% if deployment_type == 'prod' %}	38	sqlalchemy.url = 'postgresql+psycopg2://{{ execdb_db_user }}:{{ execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
39	sqlalchemy.url = 'postgresql+psycopg2://{{ prod_execdb_db_user }}:{{ prod_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
40	{% endif %}
41	{% if deployment_type == 'stg' %}
42	sqlalchemy.url = 'postgresql+psycopg2://{{ stg_execdb_db_user }}:{{ stg_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
43	{% endif %}
44	{% if deployment_type == 'dev' %}
45	sqlalchemy.url = 'postgresql+psycopg2://{{ dev_execdb_db_user }}:{{ dev_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
46	{% endif %}
47	{% if deployment_type == 'local' %}
48	sqlalchemy.url = 'postgresql+psycopg2://{{ local_execdb_db_user }}:{{ local_execdb_db_password }}@127.0.0.1:{{ execdb_db_port }}/{{ execdb_db_name }}'
49	{% endif %}
50		39
51	# Logging configuration	40	# Logging configuration
52	[loggers]	41	[loggers]
Context not available.

View Options

roles/taskotron/execdb/templates/settings.py.j2

1	{%- if deployment_type == 'prod' %}
2	SECRET_KEY = '{{ prod_execdb_secret_key }}'
3	SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ prod_execdb_db_user }}:{{ prod_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
4	{% endif %}
5	{%- if deployment_type == 'stg' %}
6	SECRET_KEY = '{{ stg_execdb_secret_key }}'
7	SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ stg_execdb_db_user }}:{{ stg_execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
8	{% endif %}
9	{%- if deployment_type == 'dev' or deployment_type == 'local' %}
10	SECRET_KEY = '{{ execdb_secret_key }}'	1	SECRET_KEY = '{{ execdb_secret_key }}'
11	SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ execdb_db_user }}:{{ execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'	2	SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ execdb_db_user }}:{{ execdb_db_password }}@{{ execdb_db_host }}:{{ execdb_db_port }}/{{ execdb_db_name }}'
12	{% endif %}	3
13	FILE_LOGGING = False	4	FILE_LOGGING = False
14	LOGFILR = '/var/log/execdb/execdb.log'	5	LOGFILR = '/var/log/execdb/execdb.log'
15	SYSLOG_LOGGING = False	6	SYSLOG_LOGGING = False

View Options

roles/taskotron/resultsdb-backend/tasks/main.yml

Context not available.
24	postgresql_db: db={{ resultsdb_db_name }}	24	postgresql_db: db={{ resultsdb_db_name }}
25		25
26	- name: ensure resultsdb db user has access to database	26	- name: ensure resultsdb db user has access to database
27	when: deployment_type == 'dev' or deployment_type == 'local'
28	delegate_to: "{{ resultsdb_db_host_machine }}"	27	delegate_to: "{{ resultsdb_db_host_machine }}"
29	become_user: postgres	28	become_user: postgres
30	become: true	29	become: true
31	postgresql_user: db={{ resultsdb_db_name }} user={{ resultsdb_db_user }} password={{ resultsdb_db_password }} role_attr_flags=NOSUPERUSER	30	postgresql_user: db={{ resultsdb_db_name }} user={{ resultsdb_db_user }} password={{ resultsdb_db_password }} role_attr_flags=NOSUPERUSER
32		31
33	- name: ensure stg resultsdb db user has access to stg database
34	when: deployment_type == 'stg'
35	delegate_to: "{{ resultsdb_db_host }}"
36	become_user: postgres
37	become: true
38	postgresql_user: db={{ resultsdb_db_name }} user={{ stg_resultsdb_db_user }} password={{ stg_resultsdb_db_password }} role_attr_flags=NOSUPERUSER
39
40	- name: ensure prod resultsdb db user has access to prod database
41	when: deployment_type == 'prod'
42	delegate_to: "{{ resultsdb_db_host }}"
43	become_user: postgres
44	become: true
45	postgresql_user: db={{ resultsdb_db_name }} user={{ prod_resultsdb_db_user }} password={{ prod_resultsdb_db_password }} role_attr_flags=NOSUPERUSER
46
47	- name: ensure selinux lets httpd talk to postgres	32	- name: ensure selinux lets httpd talk to postgres
48	seboolean: name=httpd_can_network_connect_db persistent=yes state=yes	33	seboolean: name=httpd_can_network_connect_db persistent=yes state=yes
49		34
Context not available.

View Options

roles/taskotron/resultsdb-backend/templates/settings.py.j2

1	{%- if deployment_type == 'prod' %}
2	SECRET_KEY = '{{ prod_resultsdb_secret_key }}'
3	SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ prod_resultsdb_db_user }}:{{ prod_resultsdb_db_password }}@{{ resultsdb_db_host }}:{{ resultsdb_db_port }}/{{ resultsdb_db_name }}'
4	FEDMSG_PUBLISH = True
5	{% endif %}
6	{%- if deployment_type == 'stg' %}
7	SECRET_KEY = '{{ stg_resultsdb_secret_key }}'
8	SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ stg_resultsdb_db_user }}:{{ stg_resultsdb_db_password }}@{{ resultsdb_db_host }}:{{ resultsdb_db_port }}/{{ resultsdb_db_name }}'
9	FEDMSG_PUBLISH = True
10	{% endif %}
11	{%- if deployment_type == 'local' or deployment_type == 'dev'%}
12	SECRET_KEY = '{{ resultsdb_secret_key }}'	1		SECRET_KEY = '{{ resultsdb_secret_key }}'
13	SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ resultsdb_db_user }}:{{ resultsdb_db_password }}@{{ resultsdb_db_host }}:{{ resultsdb_db_port }}/{{ resultsdb_db_name }}'	2		SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ resultsdb_db_user }}:{{ resultsdb_db_password }}@{{ resultsdb_db_host }}:{{ resultsdb_db_port }}/{{ resultsdb_db_name }}'
14	FEDMSG_PUBLISH = False
15	{% endif %}
16	FILE_LOGGING = False	3		FILE_LOGGING = False
17	LOGFILR = '/var/log/resultsdb/resultsdb.log'	4		LOGFILR = '/var/log/resultsdb/resultsdb.log'
18	SYSLOG_LOGGING = False	5		SYSLOG_LOGGING = False
19	STREAM_LOGGING = True	6		STREAM_LOGGING = True
		7
20	FEDMSG_MODNAME = 'taskotron'	8		FEDMSG_MODNAME = 'taskotron'
		9		FEDMSG_PUBLISH = True
		10		{% if deployment_type == 'local' or deployment_type == 'dev'%}
			tflinkUnsubmitted Not Done When we have more than one of these, I'd prefer to see `{% when deployment_type in ['local', 'dev'] %}`
		11		FEDMSG_PUBLISH = False
		12		{% endif %}
		13

View Options

roles/taskotron/resultsdb-frontend/templates/settings.py.j2

1	# while you can use this as a template, we recommend that you use the blockerbugs	1	# while you can use this as a template, we recommend that you use the blockerbugs
2	# cli to generate a config file	2	# cli to generate a config file
3	RDB_URL = 'http://127.0.0.1/{{ resultsdb_endpoint }}/api/v1.0'	3	RDB_URL = 'http://127.0.0.1/{{ resultsdb_endpoint }}/api/v1.0'
4	{% if deployment_type == 'dev' %}	4	SECRET_KEY = '{{ resultsdb_frontend_secret_key }}'
5	SECRET_KEY = '{{ dev_resultsdb_frontend_secret_key }}'
6	{% endif %}
7	{% if deployment_type == 'stg' %}
8	SECRET_KEY = '{{ stg_resultsdb_frontend_secret_key }}'
9	{% endif %}
10	{% if deployment_type == 'prod' %}
11	SECRET_KEY = '{{ prod_resultsdb_frontend_secret_key }}'
12	{% endif %}
13	{% if deployment_type == 'local' %}
14	SECRET_KEY = '{{ local_resultsdb_frontend_secret_key }}'
15	{% endif %}
16		5
17	FILE_LOGGING = False	6	FILE_LOGGING = False
18	LOGFILR = '/var/log/resultsdb_frontend/resultsdb_frontend.log'	7	LOGFILR = '/var/log/resultsdb_frontend/resultsdb_frontend.log'

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	1638		Oct 27 2015, 12:34 PM	★	★
Diff 2	1654		Nov 3 2015, 11:08 AM	★	★
Diff 3	1894		Feb 3 2016, 11:21 AM	★	★

Infra Ansible - fix ExecDB's local deployment inconsistencyClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 1894

inventory/group_vars/resultsdb-dev

inventory/group_vars/resultsdb-prod

inventory/group_vars/resultsdb-stg

inventory/host_vars/testdays.fedorainfracloud.org

roles/taskotron/execdb/tasks/main.yml

roles/taskotron/execdb/templates/alembic.ini.j2

roles/taskotron/execdb/templates/settings.py.j2

roles/taskotron/resultsdb-backend/tasks/main.yml

roles/taskotron/resultsdb-backend/templates/settings.py.j2

roles/taskotron/resultsdb-frontend/templates/settings.py.j2

Infra Ansible - fix ExecDB's local deployment inconsistency
ClosedPublic
Actions