Add Testdays app to infra-ansible
AbandonedPublic
Actions

Authored by jskladan on Jul 14 2015, 12:17 PM.

Details

Reviewers

tflink
mkrizek

Summary

Hopefully this is "it" for the Testdays playbook.

Test Plan

None - is there any reasonable way to test it?

Diff Detail

Lint

Lint Skipped

Unit

Unit Tests Skipped

jskladan retitled this revision from to Add Testdays app to infra-ansible.Jul 14 2015, 12:17 PM

jskladan updated this object.

jskladan edited the test plan for this revision. (Show Details)

jskladan added reviewers: tflink, mkrizek.

jskladan added a project: infrastructure.

Herald added a subscriber: tflink. · View Herald TranscriptJul 14 2015, 12:17 PM

@jskladan and I talked about this a bit over IRC but figured it would be good to continue the conversation in the review.

My main concern here is where the app will be hosted - as a qa app (more like taskotron) or an infra app. I think this depends on a few things:

the desired hostname (testdays.fp.o vs. testdays.qa.fp.o)
run-ability on el7

The more I think about it, I think it should be hosted like other infra apps (and blockerbugs) instead of being hosted like taskotron. The only reason I can think of that could prevent it being hosted that was was if resultsdb didn't work on el7.

However, for the short/medium term, we need to do something about the old f19 cloud instance that's hanging around. I'll look into making this into testdays.fedorainfracloud.org until we find a more permanent resting place for the app.

As a side question that I genuinely don't know the answer to - how many groups used testdays.fp.o for test days during f22?

I'm starting to think that this may be a better candidate for a fedorainfracloud.org instance so that we don't have to deal with the centos/infra bits right now. AFAIK, testdays is pretty much the only thing keeping the old cloud from being decomissioned.

I still want to see it in ansible and we need to have more than 1 person with access to the machine but I'm thinking that would be easier for now - any other thoughts?

I don't have any preference either way - I think we long since established that I'm not really the sysadmin :D
If deploying it in fedorainfracloud is feasible, then let's do it. Or I can deploy it in openstack, and we can just set the DNS properly to point in the right direction.

In D441#9005, @jskladan wrote:

I don't have any preference either way - I think we long since established that I'm not really the sysadmin :D

Even though you're the one of the two of us with a sysadmin certificate? :-D

If deploying it in fedorainfracloud is feasible, then let's do it. Or I can deploy it in openstack, and we can just set the DNS properly to point in the right direction.

It's easier and will be faster - there are a bunch of tickets that I'd have to file in order to deploy testdays as described in these ansible snippets.

If we're changing to a cloud host, some changes are needed. These are the first ones that I see

inventory/group_vars/testdays
14	I'm not sure what the defaults for `ks_repo` and `ks_url` are off the top of my head but it's better to be specific with them so that we get a known result
22	Since this'll be self-hosted, it should be `{{ inventory_hostname }}`
28	same as above, this should be `{{inventory_hostname }}`
33	testdays.fedorainfracloud.org
36	this can be changed to be more specific, no? Either way, the range will need to change if we're moving this to the new cloud instance
inventory/host_vars/db-qa01.qa.fedoraproject.org
30	if the db is self-hosted, these don't need to be backed up from db-qa01
inventory/inventory
512	this should change to testdays.fedorainfracloud.org
playbooks/groups/testdays.yml
63	This'll need postgresql_server as well
playbooks/include/proxies-reverseproxy.yml
332	All the changes in this file aren't needed for cloud stuff

This revision now requires changes to proceed.Jul 30 2015, 3:20 PM

I've added this to the infra ansible repo and have deployed testdays.fedorainfracloud.org - this review is no longer needed.

jskladan abandoned this revision.Sep 15 2015, 8:20 AM

Revision Contents

		Path
A	M	inventory/group_vars/testdays (38 lines)
M		inventory/host_vars/db-qa01.qa.fedoraproject.org (2 lines)
M		inventory/inventory (3 lines)
M		master.yml (1 line)
A	M	playbooks/groups/testdays.yml (69 lines)
M		playbooks/include/proxies-reverseproxy.yml (25 lines)
A	M	roles/taskotron/testdays/tasks/main.yml (31 lines)
A	M	roles/taskotron/testdays/templates/settings.py.j2 (11 lines)
A	M	roles/taskotron/testdays/templates/testdays.conf.j2 (18 lines)

Diff 1163

View Options

inventory/group_vars/testdays

This file was added.

1		---
2		# common items for the releng-* boxes
3		lvm_size: 50000
4		mem_size: 4096
5		num_cpus: 4
6		# for systems that do not match the above - specify the same parameter in
7		# the host_vars/$hostname file
8
9		tcp_ports: [ 80, 443, "{{ resultsdb_db_port }}", "{{ testdays_db_port }}"]
10		fas_client_groups: sysadmin-qa,sysadmin-main,fi-apprentice
11		nrpe_procs_warn: 250
12		nrpe_procs_crit: 300
13
14		virt_install_command: /usr/sbin/virt-install -n {{ inventory_hostname }} -r {{ mem_size }}
	tflinkUnsubmitted Not Done I'm not sure what the defaults for `ks_repo` and `ks_url` are off the top of my head but it's better to be specific with them so that we get a known result
15		--disk bus=virtio,path={{ volgroup }}/{{ inventory_hostname }}
16		--vcpus={{ num_cpus }} -l {{ ks_repo }} -x
17		"ksdevice=eth0 ks={{ ks_url }} console=tty0 console=ttyS0
18		hostname={{ inventory_hostname }} nameserver={{ dns }}
19		ip={{ eth0_ip }}::{{ gw }}:{{ nm }}:{{ inventory_hostname }}:eth0:none"
20		--network=bridge=br0,model=virtio --autostart --noautoconsole
21
22		resultsdb_db_host: db-qa01.qa.fedoraproject.org
	tflinkUnsubmitted Not Done Since this'll be self-hosted, it should be `{{ inventory_hostname }}`
23		resultsdb_db_port: 5432
24		resultsdb_endpoint: 'resultsdb_api'
25		resultsdb_fe_endpoint: 'resultsdb'
26		resultsdb_db_name: resultsdb_testdays
27
28		testdays_db_host: db-qa01.qa.fedoraproject.org
	tflinkUnsubmitted Not Done same as above, this should be `{{inventory_hostname }}`
29		testdays_db_port: 5432
30		testdays_endpoint: 'testdays'
31		testdays_db_name: testdays
32
33		external_hostname: testdays.fedoraproject.org
	tflinkUnsubmitted Not Done testdays.fedorainfracloud.org
34
35		allowed_hosts:
36		- 10.5.124
	tflinkUnsubmitted Not Done this can be changed to be more specific, no? Either way, the range will need to change if we're moving this to the new cloud instance
37
38		freezes: false

View Options

inventory/host_vars/db-qa01.qa.fedoraproject.org

Context not available.
27	- resultsdb	27		- resultsdb
28	- resultsdb_stg	28		- resultsdb_stg
29	- resultsdb_dev	29		- resultsdb_dev
		30		- resultsdb_testdays
			tflinkUnsubmitted Not Done if the db is self-hosted, these don't need to be backed up from db-qa01
		31		- testdays
30		32
31	mariadb_root_password: "{{ dbqa01_mysql_root_password }}"	33		mariadb_root_password: "{{ dbqa01_mysql_root_password }}"
32		34
Context not available.

View Options

inventory/inventory

Context not available.
508	[resultsdb-prod]	508		[resultsdb-prod]
509	resultsdb01.qa.fedoraproject.org	509		resultsdb01.qa.fedoraproject.org
510		510
		511		[testdays]
		512		testdays01.qa.fedoraproject.org
			tflinkUnsubmitted Not Done this should change to testdays.fedorainfracloud.org
		513
511	[smtp-mm]	514		[smtp-mm]
512	smtp-mm-ib01.fedoraproject.org	515		smtp-mm-ib01.fedoraproject.org
513	smtp-mm-coloamer01.fedoraproject.org	516		smtp-mm-coloamer01.fedoraproject.org
Context not available.

View Options

master.yml

Context not available.
84	- include: /srv/web/infra/ansible/playbooks/groups/resultsdb-prod.yml	84	- include: /srv/web/infra/ansible/playbooks/groups/resultsdb-prod.yml
85	- include: /srv/web/infra/ansible/playbooks/groups/resultsdb-dev.yml	85	- include: /srv/web/infra/ansible/playbooks/groups/resultsdb-dev.yml
86	- include: /srv/web/infra/ansible/playbooks/groups/resultsdb-stg.yml	86	- include: /srv/web/infra/ansible/playbooks/groups/resultsdb-stg.yml
		87	- include: /srv/web/infra/ansible/playbooks/groups/testdays.yml
87	- include: /srv/web/infra/ansible/playbooks/groups/retrace.yml	88	- include: /srv/web/infra/ansible/playbooks/groups/retrace.yml
88	- include: /srv/web/infra/ansible/playbooks/groups/releng-compose.yml	89	- include: /srv/web/infra/ansible/playbooks/groups/releng-compose.yml
89	- include: /srv/web/infra/ansible/playbooks/groups/secondary.yml	90	- include: /srv/web/infra/ansible/playbooks/groups/secondary.yml
Context not available.

View Options

playbooks/groups/testdays.yml

This file was added.

1		---
2		# create a new taskotron server
3		# NOTE: make sure there is room/space for this server on the vmhost
4		# NOTE: most of these vars_path come from group_vars/mirrorlist or from hostvars
5
6		- name: make testdays
7		hosts: testdays
8		user: root
9		gather_facts: False
10
11		vars_files:
12		- /srv/web/infra/ansible/vars/global.yml
13		- "/srv/private/ansible/vars.yml"
14		- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
15
16		tasks:
17		- include: "{{ tasks }}/virt_instance_create.yml"
18
19		handlers:
20		- include: "{{ handlers }}/restart_services.yml"
21
22		- name: make the box be real
23		hosts: testdays
24		user: root
25		gather_facts: True
26
27		vars_files:
28		- /srv/web/infra/ansible/vars/global.yml
29		- "/srv/private/ansible/vars.yml"
30		- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
31
32		roles:
33		- { role: base, tags: ['base'] }
34		- { role: rkhunter, tags: ['rkhunter'] }
35		- { role: nagios_client, tags: ['nagios_client'] }
36		- { role: hosts, tags: ['hosts']}
37		- { role: fas_client, tags: ['fas_client'] }
38		- { role: collectd/base, tags: ['collectd_base'] }
39		- { role: yum-cron, tags: ['yumcron'] }
40		- { role: sudo, tags: ['sudo'] }
41		- apache
42
43		tasks:
44		# this is how you include other task lists
45		- include: "{{ tasks }}/yumrepos.yml"
46		- include: "{{ tasks }}/2fa_client.yml"
47		- include: "{{ tasks }}/motd.yml"
48
49		handlers:
50		- include: "{{ handlers }}/restart_services.yml"
51
52		- name: configure resultsdb and testdays
53		hosts: testdays
54		user: root
55		gather_facts: True
56
57		vars_files:
58		- /srv/web/infra/ansible/vars/global.yml
59		- "/srv/private/ansible/vars.yml"
60		- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
61
62		roles:
63		- { role: taskotron/resultsdb-backend, tags: ['resultsdb-be'] }
	tflinkUnsubmitted Not Done This'll need postgresql_server as well
64		- { role: taskotron/resultsdb-frontend, tags: ['resultsdb-fe'] }
65		- { role: taskotron/testdays, tags: ['testdays'] }
66
67		handlers:
68		- include: "{{ handlers }}/restart_services.yml"
69

View Options

playbooks/include/proxies-reverseproxy.yml

Context not available.
332	# Talk directly to the app server, not haproxy	332		# Talk directly to the app server, not haproxy
			tflinkUnsubmitted Not Done All the changes in this file aren't needed for cloud stuff
333	proxyurl: http://log01	333		proxyurl: http://log01
334		334
		335		### Three entries for the Testdays app
		336		- role: httpd/reverseproxy
		337		website: testdays.fedoraproject.org
		338		destname: testdays-resultsdb
		339		localpath: /resultsdb
		340		remotepath: /resultsdb
		341		# Talk directly to the app server, not haproxy
		342		proxyurl: http://testdays01.vpn.fedoraproject.org
		343
		344		- role: httpd/reverseproxy
		345		website: testdays.fedoraproject.org
		346		destname: testdays-resultsdbapi
		347		localpath: /resultsdb_api
		348		remotepath: /resultsdb_api
		349		# Talk directly to the app server, not haproxy
		350		proxyurl: http://testdays01.vpn.fedoraproject.org
		351
		352		- role: httpd/reverseproxy
		353		website: testdays.fedoraproject.org
		354		destname: testdays
		355		localpath: /testdays
		356		remotepath: /testdays
		357		# Talk directly to the app server, not haproxy
		358		proxyurl: http://testdays01.vpn.fedoraproject.org
		359
335	### Four entries for taskotron for production	360		### Four entries for taskotron for production
336	- role: httpd/reverseproxy	361		- role: httpd/reverseproxy
337	website: taskotron.fedoraproject.org	362		website: taskotron.fedoraproject.org
Context not available.

View Options

roles/taskotron/testdays/tasks/main.yml

This file was added.

		1		- name: ensure packages required for testdays are installed
		2		action: yum name={{ item }} state=latest
		3		with_items:
		4		- testdays
		5		- mod_wsgi
		6		- python-psycopg2
		7
		8		- name: ensure database is created
		9		delegate_to: "{{ testdays_db_host }}"
		10		sudo_user: postgres
		11		sudo: true
		12		action: postgresql_db db={{ testdays_db_name }}
		13
		14		- name: ensure testdays db user has access to database
		15		delegate_to: "{{ testdays_db_host }}"
		16		sudo_user: postgres
		17		sudo: true
		18		action: postgresql_user db={{ testdays_db_name }} user={{ testdays_db_user }} password={{ testdays_db_password }} role_attr_flags=NOSUPERUSER
		19
		20		- name: ensure selinux lets httpd talk to postgres
		21		seboolean: name=httpd_can_network_connect_db persistent=yes state
		22
		23		- name: generate testdays config
		24		template: src=settings.py.j2 dest=/etc/testdays/settings.py owner=root group=root mode=0644
		25		notify:
		26		- restart httpd
		27
		28		- name: generate testdays apache config
		29		template: src=testdays.conf.j2 dest=/etc/httpd/conf.d/testdays.conf owner=root group=root mode=0644
		30		notify:
		31		- restart httpd

View Options

roles/taskotron/testdays/templates/settings.py.j2

This file was added.

		1		# while you can use this as a template, we recommend that you use the blockerbugs
		2		# cli to generate a config file
		3		RESULTSDB_URL = 'http://127.0.0.1/{{ resultsdb_endpoint }}/api/v1.0'
		4		SECRET_KEY = '{{ testdays_secret_key }}'
		5		SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{{ testdays_db_user }}:{{ testdays_db_password }}@{{ testdays_db_host }}:{{ testdays_db_port }}/{{ testdays_db_name }}'
		6		SHOW_DB_URI = False
		7		PRODUCTION = True
		8		FILE_LOGGING = False
		9		LOGFILR = '/var/log/testdays/testdays.log'
		10		SYSLOG_LOGGING = False
		11		STREAM_LOGGING = True

View Options

roles/taskotron/testdays/templates/testdays.conf.j2

This file was added.

		1		WSGIDaemonProcess testdays user=apache group=apache threads=5
		2		WSGIScriptAlias /{{ testdays_endpoint }} /usr/share/testdays/testdays.wsgi
		3		WSGISocketPrefix run/wsgi
		4
		5		<Directory /usr/share/testdays>
		6		WSGIProcessGroup testdays
		7		WSGIApplicationGroup %{GLOBAL}
		8		WSGIScriptReloading On
		9		<IfModule mod_authz_core.c>
		10		# Apache 2.4
		11		Require all granted
		12		</IfModule>
		13		<IfModule !mod_auth_core.c>
		14		Order allow,deny
		15		Allow from all
		16		</IfModule>
		17
		18		</Directory>

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	1163			Jul 14 2015, 12:16 PM	★	★

Add Testdays app to infra-ansibleAbandonedPublicActions

Details

Diff Detail

Revision Contents

Diff 1163

inventory/group_vars/testdays

inventory/host_vars/db-qa01.qa.fedoraproject.org

inventory/inventory

master.yml

playbooks/groups/testdays.yml

playbooks/include/proxies-reverseproxy.yml

roles/taskotron/testdays/tasks/main.yml

roles/taskotron/testdays/templates/settings.py.j2

roles/taskotron/testdays/templates/testdays.conf.j2

Add Testdays app to infra-ansible
AbandonedPublic
Actions