Differential D614

docker: document remote workers, no shared pool directory
ClosedPublic
Actions

Authored by adamwill on Oct 8 2015, 10:40 PM.

Details

Reviewers
garretraziel
jskladan
Commits
rOPENQA82c792ed2063: docker: document remote workers, no shared pool directory
Summary

This commit is all about allowing remote worker containers in
the Docker deployment scenario: worker containers running on
hosts other than the host of the web UI container.

First we ditch the /var/lib/openqa/pool volume from the web UI
container. There is actually no need ever to share this between
workers, because each worker is already isolated within its own
container and can happily have its own pool directory in there.
Effectively, even when all the worker instances are run on the
same Docker host, they're still all separate systems so far as
the web UI / scheduler is concerned. Obviously it can't be
shared this way for workers on remote hosts, so just get rid of
it.

Other than that this is all documentation changes. Mostly just
explaining how to do remote workers, but I also changed the
web UI container instructions to bind ports 80 and 443 on the
host; the worker instances expect the host to be listening on
port 80, in the single-host-box scenario this was covered with
the use of --link but that doesn't work when the worker's on
a different host.

I also updated all the docker image tags to ':latest' - this
seemed sensible, but yell if it's wrong.

Test Plan

Follow the instructions and see if you can actually
get remote workers to, well, work. I did get it going in the
infra cloud.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to docker: document remote workers, no shared pool directory.Oct 8 2015, 10:40 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: garretraziel, jskladan.
garretraziel accepted this revision.Oct 9 2015, 10:21 AM

Aside inline comments, looks good to me.

docker/README.md
8–10

There is actually no need in using tag at all (if no tag is given, :latest is downloaded automatically), but it won't hurt either.

51–53

Maybe add warning that root privileges are needed to bind on port 80 and 443?

76

I assume that you've removed worker number at the end on purpose. I'm ok with it, but will it not be a problem in WebUI (that there are several worker:1 instances)? For example in workers tab in admin etc.

This revision is now accepted and ready to land.Oct 9 2015, 10:21 AM
adamwill added inline comments.Oct 9 2015, 3:38 PM
docker/README.md
8–10

Ah, OK, I'll change that. I guess if we don't specify a tag it gives us some ability to control what will be downloaded from the other end?

51–53

Sure. It looked to me like running Docker as anything other than root was kind of a waste of time anyway...

76

No; it identifies workers as hostname:workerid . That's why we say lower down to make sure each worker instance has a different hostname.

If we're containerizing the workers I can't see any particular reason to run more than one worker in each container, I mean, why not just make it one worker per container? Nice and clean that way. The way the Dockerfiles are set up does not actually let you create more than one worker per container anyhow: even in your original one-Docker-host case, you were still effectively creating one new 'host' (Docker instance) per worker, so you had worker openqa_worker_1:1 , worker openqa_worker_2:2 , worker openqa_worker_3:3 and so on.

Closed by commit rOPENQA82c792ed2063: docker: document remote workers, no shared pool directory (authored by adamwill). · Explain WhyOct 13 2015, 2:42 AM
This revision was automatically updated to reflect the committed changes.
garretraziel added inline comments.Oct 15 2015, 11:27 AM
docker/README.md
8–10

Exactly. We are planning to tag each build with proper version (version of openQA that's inside) so we could request specific openQA version, but if you left out tag when pulling, it automatically pulls "latest" tag (and I think that that's the behavior user would want).

51–53

At least me and Josef added our accounts to docker group so you could run every command without root access.

76

Ok, I don't know how you get the idea we are planning to run more than one worker in one container, but we are certainly not :-). One worker per one container, how it should be with Docker, nice and clean :-). I only thought that workers should have different ID even when they are running on separate hosts (as long as they are connecting to same webui). But if they don't, it's only better.

Revision Contents

PathPackages
Mdocker/README.md (52 lines)
Mdocker/webui/Dockerfile (6 lines)
Mdocker/worker/Dockerfile (3 lines)

Diff 1612

View Options

docker/README.md

1# Get docker images 1​# Get docker images
2 2​
3You can either build the images locally, or get our "latest stable" version from the Docker hub. We recommend using the Docker hub option. 3​You can either build the images locally, or get our "latest stable" version from the Docker hub. We recommend using the Docker hub option.
4 4​
5## Download images from the Docker hub 5​## Download images from the Docker hub
6 6​
7 docker pull fedoraqa/openqa_data 7​ docker pull fedoraqa/openqa_data
8 docker pull fedoraqa/openqa_webui:4.1-3.12 8​ docker pull fedoraqa/openqa_webui
9 docker pull fedoraqa/openqa_worker:4.1-3.12 9​ docker pull fedoraqa/openqa_worker
10 10​
garretrazielUnsubmitted
Not Done

There is actually no need in using tag at all (if no tag is given, :latest is downloaded automatically), but it won't hurt either.

adamwillAuthorUnsubmitted
Not Done

Ah, OK, I'll change that. I guess if we don't specify a tag it gives us some ability to control what will be downloaded from the other end?

garretrazielUnsubmitted
Not Done

Exactly. We are planning to tag each build with proper version (version of openQA that's inside) so we could request specific openQA version, but if you left out tag when pulling, it automatically pulls "latest" tag (and I think that that's the behavior user would want).

11## Build images locally 11​## Build images locally
12 12​
13 docker build -t fedoraqa/openqa_webui:4.1-3.12 ./webui 13​ docker build -t fedoraqa/openqa_webui:latest ./webui
14 docker build -t fedoraqa/openqa_worker:4.1-3.12 ./worker 14​ docker build -t fedoraqa/openqa_worker:latest ./worker
15 docker build -t fedoraqa/openqa_data ./openqa_data 15​ docker build -t fedoraqa/openqa_data:latest ./openqa_data
16 16​
17# Running OpenQA 17​# Running OpenQA
18 18​
19Our intent was to create universal `webui` and `worker` containers and move all data storage and configurations to third container, 19​Our intent was to create universal `webui` and `worker` containers and move all data storage and configurations to third container,
20called `openqa_data`. `openqa_data` is so called [Data Volume Container](http://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container) 20​called `openqa_data`. `openqa_data` is so called [Data Volume Container](http://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container)
21and is used as database, results and configuration storage. During development and in production, you could update `webui` and `worker` images 21​and is used as database, results and configuration storage. During development and in production, you could update `webui` and `worker` images
22but as long as `openqa_data` is intact, you don't lose any data. 22​but as long as `openqa_data` is intact, you don't lose any data.
23 23​
Show All 19 Lines
43 43​
44It is also necessary to either run all containers in privileged mode, or set selinux properly: 44​It is also necessary to either run all containers in privileged mode, or set selinux properly:
45 45​
46 chcon -Rt svirt_sandbox_file_t data 46​ chcon -Rt svirt_sandbox_file_t data
47 47​
48## Run the Data & WebUI containers 48​## Run the Data & WebUI containers
49 49​
50 docker run -d -h openqa_data --name openqa_data -v `pwd`/data/factory:/data/factory -v `pwd`/data/tests:/data/tests fedoraqa/openqa_data 50​ docker run -d -h openqa_data --name openqa_data -v `pwd`/data/factory:/data/factory -v `pwd`/data/tests:/data/tests fedoraqa/openqa_data
51 docker run -d -h openqa_webui --volumes-from openqa_data --name openqa_webui -p 8080:443 fedoraqa/openqa_webui:4.1-3.12 51​ docker run -d -h openqa_webui --volumes-from openqa_data --name openqa_webui -p 80:80 -p 443:443 fedoraqa/openqa_webui
52​
53​You can change the `-p` parameters if you do not want the openQA instance to own ports 80 and 443, e.g. `-p 8080:80 -p 8043:443`, but this will cause problems if you wish to set up workers on other hosts (see below). You do need root privileges to bind ports 80 and 443 in this way.
garretrazielUnsubmitted
Not Done

Maybe add warning that root privileges are needed to bind on port 80 and 443?

adamwillAuthorUnsubmitted
Not Done

Sure. It looked to me like running Docker as anything other than root was kind of a waste of time anyway...

garretrazielUnsubmitted
Not Done

At least me and Josef added our accounts to docker group so you could run every command without root access.

52 54​
53It is now necessary to create and store the client keys for OpenQA. In the next two steps, you will set an OpenID provider (if necessary), 55​It is now necessary to create and store the client keys for OpenQA. In the next two steps, you will set an OpenID provider (if necessary),
54create the API keys in the OpenQA's web interface, and store the configuration in the Data Container. 56​create the API keys in the OpenQA's web interface, and store the configuration in the Data Container.
55 57​
56### Change the OpenID provider 58​### Change the OpenID provider
57 59​
58https://id.fedoraproject.org/ is set as a default OpenID provider. To change it, run: 60​https://id.fedoraproject.org/ is set as a default OpenID provider. To change it, run:
59 61​
60 docker exec -ti openqa_data /scripts/set_openid 62​ docker exec -ti openqa_data /scripts/set_openid
61 63​
62and enter the Provider's URL. 64​and enter the Provider's URL.
63 65​
64### Set API keys 66​### Set API keys
65 67​
66Go to https://localhost:8080/api_keys, generate key and secret. Then run: 68​Go to https://localhost/api_keys, generate key and secret. Then run:
67 69​
68 docker exec -ti openqa_data /scripts/set_keys 70​ docker exec -ti openqa_data /scripts/set_keys
69 71​
70and enter the Key and Secret. 72​and enter the Key and Secret.
71 73​
72## Run the Worker container 74​## Run the Worker container
73 75​
74 docker run -h openqa_worker_1 --name openqa_worker_1 -d --link openqa_webui:openqa_webui --volumes-from openqa_data --volumes-from openqa_webui --privileged fedoraqa/openqa_worker:4.1-3.12 1 76​ docker run -h openqa_worker_1 --name openqa_worker_1 -d --link openqa_webui:openqa_webui --volumes-from openqa_data --privileged fedoraqa/openqa_worker
garretrazielUnsubmitted
Not Done

I assume that you've removed worker number at the end on purpose. I'm ok with it, but will it not be a problem in WebUI (that there are several worker:1 instances)? For example in workers tab in admin etc.

adamwillAuthorUnsubmitted
Not Done

No; it identifies workers as hostname:workerid . That's why we say lower down to make sure each worker instance has a different hostname.

If we're containerizing the workers I can't see any particular reason to run more than one worker in each container, I mean, why not just make it one worker per container? Nice and clean that way. The way the Dockerfiles are set up does not actually let you create more than one worker per container anyhow: even in your original one-Docker-host case, you were still effectively creating one new 'host' (Docker instance) per worker, so you had worker openqa_worker_1:1 , worker openqa_worker_2:2 , worker openqa_worker_3:3 and so on.

garretrazielUnsubmitted
Not Done

Ok, I don't know how you get the idea we are planning to run more than one worker in one container, but we are certainly not :-). One worker per one container, how it should be with Docker, nice and clean :-). I only thought that workers should have different ID even when they are running on separate hosts (as long as they are connecting to same webui). But if they don't, it's only better.

75 77​
76Check whether the worker connected in the WebUI's administration interface. 78​Check whether the worker connected in the WebUI's administration interface.
77 79​
78To add more workers, increase number that is used in hostname, container name and at the end of command, so to add worker 2 use: 80​To add more workers, increase number that is used in hostname and container name, so to add worker 2 use:
79 81​
80 docker run -h openqa_worker_2 --name openqa_worker_2 -d --link openqa_webui:openqa_webui --volumes-from openqa_data --volumes-from openqa_webui --privileged fedoraqa/openqa_worker:4.1-3.12 2 82​ docker run -h openqa_worker_2 --name openqa_worker_2 -d --link openqa_webui:openqa_webui --volumes-from openqa_data --privileged fedoraqa/openqa_worker
81 83​
82## Enable services 84​## Enable services
83 85​
84Some systemd services are provided to start up the containers, so you don't have to keep doing it manually. To install and enable them: 86​Some systemd services are provided to start up the containers, so you don't have to keep doing it manually. To install and enable them:
85 87​
86 sudo cp systemd/*.service /etc/systemd/system 88​ sudo cp systemd/*.service /etc/systemd/system
87 sudo systemctl daemon-reload 89​ sudo systemctl daemon-reload
88 sudo systemctl enable openqa-data.service 90​ sudo systemctl enable openqa-data.service
Show All 31 Lines
120# Running jobs 122​# Running jobs
121 123​
122After performing the "setup" tasks above - do not forget about tests and ISOs - you can schedule a test like this: 124​After performing the "setup" tasks above - do not forget about tests and ISOs - you can schedule a test like this:
123 125​
124 docker exec openqa_webui /var/lib/openqa/script/client isos post ISO=Fedora-Server-netinst-x86_64-22_Beta_RC3.iso DISTRI=fedora VERSION=rawhide FLAVOR=generic_boot ARCH=x86_64 BUILD=22_Beta_RC3 126​ docker exec openqa_webui /var/lib/openqa/script/client isos post ISO=Fedora-Server-netinst-x86_64-22_Beta_RC3.iso DISTRI=fedora VERSION=rawhide FLAVOR=generic_boot ARCH=x86_64 BUILD=22_Beta_RC3
125 127​
126# Other specific cases 128​# Other specific cases
127 129​
130​## Adding workers on other hosts
131​
132​You may want to add workers on other hosts, so you don't need one powerful host to run the UI and all the workers.
133​
134​Let's assume you're setting up a new 'worker host', and it can see the web UI host system with the hostname `openqa_webui`.
135​
136​You must somehow share the `data` directory from the web UI host to each host on which you want to run workers. For instance, to use sshfs, on the new worker host, run:
137​
138​ sshfs -o context=unconfined_u:object_r:svirt_sandbox_file_t:s0 openqa_webui:/path/to/data /path/to/data
139​
140​Of course, the worker host must have an ssh key the web UI host will accept. You can add this mount to `/etc/fstab` to make it permanent.
141​
142​Then check `openqa_fedora_tools` out on the worker host and run the data container, as described above:
143​
144​ docker run -d -h openqa_data --name openqa_data -v /path/to/data/factory:/data/factory -v /path/to/data/tests:/data/tests fedoraqa/openqa_data
145​
146​and set up the API key with `docker exec -ti openqa_data /scripts/set_keys`.
147​
148​Finally create a worker container, but omit the use of `--link`. Ensure you use a hostname which is different from all other worker instances on all other hosts. The container name only has to be unique on this host, but it probably makes sense to always match the hostname to the container name:
149​
150​ docker run -h openqa_worker_3 --name openqa_worker_3 -d --volumes-from openqa_data --privileged fedoraqa/openqa_worker
151​
152​If the container will not be able to resolve the `openqa_webui` hostname (this depends on your network setup) you can use `--add-host` to add a line to `/etc/hosts` when running the container:
153​
154​ docker run -h openqa_worker_3 --name openqa_worker_3 -d --add-host="openqa_webui:10.0.0.1" --volumes-from openqa_data --privileged fedoraqa/openqa_worker
155​
156​Worker instances always expect to find the server as `openqa_webui`; if this will not work you must adjust the `/data/conf/client.conf` and `/data/conf/workers.ini` files in the data container. You will also need to adjust these files if you use non-standard ports (see above).
157​
128## Keeping all data in the Data Volume Container 158​## Keeping all data in the Data Volume Container
129 159​
130If you decided to keep all the data in the Volume Container (`openqa_data`) then instead of [Create directory structure] run: 160​If you decided to keep all the data in the Volume Container (`openqa_data`) then instead of [Create directory structure] run:
131 161​
132 docker exec openqa_data mkdir -p data/factory/{iso,hdd} data/tests 162​ docker exec openqa_data mkdir -p data/factory/{iso,hdd} data/tests
133 docker exec openqa_data chmod -R 777 data/factory/{iso,hdd} data/tests 163​ docker exec openqa_data chmod -R 777 data/factory/{iso,hdd} data/tests
134 164​
135In the [Run the Data & WebUI containers] section run the `openqa_data` container like this instead: 165​In the [Run the Data & WebUI containers] section run the `openqa_data` container like this instead:
Show All 11 Lines
147 177​
148If you want to keep all the data in the host system and you don't want to use Volume Container, then instead of [Create directory structure] run: 178​If you want to keep all the data in the host system and you don't want to use Volume Container, then instead of [Create directory structure] run:
149 179​
150 cp -a openqa_data/data.template data 180​ cp -a openqa_data/data.template data
151 chcon -Rt svirt_sandbox_file_t data 181​ chcon -Rt svirt_sandbox_file_t data
152 182​
153In the [Run the Data & WebUI containers] section don't run `openqa_data` container and run the `webui` container like this instead: 183​In the [Run the Data & WebUI containers] section don't run `openqa_data` container and run the `webui` container like this instead:
154 184​
155 docker run -d -h openqa_webui -v `pwd`/data:/data --name openqa_webui -p 8080:443 fedoraqa/openqa_webui:4.1-3.12 185​ docker run -d -h openqa_webui -v `pwd`/data:/data --name openqa_webui -p 443:443 -p 80:80 fedoraqa/openqa_webui:4.1-3.12
156 186​
157Change OpenID provider in `data/conf/openqa.ini` under `provider` in `[openid]` section and then put Key and Secret under 187​Change OpenID provider in `data/conf/openqa.ini` under `provider` in `[openid]` section and then put Key and Secret under
158both sections in `data/conf/client.conf`. 188​both sections in `data/conf/client.conf`.
159 189​
160In the [Run the Worker container] section, run worker as: 190​In the [Run the Worker container] section, run worker as:
161 191​
162 docker run -h openqa_worker_1 --name openqa_worker_1 -d --link openqa_webui:openqa_webui -v `pwd`/data:/data --volumes-from openqa_webui --privileged fedoraqa/openqa_worker:4.1-3.12 1 192​ docker run -h openqa_worker_1 --name openqa_worker_1 -d --link openqa_webui:openqa_webui -v `pwd`/data:/data --volumes-from openqa_webui --privileged fedoraqa/openqa_worker:4.1-3.12 1
163 193​
Show All 11 Lines
View Options

docker/webui/Dockerfile

Show All 18 Lines
19ADD openqa.conf /etc/apache2/vhosts.d/openqa.conf 19​ADD openqa.conf /etc/apache2/vhosts.d/openqa.conf
20ADD run_openqa.sh /root/ 20​ADD run_openqa.sh /root/
21 21​
22# set-up shared data and configuration 22​# set-up shared data and configuration
23RUN rm -rf /etc/openqa/openqa.ini /etc/openqa/client.conf \ 23​RUN rm -rf /etc/openqa/openqa.ini /etc/openqa/client.conf \
24 /var/lib/openqa/share/factory /var/lib/openqa/share/tests \ 24​ /var/lib/openqa/share/factory /var/lib/openqa/share/tests \
25 /var/lib/openqa/db/db.sqlite /var/lib/openqa/testresults && \ 25​ /var/lib/openqa/db/db.sqlite /var/lib/openqa/testresults && \
26 chmod +x /root/run_openqa.sh && \ 26​ chmod +x /root/run_openqa.sh && \
27 mkdir -p /var/lib/openqa/pool && \ ​
28 mkdir -p /run/dbus && \ 27​ mkdir -p /run/dbus && \
29 ln -s /data/conf/openqa.ini /etc/openqa/openqa.ini && \ 28​ ln -s /data/conf/openqa.ini /etc/openqa/openqa.ini && \
30 ln -s /data/conf/client.conf /etc/openqa/client.conf && \ 29​ ln -s /data/conf/client.conf /etc/openqa/client.conf && \
31 ln -s /data/factory /var/lib/openqa/share/factory && \ 30​ ln -s /data/factory /var/lib/openqa/share/factory && \
32 ln -s /data/tests /var/lib/openqa/share/tests && \ 31​ ln -s /data/tests /var/lib/openqa/share/tests && \
33 ln -s /data/testresults /var/lib/openqa/testresults && \ 32​ ln -s /data/testresults /var/lib/openqa/testresults && \
34 ln -s /data/db/db.sqlite /var/lib/openqa/db/db.sqlite && \ 33​ ln -s /data/db/db.sqlite /var/lib/openqa/db/db.sqlite
35 chmod -R 777 /var/lib/openqa/pool ​
36 ​
37VOLUME ["/var/lib/openqa/pool"] ​
38 34​
39EXPOSE 80 443 35​EXPOSE 80 443
40CMD ["/root/run_openqa.sh"] 36​CMD ["/root/run_openqa.sh"]
View Options

docker/worker/Dockerfile

Show All 10 Lines4RUN zypper --non-interactive in ca-certificates-mozilla curl && \
11 zypper --non-interactive in kmod && \ 11​ zypper --non-interactive in kmod && \
12 zypper --non-interactive in --from Virtualization qemu-ovmf-x86_64 12​ zypper --non-interactive in --from Virtualization qemu-ovmf-x86_64
13 13​
14# set-up qemu 14​# set-up qemu
15RUN mkdir -p /root/qemu 15​RUN mkdir -p /root/qemu
16ADD kvm-mknod.sh /root/qemu/kvm-mknod.sh 16​ADD kvm-mknod.sh /root/qemu/kvm-mknod.sh
17RUN chmod +x /root/qemu/*.sh && /root/qemu/kvm-mknod.sh && \ 17​RUN chmod +x /root/qemu/*.sh && /root/qemu/kvm-mknod.sh && \
18 # set-up shared data and configuration 18​ # set-up shared data and configuration
19 rm -rf /etc/openqa/client.conf /etc/openqa/workers.ini \ 19​ rm -rf /etc/openqa/client.conf /etc/openqa/workers.ini && \
20 /var/lib/openqa/pool && \ ​
21 mkdir -p /var/lib/openqa/share && \ 20​ mkdir -p /var/lib/openqa/share && \
22 ln -s /data/conf/client.conf /etc/openqa/client.conf && \ 21​ ln -s /data/conf/client.conf /etc/openqa/client.conf && \
23 ln -s /data/conf/workers.ini /etc/openqa/workers.ini && \ 22​ ln -s /data/conf/workers.ini /etc/openqa/workers.ini && \
24 ln -s /data/factory /var/lib/openqa/share/factory && \ 23​ ln -s /data/factory /var/lib/openqa/share/factory && \
25 ln -s /data/tests /var/lib/openqa/share/tests 24​ ln -s /data/tests /var/lib/openqa/share/tests
26 25​
27ENTRYPOINT ["/usr/share/openqa/script/worker", "--verbose", "--instance"] 26​ENTRYPOINT ["/usr/share/openqa/script/worker", "--verbose", "--instance"]
28CMD ["1"] 27​CMD ["1"]