Differential D615

replace set_keys with a more capable python script
ClosedPublic
Actions

Authored by adamwill on Oct 9 2015, 1:40 AM.

Details

Reviewers
garretraziel
jskladan
Commits
rOPENQA750eb7aa3650: replace set_keys with a more capable python script
Summary

This thing...does...more...stuff. Basically I was thinking
about how we can ease automated deployment of containerized
openQA in infra, and it seemed like it'd be good to have this.
It lets you use variant hostnames, set worker instances to use
HTTPS or not, flip a worker host around between different
servers (e.g. a staging and a production one), stuff like that.
It also doesn't require interaction, which is good for ansible
etc.

Test Plan

Check it actually works and the change to the doc
is correct.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to replace set_keys with a more capable python script.Oct 9 2015, 1:40 AM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: garretraziel, jskladan.
adamwill updated this revision to Diff 1605.Oct 9 2015, 1:56 AM

fix KEY and SECRET metavars, add optionxform comment

adamwill updated this revision to Diff 1606.Oct 9 2015, 2:10 AM

bit better handling of 'localhost'

God, I kinda hate how this stuff is set up, trying to handle it
sensibly.

jskladan accepted this revision.Oct 9 2015, 11:48 AM

well, I guess, why not. Seems like using a sledgehammer to crack a nut though :)

docker/openqa_data/scripts/client-conf
69–72

pls rename sec[s] to section[s]

71

what happens if args.hostname is set to 'localhost' already? I guess that it just goes through the cycle twice, setting the same thing two times, but just making sure ;)

123–159

every time I see this, I just crave for more people using https://github.com/docopt/docopt *wink wink*

This revision is now accepted and ready to land.Oct 9 2015, 11:48 AM
adamwill added inline comments.Oct 13 2015, 2:10 AM
docker/openqa_data/scripts/client-conf
71

yeah, exactly, it'd just do it twice, which causes no harm. It seemed not worth trying to handle this more cleverly.

123–159

eh, everyone seems to have a favourite. New Bodhi CLI is using some thing called 'click'. I'm just sticking with argparse as it's in core and widely used and I know how to use it now...FWIW, I don't like that docopt approach at all, I think I'd find it much harder than just using argparse...for the record, this took me like an hour to write, it's not much of a sledgehammer. :P

jskladan added inline comments.Oct 13 2015, 7:03 AM
docker/openqa_data/scripts/client-conf
123–159

It took me 15 minutes to read (just the parse_args()), which is why I prefer docopt. But this is more of a "lets yell at each other over beers" stuff than a technical quality of the code thing :)

Closed by commit rOPENQA750eb7aa3650: replace set_keys with a more capable python script (authored by adamwill). · Explain WhyOct 13 2015, 5:29 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 1615

View Options

docker/README.md

Show First 20 Lines • Show All 61 Lines • ▼ Show 20 Line(s)60https://id.fedoraproject.org/ is set as a default OpenID provider. To change it, run:
62 docker exec -ti openqa_data /scripts/set_openid 62​ docker exec -ti openqa_data /scripts/set_openid
63 63​
64and enter the Provider's URL. 64​and enter the Provider's URL.
65 65​
66### Set API keys 66​### Set API keys
67 67​
68Go to https://localhost/api_keys, generate key and secret. Then run: 68​Go to https://localhost/api_keys, generate key and secret. Then run:
69 69​
70 docker exec -ti openqa_data /scripts/set_keys 70​ docker exec -ti openqa_data /scripts/client-conf set -l (KEY) (SECRET)
71 ​
72and enter the Key and Secret. ​
73 71​
74## Run the Worker container 72​## Run the Worker container
75 73​
76 docker run -h openqa_worker_1 --name openqa_worker_1 -d --link openqa_webui:openqa_webui --volumes-from openqa_data --privileged fedoraqa/openqa_worker 74​ docker run -h openqa_worker_1 --name openqa_worker_1 -d --link openqa_webui:openqa_webui --volumes-from openqa_data --privileged fedoraqa/openqa_worker
77 75​
78Check whether the worker connected in the WebUI's administration interface. 76​Check whether the worker connected in the WebUI's administration interface.
79 77​
80To add more workers, increase number that is used in hostname and container name, so to add worker 2 use: 78​To add more workers, increase number that is used in hostname and container name, so to add worker 2 use:
▲ Show 20 Lines • Show All 124 Lines • Show Last 20 Lines
View Options

docker/openqa_data/scripts/client-conf

  • This file was added.
1​#!/bin/python
2​
3​# Copyright (C) 2015 Red Hat
4​#
5​# This script is free software; you can redistribute it and/or modify
6​# it under the terms of the GNU General Public License as published by
7​# the Free Software Foundation, either version 3 of the License, or
8​# (at your option) any later version.
9​#
10​# This program is distributed in the hope that it will be useful,
11​# but WITHOUT ANY WARRANTY; without even the implied warranty of
12​# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13​# GNU General Public License for more details.
14​#
15​# You should have received a copy of the GNU General Public License
16​# along with this program. If not, see <http://www.gnu.org/licenses/>.
17​#
18​# Author: Adam Williamson <awilliam@redhat.com>
19​
20​# Convenience tool for fiddling with the config files in Docker-ized
21​# openQA data containers. client.conf stores the API key/secret pairs,
22​# workers.ini specifies which server workers using that data container
23​# will try to connect to.
24​
25​import argparse
26​import sys
27​# Avoiding six for now, but let's be py3-compatible
28​try:
29​ import ConfigParser as configparser
30​except ImportError:
31​ import configparser
32​
33​# HELPER FUNCTIONS
34​
35​def _read_configs():
36​ """Read the config files."""
37​ cliconf = configparser.ConfigParser()
38​ cliconf.read('/data/conf/client.conf')
39​ workconf = configparser.ConfigParser()
40​ # This makes the parser case-sensitive - the HOST value is
41​ # upper-case
42​ workconf.optionxform = str
43​ workconf.read('/data/conf/workers.ini')
44​ return (cliconf, workconf)
45​
46​def _workers(workconf, hostname, secure):
47​ """Change the workers.ini HOST setting. All the subcommands
48​ may need to use this, so sharing it.
49​ """
50​ if secure:
51​ url = "https://{0}".format(hostname)
52​ else:
53​ url = "http://{0}".format(hostname)
54​ try:
55​ workconf.add_section('global')
56​ except configparser.DuplicateSectionError:
57​ pass
58​ workconf.set('global', 'HOST', url)
59​ with open('/data/conf/workers.ini', 'w') as fout:
60​ workconf.write(fout)
61​
62​# SUB-COMMAND FUNCTIONS
63​
64​def set_key(args, cliconf, workconf):
65​ """Add or update a client.conf entry and change workers.ini to
66​ point to it unless args.nodefault is set.
67​ """
68​ sections = [args.hostname]
69​ if args.localhost:
70​ sections.append('localhost')
71​ for section in sections:
jskladanUnsubmitted
Not Done

what happens if args.hostname is set to 'localhost' already? I guess that it just goes through the cycle twice, setting the same thing two times, but just making sure ;)

adamwillAuthorUnsubmitted
Not Done

yeah, exactly, it'd just do it twice, which causes no harm. It seemed not worth trying to handle this more cleverly.

72​ try:
jskladanUnsubmitted
Not Done

pls rename sec[s] to section[s]

73​ cliconf.add_section(section)
74​ except configparser.DuplicateSectionError:
75​ pass
76​ cliconf.set(section, 'key', args.key)
77​ cliconf.set(section, 'secret', args.secret)
78​ with open('/data/conf/client.conf', 'w') as fout:
79​ cliconf.write(fout)
80​ if not args.nodefault:
81​ _workers(workconf, args.hostname, args.secure)
82​
83​def remove(args, cliconf, workconf):
84​ """Remove an entry from client.conf and update workers.ini to
85​ point to another one, if it was pointing to this one and there
86​ are any others left to use.
87​ """
88​ if not cliconf.remove_section(args.hostname):
89​ sys.exit("Specified server not found in config!")
90​ with open('/data/conf/client.conf', 'w') as fout:
91​ cliconf.write(fout)
92​ if len(cliconf.sections()) < 1:
93​ print("WARNING: no servers remain in config!")
94​ sys.exit()
95​ try:
96​ curr = workconf.get('global', 'HOST').split('//')[1]
97​ except (configparser.NoOptionError, configparser.NoSectionError):
98​ sys.exit()
99​ # If we just removed the server that workers are supposed to
100​ # connect to, pick another.
101​ if curr == args.hostname:
102​ host = cliconf.sections()[0]
103​ if len(cliconf.sections()) > 1:
104​ print("WARNING: more than one server remains in config! "
105​ "Guessing which to use for workers!")
106​ # We'll prefer 'localhost' then 'openqa_webui' then just
107​ # pick the first one.
108​ if 'localhost' in cliconf.sections():
109​ host = 'localhost'
110​ elif 'openqa_webui' in cliconf.sections():
111​ host = 'openqa_webui'
112​ _workers(workconf, host, args.secure)
113​
114​def workers(args, cliconf, workconf):
115​ """Update workers.ini to point to the specified host."""
116​ if args.hostname not in cliconf.sections():
117​ print("WARNING: specified hostname not in client.conf!")
118​ _workers(workconf, args.hostname, args.secure)
119​
120​# ARG PARSING AND MAIN LOOP
121​
122​def parse_args():
123​ """Argument parsing."""
124​ parser = argparse.ArgumentParser(description=(
125​ "Set openQA client configuration in a data container."))
126​ subparsers = parser.add_subparsers()
127​
128​ parser_set = subparsers.add_parser(
129​ 'set', description="Set a server key/secret. Server will be added "
130​ "if it does not exist, modified if it does.")
131​ parser_set.add_argument(
132​ '-t', '--hostname', help="The hostname", default="openqa_webui")
133​ parser_set.add_argument(
134​ '-l', '--localhost', help="Set same key/secret for 'localhost'",
135​ action='store_true')
136​ parser_set.add_argument(
137​ '-n', '--nodefault', help="Do not make this entry the default for "
138​ "workers", action='store_true')
139​ parser_set.add_argument(
140​ 'key', help="The key", metavar="KEY")
141​ parser_set.add_argument(
142​ 'secret', help="The secret", metavar="SECRET")
143​ parser_set.set_defaults(func=set_key)
144​
145​ parser_remove = subparsers.add_parser(
146​ 'remove', description="Remove a server.")
147​ parser_remove.add_argument(
148​ 'hostname', help="The hostname")
149​ parser_remove.set_defaults(func=remove)
150​
151​ parser_workers = subparsers.add_parser(
152​ 'workers', description="Set the server workers will connect to.")
153​ parser_workers.add_argument(
154​ 'hostname', help="The hostname")
155​ parser_workers.set_defaults(func=workers)
156​
157​ parser.add_argument(
158​ '-s', '--secure', help="Use HTTPS for worker connections",
159​ action='store_true')
jskladanUnsubmitted
Not Done

every time I see this, I just crave for more people using https://github.com/docopt/docopt *wink wink*

adamwillAuthorUnsubmitted
Not Done

eh, everyone seems to have a favourite. New Bodhi CLI is using some thing called 'click'. I'm just sticking with argparse as it's in core and widely used and I know how to use it now...FWIW, I don't like that docopt approach at all, I think I'd find it much harder than just using argparse...for the record, this took me like an hour to write, it's not much of a sledgehammer. :P

jskladanUnsubmitted
Not Done

It took me 15 minutes to read (just the parse_args()), which is why I prefer docopt. But this is more of a "lets yell at each other over beers" stuff than a technical quality of the code thing :)

160​
161​ return parser.parse_args()
162​
163​def main():
164​ """Main loop."""
165​ try:
166​ args = parse_args()
167​ (cliconf, workconf) = _read_configs()
168​ args.func(args, cliconf, workconf)
169​ except KeyboardInterrupt:
170​ sys.stderr.write("Interrupted, exiting...\n")
171​ sys.exit(1)
172​
173​if __name__ == '__main__':
174​ main()
View Options

docker/openqa_data/scripts/set_keys

  • This file was deleted.
This file was completely deleted. Show File Contents