Differential D765

redo fedmsg consumer as a proper hub consumer
ClosedPublic
Actions

Authored by adamwill on Mar 4 2016, 11:47 PM.

Details

Reviewers
garretraziel
jskladan
Commits
rOPENQA4619d60cd417: redo fedmsg consumer as a proper hub consumer
Summary

We ran into issues with the check-compose consumer and the
openQA consumer colliding; it seems running two of the 'naive'
consumers (tail_messages()-based) on the same system is not
a good idea. This is the 'correct' way to build a production
consumer, according to fedmsg folks, and has various benefits
for Fedora infra deployments.

I've already re-done the relval and check-compose consumers in
this style, they seem to be working. Documentation for this
is at http://www.fedmsg.com/en/latest/consuming/ .

Test Plan

Deploy the consumer as documented in README.md,
see if it works (you'll need to wait for a real compose, or
twiddle with the code a bit to listen for dev messages for
testing...I have a 'TestConsumer' model in the relval consumer
which might be worth adopting here.)

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to redo fedmsg consumer as a proper hub consumer.Mar 4 2016, 11:47 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jskladan, garretraziel.
adamwill updated this revision to Diff 1960.Mar 5 2016, 1:55 AM

fedmsg hub consumers run as 'fedmsg', so update README.md
to explain how to give fedmsg permissions to read wikitcms
and openQA credentials.

Note this requires wikitcms 2.0.2 (I only just added the
system-wide credentials file, to solve this problem).

adamwill updated this revision to Diff 1963.Mar 6 2016, 12:40 AM

fix the logging

garretraziel accepted this revision.Mar 7 2016, 4:50 PM

Looks OK.

This revision is now accepted and ready to land.Mar 7 2016, 4:50 PM
Closed by commit rOPENQA4619d60cd417: redo fedmsg consumer as a proper hub consumer (authored by adamwill). · Explain WhyMar 7 2016, 4:59 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 1966

View Options

README.md

Show All 37 Lines
383. Set up a python-wikitcms credentials file. This file contains a FAS 38​3. Set up a python-wikitcms credentials file. This file contains a FAS
39username and password which will be used when reporting results to the 39​username and password which will be used when reporting results to the
40wiki. See the `python-wikitcms` documentation for more details on this. 40​wiki. See the `python-wikitcms` documentation for more details on this.
41Ideally this should be a dedicated account for the purpose of reporting 41​Ideally this should be a dedicated account for the purpose of reporting
42test results. This step can be skipped if you do not want to report 42​test results. This step can be skipped if you do not want to report
43results to the wiki. 43​results to the wiki.
44 44​
45 sudo su 45​ sudo su
46 mkdir ~/.fedora 46​ mkdir /etc/fedora
47 echo "fas_username fas_password" > ~/.fedora/credentials 47​ echo "fas_username fas_password" > /etc/fedora/credentials
48 chmod 0600 ~/.fedora/credentials 48​ chmod 0600 /etc/fedora/credentials
49 49​
505. [docker] If you run openQA in Docker, you have to set up credentials for 50​5. [docker] If you run openQA in Docker, you have to set up credentials for
51openQA API access on the host system. Create file `/etc/openqa/client.conf` 51​openQA API access on the host system. Create file `/etc/openqa/client.conf`
52with following content: 52​with following content:
53 53​
54 [localhost:8080] 54​ [localhost:8080]
55 key = KEY 55​ key = KEY
56 secret = SECRET 56​ secret = SECRET
Show All 9 Lines
66 66​
676. Get openqa-fedora-schedule (and associated tools) if you do not already 67​6. Get openqa-fedora-schedule (and associated tools) if you do not already
68have them, and install: 68​have them, and install:
69 69​
70 git clone https://bitbucket.org/rajcze/openqa_fedora_tools 70​ git clone https://bitbucket.org/rajcze/openqa_fedora_tools
71 cd openqa_fedora_tools/scheduler; sudo python setup.py install 71​ cd openqa_fedora_tools/scheduler; sudo python setup.py install
72 cd ../.. 72​ cd ../..
73 73​
747. Set up systemd services: 74​7. Enable fedmsg consumer:
75 75​
76 To run openQA jobs whenever a compose completes, you can do this: 76​ To run openQA jobs whenever a compose completes, you must install and
77​ enable fedmsg-hub. On Fedora:
77 78​
78 sudo systemctl daemon-reload 79​ sudo dnf install fedmsg-hub
79 sudo systemctl enable openqa-consumer.service 80​ sudo systemctl enable fedmsg-hub.service
80 sudo systemctl start openqa-consumer.service 81​
81 82​ Now enable the consumer. Create a file `/etc/fedmsg.d/openqa_consumer.py`
82 openqa-consumer is a fedmsg consumer which listens for 'compose complete' 83​ with the contents:
83 messages from release engineering and schedules openQA jobs for them. 84​
84 85​ config = {
85 If you wish to run `openqa-fedora-schedule` jobs as a different user, copy 86​ 'fedora_openqa_schedule.consumer.enabled': True,
86 `/usr/lib/systemd/system/openqa-consumer.service` to `/etc/systemd/system` 87​ }
87 and set `User=` correctly. Ensure the chosen user has a 88​
88 `~/.fedora/credentials` file and either has a 89​ Now start fedmsg-hub (or restart it if it's already running):
89 `~/.config/openqa/client.conf` file or can read `/etc/openqa/client.conf` 90​
90 (and, if you are using it, `schedule.conf`). 91​ sudo systemctl start fedmsg-hub.service
92​
93​ In the Docker configuration you should probably set up the fedmsg
94​ consumer on the host system, ensuring you have done step 5 above.
95​ The fedmsg consumer will run as user 'fedmsg', so you need to give
96​ this user access to the wikitcms and openQA credentials:
97​
98​ sudo chown root.fedmsg /etc/fedora/credentials
99​ sudo chown root.fedmsg /etc/openqa/client.conf
100​ sudo chmod 0640 /etc/fedora/credentials
101​ sudo chmod 0640 /etc/openqa/client.conf
91 102​
92You can run `fedora-openqa-schedule` manually to run tests on a specified 103​You can run `fedora-openqa-schedule` manually to run tests on a specified
93Fedora compose/release location. See the command's help for further details. 104​Fedora compose/release location. See the command's help for further details.
View Options

scheduler/fedora-openqa-consumer.py

  • This file was deleted.
PropertyOld ValueNew Value
File Mode100755null
This file was completely deleted. Show File Contents
View Options

scheduler/fedora_openqa_schedule/consumer.py

1#!/usr/bin/env python 1​# Copyright (C) 2016 John Dulaney, Adam Williamson
2 ​
3# Copyright (C) 2016 John Dulaney ​
4# 2​#
5# This file is part of fedora-openqa-schedule. 3​# This file is part of fedora-openqa-schedule.
6# 4​#
7# fedora-openqa-schedule is free software; you can redistribute it 5​# fedora-openqa-schedule is free software; you can redistribute it
8# and/or modify it under the terms of the GNU General Public License 6​# and/or modify it under the terms of the GNU General Public License
9# as published by the Free Software Foundation, either version 3 of 7​# as published by the Free Software Foundation, either version 3 of
10# the License, or (at your option) any later version. 8​# the License, or (at your option) any later version.
11# 9​#
12# This program is distributed in the hope that it will be useful, 10​# This program is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of 11​# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12​# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU General Public License for more details. 13​# GNU General Public License for more details.
16# 14​#
17# You should have received a copy of the GNU General Public License 15​# You should have received a copy of the GNU General Public License
18# along with this program. If not, see <http://www.gnu.org/licenses/>. 16​# along with this program. If not, see <http://www.gnu.org/licenses/>.
19# 17​#
20# Author(s): John Dulaney <jdulaney@fedoraproject.org> 18​# Author(s): John Dulaney <jdulaney@fedoraproject.org>
21# Adam Williamson <awilliam@redhat.com> 19​# Adam Williamson <awilliam@redhat.com>
22 20​
21​"""fedmsg consumer to schedule openQA jobs."""
23 22​
24"""fedmsg consumer to schedule openQA tests.""" ​
25 ​
26 ​
27import logging ​
28import re 23​import re
29import sys 24​import sys
30 25​
31import fedmsg 26​import fedmsg
32import fedora_openqa_schedule.schedule as schedule 27​import fedora_openqa_schedule.schedule as schedule
33 28​
34 29​
35logger = logging.getLogger(__name__) 30​class OpenQAConsumer(fedmsg.consumers.FedmsgConsumer):
36 31​ """A fedmsg consumer that schedules openQA jobs when a new compose
37 32​ appears.
38def consume(msg): 33​ """
39 """Consume incoming message.""" 34​ topic = ["org.fedoraproject.prod.pungi.compose.status.change",
40 # two-week atomic messages don't indicate status, so we'll just 35​ "org.fedoraproject.prod.compose.*"]
41 # always run for those, by setting default value 'FINISHED' 36​ config_key = "fedora_openqa_schedule.consumer.enabled"
42 status = msg.get('status', 'FINISHED') 37​
43 location = msg.get('location') 38​ def _log(self, level, message):
44 if 'FINISHED' in status and location: 39​ """Convenience function for sticking the class name on the
45 # We have a complete pungi4 compose or a 2-week atomic compose 40​ front of the log message as an identifier.
46 try: 41​ """
47 (compose, jobs) = schedule.jobs_from_compose(location) 42​ logfnc = getattr(self.log, level)
48 except schedule.TriggerException as err: 43​ logfnc("%s: %s", self.__class__.__name__, message)
49 logger.warning("No jobs run! %s", err) 44​
50 return 1 45​ def consume(message):
51 if jobs: 46​ """Consume incoming message."""
52 logger.info("Jobs run on %s: %s", compose, 47​ # as two-week atomic message topics are a bit awkward, we have
53 ' '.join(str(job) for job in jobs)) 48​ # to do a bit more filtering here
54 else: 49​ if not message['topic'] == "org.fedoraproject.prod.pungi.compose.status.change":
55 logger.warning("No jobs run!") 50​ if not message['topic'].endswith(".cloudimg-staging.done"):
56 return 1 51​ return
57 52​
58 logger.debug("finished") 53​ # two-week atomic messages don't indicate status, so we'll just
59 return 0 54​ # always run for those, by setting default value 'FINISHED'
60 55​ status = message['body']['msg'].get('status', 'FINISHED')
61 return 1 56​ location = message['body']['msg'].get('location')
62 57​ compstr = message['body']['msg'].get('compose_id', location)
63 58​
64def main(): 59​ if 'FINISHED' in status and location:
65 """Main listener loop.""" 60​ # We have a complete pungi4 compose or a 2-week atomic compose
66 logging.basicConfig(level=logging.INFO) 61​ self._log('info', "Scheduling openQA jobs for {0}".format(compstr))
67 # shut the hell up about new goddamn connections, requests 62​ try:
68 logging.getLogger('requests.packages.urllib3.connectionpool').setLevel(logging.WARNING) 63​ (compose, jobs) = schedule.jobs_from_compose(location)
69 try: 64​ except schedule.TriggerException as err:
70 # catch Pungi 4 'compose status change' messages and old-style 65​ self._log('warning', "No openQA jobs run! {0}".format(err))
71 # two-week Atomic compose 'staging.done' messages, which have 66​ return
72 # the release number in them, so we need a regex 67​ if jobs:
73 topicpatt = re.compile( 68​ self._log('warning', "openQA jobs run on %{0}: "
74 r'^org\.fedoraproject\.prod\.(pungi\.compose\.status\.change|' 69​ "{1}".format(compose, ' '.join(str(job) for job in jobs)))
75 'compose\.\d+\.cloudimg-staging\.done)$') 70​ else:
76 for (name, endpoint, topic, msg) in fedmsg.tail_messages(): 71​ self._log('warning', "No openQA jobs run!")
77 if topicpatt.match(topic): 72​ return
78 consume(msg['msg']) 73​
79 except KeyboardInterrupt: 74​ self._log('debug', "Finished")
80 sys.stderr.write("Interrupted, exiting...\n") 75​ return
81 sys.exit(1) 76​
82 77​ return
83 ​
84if __name__ == '__main__': ​
85 main() ​
86 ​
View Options

scheduler/setup.py

1import glob ​
2import os 1​import os
3from setuptools import setup, find_packages 2​from setuptools import setup, find_packages
4 3​
5# Utility function to read the README file. 4​# Utility function to read the README file.
6# Used for the long_description. It's nice, because now 1) we have a top level 5​# Used for the long_description. It's nice, because now 1) we have a top level
7# README file and 2) it's easier to type in the README file than to put a raw 6​# README file and 2) it's easier to type in the README file than to put a raw
8# string in below. Stolen from 7​# string in below. Stolen from
9# https://pythonhosted.org/an_example_pypi_project/setuptools.html 8​# https://pythonhosted.org/an_example_pypi_project/setuptools.html
10def read(fname): 9​def read(fname):
11 return open(os.path.join(os.path.dirname(__file__), "..", fname)).read() 10​ return open(os.path.join(os.path.dirname(__file__), "..", fname)).read()
12 11​
13# Allow modification of systemd install location via env var. ​
14SYSTEMDUNITPATH = os.getenv("SYSTEMDUNITPATH", '/usr/lib/systemd/system') ​
15 ​
16setup( 12​setup(
17 name = "fedora-openqa", 13​ name = "fedora-openqa",
18 version = "2.0", 14​ version = "2.0",
19 entry_points = { 15​ entry_points = {
20 'console_scripts': [ 16​ 'console_scripts': [
21 'fedora-openqa-schedule = fedora_openqa_schedule.cli:main', 17​ 'fedora-openqa-schedule = fedora_openqa_schedule.cli:main',
22 'fedora-openqa-consumer = fedora_openqa_schedule.consumer:main', 18​ ],
19​ 'moksha.consumer': [
20​ 'fedora_openqa_schedule.consumer = fedora_openqa_schedule.consumer:OpenQAConsumer',
23 ], 21​ ],
24 }, 22​ },
25 author = "Fedora QA devel team", 23​ author = "Fedora QA devel team",
26 author_email = "qa-devel@lists.fedoraproject.org", 24​ author_email = "qa-devel@lists.fedoraproject.org",
27 description = "Fedora openQA scheduler", 25​ description = "Fedora openQA scheduler",
28 license = "GPLv3+", 26​ license = "GPLv3+",
29 keywords = "fedora openqa test qa", 27​ keywords = "fedora openqa test qa",
30 url = "https://bitbucket.org/rajcze/openqa_fedora_tools", 28​ url = "https://bitbucket.org/rajcze/openqa_fedora_tools",
31 packages = ["fedora_openqa_schedule"], 29​ packages = ["fedora_openqa_schedule"],
32 install_requires = ['fedfind>=1.5', 'fedmsg', 'openqa-client', 'setuptools', 'six'], 30​ install_requires = ['fedfind>=1.5', 'fedmsg', 'openqa-client', 'setuptools', 'six'],
33 long_description=read('README.md'), 31​ long_description=read('README.md'),
34 classifiers=[ 32​ classifiers=[
35 "Development Status :: 3 - Alpha", 33​ "Development Status :: 3 - Alpha",
36 "Topic :: Utilities", 34​ "Topic :: Utilities",
37 "License :: OSI Approved :: GNU General Public License v3 or later " 35​ "License :: OSI Approved :: GNU General Public License v3 or later "
38 "(GPLv3+)", 36​ "(GPLv3+)",
39 ], 37​ ],
40 data_files=[ ​
41 (SYSTEMDUNITPATH, glob.glob('systemd/*.service')), ​
42 ], ​
43) 38​)
View Options

scheduler/systemd/openqa-consumer.service

  • This file was deleted.
This file was completely deleted. Show File Contents