Add fedmsg listener to openqa tools
AbandonedPublic
Actions

Authored by adamwill on Feb 23 2016, 10:05 AM.

Details

Reviewers

garretraziel
jdulaney

Summary

As pungi4 is about to be rolled live, we need a quick fedmsg listener/consumer to kick off fedmsg jobs. This is as per irc conversation with adamw.

Test Plan

use fedmsg-dg-replay to replay the following fedmsg:

https://apps.fedoraproject.org/datagrepper/id?id=2016-85af2c7b-be8d-4225-a238-3ca3bd2e2d5c&is_raw=true&size=extra-large

Note that this should /not/ trigger any jobs. Finding a finished pungi compose fedmsg seems to be a bit problematic; it appears many composes "failed in a horrible fire" or are "FINISHED_INCOMPLETE"

Diff Detail

Repository

rOPENQA fedora_openqa

Lint

Lint Skipped

Unit

Unit Tests Skipped

jdulaney retitled this revision from to Add fedmsg listener to openqa tools.Feb 23 2016, 10:05 AM

jdulaney updated this object.

jdulaney edited the test plan for this revision. (Show Details)

jdulaney added a reviewer: adamwill.

jdulaney set the repository for this revision to rOPENQA fedora_openqa.

jdulaney added a project: fedora_openqa.

jdulaney added subscribers: jdulaney, kparal.

Herald added a subscriber: tflink. · View Herald TranscriptFeb 23 2016, 10:05 AM

Sidenote: all finished Pungi 4 composes have "FINISHED_INCOMPLETE" in "status" (see here, all are either DOOMED or FINISHED_INCOMPLETE). Don't know why. Maybe it indicates that it was finished, but not everything was built (e. g. ARM disk images that aren't there)?

According to this, we should listen also for "FINISHED_INCOMPLETE" status change.

This revision now requires changes to proceed.Feb 23 2016, 12:09 PM

kparal removed a subscriber: kparal.Feb 23 2016, 3:20 PM

Yes, I'd say just do if 'FINISHED' in status: for now. Other than that this looks fine for now. Wiki reporting is probably going to need to be rethought, but let's do one step at a time.

Actually, thinking about it a bit harder, I think maybe drop the wiki report attempt for now. It will block the listener until all jobs are complete, which takes some time. I'd rather not do that, so that when we're getting multiple composes in a fairly short space of time (e.g. Branched and Rawhide) we aren't waiting for the jobs from one to complete before running the other.

I'll be thinking through how to make wiki reporting happen properly when we get to that.

adamwill requested changes to this revision.Feb 23 2016, 3:28 PM

I think it may not be a bad idea to log the fact that jobs are run in some form, so I switched to doing that instead of reporting to the wiki.

I merged this manually to the pungi4 branch with a couple of trivial changes, since we don't want this stuff on develop yet, we want to review all the pungi4 together as one big diff when it's ready. Thanks John!

adamwill commandeered this revision.Feb 23 2016, 8:47 PM

adamwill abandoned this revision.

adamwill edited reviewers, added: jdulaney; removed: adamwill.

Revision Contents

			Path	Packages
A	M		scheduler/fedora-openqa-fedmsg.py (69 lines)

Diff 1934

View Options

scheduler/fedora-openqa-fedmsg.py

This file was added.

Property	Old Value	New Value
File Mode	null	100755

		1		#! /usr/bin/python
		2		# Copyright (C) 2016 John Dulaney
		3		#
		4		# This file is part of fedora-openqa-schedule.
		5		#
		6		# fedora-openqa-schedule is free software; you can redistribute it
		7		# and/or modify it under the terms of the GNU General Public License
		8		# as published by the Free Software Foundation, either version 3 of
		9		# the License, or (at your option) any later version.
		10		#
		11		# This program is distributed in the hope that it will be useful,
		12		# but WITHOUT ANY WARRANTY; without even the implied warranty of
		13		# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
		14		# GNU General Public License for more details.
		15		#
		16		# You should have received a copy of the GNU General Public License
		17		# along with this program. If not, see <http://www.gnu.org/licenses/>.
		18		#
		19		# Author(s): John Dulaney <jdulaney@fedoraproject.org>
		20
		21
		22		"""fedmsg consumer for fedora-openqa"""
		23
		24
		25		import logging
		26		import fedmsg
		27
		28		import fedora_openqa_schedule.schedule as schedule
		29		import fedora_openqa_schedule.report as report
		30
		31
		32		logger = logging.getLogger(__name__)
		33
		34
		35		def consume(msg):
		36		"""Consume incoming message"""
		37		wiki_url = "fedoraproject.org"
		38		status = msg['status']
		39		location = msg['location']
		40		if 'FINISHED' in status:
		41		# We have a complete pungi4 compose!
		42		try:
		43		(compose, jobs) = schedule.jobs_from_compose(location)
		44		except schedule.TriggerException as err:
		45		logger.warning("No jobs run! %s", err)
		46		return 1
		47		if jobs:
		48		logger.info("Jobs run on %s", compose)
		49		else:
		50		logger.warning("No jobs run!")
		51		return 1
		52
		53		logger.debug("finished")
		54		return 0
		55
		56		return 1
		57
		58
		59		def main():
		60		"""Main listener loop"""
		61
		62		for name, endpoint, topic, msg in fedmsg.tail_messages():
		63		if topic == 'org.fedoraproject.prod.pungi.compose.status.change':
		64		consume(msg)
		65
		66
		67		if __name__ == '__main__':
		68		main()
		69

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	1932		Feb 23 2016, 9:58 AM	★	★
Diff 2	1934		Feb 23 2016, 6:48 PM	★	★