download build logs, move rpmgrill setup to prepare.sh
ClosedPublic
Actions

Authored by kparal on Jul 4 2016, 2:02 PM.

Details

Reviewers

lbrabec
• ralph2
mkrizek
jskladan

Group Reviewers

task-rpmgrill

Maniphest Tasks

T812: koji directive: allow to download build logs

Commits

rRPMGR17a9645256ce: download build logs, move rpmgrill execution to a shell script

Summary

Logs have to be manually put into correct directories after koji
directive, but before rpmgrill-unpack-rpms. Instead of running all of
that from python, it seemed easier to move that code into a shell
script.

Test Plan

tested manually

Diff Detail

Repository

rRPMGR task-rpmgrill

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

kparal retitled this revision from to download build logs, move rpmgrill setup to prepare.sh.Jul 4 2016, 2:02 PM

kparal updated this object.

kparal edited the test plan for this revision. (Show Details)

kparal added reviewers: task-rpmgrill, • ralph2, mkrizek, lbrabec, jskladan.

Herald added a subscriber: tflink. · View Herald TranscriptJul 4 2016, 2:02 PM

kparal added a task: T812: koji directive: allow to download build logs.Jul 4 2016, 2:05 PM

runtask.yml
23–27	Maybe rename this to something like "Process rpmgrill results" since this doesn't run rpmgrill anymore? Also rename `prepare.sh` to `run_rpmgrill.sh` and `run_rpmgrill.py` to `process_results.py` (or something better)?

rename files

What about now?

jskladan accepted this revision.Jul 7 2016, 1:33 PM

This revision is now accepted and ready to land.Jul 7 2016, 1:33 PM

kparal added a dependency: D916: koji_utils/directive: add option to download build.log.Jul 7 2016, 2:18 PM

fail on any error in shell script, don't fail for missing build logs

mkrizek accepted this revision.Jul 8 2016, 7:31 AM

Closed by commit rRPMGR17a9645256ce: download build logs, move rpmgrill execution to a shell script (authored by kparal). · Explain WhyJul 8 2016, 7:52 AM

This revision was automatically updated to reflect the committed changes.

Revision Contents

		Path
V		{process_results.py ← run_rpmgrill.py} (39 lines)
V		{run_rpmgrill.py → process_results.py}
A	M	run_rpmgrill.sh (26 lines)
M		runtask.yml (9 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	2338	3cf5fab		Jul 4 2016, 2:02 PM	★	★
Diff 2	2350	0708f16	- rename files	Jul 7 2016, 1:23 PM	★	★
Diff 3	2353	0708f16	- fail on any error in shell script, don't fail for missing build logs	Jul 7 2016, 4:02 PM	★	★
Diff 4	2359	0708f16	rRPMGR17a9645256cea1b0e07acdb6591c64859b8536e6	Jul 8 2016, 7:51 AM	★	★

Commit	Tree	Parents	Author	Summary	Date
122842f52e7b	c13c91154a34	ebaf077c18d0	Kamil Páral	fail on any error in shell script, don't fail for missing build logs	Jul 7 2016, 4:02 PM
ebaf077c18d0	3aeef43ed44c	93b633250695	Kamil Páral	rename files	Jul 7 2016, 1:22 PM
93b633250695	65a4f53f3465	0708f16cbaa6	Kamil Páral	download build logs, move rpmgrill setup to prepare.sh (Show More…)	Jun 30 2016, 1:59 PM

	Status	Author	Revision
	Closed	kparal	D920 download build logs, move rpmgrill setup to prepare.sh
	Closed	kparal	D916 koji_utils/directive: add option to download build.log

Diff 2359

View Options

process_results.py

This file was moved from run_rpmgrill.py.

1	import logging	1		import logging
2	if __name__ == '__main__':	2		if __name__ == '__main__':
3	# Set up logging ASAP to see potential problems during import.	3		# Set up logging ASAP to see potential problems during import.
4	# Don't set it up when not running as the main script, someone else handles	4		# Don't set it up when not running as the main script, someone else handles
5	# that then.	5		# that then.
6	logging.basicConfig()	6		logging.basicConfig()
7		7
8	import json	8		import json
9	import os	9		import os
10	import re	10		import re
11	import shutil	11		import shutil
12	import subprocess
13		12
14	from libtaskotron import check	13		from libtaskotron import check
15		14
16	log = logging.getLogger('rpmgrill')	15		log = logging.getLogger('rpmgrill')
17	log.setLevel(logging.DEBUG)	16		log.setLevel(logging.DEBUG)
18	log.addHandler(logging.NullHandler())	17		log.addHandler(logging.NullHandler())
19		18
20		19
21	def camel_to_dashes(name):	20		def camel_to_dashes(name):
22	s1 = re.sub('(.)([A-Z][a-z]+)', r'\1-\2', name)	21		s1 = re.sub('(.)([A-Z][a-z]+)', r'\1-\2', name)
23	return re.sub('([a-z0-9])([A-Z])', r'\1-\2', s1).lower()	22		return re.sub('([a-z0-9])([A-Z])', r'\1-\2', s1).lower()
24		23
25		24
26	def run(koji_build, workdir='.', artifactsdir='artifacts'):	25		def run(koji_build, workdir='.', artifactsdir='artifacts'):
27	'''The main method to run from Taskotron'''	26		'''The main method to run from Taskotron'''
28		27
29	if not os.path.exists(artifactsdir):	28		if not os.path.exists(artifactsdir):
30	os.makedirs(artifactsdir)	29		os.makedirs(artifactsdir)
31		30
32	workdir = os.path.abspath(workdir)	31		workdir = os.path.abspath(workdir)
33
34	# 1. run rpmgrill-unpack-rpms on the workdir
35	run_command(['rpmgrill-unpack-rpms', workdir])
36
37	# 2. run rpmgrill on that unpacked directory
38	unpackeddir = os.path.join(workdir, 'unpacked')	32		unpackeddir = os.path.join(workdir, 'unpacked')
39	run_command(['rpmgrill', unpackeddir])
40		33
41	# 3. read in the big results file produced by rpmgrill.	34		# rpms and logs have been downloaded by koji directive and rpmgrill
		35		# has been prepared and run by run_rpmgrill.sh
		36
		37		# 1. read in the big results file produced by rpmgrill.
42	rpmgrill_results = read_results(unpackeddir)	38		rpmgrill_results = read_results(unpackeddir)
43		39
44	# 4. Store log	40		# 2. Store log
45	log_path = store_logs(unpackeddir, artifactsdir)	41		log_path = store_logs(unpackeddir, artifactsdir)
46		42
47	# 5. Massage that into a format suitable for resultsdb/taskotron	43		# 3. Massage that into a format suitable for resultsdb/taskotron
48	details = list(massage_results(rpmgrill_results, koji_build, log_path))	44		details = list(massage_results(rpmgrill_results, koji_build, log_path))
49	output = check.export_YAML(details)	45		output = check.export_YAML(details)
50	return output	46		return output
51		47
52		48
53	def read_results(unpackeddir):	49		def read_results(unpackeddir):
54	rpmgrill_results_file = os.path.join(unpackeddir, 'rpmgrill.json')	50		rpmgrill_results_file = os.path.join(unpackeddir, 'rpmgrill.json')
55	with open(rpmgrill_results_file, 'r') as f:	51		with open(rpmgrill_results_file, 'r') as f:
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Line(s)		114	yield check.CheckDetail(
120	report_type=check.ReportType.KOJI_BUILD,	116		report_type=check.ReportType.KOJI_BUILD,
121	outcome=outcome,	117		outcome=outcome,
122	note=note,	118		note=note,
123	checkname='rpmgrill',	119		checkname='rpmgrill',
124	artifact=log_path,	120		artifact=log_path,
125	)	121		)
126		122
127		123
128	def run_command(command):
129	'''Run rpmgrill command.
130	:param list command: command to run through subprocess.Popen
131	:rtype: Results
132	'''
133	log.debug('Running command: %s', command)
134	proc = subprocess.Popen(
135	command,
136	stdout=subprocess.PIPE,
137	stderr=subprocess.STDOUT,
138	bufsize=1,
139	close_fds=True,
140	)
141	for line in iter(proc.stdout.readline, ''):
142	print line,
143	stdout, stderr = proc.communicate()
144	assert not stdout and not stderr, "Extra stdout or stderr found, this shouldn't happen"
145
146	if proc.returncode != 0:
147	# This will trigger the @failsafe decorator
148	raise IOError("Failed to run %r" % command)
149
150
151	if __name__ == '__main__':	124		if __name__ == '__main__':
152	output = run('local-run')	125		output = run('local-run')
153	print output	126		print output

View Options

run_rpmgrill.py

This file was moved to process_results.py.

View Options

run_rpmgrill.sh

This file was added.

Property	Old Value	New Value
File Mode	null	100755

		1		#!/bin/bash
		2		# Accepts one argument: workdir
		3		# Expects RPM files and optionally some log files in the workdir. Prepares
		4		# the directory for rpmgrill and then runs rpmgrill on it.
		5
		6		# fail on any error
		7		set -e
		8
		9		# go to workdir
		10		cd "$1"
		11
		12		# run rpmgrill-unpack-rpms on the workdir
		13		echo "Running rpmgrill-unpack-rpms..."
		14		rpmgrill-unpack-rpms .
		15
		16		# move log files (if they exist) where rpmgrill expects them
		17		echo "Moving log files..."
		18		if ls build.log*; then
		19		for bl in build.log*; do
		20		mv -v $bl unpacked/${bl#build.log.}/build.log
		21		done
		22		fi
		23
		24		# run rpmgrill on that unpacked directory
		25		echo "Running rpmgrill..."
		26		rpmgrill --verbose unpacked

View Options

runtask.yml

Show All 12 Lines
13		13
14	actions:	14		actions:
15	- name: download rpms from koji	15		- name: download rpms from koji
16	koji:	16		koji:
17	action: download	17		action: download
18	koji_build: ${koji_build}	18		koji_build: ${koji_build}
19	arch: ['all']	19		arch: ['all']
20	src: True	20		src: True
		21		build_log: True
21		22
22	- name: run rpmgrill on downloaded rpms	23		- name: prepare directories and run rpmgrill
		24		bash:
		25		command: bash run_rpmgrill.sh ${workdir}
		26
		27		- name: process rpmgrill results
			mkrizekUnsubmitted Not Done Maybe rename this to something like "Process rpmgrill results" since this doesn't run rpmgrill anymore? Also rename `prepare.sh` to `run_rpmgrill.sh` and `run_rpmgrill.py` to `process_results.py` (or something better)?
23	python:	28		python:
24	file: run_rpmgrill.py	29		file: process_results.py
25	callable: run	30		callable: run
26	workdir: ${workdir}	31		workdir: ${workdir}
27	koji_build: ${koji_build}	32		koji_build: ${koji_build}
28	artifactsdir: ${artifactsdir}	33		artifactsdir: ${artifactsdir}
29	export: rpmgrill_output	34		export: rpmgrill_output
30		35
31	- name: report results to resultsdb	36		- name: report results to resultsdb
32	resultsdb:	37		resultsdb:
33	results: ${rpmgrill_output}	38		results: ${rpmgrill_output}

download build logs, move rpmgrill setup to prepare.shClosedPublicActions

Details

Diff Detail

Revision Contents

Diff 2359

process_results.py

run_rpmgrill.py

run_rpmgrill.sh

runtask.yml

download build logs, move rpmgrill setup to prepare.sh
ClosedPublic
Actions