Differential D805

disable network checks for binary RPMs
ClosedPublic
Actions

Authored by kparal on Apr 7 2016, 12:31 PM.

Details

Reviewers
tflink
mkrizek
Group Reviewers
libtaskotron
Maniphest Tasks
T760: rpmlint is hammering KDE infra (and probably other projects as well)
Commits
rRPMLINT2c68c372b85e: disable network checks for binary RPMs
Summary

Due to excessive network checking (for every RPM file) we now split the
execution to checking SRPMs and checking all other RPMs. We enable
network checks only for SRPMs, but not for other RPMs.

The artifact output was slightly adjusted to make it clear that there
are two separate rpmlint executions. As a bonus, the detail note is now
much more useful (shows errors and warnings count).

The artifact output now looks like this (compare to https://taskotron.fedoraproject.org/artifacts/all/b25fbde0-eef1-11e5-b4e1-525400120b80/task_output/kf5-kdbusaddons-5.20.0-1.fc24.log ):

##### SRPMs #####
kf5-kdbusaddons.src: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
1 packages and 0 specfiles checked; 0 errors, 1 warnings.

##### RPMs  #####
kf5-kdbusaddons.armv7hl: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
kf5-kdbusaddons.armv7hl: W: shared-lib-calls-exit /usr/lib/libKF5DBusAddons.so.5.20.0 exit@GLIBC_2.4
kf5-kdbusaddons.armv7hl: W: no-manual-page-for-binary kquitapp5
kf5-kdbusaddons.i686: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
kf5-kdbusaddons.i686: W: shared-lib-calls-exit /usr/lib/libKF5DBusAddons.so.5.20.0 exit@GLIBC_2.0
kf5-kdbusaddons.i686: W: no-manual-page-for-binary kquitapp5
kf5-kdbusaddons.x86_64: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
kf5-kdbusaddons.x86_64: W: shared-lib-calls-exit /usr/lib64/libKF5DBusAddons.so.5.20.0 exit@GLIBC_2.2.5
kf5-kdbusaddons.x86_64: W: no-manual-page-for-binary kquitapp5
kf5-kdbusaddons-devel.armv7hl: W: no-documentation
kf5-kdbusaddons-devel.i686: W: no-documentation
kf5-kdbusaddons-devel.x86_64: W: only-non-binary-in-usr-lib
kf5-kdbusaddons-devel.x86_64: W: no-documentation
6 packages and 0 specfiles checked; 0 errors, 13 warnings.

rpmlint PASSED for kf5-kdbusaddons-5.20.0-1.fc24 (0 errors, 14 warnings)

It's not visible, but the network checks are executed just for SRPMs. You can see it when I manually enable verbose output for rpmlint:

##### SRPMs #####
kf5-kdbusaddons.src: I: checking
kf5-kdbusaddons.src: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
kf5-kdbusaddons.src: I: checking-url http://projects.kde.org/kdbusaddons (timeout 10 seconds)
kf5-kdbusaddons.src: I: checking-url http://download.kde.org/stable/frameworks/5.20/kdbusaddons-5.20.0.tar.xz (timeout 10 seconds)
1 packages and 0 specfiles checked; 0 errors, 1 warnings.

##### RPMs  #####
kf5-kdbusaddons.armv7hl: I: checking
kf5-kdbusaddons.armv7hl: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
kf5-kdbusaddons.armv7hl: I: network-checks-disabled http://projects.kde.org/kdbusaddons
kf5-kdbusaddons.armv7hl: W: shared-lib-calls-exit /usr/lib/libKF5DBusAddons.so.5.20.0 exit@GLIBC_2.4
kf5-kdbusaddons.armv7hl: W: no-manual-page-for-binary kquitapp5
kf5-kdbusaddons.i686: I: checking
kf5-kdbusaddons.i686: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
kf5-kdbusaddons.i686: I: network-checks-disabled http://projects.kde.org/kdbusaddons
kf5-kdbusaddons.i686: W: shared-lib-calls-exit /usr/lib/libKF5DBusAddons.so.5.20.0 exit@GLIBC_2.0
kf5-kdbusaddons.i686: W: no-manual-page-for-binary kquitapp5
kf5-kdbusaddons.x86_64: I: checking
kf5-kdbusaddons.x86_64: W: spelling-error Summary(en_US) addon -> ad don, ad-don, add on
kf5-kdbusaddons.x86_64: I: network-checks-disabled http://projects.kde.org/kdbusaddons
kf5-kdbusaddons.x86_64: W: shared-lib-calls-exit /usr/lib64/libKF5DBusAddons.so.5.20.0 exit@GLIBC_2.2.5
kf5-kdbusaddons.x86_64: W: no-manual-page-for-binary kquitapp5
kf5-kdbusaddons-devel.armv7hl: I: checking
kf5-kdbusaddons-devel.armv7hl: I: network-checks-disabled http://projects.kde.org/kdbusaddons
kf5-kdbusaddons-devel.armv7hl: W: no-documentation
kf5-kdbusaddons-devel.i686: I: checking
kf5-kdbusaddons-devel.i686: I: network-checks-disabled http://projects.kde.org/kdbusaddons
kf5-kdbusaddons-devel.i686: W: no-documentation
kf5-kdbusaddons-devel.x86_64: I: checking
kf5-kdbusaddons-devel.x86_64: I: network-checks-disabled http://projects.kde.org/kdbusaddons
kf5-kdbusaddons-devel.x86_64: W: only-non-binary-in-usr-lib
kf5-kdbusaddons-devel.x86_64: W: no-documentation
6 packages and 0 specfiles checked; 0 errors, 13 warnings.

rpmlint PASSED for kf5-kdbusaddons-5.20.0-1.fc24 (0 errors, 14 warnings)
Test Plan

Manual execution works.

Diff Detail

Repository
rRPMLINT task-rpmlint
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to disable network checks for binary RPMs.Apr 7 2016, 12:31 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added a reviewer: libtaskotron.
tflink accepted this revision.Apr 7 2016, 6:10 PM
tflink added a reviewer: tflink.
This revision is now accepted and ready to land.Apr 7 2016, 6:10 PM
mkrizek accepted this revision.Apr 7 2016, 6:28 PM
mkrizek added a reviewer: mkrizek.
Closed by commit rRPMLINT2c68c372b85e: disable network checks for binary RPMs (authored by kparal). · Explain WhyApr 8 2016, 11:15 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mrun_rpmlint.py (111 lines)

Diff 2060

View Options

run_rpmlint.py

1import logging 1​import logging
2if __name__ == '__main__': 2​if __name__ == '__main__':
3 # Set up logging ASAP to see potential problems during import. 3​ # Set up logging ASAP to see potential problems during import.
4 # Don't set it up when not running as the main script, someone else handles 4​ # Don't set it up when not running as the main script, someone else handles
5 # that then. 5​ # that then.
6 logging.basicConfig() 6​ logging.basicConfig()
7 7​
8import subprocess 8​import subprocess
9import os 9​import os
10import sys 10​import re
11​from collections import namedtuple
11 12​
12from libtaskotron import check 13​from libtaskotron import check
13 14​
14log = logging.getLogger('rpmlint') 15​log = logging.getLogger('rpmlint')
15log.setLevel(logging.DEBUG) 16​log.setLevel(logging.DEBUG)
16log.addHandler(logging.NullHandler()) 17​log.addHandler(logging.NullHandler())
17 18​
19​# outcome = str keyword, output = list of str, errors = int, warnings = int
20​Result = namedtuple('Result', ['outcome', 'output', 'errors', 'warnings'])
21​
18 22​
19def run(koji_build, workdir='.', artifactsdir='artifacts'): 23​def run(koji_build, workdir='.', artifactsdir='artifacts'):
20 '''The main method to run from Taskotron''' 24​ '''The main method to run from Taskotron'''
21 25​
22 if not os.path.exists(artifactsdir): 26​ if not os.path.exists(artifactsdir):
23 os.makedirs(artifactsdir) 27​ os.makedirs(artifactsdir)
24 28​
25 command = ['rpmlint', '%s' % os.path.abspath(workdir)] 29​ workdir = os.path.abspath(workdir)
30​
31​ # find files to run on
32​ files = sorted(os.listdir(workdir))
33​ srpms = []
34​ rpms = []
35​ for file_ in files:
36​ filepath = os.path.join(workdir, file_)
37​ if file_.endswith('.src.rpm'):
38​ srpms.append(filepath)
39​ elif file_.endswith('.rpm'):
40​ rpms.append(filepath)
41​ else:
42​ log.debug('Ignoring non-rpm file: %s', filepath)
43​
44​ # run rpmlint on SRPMs
45​ if srpms:
46​ command = ['rpmlint'] + srpms
47​ srpm_result = run_rpmlint(command)
48​ else:
49​ log.warn('No .src.rpm files found in: %s', workdir)
50​ srpm_result = Result('PASSED', [], None, None)
51​
52​ # run rpmlint on RPMs
53​ # disable network checks to avoid hitting websites too much (also the network checks were
54​ # already) performed on the SRPM. See https://phab.qadevel.cloud.fedoraproject.org/T760
55​ if rpms:
56​ command = ['rpmlint', '-o', 'NetworkEnabled False'] + rpms
57​ rpm_result = run_rpmlint(command)
58​ else:
59​ log.warn('No binary rpm files found in: %s', workdir)
60​ srpm_result = Result('PASSED', [], None, None)
61​
62​ # compute overall results
63​ detail = check.CheckDetail(koji_build, check.ReportType.KOJI_BUILD, srpm_result.outcome)
64​ detail.update_outcome(rpm_result.outcome)
65​ errors = srpm_result.errors + rpm_result.errors
66​ warnings = srpm_result.warnings + rpm_result.warnings
67​ detail.note = '%d errors, %d warnings' % (errors, warnings)
68​
69​ # print some summary to console
70​ summary = 'rpmlint %s for %s (%s)' % (detail.outcome, koji_build, detail.note)
71​ log.info(summary)
72​
73​ # store logs
74​ log_path = os.path.join(artifactsdir, '%s.log' % koji_build)
75​ log.debug('Saving log to: %s', log_path)
76​ with open(log_path, 'w') as log_file:
77​ log_file.write('##### SRPMs #####\n')
78​ log_file.writelines(srpm_result.output)
79​ log_file.write('\n')
80​ log_file.write('##### RPMs #####\n')
81​ log_file.writelines(rpm_result.output)
82​ log_file.write('\n')
83​ log_file.write(summary)
84​ detail.artifact = log_path
26 85​
27 # run rpmlint 86​ output = check.export_YAML(detail)
87​ return output
88​
89​
90​def run_rpmlint(command):
91​ '''Run rpmlint command.
92​ :param list command: command to run through subprocess.Popen
93​ :rtype: Results
94​ '''
28 log.debug('Running command: %s', command) 95​ log.debug('Running command: %s', command)
29 proc = subprocess.Popen(command, stdout=subprocess.PIPE, 96​ proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1,
30 stderr=subprocess.STDOUT, bufsize=1, ​
31 close_fds=True) 97​ close_fds=True)
32 output = [] 98​ output = []
33 for line in iter(proc.stdout.readline, ''): 99​ for line in iter(proc.stdout.readline, ''):
34 print line, 100​ print line,
35 output.append(line) 101​ output.append(line)
36 stdout, stderr = proc.communicate() 102​ stdout, stderr = proc.communicate()
37 103​ assert not stdout and not stderr, "Extra stdout or stderr found, this shouldn't happen"
38 # stdout and stderr should be empty, because we printed them out ​
39 # before calling .communicate() ​
40 if stdout or stderr: ​
41 log.warn("Extra stdout or stderr found, this shouldn't happen:") ​
42 print 'STDOUT:\n%s' % stdout ​
43 print >> sys.stderr, 'STDERR:\n%s' % stderr ​
44 output.append(stdout) ​
45 output.append(stderr) ​
46 104​
47 if proc.returncode == 0: 105​ if proc.returncode == 0:
48 result = 'PASSED' 106​ outcome = 'PASSED'
49 else: 107​ else:
50 if proc.returncode == 64 or proc.returncode == 66: 108​ if proc.returncode == 64 or proc.returncode == 66:
51 result = 'FAILED' 109​ outcome = 'FAILED'
52 else: 110​ else:
53 result = 'ABORTED' 111​ outcome = 'ABORTED'
54 112​
55 summary = 'rpmlint %s for %s' % (result, koji_build) 113​ match = re.search(r'(\d+) errors, (\d+) warnings', output[-1] if output else '')
56 log.info(summary) 114​ errors = int(match.group(1))
115​ warnings = int(match.group(2))
57 116​
58 detail = check.CheckDetail(koji_build, check.ReportType.KOJI_BUILD, 117​ result = Result(outcome=outcome, output=output, errors=errors, warnings=warnings)
59 result, summary) 118​ return result
60 ​
61 # store logs ​
62 log_path = os.path.join(artifactsdir, '%s.log' % koji_build) ​
63 log.debug('Saving log to: %s', log_path) ​
64 with open(log_path, 'w') as log_file: ​
65 log_file.writelines(output) ​
66 detail.artifact = log_path ​
67 ​
68 output = check.export_YAML(detail) ​
69 return output ​
70 119​
71 120​
72if __name__ == '__main__': 121​if __name__ == '__main__':
73 output = run('local-run') 122​ output = run('local-run')
74 print output 123​ print output