Differential D866

file_utils: don't crash in progressbar on decompressed content
ClosedPublic
Actions

Authored by kparal on May 25 2016, 11:10 AM.

Details

Reviewers
mkrizek
Group Reviewers
libtaskotron
Maniphest Tasks
T755: "ValueError: Value out of range" in python-progressbar for openh264 package
Commits
rLTRNdacd0a8f4b4c: file_utils: don't crash in progressbar on decompressed content
Summary

We were computing downloaded data incorrectly (saved size might not be
equal to read size due to content decompression) and were exceeding max
value in progress bar. This patch attempts to fix this case.

Test Plan

test suite works, tested manually with some regular NVRs and also

runtask -i openh264-1.5.3-0.1.git2706e36.fc24 -t koji_build ../task-rpmlint/runtask.yml -d

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to file_utils: don't crash in progressbar on decompressed content.May 25 2016, 11:10 AM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added a reviewer: libtaskotron.
mkrizek accepted this revision.May 25 2016, 11:53 AM
mkrizek added a reviewer: mkrizek.
This revision is now accepted and ready to land.May 25 2016, 11:53 AM
Closed by commit rLTRNdacd0a8f4b4c: file_utils: don't crash in progressbar on decompressed content (authored by kparal). · Explain WhyMay 25 2016, 12:21 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mlibtaskotron/file_utils.py (10 lines)

Diff 2224

View Options

libtaskotron/file_utils.py

Show First 20 Lines • Show All 165 Lines • ▼ Show 20 Line(s)164 for chunk in r.iter_content(chunk_size):
166 if chunk: 166​ if chunk:
167 f.write(chunk) 167​ f.write(chunk)
168 else: 168​ else:
169 widgets = [progressbar.Percentage(), progressbar.Bar(), progressbar.ETA(), 169​ widgets = [progressbar.Percentage(), progressbar.Bar(), progressbar.ETA(),
170 progressbar.FileTransferSpeed()] 170​ progressbar.FileTransferSpeed()]
171 pbar = progressbar.ProgressBar(widgets=widgets, maxval=int(total_len)).start() 171​ pbar = progressbar.ProgressBar(widgets=widgets, maxval=int(total_len)).start()
172 172​
173 read = 0 173​ read = 0
174 for chunk in pbar(r.iter_content(chunk_size)): 174​ for chunk in r.iter_content(chunk_size):
175 # filter out keep-alive new lines 175​ # filter out keep-alive new lines
176 if chunk: 176​ if chunk:
177 f.write(chunk) 177​ f.write(chunk)
178 read += len(chunk) 178​ # chunk can be smaller than chunk_size (end of file) or larger than it (content
179 pbar.update(read) 179​ # decompression)
180​ read += min(len(chunk), chunk_size)
181​ # don't exceed pbar.maxval, which can happen due to content decompression (see
182​ # T755). This should not occur except for the last chunk.
183​ pbar.update(min(read, pbar.maxval))
180 184​
181 185​
182# http://stackoverflow.com/questions/11325019 186​# http://stackoverflow.com/questions/11325019
183class Tee(object): 187​class Tee(object):
184 '''Helper class for writing data to different streams.''' 188​ '''Helper class for writing data to different streams.'''
185 189​
186 def __init__(self, *files): 190​ def __init__(self, *files):
187 self._files = list(files) 191​ self._files = list(files)
Show All 16 Lines