Differential D597

download ISOs with urllib2 not urlgrabber
ClosedPublic
Actions

Authored by adamwill on Sep 19 2015, 4:36 AM.

Details

Reviewers
garretraziel
jskladan
Maniphest Tasks
T601: Come up with some alternative to `urlgrabber` for ISO downloads
Commits
rOPENQAb2bf7411ef4c: download ISOs with urllib2 not urlgrabber
Summary

urlgrabber is old, not in the stdlib, and openSUSE's current
version doesn't handle https properly. urlllib.urlretrieve()
is a possibility but urllib (in py2) doesn't do TLS right. So
we'll do this, adapted from SO, using urllib2 (py3 urllib).
This only works for HTTP(S) of course, but all fedfind URLs
are HTTPS and that's not likely to change. Fixes T601.

We could use requests since openqa_client depends on it anyway,
but it isn't actually any easier for downloading big files
than urllib2, oddly enough.

Test Plan

Check that ISO downloads still work.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to download ISOs with urllib2 not urlgrabber.Sep 19 2015, 4:36 AM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jskladan, garretraziel.
adamwill added a task: T601: Come up with some alternative to `urlgrabber` for ISO downloads.
garretraziel requested changes to this revision.Sep 21 2015, 8:47 AM
garretraziel added inline comments.
fedora_openqa_schedule/schedule.py
31

I don't think that six is a standard library - move to "external dependencies".

98

There are too few arguments here :-).

101

Document this "magic value", please - was it arbitrary chosen or does it have any meaning?

This revision now requires changes to proceed.Sep 21 2015, 8:47 AM
adamwill added inline comments.Sep 21 2015, 3:03 PM
fedora_openqa_schedule/schedule.py
98

d'oh, guess who was testing with a test script not the real code :)

adamwill updated this revision to Diff 1551.Sep 21 2015, 3:43 PM

Address garretraziel's comments (all 3 valid)

garretraziel accepted this revision.Sep 22 2015, 6:44 AM
This revision is now accepted and ready to land.Sep 22 2015, 6:44 AM
Closed by commit rOPENQAb2bf7411ef4c: download ISOs with urllib2 not urlgrabber (authored by adamwill). · Explain WhySep 22 2015, 7:30 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
MREADME.md (4 lines)
Mfedora_openqa_schedule/schedule.py (22 lines)
Msetup.py (2 lines)

Diff 1553

View Options

README.md

Show All 20 Lines
21to generate test jobs. If you have deployed openQA in docker containers, 21​to generate test jobs. If you have deployed openQA in docker containers,
22note that fedora-openqa-schedule must be installed and configured on the 22​note that fedora-openqa-schedule must be installed and configured on the
23**host**, not in any of the containers. 23​**host**, not in any of the containers.
24 24​
251. Install packaged dependencies: 25​1. Install packaged dependencies:
26 26​
27 Either: 27​ Either:
28 28​
29 zypper in python-setuptools python-six python-requests python-urlgrabber 29​ zypper in python-setuptools python-six python-requests
30 30​
31 when you have installed openQA on OpenSUSE or: 31​ when you have installed openQA on OpenSUSE or:
32 32​
33 dnf install python-setuptools python-six python-requests python-urlgrabber python-fedfind python-wikitcms 33​ dnf install python-setuptools python-six python-requests python-fedfind python-wikitcms
34 34​
35 when you have installed openQA using Docker on Fedora. 35​ when you have installed openQA using Docker on Fedora.
36 36​
372. Check out some non-packaged dependencies from git and install them. If 37​2. Check out some non-packaged dependencies from git and install them. If
38you installed python-wikitcms and python-fedfind from packages, you can skip 38​you installed python-wikitcms and python-fedfind from packages, you can skip
39them here. 39​them here.
40 40​
41 mkdir WIKITCMS; cd WIKITCMS 41​ mkdir WIKITCMS; cd WIKITCMS
▲ Show 20 Lines • Show All 99 Lines • Show Last 20 Lines
View Options

fedora_openqa_schedule/schedule.py

Show All 22 Lines
23job scheduling go here. 23​job scheduling go here.
24""" 24​"""
25 25​
26# Standard libraries 26​# Standard libraries
27import json 27​import json
28import logging 28​import logging
29import os.path 29​import os.path
30import re 30​import re
31 31​
garretrazielUnsubmitted
Not Done

I don't think that six is a standard library - move to "external dependencies".

32# External dependencies 32​# External dependencies
33import fedfind.release 33​import fedfind.release
34from openqa_client.client import OpenQA_Client 34​from openqa_client.client import OpenQA_Client
35from six.moves import configparser 35​from six.moves import configparser
36import urlgrabber 36​from six.moves.urllib.request import urlopen
37 37​
38# Internal dependencies 38​# Internal dependencies
39from fedora_openqa_schedule.config import CONFIG 39​from fedora_openqa_schedule.config import CONFIG
40 40​
41logger = logging.getLogger(__name__) 41​logger = logging.getLogger(__name__)
42 42​
43 43​
44class TriggerException(Exception): 44​class TriggerException(Exception):
Show All 40 Lines84 if not iso_path:
85 iso_path = CONFIG.get('schedule', 'iso-path') 85​ iso_path = CONFIG.get('schedule', 'iso-path')
86 ver = image.version.replace(' ', '_') 86​ ver = image.version.replace(' ', '_')
87 if image.imagetype == 'boot': 87​ if image.imagetype == 'boot':
88 isoname = "{0}_{1}_{2}_boot.iso".format(ver, image.payload, image.arch) 88​ isoname = "{0}_{1}_{2}_boot.iso".format(ver, image.payload, image.arch)
89 else: 89​ else:
90 isoname = "{0}_{1}".format(ver, image.filename) 90​ isoname = "{0}_{1}".format(ver, image.filename)
91 filename = os.path.join(iso_path, isoname) 91​ filename = os.path.join(iso_path, isoname)
92 if not os.path.isfile(filename): 92​ if not os.path.isfile(filename):
93 logger.info("downloading %s (%s) to %s", image.url, image.desc, filename) 93​ # adapted from the University of StackOverflow:
94 # Icky hack around a urlgrabber bug: 94​ # https://stackoverflow.com/questions/22676
95 # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=715416 95​ resp = urlopen(image.url)
96 urlgrabber.urlgrab(image.url.replace('https', 'http'), filename) 96​ meta = resp.info()
97​ size = int(meta.getheaders("Content-Length")[0]) // (1024 * 1024)
98​ logger.info("downloading %s (%s) to %s, %sMiB", image.url, image.desc, filename, size)
garretrazielUnsubmitted
Not Done

There are too few arguments here :-).

adamwillAuthorUnsubmitted
Not Done

d'oh, guess who was testing with a test script not the real code :)

99​ fout = open(filename, 'wb')
100​ while True:
101​ # This is the number of bytes to read between buffer
garretrazielUnsubmitted
Not Done

Document this "magic value", please - was it arbitrary chosen or does it have any meaning?

102​ # flushes. Value taken from the SO example.
103​ buffer = resp.read(8192)
104​ if not buffer:
105​ break
106​ fout.write(buffer)
107​ fout.close()
108​
97 else: 109​ else:
98 logger.info("%s already exists", filename) 110​ logger.info("%s already exists", filename)
99 return isoname 111​ return isoname
100 112​
101 113​
102def run_openqa_jobs(isoname, flavor, arch, build, force=False): 114​def run_openqa_jobs(isoname, flavor, arch, build, force=False):
103 """# run OpenQA 'isos' job on selected isoname, with given arch 115​ """# run OpenQA 'isos' job on selected isoname, with given arch
104 and a version string. **NOTE**: the version passed to OpenQA as 116​ and a version string. **NOTE**: the version passed to OpenQA as
▲ Show 20 Lines • Show All 206 Lines • Show Last 20 Lines
View Options

setup.py

Show All 18 Lines16setup(
19 entry_points = {'console_scripts': ['fedora-openqa-schedule = fedora_openqa_schedule.cli:main'],}, 19​ entry_points = {'console_scripts': ['fedora-openqa-schedule = fedora_openqa_schedule.cli:main'],},
20 author = "Fedora QA devel team", 20​ author = "Fedora QA devel team",
21 author_email = "qa-devel@lists.fedoraproject.org", 21​ author_email = "qa-devel@lists.fedoraproject.org",
22 description = "Fedora openQA scheduler", 22​ description = "Fedora openQA scheduler",
23 license = "GPLv3+", 23​ license = "GPLv3+",
24 keywords = "fedora openqa test qa", 24​ keywords = "fedora openqa test qa",
25 url = "https://bitbucket.org/rajcze/openqa_fedora_tools", 25​ url = "https://bitbucket.org/rajcze/openqa_fedora_tools",
26 packages = ["fedora_openqa_schedule"], 26​ packages = ["fedora_openqa_schedule"],
27 install_requires = ['fedfind>=1.5', 'openqa-client', 'setuptools', 'six', 'urlgrabber'], 27​ install_requires = ['fedfind>=1.5', 'openqa-client', 'setuptools', 'six'],
28 long_description=read('README.md'), 28​ long_description=read('README.md'),
29 classifiers=[ 29​ classifiers=[
30 "Development Status :: 3 - Alpha", 30​ "Development Status :: 3 - Alpha",
31 "Topic :: Utilities", 31​ "Topic :: Utilities",
32 "License :: OSI Approved :: GNU General Public License v3 or later " 32​ "License :: OSI Approved :: GNU General Public License v3 or later "
33 "(GPLv3+)", 33​ "(GPLv3+)",
34 ], 34​ ],
35 data_files=[ 35​ data_files=[
36 (SYSTEMDUNITPATH, glob.glob('systemd/*.service') + glob.glob('systemd/*.timer')), 36​ (SYSTEMDUNITPATH, glob.glob('systemd/*.service') + glob.glob('systemd/*.timer')),
37 ], 37​ ],
38) 38​)