Differential D223

download RPM headers - implement RPM binary format parsing in Python
AbandonedPublic
Actions

Authored by garretraziel on Aug 27 2014, 12:55 PM.

Details

Reviewers
kparal
tflink
Maniphest Tasks
T226: depcheck: download only rpm headers?
Summary

when requested, download only RPM header in koji directive

Test Plan

I will write tests... in the future!

Diff Detail

Repository
rLTRN libtaskotron
Branch
feature/T226-headers
Lint
No Linters Available
Unit
No Unit Test Coverage
garretraziel retitled this revision from to downloading RPM headers.Aug 27 2014, 12:55 PM
garretraziel updated this object.
garretraziel edited the test plan for this revision. (Show Details)
garretraziel added reviewers: kparal, tflink, jskladan.
garretraziel added a comment.Aug 27 2014, 12:56 PM

note: This isn't complete solution - I am only publishing this for discussion.

kparal retitled this revision from downloading RPM headers to download RPM headers - implement RPM binary format parsing in Python.Oct 9 2014, 1:51 PM
jskladan added a comment.May 6 2015, 11:28 AM

as this seems to be abanodned anyway, how about closing the revision?

jskladan removed a reviewer: jskladan.May 7 2015, 7:04 AM
jskladan added a subscriber: jskladan.
garretraziel abandoned this revision.Aug 27 2015, 12:29 PM

It looks like there are currently other things to do. I will abandon this revision, we can open it again when we will need it.

Revision Contents

Diff 638

View Options

libtaskotron/directives/koji_directive.py

Show First 20 Lines • Show All 114 Lines • ▼ Show 20 Line(s)114 else:
115 self.target_dir = input_data['target_dir'] 115​ self.target_dir = input_data['target_dir']
116 116​
117 if 'arch' not in input_data: 117​ if 'arch' not in input_data:
118 detected_args = ', '.join(input_data.keys()) 118​ detected_args = ', '.join(input_data.keys())
119 raise exc.TaskotronDirectiveError( 119​ raise exc.TaskotronDirectiveError(
120 "The koji directive requires 'arch' as an argument. Detected " 120​ "The koji directive requires 'arch' as an argument. Detected "
121 "arguments: %s" % detected_args) 121​ "arguments: %s" % detected_args)
122 122​
123​ headers_only = False
124​ if 'headers_only' in input_data:
125​ headers_only = input_data['headers_only']
126​
123 self.arches = input_data['arch'] 127​ self.arches = input_data['arch']
124 if self.arches and ('noarch' not in self.arches): 128​ if self.arches and ('noarch' not in self.arches):
125 self.arches.append('noarch') 129​ self.arches.append('noarch')
126 130​
127 self.src = input_data.get('src', False) 131​ self.src = input_data.get('src', False)
128 132​
129 # download files 133​ # download files
130 output_data = {} 134​ output_data = {}
131 135​
132 if action == 'download': 136​ if action == 'download':
133 if 'koji_build' not in input_data: 137​ if 'koji_build' not in input_data:
134 detected_args = ', '.join(input_data.keys()) 138​ detected_args = ', '.join(input_data.keys())
135 raise exc.TaskotronDirectiveError( 139​ raise exc.TaskotronDirectiveError(
136 "The koji directive requires 'koji_build' for the 'download' " 140​ "The koji directive requires 'koji_build' for the 'download' "
137 "action. Detected arguments: %s" % detected_args) 141​ "action. Detected arguments: %s" % detected_args)
138 142​
139 nvr = rpm_utils.rpmformat(input_data['koji_build'], 'nvr') 143​ nvr = rpm_utils.rpmformat(input_data['koji_build'], 'nvr')
140 144​
141 output_data['downloaded_rpms'] = self.koji.get_nvr_rpms(nvr, 145​ output_data['downloaded_rpms'] = self.koji.get_nvr_rpms(nvr,
142 self.target_dir, arches=self.arches, src=self.src) 146​ self.target_dir, arches=self.arches, src=self.src,
147​ headers_only=headers_only)
143 148​
144 elif action == 'download_tag': 149​ elif action == 'download_tag':
145 if 'koji_tag' not in input_data: 150​ if 'koji_tag' not in input_data:
146 detected_args = ', '.join(input_data.keys()) 151​ detected_args = ', '.join(input_data.keys())
147 raise exc.TaskotronDirectiveError( 152​ raise exc.TaskotronDirectiveError(
148 "The koji directive requires 'koji_tag' for the 'download_tag' " 153​ "The koji directive requires 'koji_tag' for the 'download_tag' "
149 "action. Detected arguments: %s" % detected_args) 154​ "action. Detected arguments: %s" % detected_args)
150 155​
151 koji_tag = input_data['koji_tag'] 156​ koji_tag = input_data['koji_tag']
152 157​
153 output_data['downloaded_rpms'] = self.koji.get_tagged_rpms(koji_tag, 158​ output_data['downloaded_rpms'] = self.koji.get_tagged_rpms(koji_tag,
154 self.target_dir, arches=self.arches, src=self.src) 159​ self.target_dir, arches=self.arches, src=self.src,
160​ headers_only=headers_only)
155 161​
156 return output_data 162​ return output_data
View Options

libtaskotron/exceptions.py

Show All 28 Lines
29class TaskotronDirectiveError(TaskotronError): 29​class TaskotronDirectiveError(TaskotronError):
30 '''All errors related to Taskotron directives''' 30​ '''All errors related to Taskotron directives'''
31 pass 31​ pass
32 32​
33 33​
34class TaskotronRemoteError(TaskotronError): 34​class TaskotronRemoteError(TaskotronError):
35 '''All network and remote-server related errors''' 35​ '''All network and remote-server related errors'''
36 pass 36​ pass
37​
38​class TaskotronRemoteReadError(TaskotronRemoteError):
39​ '''Error in data obtained from network'''
40​ pass
41​No newline at end of file
View Options

libtaskotron/file_utils.py

1# -*- coding: utf-8 -*- 1​# -*- coding: utf-8 -*-
2# Copyright 2009-2014, Red Hat, Inc. 2​# Copyright 2009-2014, Red Hat, Inc.
3# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+> 3​# License: GPL-2.0+ <http://spdx.org/licenses/GPL-2.0+>
4# See the LICENSE file for more details on Licensing 4​# See the LICENSE file for more details on Licensing
5 5​
6from __future__ import absolute_import 6​from __future__ import absolute_import
7import os 7​import os
8import sys 8​import sys
9​import struct
10​import urllib2
9import urlgrabber.grabber 11​import urlgrabber.grabber
10import urlgrabber.progress 12​import urlgrabber.progress
11from libtaskotron.logger import log 13​from libtaskotron.logger import log
12from libtaskotron.exceptions import TaskotronRemoteError 14​from libtaskotron.exceptions import TaskotronRemoteError, TaskotronRemoteReadError
13 15​
14 16​
15def makedirs(fullpath): 17​def makedirs(fullpath):
16 '''This is the same as :meth:`os.makedirs`, but does not raise an exception 18​ '''This is the same as :meth:`os.makedirs`, but does not raise an exception
17 when the destination directory already exists. 19​ when the destination directory already exists.
18 20​
19 :raise OSError: if directory doesn't exist and can't be created 21​ :raise OSError: if directory doesn't exist and can't be created
20 ''' 22​ '''
▲ Show 20 Lines • Show All 90 Lines • ▼ Show 20 Line(s)111 except urlgrabber.grabber.URLGrabError, e:
111 # the file can be incomplete, remove 113​ # the file can be incomplete, remove
112 if os.path.exists(dest): 114​ if os.path.exists(dest):
113 try: 115​ try:
114 os.remove(dest) 116​ os.remove(dest)
115 except OSError, e: 117​ except OSError, e:
116 log.exception('Could not delete incomplete file: %s', dest) 118​ log.exception('Could not delete incomplete file: %s', dest)
117 raise TaskotronRemoteError(e) 119​ raise TaskotronRemoteError(e)
118 120​
121​ return dest
122​
123​
124​def extract_header(source_descriptor):
125​ '''Read RPM header from socket descriptor.
126​
127​ :param source_descriptor: socket (or file) descriptor to read RPM from
128​ :return: buffer containing only RPM lead + headers
129​ :rtype: str
130​ :raise TaskotronRemoteReadError: if downloaded RPM has bad structure
131​ '''
132​ # from http://www.rpm.org/max-rpm/s1-rpm-file-format-rpm-file-format.html
133​
134​ # MAGIC
135​ # .
136​ #
137​ # .
138​ # /^\ .
139​ # /\ "V"
140​ # /__\ I O o
141​ # //..\\ I .
142​ # \].`[/ I
143​ # /l\/j\ (] . O
144​ # /. ~~ ,\/I .
145​ # \\L__j^\/I o
146​ # \/--v} I o .
147​ # | | I _________
148​ # | | I c(` ')o
149​ # | l I \. ,/
150​ # _/j L l\_! _//^---^\\_
151​ # ~~~~~~~~~~~~~~~~~~~~~~~~~
152​ lead_length = 96
153​ header_without_entries_length = 16
154​ index_entry_length = 16
155​ byte_padding = 8
156​ lead_magic_numbers = (0xed, 0xab, 0xee, 0xdb)
157​ hdr_magic_numbers = (0x8e, 0xad, 0xe8)
158​
159​ # lead
160​ # ----
161​ # struct rpmlead {
162​ # unsigned char magic[4];
163​ # unsigned char major, minor;
164​ # short type;
165​ # short archnum;
166​ # char name[66];
167​ # short osnum;
168​ # short signature_type;
169​ # char reserved[16];
170​ # } ;
171​ lead_fmt = "!4B2Bhh66shh16s"
172​ lead_fmt_size = struct.calcsize(lead_fmt)
173​ lead = source_descriptor.read(lead_fmt_size)
174​ unpacked_lead = struct.unpack(lead_fmt, lead)
175​ if unpacked_lead[:4] != lead_magic_numbers or unpacked_lead[4] != 3:
176​ raise TaskotronRemoteReadError("Unsupported RPM format")
177​
178​ # signature header
179​ # ----------------
180​ # The first three bytes (8ead e8) contain the magic number for the start of
181​ # the header structure. The next byte is the header structure's version.
182​ # The next four bytes (0000 0000) are reserved.
183​ # The four bytes after that represent the number of index entries in the
184​ # signature section. Following that are four bytes that indicate how many
185​ # bytes of data are stored in the signature.
186​ hdr_fmt = "!3BB4sii"
187​ hdr_fmt_size = struct.calcsize(hdr_fmt)
188​ hdr_sig = source_descriptor.read(hdr_fmt_size)
189​ unpacked_hdr_sig = struct.unpack(hdr_fmt, hdr_sig)
190​ if unpacked_hdr_sig[:3] != hdr_magic_numbers or unpacked_hdr_sig[3] != 1:
191​ raise TaskotronRemoteReadError("Unsupported RPM header format")
192​
193​ indexcount_sig = unpacked_hdr_sig[-2]
194​ storesize_sig = unpacked_hdr_sig[-1]
195​
196​ # calculate padding - signature header should be padded to 8 bytes
197​ lead_sig_size = lead_length + header_without_entries_length + \
198​ indexcount_sig * index_entry_length + storesize_sig
199​
200​ sig_padding = (byte_padding - (lead_sig_size % byte_padding)) if \
201​ (lead_sig_size % byte_padding) != 0 else 0
202​ remains_sign = indexcount_sig*index_entry_length + storesize_sig + sig_padding
203​
204​ # read remaining bytes in signature header
205​ hdr_sig += source_descriptor.read(remains_sign)
206​
207​ # payload header
208​ # --------------
209​ # has same structure as signature header, but isn't padded
210​ hdr_pay = source_descriptor.read(hdr_fmt_size)
211​ unpacked_hdr_pay = struct.unpack(hdr_fmt, hdr_pay)
212​ if unpacked_hdr_pay[:3] != hdr_magic_numbers or unpacked_hdr_pay[3] != 1:
213​ raise TaskotronRemoteReadError("Unsupported RPM header format")
214​
215​ indexcount_pay = unpacked_hdr_pay[-2]
216​ storesize_pay = unpacked_hdr_pay[-1]
217​
218​ remains_pay = indexcount_pay*index_entry_length + storesize_pay
219​
220​ # read remaining bytes in payload header
221​ hdr_pay += source_descriptor.read(remains_pay)
222​
223​ return lead + hdr_sig + hdr_pay
224​
225​
226​def download_rpm_header(url, dirname, filename=None, overwrite=False,
227​ grabber=None):
228​ '''Download RPM header file.
229​
230​ :param str url: RPM URL to download
231​ :param str dirname: directory path, if the directory does not exist, it gets
232​ created (and all its parent directories).
233​ :param str filename: name of downloaded file, if not provided basename is
234​ extracted from URL
235​ :param bool overwrite: if the destination file already exists, whether to
236​ overwrite or not. If ``False``, a simple check is
237​ performed whether the remote file is the same as the
238​ local file and re-downloads it anyway if they are not
239​ the same.
240​ :param grabber: custom :class:`urlgrabber.grabber.URLGrabber` instance to be
241​ used for actual downloading
242​ :return: file path of the downloaded file
243​ :rtype: str
244​ :raise TaskotronRemoteError: if download fails or directory can't be created
245​ '''
246​ if not grabber:
247​ grabber = get_urlgrabber()
248​
249​ if not filename:
250​ filename = os.path.basename(url)
251​
252​ dest = os.path.join(dirname, filename)
253​
254​ if os.path.exists(dirname):
255​ if not os.path.isdir(dirname):
256​ raise TaskotronRemoteError(
257​ "Can't create directory: %r It is an already "
258​ "existing file.", dirname)
259​ else:
260​ try:
261​ makedirs(dirname)
262​ except OSError, e:
263​ log.exception("Can't create directory: %s", dirname)
264​ raise TaskotronRemoteError(e)
265​
266​ # overwrite?
267​ if os.path.exists(dest):
268​ if overwrite:
269​ os.remove(dest)
270​ else:
271​ log.debug('Already downloaded: %s', os.path.basename(dest))
272​ return dest
273​
274​ log.debug('Downloading: %s', url)
275​ try:
276​ urlfd = urllib2.urlopen(url)
277​ hdr = extract_header(urlfd)
278​ urlfd.close()
279​ f = open(dest, 'wb')
280​ f.write(hdr)
281​ f.close()
282​ except (urllib2.URLError, TaskotronRemoteReadError), e:
283​ log.debug('RPM header download failed: ' + str(e))
284​ log.debug('falling back to standard download method')
285​ try:
286​ grabber.urlgrab(url, dest)
287​ except urlgrabber.grabber.URLGrabError, e:
288​ log.exception('Download failed for: %s', url)
289​ # the file can be incomplete, remove
290​ if os.path.exists(dest):
291​ try:
292​ os.remove(dest)
293​ except OSError, e:
294​ log.exception('Could not delete incomplete file: %s', dest)
295​ raise TaskotronRemoteError(e)
296​
119 return dest 297​ return dest
120No newline at end of file 298​No newline at end of file
View Options

libtaskotron/koji_utils.py

Show First 20 Lines • Show All 161 Lines • ▼ Show 20 Line(s)110 def nvr_to_urls(self, nvr, arches=['all'], debuginfo=False, src=True):
162 baseurl = '/'.join((config.get_config().pkg_url, info['package_name'], 162​ baseurl = '/'.join((config.get_config().pkg_url, info['package_name'],
163 info['version'], info['release'])) 163​ info['version'], info['release']))
164 urls = ['%s/%s' % (baseurl, koji.pathinfo.rpm(r)) for r in rpms] 164​ urls = ['%s/%s' % (baseurl, koji.pathinfo.rpm(r)) for r in rpms]
165 165​
166 return sorted(urls) 166​ return sorted(urls)
167 167​
168 168​
169 def get_nvr_rpms(self, nvr, dest, arches=['all'], debuginfo=False, 169​ def get_nvr_rpms(self, nvr, dest, arches=['all'], debuginfo=False,
170 src=False): 170​ src=False, headers_only=False):
171 '''Retrieve the RPMs associated with a build NVR into the specified 171​ '''Retrieve the RPMs associated with a build NVR into the specified
172 directory. 172​ directory.
173 173​
174 :param str nvr: build NVR 174​ :param str nvr: build NVR
175 :param str dest: location where to store the RPMs 175​ :param str dest: location where to store the RPMs
176 :param arches: see :meth:`nvr_to_urls` 176​ :param arches: see :meth:`nvr_to_urls`
177 :param debuginfo: see :meth:`nvr_to_urls` 177​ :param debuginfo: see :meth:`nvr_to_urls`
178 :param src: see :meth:`nvr_to_urls` 178​ :param src: see :meth:`nvr_to_urls`
179​ :param headers_only: don't download whole RPMs, download only headers
179 :return: list of local filenames of the grabbed RPMs 180​ :return: list of local filenames of the grabbed RPMs
180 :rtype: list of str 181​ :rtype: list of str
181 :raise TaskotronRemoteError: if the files can't be downloaded or saved 182​ :raise TaskotronRemoteError: if the files can't be downloaded or saved
182 :raise TaskotronValueError: if ``arches=[]`` and ``src=False``, 183​ :raise TaskotronValueError: if ``arches=[]`` and ``src=False``,
183 therefore there is nothing to download 184​ therefore there is nothing to download
184 ''' 185​ '''
185 186​
186 rpm_urls = self.nvr_to_urls(nvr, arches, debuginfo, src) 187​ rpm_urls = self.nvr_to_urls(nvr, arches, debuginfo, src)
187 188​
188 rpm_files = [] 189​ rpm_files = []
189 log.info('Fetching %s RPMs for: %s (into %s)', len(rpm_urls), nvr, dest) 190​ log.info('Fetching %s RPMs for: %s (into %s)', len(rpm_urls), nvr, dest)
190 for url in rpm_urls: 191​ for url in rpm_urls:
191 rpm_file = file_utils.download(url, dest) 192​ if headers_only:
193​ rpm_file = file_utils.download_rpm_header(url, dest)
194​ else:
195​ rpm_file = file_utils.download(url, dest)
192 rpm_files.append(rpm_file) 196​ rpm_files.append(rpm_file)
193 197​
194 return rpm_files 198​ return rpm_files
195 199​
196 200​
197 def get_tagged_rpms(self, tag, dest, arches=['all'], debuginfo=False, 201​ def get_tagged_rpms(self, tag, dest, arches=['all'], debuginfo=False,
198 src=False): 202​ src=False, headers_only=False):
199 '''Downloads all RPMs of all NVRs tagged by a specific Koji tag. 203​ '''Downloads all RPMs of all NVRs tagged by a specific Koji tag.
200 204​
201 Note: This works basically the same as :meth:`get_nvr_rpms`, it just 205​ Note: This works basically the same as :meth:`get_nvr_rpms`, it just
202 downloads a lot of builds instead of a single one. For description of 206​ downloads a lot of builds instead of a single one. For description of
203 all shared parameters and return values, please see that method. 207​ all shared parameters and return values, please see that method.
204 208​
205 :param str tag: Koji tag to be queried for available builds, e.g. 209​ :param str tag: Koji tag to be queried for available builds, e.g.
206 ``f20-updates-pending`` 210​ ``f20-updates-pending``
207 ''' 211​ '''
208 log.debug('Querying Koji for tag: %s' % tag) 212​ log.debug('Querying Koji for tag: %s' % tag)
209 tag_data = self.session.listTagged(tag) 213​ tag_data = self.session.listTagged(tag)
210 214​
211 nvrs = ["%(nvr)s" % x for x in tag_data] 215​ nvrs = ["%(nvr)s" % x for x in tag_data]
212 rpms = [] 216​ rpms = []
213 log.info("Fetching %s builds for tag: %s", len(nvrs), tag) 217​ log.info("Fetching %s builds for tag: %s", len(nvrs), tag)
214 log.debug('Builds to be downloaded:\n %s', '\n '.join(nvrs)) 218​ log.debug('Builds to be downloaded:\n %s', '\n '.join(nvrs))
215 219​
216 for nvr in nvrs: 220​ for nvr in nvrs:
217 rpms.append(self.get_nvr_rpms(nvr, dest, arches, debuginfo, src)) 221​ rpms.append(self.get_nvr_rpms(nvr, dest, arches, debuginfo, src,
222​ headers_only))
218 223​
219 return rpms 224​ return rpms
220 225​
221 226​
222def getNEVR(build): 227​def getNEVR(build):
223 '''Extract RPM version identifier in NEVR format from Koji build object 228​ '''Extract RPM version identifier in NEVR format from Koji build object
224 229​
225 :param dict build: Koji buildinfo dictionary (as returned e.g. from 230​ :param dict build: Koji buildinfo dictionary (as returned e.g. from
Show All 17 Lines