koji_utils: retry calls, enable debug logging
ClosedPublic

Authored by kparal on Apr 2 2015, 10:08 AM.

Details

Summary

This enables call retries for standard Koji calls (up to now, we've retried
just Koji downloads). It also enables debug logging from Koji library,
because it might be handy to see why the calls were retried.

This should solve a lot of problems like this:
https://taskotron.stg.fedoraproject.org/taskmaster/builders/x86_64/builds/100908/steps/runtask/logs/stdio

Because of enabled debug logging, we should then see details like this in the debug log for each failed attempt:

[koji:__init__.py:1934] 2015-04-02 09:45:43 DEBUG   Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1899, in _callMethod
    return self._sendCall(handler, headers, request)
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1810, in _sendCall
    return self._sendOneCall(handler, headers, request)
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1828, in _sendOneCall
    cnx.endheaders()
  File "/usr/lib64/python2.7/httplib.py", line 991, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 844, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 806, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 787, in connect
    self.timeout, self.source_address)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused

That is overly verbose (we don't really need the full stacktrace), but unfortunately the short INFO printout which should be also available fails:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 859, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 732, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 471, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 335, in getMessage
    msg = msg % self.args
TypeError: %d format: a number is required, not NoneType
Logged from file __init__.py, line 1935

That is a Koji error, I reported the problem and included a patch here:
https://fedorahosted.org/koji/ticket/317

Once that problem is fixed, I'll switch DEBUG logging from Koji to INFO logging, that is enough for us.

Test Plan

test suite passes and I tested it manually a bit

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to koji_utils: retry calls, enable debug logging.Apr 2 2015, 10:08 AM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added reviewers: tflink, mkrizek, jskladan.
kparal updated this revision to Diff 880.Apr 2 2015, 10:09 AM
  • tiny docstring change
kparal updated this revision to Diff 881.Apr 2 2015, 10:11 AM
  • tiny comment change
mkrizek accepted this revision.Apr 2 2015, 10:12 AM

LGTM

This revision is now accepted and ready to land.Apr 2 2015, 10:12 AM
kparal updated this revision to Diff 882.Apr 2 2015, 10:33 AM
  • add TODO for the koji bug
This revision was automatically updated to reflect the committed changes.