This enables call retries for standard Koji calls (up to now, we've retried
just Koji downloads). It also enables debug logging from Koji library,
because it might be handy to see why the calls were retried.
This should solve a lot of problems like this:
https://taskotron.stg.fedoraproject.org/taskmaster/builders/x86_64/builds/100908/steps/runtask/logs/stdio
Because of enabled debug logging, we should then see details like this in the debug log for each failed attempt:
[koji:__init__.py:1934] 2015-04-02 09:45:43 DEBUG Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1899, in _callMethod
return self._sendCall(handler, headers, request)
File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1810, in _sendCall
return self._sendOneCall(handler, headers, request)
File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1828, in _sendOneCall
cnx.endheaders()
File "/usr/lib64/python2.7/httplib.py", line 991, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 844, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 806, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 787, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 111] Connection refusedThat is overly verbose (we don't really need the full stacktrace), but unfortunately the short INFO printout which should be also available fails:
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 859, in emit
msg = self.format(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 732, in format
return fmt.format(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 471, in format
record.message = record.getMessage()
File "/usr/lib64/python2.7/logging/__init__.py", line 335, in getMessage
msg = msg % self.args
TypeError: %d format: a number is required, not NoneType
Logged from file __init__.py, line 1935That is a Koji error, I reported the problem and included a patch here:
https://fedorahosted.org/koji/ticket/317
Once that problem is fixed, I'll switch DEBUG logging from Koji to INFO logging, that is enough for us.