1. Grab the socket lock for the whole conversation.
We need to keep the lock until the whole conversation is over.
Otherwise we may have concurrency issues where communication gets intermixed
and errors in one thread can cause a thread to hang.
Here is what we observed:
thread 1: grabs lock and send a request.
thread 2: grabs lock and sends a request
server: thread 2 request causes a fatal error and the server close the
connection
thread 2: grabs the lock and waits for a reply.
thread 2: gets the error and returns to caller with it (connection is closed).
thread 1: grabs the lock (which reopens the closed channel) and reads ...
... forever as the server has already killed all the previous state.
2. Fail immediately on short reads for the initial 4 byte length header.
If the first 4 bytes do not come at once don't bother retrying. In 99.9% of the
cases what we are witnessing here is a fatal error from the proxy that closed
the socket. Reopening the scket cannot accomplish anything as the request sent
down the channel is tied to the specific socket, so once the socket is closed
there is no hope to ever get back a reply.
Signed-off-by: Simo Sorce <simo@redhat.com>
Reviewed-by: Günther Deschner <gdeschner@redhat.com>