remote_exec: speed up minion->host copy operations a bit
ClosedPublic

Authored by kparal on Mar 8 2016, 3:10 PM.

Details

Summary

This gets attributes for all files in a directory as a batch, instead of
one by one, which slightly increases speed of copy operations.

This also makes all errors more readable, by printing the whole
exception, and not just its strerror attribute.

Additional info:
This diff was started when I tested D767, which creates 26 MB / 5400
files of artifacts. For some reason, copying this dir to the minion
takes 24 seconds, while copying it back to the host (unchanged) takes 65
seconds. I tried to profile it and came up with this patch, which
reduces it to 57 seconds, but I can't find any other bottleneck. It
seems it's related to Paramiko or SFTP. One of the slowdowns is that we
need to query for file attributes remotely, which is slower than
locally, but it does not explain all the difference. It seems we'll need
to live with it.

Test Plan

test suite works, and I manually tested --libvirt and --ssh and verified the copy-out speed increased a bit

Diff Detail

Repository
rLTRN libtaskotron
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
kparal retitled this revision from to remote_exec: speed up minion->host copy operations a bit.Mar 8 2016, 3:10 PM
kparal updated this object.
kparal edited the test plan for this revision. (Show Details)
kparal added a reviewer: libtaskotron.
kparal updated this revision to Diff 1969.Mar 8 2016, 3:14 PM
  • improve docstring
mkrizek accepted this revision.Mar 9 2016, 8:27 AM
mkrizek added a reviewer: mkrizek.
This revision is now accepted and ready to land.Mar 9 2016, 8:27 AM
This revision was automatically updated to reflect the committed changes.