This changes 'all' into 'nightlies': it doesn't run on the
'current' compose any more. In practice we run 'current' every
hour and 'all' (now 'nightlies') every day, so there's really
not much point doing 'current' in 'all', and this cleans it up
a bit.
Then the fun starts! Using some very new fedfind functionality,
'nightlies' can now wait for composes to complete before firing
jobs. By default it will wait 8 hours; the -w parameter lets
you specify a different wait time (in minutes).
It does the waiting and job scheduling in parallel, using
multiprocessing.Pool (I actually did it with threading first,
but this is a lot simpler and just as fast). So as soon as
either compose appears it will download the ISOs and schedule
the jobs; it doesn't wait for one then wait for the other.
Upshot, we can schedule this to run every night an hour or two
before the earliest time the compose is likely to complete and
it should fire off the jobs as the composes complete. If we
want to get aggressive we can probably in theory tweak more so
we run the boot.iso tests as soon as it appears and run the
lives later, but it'd require some refactoring of jobs_from_
fedfind.
I still think that using Pool inside a Pool is a bad idea - it seems to me that it's like running event loop inside event loop. I think that I will just put up with it, because code in trigger looks reasonable and second Pool is running inside fedfind and that's something I'm not reviewing :-).
We should monitor how it behaves (if it isn't spawning a lot of zombies for example) and when there will be a problem, we can use something complex (like Celery, coincidentally I was reading about it few weeks ago) or something easy and straightforward (like running two, singleprocess jobs by cron).