scheduler: restart the consumer when it crashes
ClosedPublic

Authored by adamwill on Mar 1 2016, 8:34 PM.

Details

Summary

The fedmsg consumer crashed last night; apparently fedmsg
tail_messages() will crash by design if it can't retrieve the
CRL (for Security Purposes). Apparently the box's network went
down for a short time or something. This is obviously Bad -
we rely on the consumer running or else we'll miss scheduling
jobs for new composes. So let's have the unit file restart it
on failure. Just in case we write some dumb code that makes it
crash *all the time*, set RestartSec to 10 so it'll only try
and restart every ten seconds and won't kill the system or spam
the logs too badly.

Test Plan

Start the consumer running, make it die somehow
(probably just killing it will do the trick), check it comes
back.

Diff Detail

Repository
rOPENQA fedora_openqa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
adamwill retitled this revision from to scheduler: restart the consumer when it crashes.Mar 1 2016, 8:34 PM
adamwill updated this object.
adamwill edited the test plan for this revision. (Show Details)
adamwill added reviewers: jskladan, garretraziel.
This revision is now accepted and ready to land.Mar 2 2016, 12:09 PM
This revision was automatically updated to reflect the committed changes.