Stop looping if all workers have died

If the workers are crashing and the restart limit has been met,
we need to stop listening for events and trigger an internal
error.
This commit is contained in:
Tim Jenness
2017-10-04 08:32:12 -07:00
parent e4f36b18f3
commit 9e59d07947
3 changed files with 16 additions and 0 deletions

View File

@@ -120,6 +120,10 @@ class DSession:
def loop_once(self):
"""Process one callback from one of the slaves."""
while 1:
if not self._active_nodes:
# If everything has died stop looping
self.triggershutdown()
raise RuntimeError("Unexpectedly no active workers available")
try:
eventcall = self.queue.get(timeout=2.0)
break