Python Celery Task finished without backend

My main task should now wait until the subtask has finished.

You should never wait for a subtask as this may lead to resource starvation and deadlock (all tasks waiting for another task, but no more workers to process them).

Instead you should use a callback to do additional actions after the subtask completes (see the Canvas guide in the Celery user guide).

I'm monitoring the whole application with Celery Flower and I can see that subtask is successfuelly finishing. How can Celery detect that state? I browsed their code but couldn't find out how they do the detection.

Flower and other monitors does not use results (task state), instead they use what we call events.

Event messages are emitted when certain actions occur in the worker, and this becomes a transient stream of messages. Processes can subscribe to certain events (or all of them) to monitor the cluster.

The events are separate from task states because,

• Events are not persistent (transient)

  Missing an event is not regarded as a critical failure.

• Complex fields are not serialized

  Events are for diagnostic and informational purposes, and should not be used to introspect task return values or exceptions for example, as only the repr() of these is stored to make sure monitors can be written in other languages, and big fields may be truncated to ensure faster transmission.