Health check times out during long-running background task
During a long-running background task, like importing a big file in CSVImport
, the health check times out checking the celery result backend:
ERROR 2022-01-04 20:36:05,838 backends: unavailable: TimeoutError: The task took too long to return a result
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/health_check/contrib/celery/backends.py", line 24, in check_status
result.get(timeout=result_timeout)
File "/usr/local/lib/python3.9/dist-packages/celery/result.py", line 224, in get
return self.backend.wait_for_pending(
File "/usr/local/lib/python3.9/dist-packages/celery/backends/base.py", line 756, in wait_for_pending
meta = self.wait_for(
File "/usr/local/lib/python3.9/dist-packages/celery/backends/base.py", line 792, in wait_for
raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.
On Kubernetes, this causes the liveness probe to fail, and the container to be restarted mid-import.