title: Django celery worker to send real-time status and result messages to front end
body: <p>In a django app I'm running async tasks and would like to show progress, errors etc to the user. If there are errors, the user should be redirect to a page where additional input or some action is required to fix the problem. What is the best way to communicate from the celery work back to the front end?</p>
  
  <p>Here's a basic structure in pseudo code:</p>
  
  <pre><code># views.py
  from tasks import run_task
  
  def view_task():
      run_task.delay()
      return render(request, 'template.html')
  
  # tasks.py
  from compute_module import compute_fct
  
  @shared_task
  def run_task():
      result = compute_fct()
  
      # how to catch status update messages from compute_module while compute_fct is running??
  
      if result == 'error':
          handle_error()
      else:
          handle_succes()     
  
  # compute_module
  import pandas as pd
  
  def compute_fct():
      # send message: status = loading file
      df = pd.read_csv('test.csv')
      # send message: status = computing
      val = df['col'].mean()
  
      if val is None:
          return {'status':'error'}
      else:
          return {'status':'success','val':val}
  </code></pre>
  
  <p>What I would ideally want:</p>
  
  <ul>
  <li><code>compute_module.py</code> module uses python native logger. By separation of duties I want to keep the logging as generic as possible and use the standard python/django loggers. But they don't seem to be designed to send messages to front end. </li>
  <li>celery task somehow handles the logs and instead of displaying them on stdout redirects them to pusher</li>
  <li>front-end js shows and handles the messages</li>
  </ul>
  
  <p>There might be standard ways of communicating between celery worker and front end that I'm not aware off. this scenario must happen often and I am surprised it's so difficult to implement. in a way the rabbitmq message queue or aws sns should be designed for this. below are resources that I looked at but don't feel either of them work very well but maybe I am just confused.</p>
  
  <p>logging: this seems to be more about logging on the server side, not sending messages to user</p>
  
  <ul>
  <li><a href="http://docs.celeryproject.org/en/latest/userguide/tasks.html#logging" rel="noreferrer">http://docs.celeryproject.org/en/latest/userguide/tasks.html#logging</a> </li>
  <li><a href="https://docs.djangoproject.com/en/2.0/topics/logging/" rel="noreferrer">https://docs.djangoproject.com/en/2.0/topics/logging/</a> </li>
  <li><a href="http://oddbird.net/2017/04/17/async-notifications/" rel="noreferrer">http://oddbird.net/2017/04/17/async-notifications/</a></li>
  <li><a href="https://www.google.com/search?q=celery+worker+send+message+to+front+end" rel="noreferrer">https://www.google.com/search?q=celery+worker+send+message+to+front+end</a>           </li>
  </ul>
  
  <p>Celery cam seems to be about admin monitoring tasks, not sending messages to user</p>
  
  <ul>
  <li><a href="http://docs.celeryproject.org/en/latest/userguide/monitoring.html" rel="noreferrer">http://docs.celeryproject.org/en/latest/userguide/monitoring.html</a></li>
  </ul>
  
  <p>pusher I like but I don't want to have <code>compute_module.py</code> deal with it. That is For example I would prefer not to do any pusher.com integration inside <code>compute_module.py</code>. Guess I could pass a pusher object that has already been instantiated so the module can just push messages but again I would prefer it to be generic</p>
  
  <ul>
  <li><a href="https://blog.pusher.com/improve-user-experience-app-real-time-progress-bar-tutorial/" rel="noreferrer">https://blog.pusher.com/improve-user-experience-app-real-time-progress-bar-tutorial/</a></li>
  <li><a href="https://blog.pusher.com/django-pusherable/" rel="noreferrer">https://blog.pusher.com/django-pusherable/</a></li>
  </ul>