title: How to schedule my crawler function in django periodically using celery?
body: <p>Here I have a view <code>CrawlerHomeView</code> which is used to create the task object from a form now I want to schedule this task periodically with celery.</p>
  
  <p>I want to schedule this <code>CrawlerHomeView</code> process with the  task object search_frequency and by checking some task object fields. </p>
  
  <p>Task Model</p>
  
  <pre><code>class Task(models.Model):
      INITIAL = 0
      STARTED = 1
      COMPLETED = 2
  
      task_status = (
          (INITIAL, 'running'),
          (STARTED, 'running'),
          (COMPLETED, 'completed'),
          (ERROR, 'error')
      )
  
      FREQUENCY = (
          ('1', '1 hrs'),
          ('2', '2 hrs'),
          ('6', '6 hrs'),
          ('8', '8 hrs'),
          ('10', '10 hrs'),
      )
  
      name = models.CharField(max_length=255)
      scraping_end_date = models.DateField(null=True, blank=True)
      search_frequency = models.CharField(max_length=5, null=True, blank=True, choices=FREQUENCY)
      status = models.IntegerField(choices=task_status)
  </code></pre>
  
  <p>tasks.py</p>
  
  <p>I want to run the view below posted periodically [period=(task's search_frequency time] if the task status is 0 or 1 and not crossed the task scraping end date. But I got stuck here. How can I do this?</p>
  
  <pre><code>@periodic_task(run_every=crontab(hour="task.search_frequency"))  # how to do  with task search_frequency value
  def schedule_task(pk):
      task = Task.objects.get(pk=pk)
      if task.status == 0 or task.status == 1 and not datetime.date.today() &gt; task.scraping_end_date:
  
          # perform the crawl function ---&gt; def crawl() how ??
  
          if task.scraping_end_date == datetime.date.today():
              task.status = 2
              task.save()  # change the task status as complete.
  </code></pre>
  
  <p>views.py</p>
  
  <p>I want to run this view periodically.How can I do it?</p>
  
  <pre><code>class CrawlerHomeView(LoginRequiredMixin, View):
      login_url = 'users:login'
  
      def get(self, request, *args, **kwargs):
          # all_task = Task.objects.all().order_by('-id')
          frequency = Task()
          categories = Category.objects.all()
          targets = TargetSite.objects.all()
          keywords = Keyword.objects.all()
  
          form = CreateTaskForm()
          context = {
              'targets': targets,
              'keywords': keywords,
              'frequency': frequency,
              'form':form,
              'categories': categories,
          }
          return render(request, 'index.html', context)
  
      def post(self, request, *args, **kwargs):
  
          form = CreateTaskForm(request.POST)
          if form.is_valid():
  
              # try:
              unique_id = str(uuid4()) # create a unique ID. 
              obj = form.save(commit=False)
  
              # obj.keywords = keywords
              obj.created_by = request.user
              obj.unique_id = unique_id
              obj.status = 0
              obj.save()
              form.save_m2m()
  
              keywords = ''
              # for keys in ast.literal_eval(obj.keywords.all()): #keywords change to csv
              for keys in obj.keywords.all():
                  if keywords:
                      keywords += ', ' + keys.title
                  else:
                      keywords += keys.title
              # tasks = request.POST.get('targets')
              # targets = ['thehimalayantimes', 'kathmandupost']
              # print('$$$$$$$$$$$$$$$ keywords', keywords)
  
              task_ids = [] #one Task/Project contains one or multiple scrapy task
  
              settings = {
                  'spider_count' : len(obj.targets.all()),
                  'keywords' : keywords,
                  'unique_id': unique_id, # unique ID for each record for DB
                  'USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
              }
  
              # res = ast.literal_eval(ini_list) 
  
              for site_url in obj.targets.all():
                  domain = urlparse(site_url.address).netloc # parse the url and extract the domain
                  spider_name = domain.replace('.com', '')
                  task = scrapyd.schedule('default', spider_name, settings=settings, url=site_url.address, domain=domain, keywords=keywords)
  
              # task = scrapyd.schedule('default', spider_name , settings=settings, url=obj.targets, domain=domain, keywords=obj.keywords)
              return redirect('crawler:task-list')
              # except:
              #     return render(request, 'index.html', {'form':form})
          return render(request, 'index.html', {'form':form, 'errors':form.errors})
  </code></pre>
  
  <p><strong>Any Suggestions or answer is there for this problem ?</strong></p>