title: How to create an api with FastAPI and scrapy?
body: <p>I am working on a web application project which will allow user to search from keywords and get a set of results based on the keywords entered from scraping. For this I use scrapy for scraping results from a web search engine. I wrote some code to pass the keywords to the scrapy file and display the scrapy results on a webpage. However, I'm having trouble passing the keywords to scrapy using FastAPI, because when I run my api code, I always get a set of errors from Scrapy. Here is the gist containing the runtime terminal <a href="https://gist.github.com/gouneken/8542b1c5228c6f80490cf3e6a15e1110" rel="nofollow noreferrer">output</a>. I don't understand the problem, yet my scrapy code was working perfectly before I connected it to the api, I'm a beginner on creating APIs, so I ask for your help. Here is the code for my web scraper:</p>
  <pre><code>import scrapy
  import datetime
  from requests_html import HTMLSession 
  
  class PagesearchSpider(scrapy.Spider):
  
      name = 'pageSearch'
  
  
      def start_requests(self,query):
          #queries = [ 'investissement']
          #for query in queries:
              url = f'https://www.ask.com/web?q={query}'
              s = HTMLSession()
              r = s.get
              print(r.status_code)
              qlist[&quot;&quot;]
              
              yield scrapy.Request(url, callback=self.parse, meta={'pos': 0})
  
  
  
  
      def parse(self, response):
  
          print('url:', response.url)
          
          start_pos = response.meta['pos']
          print('start pos:', start_pos)
  
          dt = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')    
          
          items = response.css('div.PartialSearchResults-item')
          
          for pos, result in enumerate(items, start_pos+1):
              yield {
                  'title':    result.css('a.PartialSearchResults-item-title-link.result-link::text').get().strip(), 
                  'snippet':  result.css('p.PartialSearchResults-item-abstract::text').get().strip(), 
                  'link':     result.css('a.PartialSearchResults-item-title-link.result-link').attrib.get('href'), 
                  'position': pos, 
                  'date':     dt,
              }
  
              qlist.append(items)
  
          # --- after loop ---
          
          next_page = response.css('.PartialWebPagination-next a')
          
          if next_page:
              url = next_page.attrib.get('href')
              print('next_page:', url)  # relative URL
              # use `follow()` to add `https://www.ask.com/` to URL and create absolute URL
              yield response.follow(url, callback=self.parse, meta={'pos': pos+1})
  
  
  # --- run without project, and save in file ---
  
  from scrapy.crawler import CrawlerProcess
  
  c = CrawlerProcess({
      #'USER_AGENT': 'Mozilla/5.0',
      # save in file CSV, JSON or XML
      'FEEDS': {'test.json': {'format': 'json'}},
      #'ROBOTSTXT_OBEY': True,  # this stop scraping
  })
  c.crawl(PagesearchSpider)
  c.start() 
  </code></pre>
  <p>the code allowing the functioning of my api:</p>
  <pre><code>from fastapi import FastAPI
  from script import PagesearchSpider
  
  app = FastAPI()
  request = PagesearchSpider()
  
  @app.get(&quot;/{cat}&quot;)
  async def read_item(cat):
      return request.start_requests('cat')
  
  </code></pre>
  <p>I changed part of my scraper code, viz:</p>
  <pre><code>def start_requests(self,query):
          #queries = [ 'investissement']
          #for query in queries:
              url = f'https://www.ask.com/web?q={query}'
              
              if __name__ == '__main__':
                                          s = HTMLSession()
                                          r = s.get
                                          print(r.status_code)
                                          qlist[&quot;&quot;]
  </code></pre>
  <p>but I still get the same errors. Unless I got the wrong crawler code, I'm not very good at scrapy
  I also rewrote the code below my function...</p>
  <pre><code># --- run without project, and save in file ---
  if __name__ == &quot;__main__&quot;:
      from scrapy.crawler import CrawlerProcess
      c = CrawlerProcess({
      #'USER_AGENT': 'Mozilla/5.0',
      # save in file CSV, JSON or XML
      #'FEEDS': {'test.json': {'format': 'json'}},
      #'ROBOTSTXT_OBEY': True,  # this stop scraping
  })
      c.crawl(PagesearchSpider)
      c.start() 
  </code></pre>
  <p>and I executed this command <code>python -m uvicorn main:app --reload</code> to start server and I had the following results in the command line :</p>
  <pre><code>←[32mINFO←[0m:     Will watch for changes in these directories: ['C:\\Users\\user\\Documents\\AAprojects\\Whelpsgroups1\\searchApi\\apiFast']
  ←[32mINFO←[0m:     Uvicorn running on ←[1mhttp://127.0.0.1:8000←[0m (Press CTRL+C to quit)
  ←[32mINFO←[0m:     Started reloader process [←[36m←[1m10956←[0m] using ←[36m←[1mstatreload←[0m
  ←[33mWARNING←[0m:  The --reload flag should not be used in production on Windows.
  ←[32mINFO←[0m:     Started server process [←[36m6724←[0m]
  ←[32mINFO←[0m:     Waiting for application startup.
  ←[32mINFO←[0m:     Application startup complete.
  ←[33mWARNING←[0m:  StatReload detected file change in 'main.py'. Reloading...
  ←[33mWARNING←[0m:  The --reload flag should not be used in production on Windows.
  ←[32mINFO←[0m:     Started server process [←[36m2720←[0m]
  ←[32mINFO←[0m:     Waiting for application startup.
  ←[32mINFO←[0m:     Application startup complete.
  
  </code></pre>
  <p>But when I click from the command line on the link of the address my server started at, it opens my file explorer on windows 10 and when I manually write that link i.e. <code>http://127.0.0.0:8000/</code> in my browser search bar it says <code>127.0.0.0</code> took too long to respond. Yet I didn't change any of my files, just the command I was using in console line so I don't know why this error.</p>
  <p>I looked at the stack overflow questions, but they weren't directly related to the problem of difficulty sharing data between an api and a web scraper and on the internet I couldn't find any relevant answers. So I hope you could help me, I look forward to your answers, thank you!</p>