title: Trying to response Amazon's Captcha with scrapy, strange behavior on spider generator
body: <p>I'm creating a crawler to Amazon for study reason, but it is being caught by their captcha.
  So I have made a captcha solver, but I'm having trouble to response the captcha form.
  The problem is that if I put a yeild FormRequest in the method, it seems to not be called.</p>
  
  <pre class="lang-py prettyprint-override"><code>class Havaianas2Spider(scrapy.Spider):
      name = 'coleta_dados_grafo'
      rank_path = sorted([x for x in os.listdir('scraps') if 'links_base' in x], reverse=True)[0]
      lista_links = pd.read_csv('scraps/' + rank_path)
      start_urls = lista_links['links'].values
      custom_settings = {'FEED_URI': "scraps/produtos_%(time)s.csv",
                         'FEED_FORMAT': 'csv'}
      final_path = '/?th=1&amp;psc=1'
  
      def base_path_get(self, response):
          dp_idx = response.request.url.find('/dp/') + 4
          base_path = response.request.url[:dp_idx]
          return base_path
  
      def solve_captcha(self, response, origin_method):
          self.logger.info('SOLVING CAPTCHA!')
          captcha_url = response.xpath('//div[@class="a-row a-text-center"]/img/@src').extract_first()
          img = load_url(captcha_url)
          captcha_string = break_captcha(img)
          img.save('C:/Users/Bruno Aquino/Documents/ecom_scraper/amazon_scraper/amazon_scraper/captchas/{}.jpg'.format(
              captcha_string))
          yield FormRequest.from_response(response,
                                          formdata={'field-keywords': captcha_string},
                                          callback=origin_method)
  
      def verify_if_captcha(self, response):
          captcha_url = response.xpath('//div[@class="a-row a-text-center"]/img/@src').extract_first()
  
          if captcha_url:
              self.logger.info('PAGE {} GOT BY CAPTCHA!'.format(response.request.url))
              return True
          else:
              return False
  
      def parse(self, response):
  
          captcha = self.verify_if_captcha(response)
  
          if captcha:
              self.solve_captcha(response, self.parse)
  
          else:
              base_path = self.base_path_get(response)
  
              asin_colors = response.xpath('//div[@id="cerberus-data-metrics"]/@data-asin').extract() +\
              [x[4:14] for x in response.xpath('//li[contains(@id,"color_name_")]/@data-dp-url').extract() if '/dp/' in x]
  
              for asin in asin_colors:
                  new_path = base_path + asin + self.final_path
  
                  if asin:
                      yield scrapy.Request(
                          response.urljoin(new_path),
                          callback=self.parse_l2)
  </code></pre>
  
  <p>below the Amazon's captcha form</p>
  
  <pre class="lang-html prettyprint-override"><code>&lt;form method="get" action="/errors/validateCaptcha" name=""&gt;
                              &lt;input type=hidden name="amzn" value="Xnnhl7YtGcH60X2yPaN7eA==" /&gt;&lt;input type=hidden name="amzn-r" value="&amp;#047;Capodarte&amp;#045;Chinelo&amp;#045;Preto&amp;#045;38&amp;#047;dp&amp;#047;B07N13Q5F2&amp;#047;?th&amp;#061;1&amp;amp;psc&amp;#061;1" /&gt;
                              &lt;div class="a-row a-spacing-large"&gt;
                                  &lt;div class="a-box"&gt;
                                      &lt;div class="a-box-inner"&gt;
                                          &lt;h4&gt;Type the characters you see in this image:&lt;/h4&gt;
                                          &lt;div class="a-row a-text-center"&gt;
                                              &lt;img src="https://images-na.ssl-images-amazon.com/captcha/yniigayf/Captcha_kbknwlcmvm.jpg"&gt;
                                          &lt;/div&gt;
                                          &lt;div class="a-row a-spacing-base"&gt;
                                              &lt;div class="a-row"&gt;
                                                  &lt;div class="a-column a-span6"&gt;
                                                      &lt;label for="captchacharacters"&gt;Type characters&lt;/label&gt;
                                                  &lt;/div&gt;
                                                  &lt;div class="a-column a-span6 a-span-last a-text-right"&gt;
                                                      &lt;a onclick="window.location.reload()"&gt;Try different image&lt;/a&gt;
                                                  &lt;/div&gt;
                                              &lt;/div&gt;
                                              &lt;input autocomplete="off" spellcheck="false" id="captchacharacters" name="field-keywords" class="a-span12" autocapitalize="off" autocorrect="off" type="text"&gt;
                                          &lt;/div&gt;
                                      &lt;/div&gt;
                                  &lt;/div&gt;
                              &lt;/div&gt;
  
                              &lt;div class="a-section a-spacing-extra-large"&gt;
  
                                  &lt;div class="a-row"&gt;
                                      &lt;span class="a-button a-button-primary a-span12"&gt;
                                          &lt;span class="a-button-inner"&gt;
                                              &lt;button type="submit" class="a-button-text"&gt;Continue shopping&lt;/button&gt;
                                          &lt;/span&gt;
                                      &lt;/span&gt;
                                  &lt;/div&gt;
  
                              &lt;/div&gt;
                          &lt;/form&gt;
  </code></pre>
  
  <p>I put two logs into the code.</p>
  
  <p>The first on inside the verify_if_captcha:</p>
  
  <pre class="lang-py prettyprint-override"><code>self.logger.info('PAGE {} GOT BY CAPTCHA!'.format(response.request.url))
  </code></pre>
  
  <p>This one is printed</p>
  
  <p>The second inside the solve_captcha:</p>
  
  <pre class="lang-py prettyprint-override"><code>self.logger.info('SOLVING CAPTCHA!')
  </code></pre>
  
  <p>This one is never printed</p>
  
  <p>Can some one help me, please?</p>