title: Scrapy, await before pipeline
body: <p>(I'm new to scrapy).</p>
  <h2>Context</h2>
  <p>Suppose there is a site <code>https://example.com</code> and I want to scrape it.</p>
  <p>It is structured like this:</p>
  <pre><code>&lt;body&gt;
  &lt;ul&gt;
      &lt;li&gt;
          title_foo
          &lt;a href=&quot;https://example.com/title_foo&quot;&gt;a desription&lt;/a&gt;
      &lt;/li&gt;
      &lt;li&gt;
          title_bar
          &lt;a href=&quot;https://example.com/title_bar&quot;&gt;an another desription&lt;/a&gt;
      &lt;/li&gt;
  &lt;/ul&gt;
  &lt;/body&gt;
  </code></pre>
  <p>following the <code>&lt;a&gt;</code> link, I can get a description that I need before creating my items and send them to my pipeline which will store them into my db.</p>
  <p>For instance, let's say that when I follow <a href="https://example.com/title_foo" rel="nofollow noreferrer">https://example.com/title_foo</a>, I retreive that description into</p>
  <pre><code>&lt;div class=&quot;a-descrption&quot;&gt;
      a description
  &lt;/div&gt;
  </code></pre>
  <p>In <code>items.py</code>, I've:</p>
  <pre><code>class MyItem(scrapy.Item):
      title = scrapy.Field()
      description = scrapy.Field()
  </code></pre>
  <p>And my spider should looks like that:</p>
  <pre><code>import scrapy
  from scrapy_project.items import MyItem
  
  class MySpider(scrapy.Spider):
      name = &quot;my_spider&quot;
  
      def start_requests(self):
          urls = [
              'https://example.com',
          ]
          for url in urls:
              yield scrapy.Request(url=url, callback=self.parse)
  
      def parse(self, response):
          for li in response.xpath('//li/text()').getall()
              yield {&quot;title&quot;: li }
  </code></pre>
  <p>At least, I hope, I didn't tested it, correct me if something is wrong.</p>
  <h2>Question</h2>
  <p>There is a kind of &quot;await&quot; mechanism here. I mean, I need to tell to scrapy to wait that  link has been visited before yielding my item. That seems to be a common problem and I'm sure that some practices exists in order to solve that problem but wasn't able to found how.</p>
  <p>How do you do ?</p>