title: Scrapy - making selections from dropdown(e.g.date) on webpage
body: <p>I'm new to scrapy and python and i am trying to scrape data off the following start url .</p>
  
  <p>After login, this is my start url---> </p>
  
  <p>start_urls = ["<a href="http://www.flightstats.com/go/HistoricalFlightStatus/flightStatusByFlight.do" rel="nofollow">http://www.flightstats.com/go/HistoricalFlightStatus/flightStatusByFlight.do</a>?"]</p>
  
  <p>(a) from there i need to interact with the webpage to select ---by-airport---</p>
  
  <p>and then make ---airport, date, time period selection---</p>
  
  <p>how can i do that? i would like to loop over all time periods and past dates..</p>
  
  <ul>
  <li><p>I have used firebug to see the source, I cannot show here as I do not have enough points to post images..</p></li>
  <li><p>i read a post mentioning the use of Splinter..</p></li>
  </ul>
  
  <p>(b) after the selections it will lead me to a page where there are links to the eventual page with the information i want. How do i populate the links and make scrapy look into every one to extract the information?</p>
  
  <p>-using rules? where should i insert the rules/ linkextractor function?</p>
  
  <p>I am willing to try myself, hope help can be given to find posts that can guide me.. I am a student and I have spent more than a week on this.. I have done the scrapy tutorial, python tutorial, read the scrapy documentation and searched for previous posts in stackoverflow but I did not manage to find posts that cover this.</p>
  
  <p>a million thanks.</p>
  
  <p>my code so far to log-in and the items to scrape via xpath from the eventual target site:</p>
  
  <pre><code>`import scrapy
  
  from tutorial.items import FlightItem
  from scrapy.http import FormRequest
  
  class flightSpider(scrapy.Spider):
      name = "flight"
      allowed_domains = ["flightstats.com"]
      login_page =     'https://www.flightstats.com/go/Login/login_input.do;jsessionid=0DD6083A334AADE3FD6923ACB8DDCAA2.web1:8009?'
      start_urls = [
      "http://www.flightstats.com/go/HistoricalFlightStatus/flightStatusByFlight.do?"]
  
  def init_request(self):
      #"""This function is called before crawling starts."""
      return Request(url=self.login_page, callback=self.login)
  
  def login(self, response):
   #"""Generate a login request."""
      return FormRequest.from_response(response,formdata= {'loginForm_email': 'marvxxxxxx@hotmail.com', 'password': 'xxxxxxxx'},callback=self.check_login_response)
  
  def check_login_response(self, response):
          #"""Check the response returned by a login request to see if we aresuccessfully logged in."""
      if "Sign Out" in response.body:
          self.log("\n\n\nSuccessfully logged in. Let's start crawling!\n\n\n")
           # Now the crawling can begin..
  
      return self.initialized() # ****THIS LINE FIXED THE LAST PROBLEM*****
  
      else:
              self.log("\n\n\nFailed, Bad times :(\n\n\n")
           # Something went wrong, we couldn't log in, so nothing happens.
  
  def parse(self, response):
      for sel in response.xpath('/html/body/div[2]/div[2]/div'):
          item = flightstatsItem()
          item['flight_number'] = sel.xpath('/div[1]/div[1]/h2').extract()
          item['aircraft_make'] = sel.xpath('/div[4]/div[2]/div[2]/div[2]').extract()
          item['dep_date'] = sel.xpath('/div[2]/div[1]/div').extract()
          item['dep_airport'] = sel.xpath('/div[1]/div[2]/div[2]/div[1]').extract()
          item['arr_airport'] = sel.xpath('/div[1]/div[2]/div[2]/div[2]').extract()
          item['dep_gate_scheduled'] = sel.xpath('/div[2]/div[2]/div[1]/div[2]/div[2]').extract()
          item['dep_gate_actual'] = sel.xpath('/div[2]/div[2]/div[1]/div[3]/div[2]').extract()
          item['dep_runway_actual'] = sel.xpath('/div[2]/div[2]/div[2]/div[3]/div[2]').extract()
          item['dep_terminal'] = sel.xpath('/div[2]/div[2]/div[3]/div[2]/div[1]').extract()
          item['dep_gate'] = sel.xpath('/div[2]/div[2]/div[3]/div[2]/div[2]').extract()
          item['arr_gate_scheduled'] = sel.xpath('/div[3]/div[2]/div[1]/div[2]/div[2]').extract()
          item['arr_gate_actual'] = sel.xpath('/div[3]/div[2]/div[1]/div[3]/div[2]').extract()
          item['arr_terminal'] = sel.xpath('/div[3]/div[2]/div[3]/div[2]/div[1]').extract()
          item['arr_gate'] = sel.xpath('/div[3]/div[2]/div[3]/div[2]/div[2]').extract()
  
          yield item`
  </code></pre>