title: FlickrAPI only returns an incomplete number of results
body: <p>My goal is to extract geodata (lat and lon values), views, photo id's, url's and the date posted from the Flickr database inside the geographic boundaries of the city of Cologne, Germany. The data is then written in a csv file. The total number of results using just <code>tags='Köln'</code> is around 110.000. I want to extract <em>at least</em> a 5-digit number of data points from that. To achieve that, I set three delimiters: the tag, the maximum upload date and the minimum upload date.</p>
  <p>What already works: The data is successfully written into the csv.</p>
  <p>What does not work yet: When I return the search results using <code>xml.etree.ElementTree.dump()</code>, I can see that around 3,700 results are found for the respective search parameters. As far as I know, this number is inside the limit of 4,000 results per query set by Flickr. However, only between 700 and 1,000 data points are written into the csv file. The number is never the same and varies with every execution, which is weird, because I clearly defined the time frame. Also, despite adding a timer between calls using <code>time.sleep(1)</code> I still get kicked out by the server from time to time (error code 500). After struggling a lot with the barely documented limits, I really can't tell why my code is still not working as intended.</p>
  <p>The code I used is as follows:</p>
  <pre><code>import flickrapi
  import os
  import datetime
  import time
  
  ## Only needed to explore the xml tree
  # import xml 
  
  ## API key and secret provided by Flickr
  api_key = 'api key'
  api_secret = 'api secret'
  
  ## Approximate geographic coordinates of the administrational boundaries of the city of Cologne
  boundaries = '6.8064182,50.8300729,7.1528453,51.0837915'
  
  ## Counter for the ID column required by GIS software
  id_count = 1
  
  ## Creation of an editable csv file and its top row
  csv = open('flickr_data.csv', mode='a')
  if(os.stat('flickr_data.csv').st_size == 0):
      csv.write('ID,Photo_ID,Lat,Lon,Views,Taken_Unix,Taken,URL \n')
  
  ## Authentication of the Flickr API
  flickr = flickrapi.FlickrAPI(api_key, api_secret)
  
  ## Page counter
  page_number = 1
  
  ## Only needed to explore the xml tree
  # test_list = flickr.photos_search(max_upload_date = '2020-07-09 23:59:59',min_upload_date = '2020-01-15 0:00:00',tags = 'Köln',bbox = boundaries,has_geo = '1',page = 1,extras = 'views',per_page = '250')
  # xml.etree.ElementTree.dump(test_list)
  
  ## While loop keeps running until page 16 is reached. The total number of pages for the wanted search query is 452.
  ## However, Flickr only returns a number of photos equivalent to 16 pages of 250 results.
  ## At this point, the code is reiterated until the maximum number of pages is reached.
  while page_number &lt; 17:
      
      ## Flickr search for the geographic boundaries of Cologne, Germany.
      photo_list = flickr.photos_search(tags = 'Köln',
                                        max_upload_date = '2020-07-09 23:59:59',
                                        min_upload_date = '2020-01-15 00:00:00',
                                        bbox = boundaries,
                                        has_geo = '1',
                                        page = page_number,
                                        extras = 'views',
                                        per_page = '250') ## maximum allowed photos per page for bbox-delimited requests
      
      ## For loop keeps running as long as there are photos on a page
      for photo in photo_list[0]:
          ## extraction of latitude and longitude data from the search results
          geodata = flickr.photos_geo_getLocation(photo_id = photo.attrib['id'])
          lat = geodata[0][0].attrib['latitude']
          lon = geodata[0][0].attrib['longitude']
          
          ## extraction of views from the search results
          views = photo.get('views')
  
          ## extraction and conversion of upload dates
          photo_info = flickr.photos.getInfo(photo_id = photo.attrib['id'])
          date_unix = int(photo_info[0][4].attrib['posted'])
          date = datetime.datetime.utcfromtimestamp(date_unix).strftime('%Y-%m-%d %H:%M:%S')
          url = 'https://www.flickr.com/photos/' + photo.attrib['owner'] + '/' + photo.attrib['id']
  
          
          
          ## the csv is filled with the acquired information
          csv.write('%s,%s,%s,%s,%s,%s,%s,%s \n' % (id_count,
                                                    photo.attrib['id'],
                                                    lat,
                                                    lon,
                                                    views,
                                                    date_unix,
                                                    date,
                                                    url))
          id_count += 1
          ## 1 second wait time between calls to prevent error code 500
          time.sleep(1)
      ## Turns the page
      page_number += page_number
  
  ## Total number of photos searched
  print(sum(1 for line in open(flickr_data.csv))-1)
  
  csv.close()
  </code></pre>
  <p>The following is an excerpt of the xml that gets returned by <code>flickr.photos_search</code></p>
  <pre><code>&lt;rsp stat=&quot;ok&quot;&gt;
  &lt;photos page=&quot;1&quot; pages=&quot;16&quot; perpage=&quot;250&quot; total=&quot;3755&quot;&gt;
      &lt;photo id=&quot;50094525552&quot; owner=&quot;98355876@N00&quot; secret=&quot;6d66d421af&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;-&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;250&quot; /&gt;
      &lt;photo id=&quot;50093709173&quot; owner=&quot;98355876@N00&quot; secret=&quot;90c31cac1d&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;-&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;260&quot; /&gt;
      &lt;photo id=&quot;50093706783&quot; owner=&quot;98355876@N00&quot; secret=&quot;9521b8ba7d&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;-&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;224&quot; /&gt;
      &lt;photo id=&quot;50093641658&quot; owner=&quot;82692690@N02&quot; secret=&quot;e26afb1e79&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;Cabecera. Catedral gótica de Colonia. JX3.&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;201&quot; /&gt;
      &lt;photo id=&quot;50090280721&quot; owner=&quot;98355876@N00&quot; secret=&quot;cc0e2d7b8b&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;-&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;295&quot; /&gt;
      &lt;photo id=&quot;50090278631&quot; owner=&quot;98355876@N00&quot; secret=&quot;8113aaa628&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;-&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;280&quot; /&gt;
      &lt;photo id=&quot;50090277186&quot; owner=&quot;98355876@N00&quot; secret=&quot;73753c811d&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;-&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;320&quot; /&gt;
      &lt;photo id=&quot;50090150901&quot; owner=&quot;136678496@N04&quot; secret=&quot;6de14ca572&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;Good Morning&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;104&quot; /&gt;
      &lt;photo id=&quot;50089819277&quot; owner=&quot;7283893@N05&quot; secret=&quot;43e5290b07&quot; server=&quot;65535&quot; farm=&quot;66&quot; title=&quot;Der Chef / The Boss&quot; ispublic=&quot;1&quot; isfriend=&quot;0&quot; isfamily=&quot;0&quot; views=&quot;421&quot; /&gt;
  </code></pre>
  <p>The following is the output of the script with the ID count printed at the end of every for loop and the page number printed with every while loop:</p>
  <pre><code>1
  2
  3
  4
  5
  6
  7
  8
  9
  
  (...)
  
  245
  246
  247
  248
  249
  250
  -------- PAGE 2 --------
  251
  252
  253
  254
  255
  256
  257
  
  (...)
  
  493
  494
  495
  496
  497
  498
  499
  500
  -------- PAGE 4 --------
  501
  502
  503
  504
  505
  506
  507
  
  (...)
  
  743
  744
  745
  746
  747
  748
  749
  750
  -------- PAGE 8 --------
  751
  752
  753
  754
  755
  756
  757
  758
  759
  
  (...)
  
  990
  991
  992
  993
  994
  995
  996
  997
  998
  999
  1000
  -------- PAGE 16 --------
  1001
  1002
  1003
  1004
  1005
  -------- PAGE 32 --------
  </code></pre>