title: Optimize returning top results from intersecting Sorted Set and Set in Redis
body: <p>I am trying to optimize my Redis code and I am currently having some difficulties scaling my solution. Redis is hosted on Redis Cloud, the app on Heroku and I am using Ruby for my language. </p>
  
  <p><strong>My Redis setup:</strong></p>
  
  <p>I have a series of sorted sets which each contains approx 1,000 scored members and a corresponding set (which could be a hash, string, list, set, sorted set or any other structure) for each user in the system. </p>
  
  <p>For example in the news:sports key I have the following structure. Other keys examples are news:entertainment, news:business.</p>
  
  <p><em>news:sports -- Member Score</em></p>
  
  <pre><code>StoryOID1 1000
  StoryOID2 999
  StoryOID3 998
  StoryOID4 997
  StoryOID5 996
  ...
  </code></pre>
  
  <p>I also have a user specific key (a set) for each of the master sorted sets (user1:news:sports) that contain a list of stories the user has already seen. ie </p>
  
  <pre><code>seen:user1:sports StoryOID2 
  </code></pre>
  
  <p><strong>My challenge:</strong></p>
  
  <p>On each user request, I need to return the top 3 members (with the highest scores, although I don't need to know the scores) from a specific sorted set that the user hasn't already seen. I prefer not to save the result in Redis as have no long term use and I will just have to delete the key. </p>
  
  <p>Given the example above, User1 requests sports:news, I would return:</p>
  
  <pre><code>StoryOID1
  StoryOID3
  StoryOID4
  </code></pre>
  
  <p>As part of my code, I loop through 10 sorted sets (10 genres) returning the top 3 unseen stories from each for a total of 30 OIDS being returned for each request. </p>
  
  <p>For each user request:</p>
  
  <pre><code>Do this 10 times:         
       ZRANGEBYSCORE top 24 members
       loop user genres key using SISMEMBER until I return 3 unseen members
  end
  </code></pre>
  
  <p>Benchmarking at 60 dynos (heroku) on I can only get to 500 simultaneous connections (and at 1000 ms response time) with the Redis loop below being the bottleneck. I am aiming to scale a few multiples over my current setup. Anything and Everything can be change to scale this process. </p>
  
  <p>My Current Process (in Ruby):</p>
  
  <pre><code>def newslist (userOID,genres)
  
      #pull top 24 stories for the given news:genres -- 24 could be replaced by 1,3,6,12 etc
      newsscores = @@redis.zrevrangebyscore("news:#{genres}", "+inf", "-inf", :limit =&gt; [0, 24],:with_scores =&gt; true)  
  
      newsstories = Array.new(3)
      i = 0 #news acceptance counter
      loopcnt = 0 #loop counter
      while i &lt; 3 
        if newsscores.count == loopcnt - 1 #loop to the max number of news returned in news news
          break #breakout of loop
        end
        seen = @@redis.sismember("seen:#{userOID}:#{genres}", newsscores[loopcnt][0])
        if seen == false
          newsstories[i] = newsscores[loopcnt][0]
          i+=1
        end
        loopcnt += 1
      end
      if i==3
        return newsstories #return 3 news newss
      else
        return 0 #return 0 -- this should cause a repick
      end
      return 0 #return 0 -- this should cause a repick
  end
  </code></pre>
  
  <p>I know I am paying a large costs the large number of Redis requets. My current thought process is to basically translate the above into a Lua script I can run on the server side but I can't help but feel there is a more elegant solution that will scale far better.</p>
  
  <p>Is there a better way?</p>