title: What's the best way to count unique visitors with Hadoop?
body: <p>hey all, just getting started on hadoop and curious what the best way in mapreduce would be to count unique visitors if your logfiles looked like this...</p>
  
  <pre><code>DATE       siteID  action   username
  05-05-2010 siteA   pageview jim
  05-05-2010 siteB   pageview tom
  05-05-2010 siteA   pageview jim
  05-05-2010 siteB   pageview bob
  05-05-2010 siteA   pageview mike
  </code></pre>
  
  <p>and for each site you wanted to find out the unique visitors for each site?</p>
  
  <p>I was thinking the mapper would emit siteID \t username
  and the reducer would keep a set() of the unique usersnames per key and then emit the length of that set. However that would be potentially storing millions of usernames in memory which doesn't seem right. Anyone have a better way?</p>
  
  <p>I'm using python streaming by the way</p>
  
  <p>thanks</p>