title: Python Pandas - Searching two dataframes for matching strings within a specific column
body: <p>thanks for taking the time to read my question.</p>
  
  <p>I'm working on a system that performs customized web traffic monitoring in real time. The system (in its current alpha iteration) takes a .txt dump of traffic every few minutes. I'll call it df_webtraffic. This .txt has several columns, one of which is the url field for the web traffic. What I need to do is, for each row's url entry (df_webtraffic.url), scan another df (or series, I don't know how much it matters since it's just one column) of prohibited websites (I'll call it blacklist). The blacklist is large (nearly a million entries), so efficiency is highly important.</p>
  
  <p>I've read dozens of forum posts, but most topics seem related to finding a specific string rather than finding a match within a series. How do I scan a column in one datafield and look for corresponding matches in another datafield or series? </p>
  
  <p>I've written this snippet: <code>print y[y['URL'].str.contains[(blacklist)]</code> and it appears to work... but it never completes. It's just processing... and processing... and processing - which I would expect, but system monitor shows that my processor is hardly ever breaking 5% for the python process, never more than 10%, and sits at 0% regularly. For something this demanding I'd expect it to be consuming a lot of CPU.</p>
  
  <p>Thoughts?</p>
  
  <p>Drew</p>
  
  <p>UPDATE:</p>
  
  <p>One thing I wasn't clear on is that a domain in a blacklist isn't always going to be an exact match to a clicked url - someone may click a url that directs to a subdirectory within a domain (like www.website.com/subdir/link.html), however the blacklist entry that I need to correspond this to may only be www.website.com. Thus, I need to see if an entry in the clicked url's contains any entry in the blacklist, even if it doesn't explicitly match it.</p>