title: making matplotlib scatter plots from dataframes in Python's pandas
body: <p>What is the best way to make a series of scatter plots using <code>matplotlib</code> from a <code>pandas</code> dataframe in Python? </p>
  
  <p>For example, if I have a dataframe <code>df</code> that has some columns of interest, I find myself typically converting everything to arrays:</p>
  
  <pre><code>import matplotlib.pylab as plt
  # df is a DataFrame: fetch col1 and col2 
  # and drop na rows if any of the columns are NA
  mydata = df[["col1", "col2"]].dropna(how="any")
  # Now plot with matplotlib
  vals = mydata.values
  plt.scatter(vals[:, 0], vals[:, 1])
  </code></pre>
  
  <p>The problem with converting everything to array before plotting is that it forces you to break out of dataframes.</p>
  
  <p>Consider these two use cases where having the full dataframe is essential to plotting:</p>
  
  <ol>
  <li><p>For example, what if you wanted to now look at all the values of <code>col3</code> for the corresponding values that you plotted in the call to <code>scatter</code>, and color each point (or size) it by that value? You'd have to go back, pull out the non-na values of <code>col1,col2</code> and check what their corresponding values.</p>
  
  <p>Is there a way to plot while preserving the dataframe? For example:</p>
  
  <pre><code>mydata = df.dropna(how="any", subset=["col1", "col2"])
  # plot a scatter of col1 by col2, with sizes according to col3
  scatter(mydata(["col1", "col2"]), s=mydata["col3"])
  </code></pre></li>
  <li><p>Similarly, imagine that you wanted to filter or color each point differently depending on the values of some of its columns. E.g. what if you wanted to automatically plot the labels of the points that meet a certain cutoff on <code>col1, col2</code> alongside them (where the labels are stored in another column of the df), or color these points differently, like people do with dataframes in R.  For example:</p>
  
  <pre><code>mydata = df.dropna(how="any", subset=["col1", "col2"]) 
  myscatter = scatter(mydata[["col1", "col2"]], s=1)
  # Plot in red, with smaller size, all the points that 
  # have a col2 value greater than 0.5
  myscatter.replot(mydata["col2"] &gt; 0.5, color="red", s=0.5)
  </code></pre></li>
  </ol>
  
  <p>How can this be done?</p>
  
  <p><strong>EDIT</strong> Reply to crewbum:</p>
  
  <p>You say that the best way is to plot each condition (like <code>subset_a</code>, <code>subset_b</code>) separately. What if you have many conditions, e.g. you want to split up the scatters into 4 types of points or even more, plotting each in different shape/color. How can you elegantly apply condition a, b, c, etc. and make sure you then plot "the rest" (things not in any of these conditions) as the last step? </p>
  
  <p>Similarly in your example where you plot <code>col1,col2</code> differently based on <code>col3</code>, what if there are NA values that break the association between <code>col1,col2,col3</code>? For example if you want to plot all <code>col2</code> values based on their <code>col3</code> values, but some rows have an NA value in either <code>col1</code> or <code>col3</code>, forcing you to use <code>dropna</code> first. So you would do:</p>
  
  <pre><code>mydata = df.dropna(how="any", subset=["col1", "col2", "col3")
  </code></pre>
  
  <p>then you can plot using <code>mydata</code> like you show -- plotting the scatter between <code>col1,col2</code> using the values of <code>col3</code>. But <code>mydata</code> will be missing some points that have values for <code>col1,col2</code> but are NA for <code>col3</code>, and those still have to be plotted... so how would you basically plot "the rest" of the data, i.e. the points that are <em>not</em> in the filtered set <code>mydata</code>?</p>