title: What's the fastest way to acces a Pandas DataFrame?
body: <p>I have a DataFrame df with 541 columns, and I need to save all unique pairs of its column names into the rows of a separate DataFrame, repeated 8 times each.</p>
  
  <p>I thought I would create an empty DataFrame fp, double loop through df's column names, insert into every 8th row, and fill in the blanks with the last available value.</p>
  
  <p>When I tried to do this though I was baffled by how long it's taking. With 541 columns I only have to write 146,611 times yet it's taking well over 20 minutes. This seems egregious for just data access. Where is the problem and how can I solve it? It takes less time than that for Pandas to produce a correlation matrix with the columns so I must me doing something wrong.</p>
  
  <p>Here's a reproducible example of what I mean:</p>
  
  <pre><code>fp = np.empty(shape = (146611, 10))
  fp.fill(np.nan)
  
  fp = pd.DataFrame(fp)
  
  %timeit for idx in range(0, len(fp)): fp.iloc[idx, 0] = idx
  
  # 1 loop, best of 3: 22.3 s per loop
  </code></pre>