title: Python Pandas and Generator to Process Lines in CSV
body: <p>Hoping that this is an allowable SO question but I am hoping to get some advice on how to convert the below code which processes lines in a file to produce a dataframe into one that uses generators and yields because this implementation using list and append is far too slow. </p>
  
  <p>Here is the solution I came up with but I was really hoping to avoid using very slow lists and append operation. I was hoping for cool generator and yield solution instead but not comfortable enough yet working with generators. </p>
  
  <p>Sample lines in file:</p>
  
  <pre><code>"USNC3255","27","US","NC","LANDS   END","72305006","KNJM","KNCA","KNKT","T72305006","","","NCC031","NCZ095","","545","28594","America/New_York","34.65266","-77.07661","7","RDU","893727","
  "USNC3256","27","US","NC","LANDSDOWN","72314058","KEHO","KAKH","KIPJ","T72314058","","","NCC045","NCZ068","sc007","517","28150","America/New_York","35.29374","-81.46537","797","CLT","317845","
  </code></pre>
  
  <p>Current Solution:</p>
  
  <pre><code>def parse_file(filename):
  
      newline = []
  
      with open(filename, 'rb') as f:
          reader = csv.reader(f, quoting=csv.QUOTE_NONE)
          for row in reader:
              newline.append([s.strip('"') for s in row[:-1]])
      df = pd.DataFrame(newline)
      df = df.applymap(lambda x: nan if len(x) == 0 else x).astype(object)
      return df
  
  df = parse_file(filename)
  </code></pre>
  
  <p>Output is just a dataframe with 23 columns and two rows if used against the sample lines above. </p>