title: Python file.write slows down as the output file grows bigger
body: <p>I'm very new to Python, but I have an aching problem.
  I have received a program which reads an infile (text), changes some values, and writes an outfile (also text).
  As the outfile grows bigger, the writes get slower and slower, making it unbearably slow after some 2 MB.
  Why can this be? I have tried altering the code to use buffers of different sizes, and I have changed it to cache the data into larger chunks (a string) before writing. I also tried join instead of += to create the string to be written. NONE of these do any difference at all to performance - except writing bigger chunks, which actually made the code SLOWER.(!!!)</p>
  
  <p>Here is the method that writes the outfile. I moved the write portion from a separate method to inline:</p>
  
  <pre><code>for ifile in _file_stripper(f_in):
      parse_infile(ifile)
      date = variable_data['arkiveringsdatum']
      variable_data['arkiveringsdatum'] = datetime( int(date[0:4]), int(date[4:6]), int(date[6:8]), tzinfo=TZ()).isoformat('T')
      _personnr= variable_data['personnr'].replace('-', '').split(' ')[0]
      tmplist = ['&lt;utskriftsstatus&gt;&lt;brevid_kalla&gt;', variable_data['brevid_kalla'], '&lt;/brevid_kalla&gt;&lt;mapp&gt;Se Allt&lt;/mapp&gt;&lt;tidpunkt&gt;', variable_data['arkiveringsdatum'], '&lt;/tidpunkt&gt;&lt;dokumentpaket&gt;', variable_data['dokumenttyp'], '&lt;/dokumentpaket&gt;&lt;status&gt;Utskriven&lt;/status&gt;&lt;rensningsdatum&gt;999999&lt;/rensningsdatum&gt;&lt;kundid_gdb&gt;', variable_data['kundid_gdb'], '&lt;/kundid_gdb&gt;&lt;personnr&gt;', _personnr, '&lt;/personnr&gt;&lt;/utskriftsstatus&gt;']
      f_out.write(''.join(tmplist))
  </code></pre>
  
  <p>Method <code>_file_stripper</code> splits a big file into records.
  Infiles are 5-21 MB.</p>
  
  <p>Please advice where to look for the error.
  When I talk about slowdown, the write speed falls down below 4KB written/second after around 1 MB has been written, and it keeps falling as the outfile grows bigger.</p>
  
  <p>EDIT: On request, here is parse_infile and _file_stripper:</p>
  
  <pre><code>def parse_infile(f_in):
     index = ""     #variabel som håller vilken ondemandvariabel vi läser in
     found_data = 0  #1 ifall vi hittat det vi letar efter annars 0
     for row in f_in:
        if( 'personnr' in row):
           found_data=1
           index = "personnr"
        elif( 'kundid_gdb' in row):
           found_data=1
           index = "kundid_gdb"
        elif( 'brevid_kalla' in row):
           found_data=1
           index = "brevid_kalla"
        elif( 'arkiveringsdatum' in row):
           found_data=1
           index = "arkiveringsdatum"
        elif( 'GROUP_FILENAME' in row ):
           variable_data['dokumenttyp'] = row.split(':')[-1].split('.')[2].capitalize()
        elif(found_data==1):
           variable_data[index] = row.split(':')[1].strip() 
           index = ""  #Nollställ index ifall värden saknas i filen
           found_data=0
        else:
           pass
  
  def _file_stripper(tot_file):
     try:
        myfile = []
        for rows in tot_file:
           if not 'GROUP_FILENAME' in rows:
              myfile.append(rows)
           else:
              myfile.append(rows)
              yield myfile
     except Exception:
        pass
  
  variable_data = { "brevid_kalla": "", "arkiveringsdatum": "", 
            "kundid_gdb": "", "personnr": "",
            "dokumenttyp": "" }
  </code></pre>