title: Why does Python write one wrongly encoded line every two lines?
body: <p><strong>I am trying</strong> to dump the content of a table column from SQL Server 2K into text files, that I want to later treat with Python and output new text files. </p>
  
  <p><strong>My problem</strong> is that I can't get python to use the correct encoding, and while the input files appear fine on my text editor the output ones have broken characters apparently once every two lines.</p>
  
  <p>My python code can be reduced to:</p>
  
  <pre><code>input = open('input', 'r')
  string = input.read()
  # Do stuff
  output = open('output', 'w+')
  output.write(string)
  </code></pre>
  
  <p>Printing this string in the windows shell gives me the expected characters, though separated from each other by one space too many.</p>
  
  <p>But when I open the output file, once every two line everything is broken (though the "added" whitespaces have disappeared)</p>
  
  <p><strong>Some context:</strong> To dump the column to files, I'm using this script: <a href="https://www.simple-talk.com/code/WorkingWithFiles/spWriteStringTofile.txt" rel="nofollow"><code>spWriteStringTofile</code></a> which I believe is using the default server encoding.</p>
  
  <p>After some research, it appears that this encoding is <code>SQL_Latin1_General_CP1_CI_AS</code>. I tried adding <code># -*- coding: latin_1 -*</code> at the beginning of the script, I tried converting the encoding inside SQL Server to <code>Latin1_General_CI_AS</code>, I tried to <code>string.decode('latin_1').encode('utf8')</code> but it didn't change a thing (except the last attempt which output only broken characters).</p>
  
  <p>What can I try ?</p>
  
  <hr>
  
  <p><strong>EDIT 2:</strong> I tried the <code>newFile.write(line.decode('utf-16-be').encode('utf-16-le'))</code> solution, with throws an error at the first line of my file. From the python GUI:</p>
  
  <pre><code>(Pdb) print line
  ÿþ
  
  (Pdb) print repr(line)
  '\xff\xfe\n'
  (Pdb) line.decode('utf-16-be').encode('utf-16-le')
  *** UnicodeDecodeError: 'utf16' codec can't decode byte 0x0a in position 2: truncated data
  </code></pre>
  
  <p>Only a newline appear in Sublime Text 2 for this first line...</p>
  
  <p>When I bypass it (<code>try: ... except: pass</code>, quick&amp;dirty), a newline is added between correct and incorrect lines, but the broken characters are still here.</p>
  
  <hr>
  
  <p><strong>EDIT:</strong> I went through the document line by line</p>
  
  <pre><code>newFile = open('newfile', 'a+')
  with open('input') as fp:
      for line in fp:
          import pdb
          pdb.set_trace()
          newFile.write(line)
  </code></pre>
  
  <p>In the pdb, on a faulty line:</p>
  
  <pre><code>(Pdb) print line
                             a s  S o l d D e b i t o r , # &lt;-- Not actual copy paste
  (Pdb) print repr(line)
  '\x00\t\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00a\x00s\x00 \x00S\x00o\x00l\x00d\x00D\x00e\x00b\x00i\x00t\x00o\x00r\x00,\x00\r\x00\n'
  </code></pre>
  
  <p><em>However</em> for some reason I couldn't copy/paste the <code>print line</code> value: I can copy the individual alphabetic character but not when I select the "whitespace" that is betzeen them...</p>
  
  <hr>
  
  <p>Input:</p>
  
  <pre><code>r &lt;= @Data2 then (case when @Deviza='' or @Deviza=@sMoneda 
      then isnull(Debit,0) else isnull(DevDebit,0) end)
      else 0 end) 
       - Sum(case when DataInr &gt;= @BeginDate and DataInr &lt;= @Data2 
         then  (case when @Deviza='' or @Deviza=@sMoneda 
         then  isnull(Credit,0) else isnull(DevCredit,0) end)
         else 0 end) 
         else 0 end
      as SoldDebitor,
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>r &lt;= @Data2 then (case when @Deviza='' or @Deviza=@sMoneda 
              then  isnull(Debit,0) else isnull(DevDebit,0) end)
  ਍ऀ                       攀氀猀攀 　 攀渀搀⤀ ഀഀ
        - Sum(case when DataInr &gt;= @BeginDate and DataInr &lt;= @Data2 
              then  (case when @Deviza='' or @Deviza=@sMoneda
              then  isnull(Credit,0) else isnull(DevCredit,0) end)
  ਍ऀ                       攀氀猀攀 　 攀渀搀⤀ ഀഀ
          else 0 end
  ਍ऀ                 愀猀 匀漀氀搀䐀攀戀椀琀漀爀Ⰰഀഀ
  </code></pre>