title: How use perl to process a file whose format is similar to unicode?
body: <p>I have a legacy program, and after running it, it will generate a log file. Now I need to analysis this log file. </p>
  
  <p>But the file format is very strange. Please see the following,I used vi to open it, it looks like an unicode file, but it is not FFFE started. after I used notepad open it, save it and open again, I found that the FFFE is added by notepad. Then I can use command 'type log.txt > log1.txt" to convert the whole file to ANSI format. Later in perl, I can use /TDD/ in perl to search what I need. </p>
  
  <p>But now, I can't deal with this file format. </p>
  
  <p>Any comment or idea will be very appreciated. </p>
  
  <pre><code>0000000: 5400 4400 4400 3e00 2000 4c00 6f00 6100  T.D.D.&gt;. .L.o.a.
  </code></pre>
  
  <p>After notepad save it</p>
  
  <pre><code>0000000: fffe 5400 4400 4400 3e00 2000 4c00 6f00  ..T.D.D.&gt;. .L.o.
  
  open STDIN, "&lt; log.txt";
  while(&lt;&gt;)
  {
    if (/TDD/)
    {
      # Add my logic.
    }
  }
  </code></pre>
  
  <p>I have read the thread which is very useful, but still can't resolve my problem.
  <a href="https://stackoverflow.com/questions/2461737/how-can-i-open-a-unicode-file-with-perl">How can I open a Unicode file with Perl?</a></p>
  
  <p>I can't add answer, so I edit my thread. </p>
  
  <p>Thanks Michael, 
  I tried your script but got the following error. I checked my perl version is 5.1, OS is windows 2008. </p>
  
  <pre><code>* ascii
  * ascii-ctrl
  * iso-8859-1
  * null
  * utf-8-strict
  * utf8
  UTF-16:Unrecognised BOM 5400 at test.pl line 12.
  </code></pre>
  
  <p><strong>Update</strong></p>
  
  <p>I tried the UTF-16LE with the command: </p>
  
  <pre><code>perl.exe open.pl utf-16le utf-16 &lt;my log file&gt;.txt
  </code></pre>
  
  <p>but I still got the error like </p>
  
  <pre><code>UTF-16LE:Partial character at open.pl line 18, &lt;$fh&gt; line 1824.
  </code></pre>
  
  <p>also, I tried utf-16be, got the same error. </p>
  
  <p>If I used utf-16, I will got the error </p>
  
  <pre><code>UTF-16:Unrecognised BOM 5400 at open.pl line 18.
  </code></pre>
  
  <p>open.pl line 18 </p>
  
  <pre><code>is "print while &lt;$fh&gt;;"
  </code></pre>
  
  <p>Any idea? </p>
  
  <p>Updated: 5/11/2011. 
  Thank you guys for your help. I resolved the problem. 
  I found that the data in log file are not UTF-16 after all. So, I had to write a .net project by visual studio. It will read the log file with UTF-16 and write to a new file with UTF-8. And then I used perl script to parse the file and generate result data. It worked now. </p>
  
  <p>So, if any of you know how to use perl read a file with many garbage data, please tell me, thank you very much. </p>
  
  <p>e.g. garbage data sample</p>
  
  <pre><code>tests.cpp:34)
  ਍吀䐀䐀㸀 䰀漀愀搀椀渀最 挀挀洀挀漀爀攀⸀搀氀
  </code></pre>
  
  <p>use hex reader to open it: </p>
  
  <pre><code>0000070: a88d e590 80e4 9080 e490 80e3 b880 e280  ................
  0000080: 80e4 b080 e6bc 80e6 8480 e690 80e6 a480  ................
  0000090: e6b8 80e6 9c80 e280 80e6 8c80 e68c 80e6  ................
  00000a0: b480 e68c 80e6 bc80 e788 80e6 9480 e2b8  ................
  </code></pre>