title: Detecting utf-8 without BOM or with BOM
body: <p>I'm building a compression program. I want to use LWZ for utf-8 files (any urf-8 files) and BZip for others (usually random binary files). I can't find method to define is file utf8 or not.</p>
  
  <p>I tried <a href="https://stackoverflow.com/questions/5657738/how-can-i-identify-different-encodings-against-files-without-the-use-of-a-bom-an">this</a> and many other methods all over stackoverflow but they can't do it for me.
  I can share examples of files that should be recognized as utf 8 and files that should be recognized as "others"</p>
  
  <pre><code> else if (args[0] != null &amp;&amp; args[1] != null)
          {
  
              if (random binary detected)
              {
                  Console.WriteLine("Started Bzip");
                  byte[] res = new Bzip2Compressor(65).Compress(File.ReadAllBytes(args[0]));
                  File.WriteAllBytes(args[1], res);
                  Console.WriteLine("Done!");
                  return;
              }
              else //for utf 8 cases (both with bom and without)
              {
                  Console.WriteLine("Started LZW");
                  byte[] res = LZWCompressor.Compress(File.ReadAllBytes(args[0]));
                  File.WriteAllBytes(args[1], res);
                  Console.WriteLine("Done");
                  return;
              }
          }
  </code></pre>
  
  <p>Note: i only need to separate utf-8 and all others</p>
  
  <p>EDIT: so i would like to check first n symbols to be invalid utf 8;</p>
  
  <pre><code>var bytes = new byte[1024 * 1024];
  new Random().NextBytes(bytes);
  File.WriteAllBytes(@"PATH", bytes);
  </code></pre>
  
  <p>General goal is to detected files cerated like in code above as NOT utf-8 files</p>