title: What is the encoding to read and write files with special characters such as en dash, left quotes, etc?
body: <p>I'm reading csv files that contain special characters such as the long en dash –, left double quotes “, and right double quotes ” and I can't figure out the proper way to read and write these correctly. I thought it was UTF8 or Unicode but it reads and writes them as a square or ? with a diamond. Opening the files in notepad++ to confirm. Maybe another specific encoding is needed? Here is the code I've been using so far, tried a few variations of this with different encoding.:</p>
  
  <pre><code>string[] lines = File.ReadAllLines(filePathTxt.Text, Encoding.UTF8);
  ...
  Stream s = new FileStream(filePath, FileMode.Append);
  StreamWriter sw = new StreamWriter(s, Encoding.UTF8, 1000, true);
  </code></pre>
  
  <p>Input of: </p>
  
  <p>Surveys – Public</p>
  
  <p>Documents:,“A”</p>
  
  <p>comes out as</p>
  
  <p>Surveys � Public</p>
  
  <p>Documents:,�A�</p>
  
  <p>Also shows problems in debugger as soon as it's read into the string array.</p>
  
  <p>Edit: I've tried Unicode also. I'm using NotePad++, Win 10. The problem is definitely in the Read step, because if I add the following line to manually write a line of data, like so:</p>
  
  <pre><code> sw.WriteLine("Surveys – Public");
  </code></pre>
  
  <p>That line writes the dash fine, so it's on the initial read of the file from the source csv where the characters get messed up. I've tried reading with a few different encodings, and NotePad++ just shows the csv as being ANSI.</p>