title: How to distinguish between a correct and a botched unicode encoded string in Python?
body: <p>I have string data in various languages where parts of the strings have seen some wrong encoding/decoding while others are correct, I need to fix the wrong ones:</p>
  
  <p>Here's an example for the german word "Zubehör":</p>
  
  <pre><code>correct = "ZUBEHÖR"
  incorrect = "ZUBEHÃ\x96R"
  </code></pre>
  
  <p>I already found out that I can correct the errors like this:</p>
  
  <pre><code>incorrect.encode("raw_unicode_escape").decode("utf8")
  </code></pre>
  
  <p>However using this on the correct strings yields an error. I could iterate over all strings and use a <code>try</code>-statement, but I don't know if this will work reliably and I'd like to know a more elegant way.</p>
  
  <p>Also while the <code>\x96</code> is written out when printing it's actually only one character:</p>
  
  <pre><code>incorrect[-3]
  Out[34]: 'Ã'
  
  incorrect[-2]
  Out[33]: '\x96'
  </code></pre>
  
  <p>How can I reliably only find those strings that have these odd unicode characters in them like <code>ZUBEHÃ\x96R</code>?</p>
  
  <p>EDIT: </p>
  
  <p>Here's something else I stumbled upon while experimenting:</p>
  
  <p>When I do <code>incorrect.encode("raw_unicode_escape")</code> then the result is <code>b'ZUBEH\xc3\x96R'</code>.</p>
  
  <p>But when I do this with e.g. a cyrillic word like this:</p>
  
  <pre><code>"Персонализированные".encode("raw_unicode_escape")
  </code></pre>
  
  <p>Then the result is <code>b'\\u041f\\u0435\\u0440\\u0441\\u043e\\u043d\\u0430\\u043b\\u0438\\u0437\\u0438\\u0440\\u043e\\u0432\\u0430\\u043d\\u043d\\u044b\\u0435'</code></p>
  
  <p>Why am I getting <code>\x</code>-escapes in the first case and <code>\u</code>-escapes in the second case while doing the exact same thing?</p>
  
  <p>And why can I <code>.decode("utf8")</code> back the <code>\x</code>-escapes into a readable format but not the <code>\u</code>-escapes?</p>