also specifies the encoding, since the attacker can then choose a clever way to hide malicious text in the encoded bytestream. The StreamRecoder class can transparently convert between encodings, taking a stream that returns data in encoding #1 and behaving like a stream returning data in encoding #2. For example, if you have an input file f that’s in Latin-1, you can wrap it with a$_{StreamRecoder}$ to return bytes encoded in UTF-8: What can you do if you need to make a change to a file, but don’t know the file’s encoding? If you know the encoding is ASCII-compatible and only want to examine or modify the ASCII parts, you can open the file with the $_{surrogateescape}$error handler: The $_{surrogateescape}$error handler will decode any non-ASCII bytes as code points in a special range running from U+DC80 to U+DCFF. These code points will then turn back into the same bytes when the $_{surrogateescape}$error handler is used to encode the data and write it back out. One section of Mastering Python 3 Input/Output, a PyCon 2010 talk by David Beazley, discusses text processing and binary data handling. The PDF slides for Marc-André Lemburg’s presentation “Writing Unicode-aware Applications in Python” discuss questions of character encodings as well as how to internationalize and localize an application. These slides cover Python 2.x only. The Guts of Unicode in Python is a PyCon 2013 talk by Benjamin Peterson that discusses the internal Unicode representation in Python 3.3. 3.3 References Files in an Unknown Encoding Converting Between File Encodings with open(fname, 'r', encoding="ascii", errors="surrogateescape") as f: data =f.read() # make changes to the string 'data' with open(fname+'.new','w', encoding="ascii", errors="surrogateescape") as f: f.write(data) new_f=codecs.StreamRecoder(f, # en/decoder: used by read() to encode its results and # by write() to decode its input. codecs.getencoder( 'utf-8 '), codecs .getdecoder('utf-8 '), # reader/writer: used to read and write to the stream. codecs.getreader('latin-1 '), codecs .getwriter('latin-1 ') ) 11