Take the 2-minute tour ×

While making my App Engine app I suddenly ran into an error which shows every couple of requests:

  File "/home/ubuntu/Programs/google/google_appengine/google/appengine/ext/webapp/util.py", line 98, in run_wsgi_app
  File "/home/ubuntu/Programs/google/google_appengine/google/appengine/ext/webapp/util.py", line 118, in run_bare_wsgi_app
    for data in result:
  File "/home/ubuntu/Programs/google/google_appengine/google/appengine/ext/appstats/recording.py", line 897, in appstats_wsgi_wrapper
    result = app(environ, appstats_start_response)
  File "/home/ubuntu/Programs/google/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 717, in __call__
    handler.handle_exception(e, self.__debug)
  File "/home/ubuntu/Programs/google/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 463, in handle_exception
  File "/home/ubuntu/Programs/google/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 436, in error
  File "/home/ubuntu/Programs/google/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 288, in clear
  File "/usr/lib/python2.7/StringIO.py", line 106, in seek
    self.buf += ''.join(self.buflist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 208: ordinal not in range(128)

I really have no idea where this could be, it only happens when I use a specific function but it's impossible to track all string I have. It's possible this byte is a character like ' " [ ] etc, but only in another language

How can I find this byte and possibly other ones?

I am running GAE with python 2.7 in ubuntu 11.04



This is the code I ended up using: from codecs import BOM_UTF8 from os import listdir, path p = "path"

def loopPath(p, times=0):
    for fname in listdir(p):
        filePath = path.join(p, fname)
        if path.isdir(filePath):
            return loopPath(filePath, times+1)

        if fname.split('.', 1)[1] != 'py': continue

        f = open(filePath, 'r')
        ln = 0
        for line in f:
            #print line[:3] == BOM_UTF8
            if not ln and line[:3] == BOM_UTF8:
                line = line[4:]
            col = 0
            for c in list(line):
                if ord(c) > 128:
                    raise Exception('Found "'+line[c]+'" line %d column %d in %s' % (ln+1, col, filePath))
                col += 1
            ln += 1

share|improve this question
Have you tried looking at byte '\0xd7' at position 208 of the buffer as the error obviously pointed out? –  Jeff Mercado Sep 17 '11 at 18:05
That buffer is an internal variable of the StringIO which is really deep inside GAE's code. and the buffer won't show me exactly where it is in my code, only a lot of text... –  Shedokan Sep 17 '11 at 21:22

5 Answers 5

up vote 3 down vote accepted

Just goes through every character in each line of code. Something like that:

# -*- coding: utf-8 -*-
import sys

data = open(sys.argv[1])
line = 0
for l in data:
    line += 1
    char = 0
    for s in list(unicode(l,'utf-8')):
        char += 1
            print 'Non ASCII character at line:%s char:%s' % (line,char)
share|improve this answer
it can be simplified by using ord(s) > 128 –  Shedokan Sep 17 '11 at 21:34

When I translated UTF-8 files to latin1 LaTeX I had similar problems. I wanted a list of all evil unicode characters in my files.

It is probably even more you need, but I used this:


def fortex(exc):
    import unicodedata, exceptions
    if not isinstance(exc, exceptions.UnicodeEncodeError):
        raise TypeError("don't know how to handle %r" % exc)
    l = []
    print >>sys.stderr, "   UNICODE:", repr(exc.object[max(0,exc.start-20):exc.end+20])
    for c in exc.object[exc.start:exc.end]:
        uname = unicodedata.name(c, u"0x%x" % ord(c))
        key = repr(c)
        if not UNICODE_ERRORS.has_key(key): UNICODE_ERRORS[key] = [ 1, uname ]
        else: UNICODE_ERRORS[key][0] += 1
    return (u"\\gpTastatur{%s}" % u", ".join(l), exc.end)

def main():    
    codecs.register_error("fortex", fortex)
    fileout = codecs.open(filepath, 'w', DEFAULT_CHARSET, 'fortex')


Here is the matching excerpt from the Python doc:

codecs.register_error(name, error_handler) Register the error handling function error_handler under the name name. error_handler will be called during encoding and decoding in case of an error, when name is specified as the errors parameter.

For encoding error_handler will be called with a UnicodeEncodeError instance, which contains information about the location of the error. The error handler must either raise this or a different exception or return a tuple with a replacement for the unencodable part of the input and a position where encoding should continue. The encoder will encode the replacement and continue encoding the original input at the specified position. Negative position values will be treated as being relative to the end of the input string. If the resulting position is out of bound an IndexError will be raised.

share|improve this answer
+1 for saving all errors, but I don't see if it saves where the error was found –  Shedokan Sep 17 '11 at 21:34
use exc.start if you want to track that. –  towi Sep 18 '11 at 10:31

This should list the offending lines:

grep -v [:alnum:] dodgy_file

$ cat test

$ cat test2

$ grep -v [:alnum:] test

$ grep -v [:alnum:] test2
share|improve this answer
Don't know if it works, but I dislike command lines –  Shedokan Sep 17 '11 at 21:18
@shedokan - care to explain why? cmd-line can be your greatest friend! It really pays off to learn to use it. Learning some standard linux tools like grep, sed, awk, sort, uniq and connecting them using pipes gives you a tool-set that is unrivaled by any GUI-program in the world! –  Fredrik Pihl Sep 17 '11 at 21:21
I'm a windows guy and used to not having to see any white on black text :) I'm only using linux for GAE nothing more. Even though they can be powerful I don't need them so no point in learning them, even though they can sometimes be useful –  Shedokan Sep 17 '11 at 21:27
but you do need then, even on windows! install cygwin; they are VERY useful so please reconsider your path to technical enlightenment. More tools in your utility-belt means you are better equipped to handle new challenges. Just my 10c –  Fredrik Pihl Sep 17 '11 at 21:47

You can use the command:

grep --color='auto' -P -n "[\x80-\xFF]" file.xml

This will give you the line number, and will highlight non-ascii chars in red.

Copied from How Do I grep For non-ASCII Characters in UNIX. Fredrik's answer is good but not quite right because it also finds ASCII chars that are not alphanumeric.

share|improve this answer

This Python script gives the offending character and its index in the text when that text is viewed as a single line:

[(index, char) for (index, char) in enumerate(open('myfile').read()) if ord(char) > 127]
share|improve this answer

Your Answer


