Parse File In Robust Way With Python 3

I have a log file that I need to go through line by line, and apparently it contains some 'bad bytes'. I get an error message along the following lines: UnicodeDecodeError: 'utf-

Solution 1:

You can also use the codecs module. When you use the codecs.open() function, you can specify how it handles errors using the errors argument:

codecs.open(filename, mode[, encoding[, errors[, buffering]]])

The errors argument can be one of several different keywords that specify how you want Python to behave when it attempts to decode a character that is invalid for the current encoding. You'll probably be most interested in codecs.ignore_errors or codecs.replace_errors, which cause invalid characters to be either ignored or replaced with a default character, respectively.

This method can be a good alternative when you know you have corrupt data that will cause the UnicodeDecodeError to be raised even when you specify the correct encoding.

Example:

with codecs.open('file.txt', mode='r', errors='ignore'):
    # ...stuff...# Even if there is corrupt data and invalid characters for the default# encoding, this open() will still succeed

Solution 2:

So apparently your file does not contain valid UTF-8 (which is the default encoding). If you know, what encoding is used (e.g. iso-8859-1 which was afaik the python2 default), you can specify it when opening by using

open(sys.argv[1], mode='r', encoding='iso-8859-1')

If the encoding is unknown or not valid as all, you can open the file as binary.

open(sys.argv[1], mode='rb')

This will make the content accessible as bytes rather than trying to interpret them as characters.

Solution 3:

In python <=2.7, strings (str) are arrays of 8 bits characters. So when reading a file composed of 8 bits characters or bytes, you get the bytes without problem, no matter what the actual encoding is. Simply, you may read them with a wrong representation, but it will never throw any exception.

In python >=3,strings are unicode strings (16 bits per character). So when reading a file python has to decode the file, and by default it uses system encoding - not necessarily UTF-8. In your case, it seems to assume UTF-8 encoding, when your log file is not UTF-8 encoding so the exception.

If not sure of the encoding you may reasonably try to use ISO-8859-1 with

open(sys.argv[1], 'r', encoding='iso-8859-1')