Python: Traceback Codecs.charmap_decode(input,self.errors,decoding_table)[0]

Following is sample code, aim is just to merges text files from give folder and it's sub folder. i am getting Traceback occasionally so not sure where to look. also need some help

Solution 1:

The error is thrown because Python 3 opens your files with a default encoding that doesn't match the contents.

If all you are doing is copying file contents, you'd be better off using the shutil.copyfileobj() function together with opening the files in binary mode. That way you avoid encoding issues altogether (as long as all your source files are the same encoding of course, so you don't end up with a target file with mixed encodings):

import shutil
import os.path

with open('C:\\Dropbox\\Python\\master.txt','wb') as output:
    forpath, f_name in files:
        with open(os.path.join(path, f_name), 'rb') as input:
            shutil.copyfileobj(input, output)
        output.write(b'\n') # insert extra newline between files

I've cleaned up the code a little to use context managers (so your files get closed automatically when done) and to use os.path to create the full path for your files.

If you do need to process your input line by line you'll need to tell Python what encoding to expect, so it can decode the file contents to python string objects:

open(path, mode, encoding='UTF8')

Note that this requires you to know up front what encoding the files use.

Read up on the Python Unicode HOWTO if you have further questions about python 3, files and encodings.

Solution 2:

I faced the similar issue while removing the file using os module remove function.

The required changes i performed is:

file = open(filename)

to

file = open(filename, encoding="utf8")

Add an encoding=“utf-8”

UTF-8 is one of the most commonly used encodings, and Python often defaults to using it. UTF stands for “Unicode Transformation Format”, and the '8' means that 8-bit values are used in the encoding. ... UTF-8 uses the following rules: If the code point is < 128, it's represented by the corresponding byte value.