Does The Python "open" Function Save Its Content In Memory Or In A Temp File?

For the following Python code: fp = open('output.txt', 'wb') # Very big file, writes a lot of lines, n is a very large number for i in range(1, n): fp.write('something' * n) fp

Solution 1:

It's stored in the operating system's disk cache in memory until it is flushed to disk, either implicitly due to timing or space issues, or explicitly via fp.flush().

Solution 2:

There will be write buffering in the Linux kernel, but at (ir)regular intervals they will be flushed to disk. Running out of such buffer space should never cause an application-level memory error; the buffers should empty before that happens, pausing the application while doing so.

Solution 3:

Building on ataylor's comment to the question:

You might want to nest your loop. Something like

for i in range(1,n):
    for each in range n:
        fp.write('something')
fp.close()

That way, the only thing that gets put into memory is the string "something", not "something" * n.

Solution 4:

If you a writing out a large file for which the writes might fail you a better off flushing the file to disk yourself at regular intervals using fp.flush(). This way the file will be in a location of your choosing that you can easily get to rather than being at the mercy of the OS:

fp = open('output.txt', 'wb')
counter = 0for line in many_lines:
    file.write(line)
    counter += 1if counter > 999:
        fp.flush()
fp.close()

This will flush the file to disk every 1000 lines.

Solution 5:

If you write line by line, it should not be a problem. You should show the code of what you are doing before the write. For a start you can try to delete objects where not necessary, use fp.flush() etc..