Python, Ctypes And Mmap

I am wondering if it is possible for the ctypes package to interface with mmap. Currently, my module allocates a buffer (with create_string_buffer) and then passes that using byref

Solution 1:

An mmap object "supports the writable buffer interface", therefore you can use the from_buffer class method, which all ctypes classes have, with the mmap instance as the argument, to create a ctypes object just like you want, i.e., sharing the memory (and therefore the underlying file) that the mmap instance has mapped. I imagine, in specific, that you'll want a suitable ctypesarray.

Solution 2:

Be aware that the operating system is going to be doing readahead for read() anyway. You're going to be blocking either in read() or write()--one or the other will bottleneck the operation--but even though you're blocking in one, that doesn't mean the other isn't taking place for you behind the scenes. That's the job of every multitasking operating system.

If you use mmap for this, you're very likely making things more complicated for the OS--making it harder for it to determine that you're really just streaming data in and out, and making it more complicated for it to do read-ahead. It may still figure it out (operating systems are very good at this), but you're probably not helping.

The only benefit in principle is avoiding the cost of a memory copy, but it doesn't sound like that's your goal here (and unless profiling clearly says otherwise, I'd strongly doubt that would help performance).