How Do I Use The Correct Dll Files To Enable 3rd Party C Libraries In A Cython C Extension?

I have a C function that involves decompressing data using zstd. I am attempting to call that function using Cython. Using this page from the docs as a guide I can compile and run

Solution 1:

We link our cython-extension against a windows-dll, that means:

• *.lib-file (i.e. zstd.lib) is needed in "path/to/zstd/lib" during the compile time
• *.dll-file (i.e. zstd.dll) is needed somewhere where Windows can find it when the module is imported.

Normally, Windows will not look in the "path/to/zstd/lib". And so we get a somewhat cryptic error message:

ImportError: DLL load failed: The specified module could not be found.

Which doesn't mean there is something wrong with the module - it just happens to depend on a dll which cannot be found.

While linux has -rpath-option for the linker with which "path/to/zstd/lib" could be passed (it can be added with runtime_library_dirs-argument to Extension), there is no such option on Windows. The dll-search-algorithmus for Windows can be found here. In a nutshell, dll is searched in (possible in another order as presented here)

• The directory from which the module is loaded.
• The current directory.
• system-directory (e.g. C:\Windows\System32)
• windows-directory(e.g. C:\Windows)
• directories that are listed in the PATH-variable
• others

Putting the dll into system- or windows-directory doesn't sound too appealing, which leave us with the following options:

• (the easiest?) to copy the zstd.dll next to the compiled extension
• to add the zstd-path to the PATH-variable, e.g. set PATH="path/to/zstd/lib";%PATH%

---

Another option is somewhat more tricky: Given that

If a DLL with the same module name is already loaded in memory, the system checks only for redirection and a manifest before resolving to the loaded DLL, no matter which directory it is in. The system does not search for the DLL.

we can use ctypes to "preload" the right dll, which will be used (without the need to search for it on the disc) when the wrapper-module is imported, i.e.:

import ctypes; 
ctypes.CDLL("path/to/zstd/lib/zstd.dll"); # we preload with the full pathimport hello_wrapper  # works now!

---

The above applies if the extension is built and used on the same system (e.g. via build_ext --inplace). installation/distribution is somewhat more cumbersome (this is covered by this SO-post), one idea would be:

• to put *.h-, *.lib- and *.dll-files into 'package_data' (it seems to happen automatically anyway)
• the right relative library_path (or programmatically the absolute path) can be set in the setup.py so *.lib is found by the linker.
• dll will be put next to the compiled *.pyd-file in the installation.

An example could be the following more or less minimal setup.py, where everything (pyx-file, h-files, lib-file, dll-file) are put into a package/folder src/zstd:

from setuptools import setup, Extension, find_packages
from Cython.Build import cythonize

ext_modules = [
    Extension(
        "zstd.zstdwrapper",
        ["src/zstd/zstdwrapper.pyx"],
        libraries=["zstd"],
        library_dirs=["src/zstd"],
        include_dirs=[], # set automatically to src/zstd during the build
    )
]

print(find_packages(where='src'))

setup(
    name = 'zstdwrapper',
    ext_modules = cythonize(ext_modules),
    packages = find_packages(where='src'),
    package_dir = {"": "src"},
)

And now it can be installed with python setup.py install or used to create e.g. a source-distribution via python setup.py sdist which then can be installed via pip.