Hi! I am tinkering with PyOxidizer plugin and ran ...
# general
c
Hi! I am tinkering with PyOxidizer plugin and ran into an issue. Would appreciate help. Details in the thread.
I was able to build the binary and the plugin was smart enough to package some third party shared libraries into a lib/ in the same parent dir as the binary. However, when I run the binary I get
Copy code
Traceback (most recent call last):
  File "runpy", line 197, in _run_module_as_main
  File "runpy", line 87, in _run_code
  File "pushdown_service.service", line 15, in <module>
  File "db_dtypes", line 22, in <module>
  File "numpy", line 125, in <module>
  File "numpy.__config__", line 9, in <module>
NameError: name '__file__' is not defined
Okay, i'll start by adding numpy to
filesystem_resources
. Still, no dice.
Copy code
Traceback (most recent call last):
  File "numpy.core", line 23, in <module>
  File "numpy.core.multiarray", line 10, in <module>
  File "numpy.core.overrides", line 6, in <module>
ImportError: libopenblas64_p-r0-15028c96.3.21.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "runpy", line 197, in _run_module_as_main
  File "runpy", line 87, in _run_code
  File "pushdown_service.service", line 15, in <module>
  File "db_dtypes", line 22, in <module>
  File "numpy", line 141, in <module>
  File "numpy.core", line 49, in <module>
ImportError: ...
Wait a minute, it's there!
Copy code
soda$ ls dist/service_bin/x86_64-unknown-linux-gnu/debug/install/lib/numpy/libs/libopenblas64_p-r0-15028c96/3/21.so 
dist/service_bin/x86_64-unknown-linux-gnu/debug/install/lib/numpy/libs/libopenblas64_p-r0-15028c96/3/21.so
Looking at pyoxidizer docs, it seems one needs to add
module_search_paths
so
filesystem-relative:lib
is in the search path. In https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/packaging/pyoxidizer/config.py I see that this is not done. Am I missing something? Is this expected/are the default parameters expected to work without this config? I can try generating the wheel in pants and then going to pyoxidizer and trying to get this to work; however, would prefer to stay within pants if possible. p.s: I also tried -keep-sanboxes=always to see if I can hack the
bzl
file and couldn't find one.
I dug into this a bit more. The existing template for the
pyoxidizer_binary
target works well for all cases I checked except with a dependency on
numpy
; seems to be an issue with PyOxidizer. I went ahead and created this for repro: https://github.com/csnatarajan/pants-pyoxy Using the template params (
policy.allow_files = True
,
policy.include_file_resources = True
, `policy.file_scanner_emit_files = True`; policy = PythonPackagingPolicy) seems necessary. On a different note though, the lib (filesystem-resource) it produces is 100M and the binary is 200M+; this is bordering on criminal for using (only) numpy as a dependency! TIme to checkout scie!