Related to the above, when I try to run my pex I get

can't find '__main__' module

despite having set

entry_point

for my

pex_binary

rule. That appears to be related to https://github.com/pex-tool/pex/issues/958 but I also set

execution_mode = venv

which is supposed to be the workaround for this. Any idea what that's not working?

Sorry for the trouble. Just to confirm the moving parts: 1. What’s the BUILD definition of the PEX in question? 2. Are the files you expect in the final pex zip?

I have a custom build macro:

def cpy_std_pex(**kwargs):
    python_sources(name='lib')
    if 'entry_point' not in kwargs:
        kwargs['entry_point'] = 'main.py'
    # We set the execution mode to venv for 2 reasons: (1) this sidesteps a restriction that .zip files can't otherwise
    # be bigger than 2GB (see <https://github.com/pex-tool/pex/issues/958>) which can be an issue for executables that
    # include pytorch and (2) venv executions actually start up faster after the 1st run (though they're slower on first
    # run).
    if 'execution_mode' not in kwargs:
        kwargs['execution_mode'] = 'venv'
    pex_binary(**kwargs)

which in this case I'm calling without arguments. And yes, all the files, including my

main.py

are in there. And the

entry_point

in the unziiped

PEX-INFO

is correct.

I think Python is having trouble opening the zip (https://github.com/python/cpython/issues/77140) and thus can't even execute the PEX start-up code in

/__main__.py

and I believe

execution_mode = venv

will only work with either: • a "sh boot" PEX that uses a shell script for set-up. This requires Pants support which is in the upcoming 2.20 series (https://github.com/pantsbuild/pants/pull/19925). • a PEX where Python can successfully execute

__main__.py

, to do the setup Another option would be avoiding the zip, with a "packed" layout PEX, which is supported in Pants via

layout="packed"

.

Thanks @broad-processor-92400! Yes, it works with

layout="packed"

. Well, it kind of works. I have a transitive dependency on

nvidia_cudnn_cu11-8.7.0.84-py3-none-manylinux1_x86_64.whl

and if you look inside that whl file you see some shared libraries, including

libcudnn.so.8

:

$ unzip -l runner.pex/.deps/nvidia_cudnn_cu11-8.7.0.84-py3-none-manylinux1_x86_64.whl | grep libcudnn.so.8
   150200  1980-01-01 00:00   nvidia/cudnn/lib/libcudnn.so.8

but when I run the pex I get an error that it can't find the shared lib:

$ python runner.pex/ -h
  File "/home/oliver_companionprofessional_com/.pyenv/versions/3.10.6/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/oliver_companionprofessional_com/.pyenv/versions/3.10.6/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/__main__.py", line 106, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/.bootstrap/pex/pex_bootstrapper.py", line 618, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/.bootstrap/pex/pex.py", line 560, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/.bootstrap/pex/pex.py", line 467, in _wrap_coverage
    return runner(*args)
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/.bootstrap/pex/pex.py", line 498, in _wrap_profiling
    return runner(*args)
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/.bootstrap/pex/pex.py", line 603, in _execute
    return self.execute_entry(
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/.bootstrap/pex/pex.py", line 800, in execute_entry
    return self.execute_module(entry_point.module)
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/.bootstrap/pex/pex.py", line 808, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/home/oliver_companionprofessional_com/.pyenv/versions/3.10.6/lib/python3.10/runpy.py", line 224, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/oliver_companionprofessional_com/.pyenv/versions/3.10.6/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/oliver_companionprofessional_com/.pyenv/versions/3.10.6/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/oliver_companionprofessional_com/.pex/unzipped_pexes/c6bc5fab13b3ea6c3b2058bbee01a89110307fbc/exec/gait/runner/main.py", line 7, in <module>
    import torch
  File "/home/oliver_companionprofessional_com/.pex/installed_wheels/e91f71c0d14f1a88ef322bf99c2fda6753f27209224e875652cd5c9d1e316b94/torch-2.2.1+cu118-cp310-cp310-linux_x86_64.whl/torch/__init__.py", line 237, in <module>
    from torch._C import *  # noqa: F403
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory

shouldn't pex be unzipped that, setting up

LD_LIBRARY_PATH

(or similar), etc. Any ideas?

Hm, I'm not sure about that one. I know there's been various discussion both here in slack and in the pants issue tracker about pytorch that might provide insight (I'm having to dash off so can't confirm, though)

😞 thanks.

https://github.com/pantsbuild/pants/issues/20205 looks related, I don't know if it lists work-arounds

THANK YOU! I spent hours searching for bugs/solutions and somehow didn't find that one. The

venv_site_packages_copies

trick seems to fix it.

I've updated that ticket with the error message here, so hopefully it's easier find for the next person