https://pantsbuild.org/ logo
Join Slack
Powered by
Hi All. I need help with an issue I have. I am bui...
# general
s
Hi All. I need help with an issue I have. I am building a pex with no dependencies. My environment is supposed to have them. I am testing this locally by using pyenv, when I build my image the image is build without dependencies. it seems like the pex setups its environment so that it runs in an isolated environment. I have dependecy errors but the interpreter I am using does contain the dependencies. Is there a way to correctly do this?
e
If you're using a Pants 
pex_binary
target to define the PEX, set `inherit_path`: https://www.pantsbuild.org/docs/reference-pex_binary#codeinherit_pathcode If you're using 
pex
directly to create the PEX, use the 
--inherit-path
option (see 
pex --help
). If you have a random PEX you did not build, run it with 
PEX_INHERIT_PATH=... my.pex ...
(see 
pex --help-variables).
🙌 1
To make sure you inherit the intended Python environment dependencies, it's safest to run 
/this/python my.pex
instead of 
my.pex
.
s
I have tried this ~/.pyenv/versions/pipelines/bin/python3 dist/valoreo.mains.sync_yuju/main.pex but it doesnt work 😞 I know dependencies are there because if I do ~/.pyenv/versions/pipelines/bin/python3 dist/valoreo.mains.sync_yuju/main.pex -i and import my dependencies theres no error. Any idea what could be wrong? My config is the following
Copy code
pex_binary(
    name="main",
    entry_point="main.py",
    include_requirements=False,
    dependencies=[":docker_config"],
    inherit_path="prefer"
)
e
You'll need to supply some details to kick off the remote debug session. Perhaps provide the full command line output from your two commands above as a starting point.
s
sure, the output is the following:
Copy code
Traceback (most recent call last):
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 517, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 422, in _wrap_coverage
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 453, in _wrap_profiling
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 575, in _execute
    return self.execute_entry(
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 768, in execute_entry
    return self.execute_module(entry_point.module, alter_sys)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 780, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/mains/sync_yuju/main.py", line 4, in <module>
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
I just realized that paths in this output point to /Users/glserran/.pyenv/versions/3.9.7/ even when I invoked with ~/.pyenv/versions/pipelines/bin/python3 , indeed /Users/glserran/.pyenv/versions/3.9.7/ has no apache_beam.
e
Ok. That's not quite enough information to debug this. Can you please provide the full command line + output all in one and then also a listing of the site-packages directory of the python interpreter used to execute the command.
s
Copy code
~/.pyenv/versions/pipelines/bin/python3 dist/valoreo.mains.sync_yuju/main.pex
Traceback (most recent call last):
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 517, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 422, in _wrap_coverage
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 453, in _wrap_profiling
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 575, in _execute
    return self.execute_entry(
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 768, in execute_entry
    return self.execute_module(entry_point.module, alter_sys)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 780, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/mains/sync_yuju/main.py", line 4, in <module>
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'



ls ./3.9.7/envs/pipelines/lib/python3.9/site-packages
__pycache__                             docopt-0.6.2-py3.9.egg-info             orjson                                  pyparsing
_distutils_hack                         docopt.py                               orjson-3.7.8.dist-info                  pyparsing-3.0.9.dist-info
apache_beam                             dot_parser.py                           packaging                               python_dateutil-2.8.2.dist-info
apache_beam-2.40.0-py3.9.egg-info       fastavro                                packaging-21.3.dist-info                pytz
bson                                    fastavro-1.5.3.dist-info                pip                                     pytz-2022.1.dist-info
certifi                                 google                                  pip-21.2.3.dist-info                    requests
certifi-2022.6.15.dist-info             gridfs                                  pkg_resources                           requests-2.28.1.dist-info
charset_normalizer                      grpc                                    proto                                   setuptools
charset_normalizer-2.1.0.dist-info      grpcio-1.47.0-py3.9.egg-info            proto_plus-1.20.6.dist-info             setuptools-57.4.0.dist-info
cloudpickle                             hdfs                                    protobuf-3.20.1-nspkg.pth               six-1.16.0.dist-info
cloudpickle-2.1.0.dist-info             hdfs-2.7.0.dist-info                    protobuf-3.20.1.dist-info               six.py
crcmod                                  httplib2                                pyarrow                                 typing_extensions-4.3.0.dist-info
crcmod-1.7-py3.9.egg-info               httplib2-0.20.4.dist-info               pyarrow-7.0.0.dist-info                 typing_extensions.py
dateutil                                idna                                    pydot-1.4.2.dist-info                   urllib3
dill                                    idna-3.3.dist-info                      pydot.py                                urllib3-1.26.10.dist-info
dill-0.3.1.1-py3.9.egg-info             numpy                                   pymongo
distutils-precedence.pth                numpy-1.22.4.dist-info                  pymongo-3.12.3-py3.9.egg-info
e
So, you 
ls ./3.9.7/envs/pipelines/lib/python3.9/site-packages
but you run with 
~/.pyenv/versions/pipelines/bin/python3
instead of 
./3.9.7/envs/pipelines/bin/python
. You must run the PEX with the interpreter whose site-packages you want it to inherit. Can you please try this and report back with full command line + output again?:
Copy code
PEX_VERBOSE=3 PEX_INHERIT_PATH=prefer ./3.9.7/envs/pipelines/bin/python dist/valoreo.mains.sync_yuju/main.pex
s
Got it. This makes it work. The remaining errors are unrelated.
Copy code
PEX_VERBOSE=3 PEX_INHERIT_PATH=prefer ~/.pyenv/versions/3.9.7/envs/pipelines/bin/python dist/valoreo.mains.sync_yuju/main.pex
pex: Laying out PEX zipfile /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex: 0.1ms
pex: Executing installed PEX for /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex at /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex: Installed VendorImporter(root='/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Searching for pyenv root...: 0.0ms
pex: Using $PATH constrained by ['CPython==3.9.7']
pex: Constraints on interpreters: ['CPython==3.9.7'], Matching Interpreter: /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9
pex: Selecting runtime interpreter :: Testing /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9 can resolve PEX at /Users/glserran/.pex/unzipped_pexes/8b65pex: Selecting runtime interpreter :: Testing /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9 can resolve PEX at /Users/glserran/.pex/unzipped_pexes/8b65pex: Selecting runtime interpreter: 0.8ms                                                                                                                                                                                                                                                                                         
pex:   Testing /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9 can resolve PEX at /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8: 0.2ms
pex:     Searching dependency cache: /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.deps: 0.0ms
pex: Using the current interpreter /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python since it matches constraints and PYTHONPATH is not set.
pex: Found site-library: /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/lib/python3.9/site-packages
pex: Found .pth file: /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/lib/python3.9/site-packages/distutils-precedence.pth
pex: Found .pth file: /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/lib/python3.9/site-packages/protobuf-3.20.1-nspkg.pth
pex: Not a tainted path element: /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap
pex: Not a tainted path element: /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex: Not a tainted path element: /Users/glserran/.pyenv/versions/3.9.7/lib/python39.zip
pex: Not a tainted path element: /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9
pex: Not a tainted path element: /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload
pex: Tainted path element: /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/lib/python3.9/site-packages
pex: Dropping google
pex: New sys.path: ['/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap', '/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8', '/Users/glserran/.pyenv/versions/3.9.7/lib/python39.zip', '/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9', '/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload', '/Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/lib/python3.9/site-packages']
pex: Activating PEX virtual environment from /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8: 0.1ms
pex: Not setting process title since setproctitle is not available in /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex
pex: Bootstrap complete, performing final sys.path modifications...
pex: Uninstalled VendorImporter(root='/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Demoting code from Bootstrap(sys_path_entry='/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap')
pex: Re-imported pex from ['/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex']
pex: PYTHONPATH contains:
pex:     /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex:   * /Users/glserran/.pyenv/versions/3.9.7/lib/python39.zip
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload
pex:     /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/lib/python3.9/site-packages
pex:     /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Traceback (most recent call last):
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 517, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 422, in _wrap_coverage
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 453, in _wrap_profiling
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 575, in _execute
    return self.execute_entry(
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 768, in execute_entry
    return self.execute_module(entry_point.module, alter_sys)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 780, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/mains/sync_yuju/main.py", line 13, in <module>
    from pipelines.orders.sync_yuju_orders.sync_yuju_orders_pipeline import query_market_orders_in_range, \
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/pipelines/orders/sync_yuju_orders/sync_yuju_orders_pipeline.py", line 3, in <module>
    from clients.yuju_client import yuju_client, ShopMarket, Query
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/clients/yuju_client.py", line 78, in <module>
    yuju_client = YujuClient("<https://api.software.madkting.com>")
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/clients/yuju_client.py", line 37, in __init__
    self.session.headers.update({"Authorization": environ['YUJU-AUTH-TOKEN']})
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'YUJU-AUTH-TOKEN'
However if I run ~/.pyenv/versions/3.9.7/envs/pipelines/bin/python dist/valoreo.mains.sync_yuju/main.pex there is still the same issue:
Copy code
~/.pyenv/versions/3.9.7/envs/pipelines/bin/python dist/valoreo.mains.sync_yuju/main.pex 
Traceback (most recent call last):
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 517, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 422, in _wrap_coverage
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 453, in _wrap_profiling
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 575, in _execute
    return self.execute_entry(
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 768, in execute_entry
    return self.execute_module(entry_point.module, alter_sys)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 780, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/mains/sync_yuju/main.py", line 4, in <module>
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
it seems to me that inherit_path config is not being applied into the pex.
e
What Pants version is this?
s
2.12.0
e
Ok. Two more things to try then: 1. Inspect the PEX info: 
unzip -qc dist/valoreo.mains.sync_yuju/main.pex PEX-INFO | python -mjson.tool
2. If step 1 shows an unexpected value for 
"inherit_path"
embedded in the PEX, then re-run the 
./pants package
command using 
./pants -ldebug --no-local-cahe package ...
and report full command line + output again.
s
Copy code
unzip -qc dist/valoreo.mains.sync_yuju/main.pex PEX-INFO | python -mjson.tool | grep inherit_path
    "inherit_path": "prefer",
seems fine
e
Can you please run this and report full output?:
Copy code
PEX_VERBOSE=3 ~/.pyenv/versions/3.9.7/envs/pipelines/bin/python dist/valoreo.mains.sync_yuju/main.pex
s
Yes, thanks a lot for the help
Copy code
PEX_VERBOSE=3 ~/.pyenv/versions/3.9.7/envs/pipelines/bin/python dist/valoreo.mains.sync_yuju/main.pex
pex: Laying out PEX zipfile /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex: 0.1ms
pex: Executing installed PEX for /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex at /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex: Installed VendorImporter(root='/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Searching for pyenv root...: 0.1ms
pex: Using $PATH constrained by ['CPython==3.9.7']
pex: Constraints on interpreters: ['CPython==3.9.7'], Matching Interpreter: /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9
pex: Selecting runtime interpreter :: Testing /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9 can resolve PEX at /Users/glserran/.pex/unzipped_pexes/8b65pex: Selecting runtime interpreter :: Testing /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9 can resolve PEX at /Users/glserran/.pex/unzipped_pexes/8b65pex: Selecting runtime interpreter: 1.1ms                                                                                                                                                                                                                                                                                         
pex:   Testing /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python3.9 can resolve PEX at /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8: 0.2ms
pex:     Searching dependency cache: /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.deps: 0.0ms
pex: Resolved base interpreter of PythonInterpreter('/Users/glserran/.pyenv/versions/3.9.7/bin/python3.9', PythonIdentity('/Users/glserran/.pyenv/versions/3.9.7/bin/python3.9', 'cp39', 'cp39', 'macosx_12_0_arm64', (3, 9, 7))) from virtual environment at /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines
pex: Re-executing: cmdline='/Users/glserran/.pyenv/versions/3.9.7/bin/python3.9 /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex', sys.executable='/Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python', PEX_PYTHON=None, PEX_PYTHON_PATH=None, interpreter_constraints=['CPython==3.9.7']
pex: Laying out PEX zipfile /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex: 0.1ms
pex: Executing installed PEX for /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex at /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex: Installed VendorImporter(root='/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Searching for pyenv root...: 0.2ms
pex: Found site-library: /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/site-packages
pex: Found .pth file: /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/site-packages/distutils-precedence.pth
pex: Not a tainted path element: /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap
pex: Not a tainted path element: /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex: Not a tainted path element: /Users/glserran/.pyenv/versions/3.9.7/lib/python39.zip
pex: Not a tainted path element: /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9
pex: Not a tainted path element: /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload
pex: Tainted path element: /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/site-packages
pex: New sys.path: ['/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap', '/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8', '/Users/glserran/.pyenv/versions/3.9.7/lib/python39.zip', '/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9', '/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload', '/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/site-packages']
pex: Activating PEX virtual environment from /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8 :: Searching dependency cache: /Users/glserran/.pepex: Activating PEX virtual environment from /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8: 0.1ms                                                                                                           
pex:   Searching dependency cache: /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.deps: 0.0ms
pex: Not setting process title since setproctitle is not available in /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex
pex: Bootstrap complete, performing final sys.path modifications...
pex: Uninstalled VendorImporter(root='/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Demoting code from Bootstrap(sys_path_entry='/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap')
pex: Re-imported pex from ['/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex']
pex: PYTHONPATH contains:
pex:     /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex:   * /Users/glserran/.pyenv/versions/3.9.7/lib/python39.zip
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/site-packages
pex:     /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Traceback (most recent call last):
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 517, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 422, in _wrap_coverage
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 453, in _wrap_profiling
    return runner(*args)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 575, in _execute
    return self.execute_entry(
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 768, in execute_entry
    return self.execute_module(entry_point.module, alter_sys)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap/pex/pex.py", line 780, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/mains/sync_yuju/main.py", line 4, in <module>
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
e
Aha:
Copy code
pex: Resolved base interpreter of PythonInterpreter('/Users/glserran/.pyenv/versions/3.9.7/bin/python3.9', PythonIdentity('/Users/glserran/.pyenv/versions/3.9.7/bin/python3.9', 'cp39', 'cp39', 'macosx_12_0_arm64', (3, 9, 7))) from virtual environment at /Users/glserran/.pyenv/versions/3.9.7/envs/pipelines
pex: Re-executing: cmdline='/Users/glserran/.pyenv/versions/3.9.7/bin/python3.9 /Users/glserran/src/valoreo-pipelines/dist/valoreo.mains.sync_yuju/main.pex', sys.executable='/Users/glserran/.pyenv/versions/3.9.7/envs/pipelines/bin/python', PEX_PYTHON=None, PEX_PYTHON_PATH=None, interpreter_constraints=['CPython==3.9.7']
...
pex: PYTHONPATH contains:
pex:     /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8
pex:   * /Users/glserran/.pyenv/versions/3.9.7/lib/python39.zip
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload
pex:     /Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/site-packages
pex:     /Users/glserran/.pex/unzipped_pexes/8b65f17a2cce6186e843cd4259b56c7e7b76e1d8/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Note that the PEX re-executes itself using 
/Users/glserran/.pyenv/versions/3.9.7/bin/python3.9
and the resulting 
sys.path
has 
/Users/glserran/.pyenv/versions/3.9.7/lib/python3.9/site-packages
and not 
~/.pyenv/versions/3.9.7/envs/pipelines/lib/python3.9/site-packages
.
For whatever reason, the 
PEX_INHERIT_PATH=prefer ...
approach did not lead to a re-execution if you examine its logs. I'll need to try to reproduce this difference locally and I'll file a Pex bug here shortly if I can. Thanks for the detailed debug information @strong-refrigerator-32393. In the mean-time can you work around by using the 
PEX_INHERIT_PATH=prefer ...
technique?
s
go it. Thanks for the help. Using this method is viable for me! If you end up logging a bug. Please let me know.
e
This is now fixed in Pex 2.1.102.
3 Views
Open in Slack
PreviousNext
Powered by