https://pantsbuild.org/ logo
#general
Title
# general
c

cuddly-egg-75110

11/19/2020, 9:49 PM
Hi all! Whenever I try to do something with my source like
./pants lint ::
or
./pants repl ::
I get a
ModuleNotFoundError: No module named 'pex.bin'
, has anyone seen this before? Google isn't giving me any hints. Full stack-trace in thread, thanks for the help and the awesome product!
Copy code
13:48:37.99 [INFO] initialization options changed: reinitializing pantsd...
13:48:38.33 [INFO] pantsd initialized.
13:48:38.41 [INFO] No pyenv binary found. Will not use pyenv interpreters.
13:48:39.38 [INFO] Completed: Building pylint.pex with 1 requirement: pylint>=2.4.4,<2.5
13:48:39.39 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 255, in run
    engine_result = self._run_v2(goals)
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 182, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 204, in _maybe_run_v2_body
    poll_delay=(0.1 if poll else None),
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 126, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pylint.pex with 1 requirement: pylint>=2.4.4,<2.5' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 446, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 378, in _wrap_coverage
    return runner(*args)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 409, in _wrap_profiling
    return runner(*args)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 508, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 610, in execute_entry
    return runner(entry_point)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 625, in execute_pkg_resources
    runner = entry.resolve()
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'pex.bin'
I've tried completely wiping
~/.cache/pants
and all the temp Pants directories in my project, too.
h

hundreds-father-404

11/19/2020, 9:56 PM
Hey Ziyad! Welcome, and sorry you’re running into this issue. I have not seen that before and am trying to think through what’s going on. Could you please rerun with
--pex-verbosity=9
, then upload here or copy into a gist? (Warning it’ll be a lot of text)
c

cuddly-egg-75110

11/19/2020, 10:00 PM
Thanks for replying so quickly, here is a Gist: https://gist.github.com/ziyadedher/72575d5477d5934ac78de748f8fe9403
w

witty-crayon-22786

11/19/2020, 10:01 PM
thanks: i can take a look at this
❤️ 1
c

cuddly-egg-75110

11/19/2020, 10:10 PM
Interesting, it works if I have the following in my
pants.toml
,
3.7
also does not work
Copy code
[python-setup]
interpreter_constraints = ["CPython>=3.8"]
w

witty-crayon-22786

11/19/2020, 10:11 PM
from your verbose output, it’s choosing to use
/usr/bin/python3.8
Copy code
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
it probably wouldn’t hurt to try out 3.6 or 3.7, but to do that you’d need to set something like
["CPython>=3.6,<3.7"]
etc
h

hundreds-father-404

11/19/2020, 10:14 PM
But later it says:
Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/local/bin/python3.7
, and that’s the Pex process that fails So I suspect there’s an issue with
/usr/local/bin/python3.7
. And that you avoid it by setting
>=3.8
for your constraints because that interpreter ends up getting skipped (When running Pylint, we use whatever interpreter corresponds to the code you have, as it determines what AST Pylint uses)
w

witty-crayon-22786

11/19/2020, 10:14 PM
ah, yea. sorry.
h

hundreds-father-404

11/19/2020, 10:15 PM
So now the question is why does
/usr/local/bin/python3.7
not play nicely
c

cuddly-egg-75110

11/19/2020, 10:15 PM
Ahhh that makes sense! Thank you! Using 3.8 is fine so I can just set the interpreter constraint, but I can investigate and see why 3.7 isn't happy, let me know if you have any hunches
w

witty-crayon-22786

11/19/2020, 10:17 PM
@cuddly-egg-75110: if this code isn’t sensitive, do you think you could zip up the content of
/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap
?
💯 1
h

hundreds-father-404

11/19/2020, 10:17 PM
I’m glad you were able to get it working! But yes I’d love to figure out what’s going on Could you also please run
/usr/local/bin/python3.7 -c 'import sys; print(sys.implementation)'
?
c

cuddly-egg-75110

11/19/2020, 10:21 PM
@hundreds-father-404:
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-37', hexversion=50791920, name='cpython', version=sys.version_info(major=3, minor=7, micro=5, releaselevel='final', serial=0))
@witty-crayon-22786: Sure thing, its just a toy codebase I am trying this out on right now. Attached!
❤️ 1
w

witty-crayon-22786

11/19/2020, 10:26 PM
…indeed,
pex/bin/**
is not in there.
😂 1
👀 1
c

cuddly-egg-75110

11/19/2020, 10:28 PM
I'm heading into a few meetings right now, I'll try to stay active here but I might take a bit longer to reply! Thanks again for your help!
h

hundreds-father-404

11/19/2020, 10:30 PM
Sounds good! Thanks for this debug info. And let us know if there’s anything else we can help with or any feedback on things like docs or ideas to make Pants better 🙂
💯 1
w

witty-crayon-22786

11/19/2020, 11:18 PM
@cuddly-egg-75110: hey, would you mind zipping up
/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6
when you get the chance? sorry about that
c

cuddly-egg-75110

11/19/2020, 11:20 PM
@witty-crayon-22786 Sure thing, here it is!
👍 1
w

witty-crayon-22786

11/19/2020, 11:47 PM
@cuddly-egg-75110: thanks for the help: making progress i think! are you able to share the contents of your requirements file?
and if you use a constraints.txt, that as well?
filed https://github.com/pantsbuild/pants/issues/11211 about this… if you’re comfortable including your requirements.txt or constraints.txt there, or any other information that might be helpful reproducing, it would be awesome.
👍 1
h

hundreds-father-404

11/20/2020, 12:00 AM
Note that pylint.pex requirements don’t come from requirements.txt - it comes from options on
[pylint]
. The constraints.txt can be used though
w

witty-crayon-22786

11/20/2020, 12:01 AM
…true.
c

cuddly-egg-75110

11/20/2020, 12:12 AM
This happens even if my
requirements.txt
is empty! (And I don't use
constraints.txt
yet, waiting on that
poetry
native support :P)
h

hundreds-father-404

11/20/2020, 12:13 AM
Ziyad, would you be willing to run with both
--pex-verbosity=9
and
-ldebug
please? (And temporarily reverting the interpreter constraints workaround) Stu, that’ll give us the argv being used to create pylint.pex
w

witty-crayon-22786

11/20/2020, 12:14 AM
👍
c

cuddly-egg-75110

11/20/2020, 12:17 AM
Sure thing, here's a Gist for the output of that: https://gist.github.com/ziyadedher/6802dd25dcd059ef18d68eae09e02b87
❤️ 1
Attached is a minimal example that produces the error for me, and the output of
./pants repl :: --pex-verbosity=9 -ldebug
in that example is https://gist.github.com/ziyadedher/4d2e99e2cc0e44bb2379b6e07f7754f8
🔥 1
💯 1
w

witty-crayon-22786

11/20/2020, 1:08 AM
thanks a ton! i’ll look at this tomorrow.
c

cuddly-egg-75110

11/20/2020, 8:30 PM
Strange, I am now getting issues with 3.8 as well if I use my system install, but not if I use my virtual env install.
w

witty-crayon-22786

11/20/2020, 8:32 PM
when you say system install, what do you mean? is this a linux?
h

hundreds-father-404

11/20/2020, 8:32 PM
Also see https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path for how to change the interpreter search path to avoid system python
c

cuddly-egg-75110

11/20/2020, 8:37 PM
Yeah running on Arch Linux, system install is in
/usr/bin
. That workaround works @hundreds-father-404 though I need to use the
PATH
env variable rather than the config flag since it doesn't support relative paths
w

witty-crayon-22786

11/20/2020, 8:39 PM
would you mind re-running
Copy code
/usr/bin/$BIN -c 'import sys; print(sys.implementation)'
… for the relevant python 3.8 $BIN ?
c

cuddly-egg-75110

11/20/2020, 8:41 PM
Sure thing, I hope I am not taking up too much of your time! I really appreciate the help!
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50857712, name='cpython', version=sys.version_info(major=3, minor=8, micro=6, releaselevel='final', serial=0))
w

witty-crayon-22786

11/20/2020, 8:41 PM
and are you running pants under a manually created venv?
c

cuddly-egg-75110

11/20/2020, 8:41 PM
It succeeds when I run it under the manually created venv, but not under that binary under
/usr/bin
i.e. I add something like
Copy code
interpreter_search_paths = [
    "/path/to/my/project/.venv/bin",
]
it works but if I leave it out it fails, both with
interpreter_constraints = ["CPython>=3.8"]
And this is the output of
sys.implementation
from my
.venv
Python:
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50857712, name='cpython', version=sys.version_info(major=3, minor=8, micro=6, releaselevel='final', serial=0))
It looks the same
w

witty-crayon-22786

11/20/2020, 8:45 PM
interesting. do your system installs have anything interesting in
pip list
…? i’m wondering whether the venv is isolating you from some sort of package collision with site-packages
c

cuddly-egg-75110

11/20/2020, 8:48 PM
Nothing that catches my eye, there are a few packages though, anything I should be looking out for?
w

witty-crayon-22786

11/20/2020, 8:48 PM
things like
setuptools
…being at … odd versions? or anything related to packaging, essentially. pex tries to vendor everything it needs, but
c

cuddly-egg-75110

11/20/2020, 8:49 PM
setuptools 50.3.2
is in there, but its at the same version as my venv
w

witty-crayon-22786

11/20/2020, 8:49 PM
yea, true.
possibly relevant are:
packaging
,
wheel
,
setuptools
c

cuddly-egg-75110

11/20/2020, 8:51 PM
all same version too 😕
Let me try generating a new clean venv and see what happens
If it helps at all I am using
poetry
to manage my virtual env
h

hundreds-father-404

11/20/2020, 8:52 PM
Thanks! Also what Linux distro? We’ve known macOS to have many issues with system Python, but Linux usually behaves well
c

cuddly-egg-75110

11/20/2020, 8:52 PM
I'm using Arch, which could be the reason why. I know MacOS has some Path resolving stuff that doesn't perform nicely is there anything else maybe MacOS related that I can try here?
w

witty-crayon-22786

11/20/2020, 8:52 PM
this wouldn’t be the first isolation issue we’ve seen though… https://github.com/pantsbuild/pex/issues/1031 comes to mind
👍 1
but the fact that is so widespread makes me suspect that it should be reproducible in a docker image with the right set of packages globally pip installed
c

cuddly-egg-75110

11/20/2020, 8:57 PM
Is getting a path like
File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 255, in run
as part of my stacktrace normal if my
interpreter_constraints
is
CPython>=3.8
?
w

witty-crayon-22786

11/20/2020, 8:58 PM
yes: that is the python that pants is using to run, as opposed to the python it is running your code with
💯 1
there are two stacktraces: the outer one is pants failing because of the inner one (pex failing in a subprocess in a different interpreter)
c

cuddly-egg-75110

11/20/2020, 9:00 PM
Yep, makes sense
w

witty-crayon-22786

11/20/2020, 9:00 PM
@cuddly-egg-75110: ok, sorry. we’ve been all over the place here. if you’re able to describe on https://github.com/pantsbuild/pants/issues/11211 : 1. which Arch version you are using 2. which system interpreters cause issues 3. roughly what is pip installed for those interpreters … that feels like maybe a way for us to reproduce
thanks a ton for your time here.
❤️ 1
c

cuddly-egg-75110

11/20/2020, 9:00 PM
Sure thing
Thank you! I know its super annoying to debug things remotely hahaha
👍 1
Added a comment. Theres something that's happening that really confuses me now, running
./pants repl
if I have
CPython>=3.8
passes, but
./pants lint ::
fails with the error without changing anything else.
👍 1
e

enough-analyst-54434

11/25/2020, 4:02 AM
@cuddly-egg-75110 I have a docker-based repro attempt documented here: https://github.com/pantsbuild/pants/issues/11211#issuecomment-733448253 If you get a chance to look at that and get the missing ingredients added to the Dockerfile that can make this reproducibly repro, I'd be grateful.
I will note that there is no need / point to creating venvs for Pants to use. Pants will make isolated venvs (using Pex) for any interpreters you point it to.
As such the
pip list
debugging, which would make sense in the absence of Pex use, should have been irrelevant.
@cuddly-egg-75110 you provided some zips here that are not attached to the issue. If it's OK to attach those let me know and I'll do it. In the meantime though, my docker experiment backs up Eric's conclusion that your /usr/local/bin/python3.7 is the place to be looking. As such, can you run the equivalent of this?:
Copy code
jsirois@gill ~/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6 $ PEX_VERBOSE=1 python3.7 . --version
pex: Using the current interpreter /usr/bin/python3.7 since it matches constraints and PYTHONPATH is not set.
pex: Found site-library: /usr/lib/python3.7/site-packages
pex: Tainted path element: /usr/lib/python3.7/site-packages
pex: Scrubbing from user site: /home/jsirois/.local/lib/python3.7/site-packages
pex: Scrubbing from site-packages: /usr/lib/python3.7/site-packages
pex: New sys.path: ['/home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap', '/home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload']
pex: Activating PEX virtual environment from .: 8.8ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:     /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6
pex:   * /usr/lib/python37.zip
pex:     /usr/lib/python3.7
pex:     /usr/lib/python3.7/lib-dynload
pex:     /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.deps/pex-2.1.21-py2.py3-none-any.whl
pex:     /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
. 2.1.21
The key thing to observe in this good result is the `pex: PYTHONPATH contains:`list which critically contains
.deps/pex-2.1.21-py2.py3-none-an.whl
(pex: /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.deps/pex-2.1.21-py2.py3-none-any.whl). That wheen is the source for the
pex.bin
package your setup cannot find.
N.B.: the
PEX_VERBOSE=1 python3.7 . --version
must be run from the
.../42d4f75d046397e5f014ec8afd1a39b3ba8787c6/
directory for this to work verbatim.
c

cuddly-egg-75110

11/27/2020, 12:17 AM
Hi, sorry for the late reply on this. I have been trying to reproduce this in my docker container to no avail. I assume there is something very weird that is happening with my local Python version (I think I might have compiled my local Python 3.7 from source?) I am going to give it another shot tonight and I will update you if I find anything interesting!
4 Views