https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all! Whenever I try to do something with my sou...
# general
c
Hi all! Whenever I try to do something with my source like 
./pants lint ::
or 
./pants repl ::
I get a 
ModuleNotFoundError: No module named 'pex.bin'
, has anyone seen this before? Google isn't giving me any hints. Full stack-trace in thread, thanks for the help and the awesome product!
Copy code
13:48:37.99 [INFO] initialization options changed: reinitializing pantsd...
13:48:38.33 [INFO] pantsd initialized.
13:48:38.41 [INFO] No pyenv binary found. Will not use pyenv interpreters.
13:48:39.38 [INFO] Completed: Building pylint.pex with 1 requirement: pylint>=2.4.4,<2.5
13:48:39.39 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 255, in run
    engine_result = self._run_v2(goals)
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 182, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 204, in _maybe_run_v2_body
    poll_delay=(0.1 if poll else None),
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 126, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pylint.pex with 1 requirement: pylint>=2.4.4,<2.5' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 446, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 378, in _wrap_coverage
    return runner(*args)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 409, in _wrap_profiling
    return runner(*args)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 508, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 610, in execute_entry
    return runner(entry_point)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 625, in execute_pkg_resources
    runner = entry.resolve()
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'pex.bin'
I've tried completely wiping 
~/.cache/pants
and all the temp Pants directories in my project, too.
h
Hey Ziyad! Welcome, and sorry you’re running into this issue. I have not seen that before and am trying to think through what’s going on. Could you please rerun with 
--pex-verbosity=9
, then upload here or copy into a gist? (Warning it’ll be a lot of text)
c
Thanks for replying so quickly, here is a Gist: https://gist.github.com/ziyadedher/72575d5477d5934ac78de748f8fe9403
w
thanks: i can take a look at this
❤️ 1
c
Interesting, it works if I have the following in my 
pants.toml
, 
3.7
also does not work
Copy code
[python-setup]
interpreter_constraints = ["CPython>=3.8"]
w
from your verbose output, it’s choosing to use 
/usr/bin/python3.8
Copy code
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
it probably wouldn’t hurt to try out 3.6 or 3.7, but to do that you’d need to set something like 
["CPython>=3.6,<3.7"]
etc
h
But later it says: 
Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/local/bin/python3.7
, and that’s the Pex process that fails So I suspect there’s an issue with 
/usr/local/bin/python3.7
. And that you avoid it by setting 
>=3.8
for your constraints because that interpreter ends up getting skipped (When running Pylint, we use whatever interpreter corresponds to the code you have, as it determines what AST Pylint uses)
w
ah, yea. sorry.
h
So now the question is why does 
/usr/local/bin/python3.7
not play nicely
c
Ahhh that makes sense! Thank you! Using 3.8 is fine so I can just set the interpreter constraint, but I can investigate and see why 3.7 isn't happy, let me know if you have any hunches
w
@cuddly-egg-75110: if this code isn’t sensitive, do you think you could zip up the content of 
/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap
?
💯 1
h
I’m glad you were able to get it working! But yes I’d love to figure out what’s going on Could you also please run 
/usr/local/bin/python3.7 -c 'import sys; print(sys.implementation)'
?
c
@hundreds-father-404: 
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-37', hexversion=50791920, name='cpython', version=sys.version_info(major=3, minor=7, micro=5, releaselevel='final', serial=0))
@witty-crayon-22786: Sure thing, its just a toy codebase I am trying this out on right now. Attached!
pants.zip
❤️ 1
w
…indeed, 
pex/bin/**
is not in there.
😂 1
👀 1
c
I'm heading into a few meetings right now, I'll try to stay active here but I might take a bit longer to reply! Thanks again for your help!
h
Sounds good! Thanks for this debug info. And let us know if there’s anything else we can help with or any feedback on things like docs or ideas to make Pants better 🙂
💯 1
w
@cuddly-egg-75110: hey, would you mind zipping up 
/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6
when you get the chance? sorry about that
c
@witty-crayon-22786 Sure thing, here it is!
pants.zip
👍 1
w
@cuddly-egg-75110: thanks for the help: making progress i think! are you able to share the contents of your requirements file?
and if you use a constraints.txt, that as well?
filed https://github.com/pantsbuild/pants/issues/11211 about this… if you’re comfortable including your requirements.txt or constraints.txt there, or any other information that might be helpful reproducing, it would be awesome.
👍 1
h
Note that pylint.pex requirements don’t come from requirements.txt - it comes from options on 
[pylint]
. The constraints.txt can be used though
w
…true.
c
This happens even if my 
requirements.txt
is empty! (And I don't use 
constraints.txt
yet, waiting on that 
poetry
native support :P)
h
Ziyad, would you be willing to run with both 
--pex-verbosity=9
and 
-ldebug
please? (And temporarily reverting the interpreter constraints workaround) Stu, that’ll give us the argv being used to create pylint.pex
w
👍
c
Sure thing, here's a Gist for the output of that: https://gist.github.com/ziyadedher/6802dd25dcd059ef18d68eae09e02b87
❤️ 1
Attached is a minimal example that produces the error for me, and the output of 
./pants repl :: --pex-verbosity=9 -ldebug
in that example is https://gist.github.com/ziyadedher/4d2e99e2cc0e44bb2379b6e07f7754f8
mre.zip
🔥 1
💯 1
w
thanks a ton! i’ll look at this tomorrow.
c
Strange, I am now getting issues with 3.8 as well if I use my system install, but not if I use my virtual env install.
w
when you say system install, what do you mean? is this a linux?
h
Also see https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path for how to change the interpreter search path to avoid system python
c
Yeah running on Arch Linux, system install is in 
/usr/bin
. That workaround works @hundreds-father-404 though I need to use the 
PATH
env variable rather than the config flag since it doesn't support relative paths
w
would you mind re-running
Copy code
/usr/bin/$BIN -c 'import sys; print(sys.implementation)'
… for the relevant python 3.8 $BIN ?
c
Sure thing, I hope I am not taking up too much of your time! I really appreciate the help! 
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50857712, name='cpython', version=sys.version_info(major=3, minor=8, micro=6, releaselevel='final', serial=0))
w
and are you running pants under a manually created venv?
c
It succeeds when I run it under the manually created venv, but not under that binary under 
/usr/bin
i.e. I add something like
Copy code
interpreter_search_paths = [
    "/path/to/my/project/.venv/bin",
]
it works but if I leave it out it fails, both with 
interpreter_constraints = ["CPython>=3.8"]
And this is the output of 
sys.implementation
from my 
.venv
Python: 
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50857712, name='cpython', version=sys.version_info(major=3, minor=8, micro=6, releaselevel='final', serial=0))
It looks the same
w
interesting. do your system installs have anything interesting in 
pip list
…? i’m wondering whether the venv is isolating you from some sort of package collision with site-packages
c
Nothing that catches my eye, there are a few packages though, anything I should be looking out for?
w
things like 
setuptools
…being at … odd versions? or anything related to packaging, essentially. pex tries to vendor everything it needs, but
c
setuptools 50.3.2
is in there, but its at the same version as my venv
w
yea, true.
possibly relevant are: 
packaging
, 
wheel
, 
setuptools
c
all same version too 😕
Let me try generating a new clean venv and see what happens
If it helps at all I am using 
poetry
to manage my virtual env
h
Thanks! Also what Linux distro? We’ve known macOS to have many issues with system Python, but Linux usually behaves well
c
I'm using Arch, which could be the reason why. I know MacOS has some Path resolving stuff that doesn't perform nicely is there anything else maybe MacOS related that I can try here?
w
this wouldn’t be the first isolation issue we’ve seen though… https://github.com/pantsbuild/pex/issues/1031 comes to mind
👍 1
but the fact that is so widespread makes me suspect that it should be reproducible in a docker image with the right set of packages globally pip installed
c
Is getting a path like 
File "/home/ziyadedher/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 255, in run
as part of my stacktrace normal if my 
interpreter_constraints
is 
CPython>=3.8
?
w
yes: that is the python that pants is using to run, as opposed to the python it is running your code with
💯 1
there are two stacktraces: the outer one is pants failing because of the inner one (pex failing in a subprocess in a different interpreter)
c
Yep, makes sense
w
@cuddly-egg-75110: ok, sorry. we’ve been all over the place here. if you’re able to describe on https://github.com/pantsbuild/pants/issues/11211 : 1. which Arch version you are using 2. which system interpreters cause issues 3. roughly what is pip installed for those interpreters … that feels like maybe a way for us to reproduce
thanks a ton for your time here.
❤️ 1
c
Sure thing
Thank you! I know its super annoying to debug things remotely hahaha
👍 1
Added a comment. Theres something that's happening that really confuses me now, running 
./pants repl
if I have 
CPython>=3.8
passes, but 
./pants lint ::
fails with the error without changing anything else.
👍 1
e
@cuddly-egg-75110 I have a docker-based repro attempt documented here: https://github.com/pantsbuild/pants/issues/11211#issuecomment-733448253 If you get a chance to look at that and get the missing ingredients added to the Dockerfile that can make this reproducibly repro, I'd be grateful.
I will note that there is no need / point to creating venvs for Pants to use. Pants will make isolated venvs (using Pex) for any interpreters you point it to.
As such the 
pip list
debugging, which would make sense in the absence of Pex use, should have been irrelevant.
@cuddly-egg-75110 you provided some zips here that are not attached to the issue. If it's OK to attach those let me know and I'll do it. In the meantime though, my docker experiment backs up Eric's conclusion that your /usr/local/bin/python3.7 is the place to be looking. As such, can you run the equivalent of this?:
Copy code
jsirois@gill ~/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6 $ PEX_VERBOSE=1 python3.7 . --version
pex: Using the current interpreter /usr/bin/python3.7 since it matches constraints and PYTHONPATH is not set.
pex: Found site-library: /usr/lib/python3.7/site-packages
pex: Tainted path element: /usr/lib/python3.7/site-packages
pex: Scrubbing from user site: /home/jsirois/.local/lib/python3.7/site-packages
pex: Scrubbing from site-packages: /usr/lib/python3.7/site-packages
pex: New sys.path: ['/home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap', '/home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload']
pex: Activating PEX virtual environment from .: 8.8ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:     /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6
pex:   * /usr/lib/python37.zip
pex:     /usr/lib/python3.7
pex:     /usr/lib/python3.7/lib-dynload
pex:     /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.deps/pex-2.1.21-py2.py3-none-any.whl
pex:     /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
. 2.1.21
The key thing to observe in this good result is the `pex: PYTHONPATH contains:`list which critically contains 
.deps/pex-2.1.21-py2.py3-none-an.whl
(pex: /home/jsirois/Desktop/issues/arch/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.deps/pex-2.1.21-py2.py3-none-any.whl). That wheen is the source for the 
pex.bin
package your setup cannot find.
N.B.: the 
PEX_VERBOSE=1 python3.7 . --version
must be run from the 
.../42d4f75d046397e5f014ec8afd1a39b3ba8787c6/
directory for this to work verbatim.
c
Hi, sorry for the late reply on this. I have been trying to reproduce this in my docker container to no avail. I assume there is something very weird that is happening with my local Python version (I think I might have compiled my local Python 3.7 from source?) I am going to give it another shot tonight and I will update you if I find anything interesting!
4 Views
Open in Slack
PreviousNext
Powered by