in the debug line where it says, "searching for python3" it is then followed by a test of python2

must be parallel process stuff. I see where it completes the python3 tests.

Yeah, we look for any of

python

,

python2

, and

python3

for the interpreter to run Pex, and test that they’re valid binaries. That all happens in parallel.

but if it is finding the python3 interpreter, how come it fails later saying "No interpreter compatible with the requested constraints was found: Version matches CPython>=3.6"?

And is this two different issues or one?

Cool, you’re right that Pants is able to find an interp to run Pex with. There are two different context we look for an interpreter: 1. what to run the Pex CLI tool with, which can be any valid 2.7 or 3.5+ interp 2. what to use for your own code, which we delegate to Pex because it understands things like interpreter constraints #1 is working as we’d expect, but #2 is not for some reason. That error message you’re getting is coming from Pex. And it looks like Pants is properly setting everything for Pex, so I suspect there could be an issue with Pex itself

@jolly-midnight-72759 can you gather debug output again but this time with

--pex-verbosity=9

?

John that won’t work, unfortunately. It’s not wired up properly. I’m pushing the cherry-pick of the fix right now

So no need for

pex-verbosity

?

Aha. Ok, @jolly-midnight-72759 do you have the ability to download pex on that machine? Say via curl?

I do. This is a POC box so I can do ANYTHING bwahahahah. But I need to step AFK for about 90 min.

K, I'll write up what you can try when you get back.

Ah, nm. I'm always too slow to pull out the right tool for this sort of debugging. With this Dockerfile:

FROM ubuntu:18.04

RUN apt update && apt upgrade -y && apt install -y locales language-pack-en
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8

RUN apt install -y curl python python3-dev git vim build-essential unzip tar

I reproduce using:

$ docker run --rm -it raul:3.6
root@f1a736edc580:/# git clone <https://github.com/pantsbuild/example-python>
Cloning into 'example-python'...
remote: Enumerating objects: 329, done.
remote: Total 329 (delta 0), reused 0 (delta 0), pack-reused 329
Receiving objects: 100% (329/329), 74.15 KiB | 1.65 MiB/s, done.
Resolving deltas: 100% (197/197), done.
root@f1a736edc580:/# cd example-python
root@f1a736edc580:/example-python# ./pants fmt lint typecheck test ::
...
New virtual environment successfully created at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36.
17:11:46.96 [INFO] initializing pantsd...
17:11:47.83 [INFO] pantsd initialized.
17:11:48.02 [INFO] No pyenv binary found. Will not use pyenv interpreters.
17:11:54.18 [INFO] Completed: Building docformatter.pex with 1 requirement: docformatter>=1.3.1,<1.4
17:11:54.53 [INFO] Completed: Format with docformatter - made no changes.
17:11:54.88 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 289, in run
    engine_result = self._run_v2()
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 195, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 217, in _maybe_run_v2_body
    poll_delay=(0.1 if poll else None),
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/site-packages/pants/init/engine_initializer.py", line 127, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython>=3.6' failed with exit code 102.
stdout:

stderr:
Could not find a compatible interpreter.

Examined the following interpreters:
1.) /usr/bin/python2.7 CPython==2.7.17

No interpreter compatible with the requested constraints was found:
  Version matches CPython>=3.6



(Use --print-stacktrace to see more error details.)
root@f1a736edc580:/example-python#

If it is reproducible do you want me to open an issue?

I can take it from here @jolly-midnight-72759 and open the issue. Thank you.

OK - this is a latent Pex bug triggered by Pants recent switch to using `--python-path`: https://github.com/pantsbuild/pex/issues/1109

Ah ha! Benjy was onto something when he found that part concerning. We agreed to not touch it in the PR adding

--python-path

to reduce # of changes

w00t! 🦉

Ok, found those comment @hundreds-father-404 and responded. Things cannot be simplified as suggested, so I cleared that up.

Is this also the root cause of the

PermissionDenied

error which started me down this path?

And the reason I don't see this bug on my mac is because

<PYENV>

comes before

<PYPATH>

, yes?

I don’t suspect they’re related. cc @witty-crayon-22786 on the PermissionDenied, Only triggers when

--pantsd

is used.

So let's start a new thread for

PermissionDenied

. 🛒

And the reason I don’t see this bug on my mac is because <PYENV> comes before <PYPATH>, yes?

The ordering doesn’t matter for

interpreter_search_paths

. It works on your mac because you have interpreters in two distinct directories, so you’re not hitting this edge case where the sibling interpreters to the current interpreter are not considered. The issue is that CI has everything in a single directory (Which, again, should be fine. Pex bug)

So that thread had two (probably unrelated issues in it). Let's leave that one for the PEX issue.

@jolly-midnight-72759 instead of a new thread for perm denied, how about you giving us a Dockerfile we can repro that with?

I'm running this on an EC2.

Its going to be awfully far afield and titchy to figure that one out without a repro case.

Ok. Let me see what I can do.