https://pantsbuild.org/ logo
Join Slack
Powered by
Anyone ever come across this before? Python 3.6.12...
# general
s
Anyone ever come across this before? Python 3.6.12 install via pyenv and 100% available from the PATH. Pants finds it to bootstrap but when it comes to finding interpreters to resolve requirements and run the code it finds nothing… Running Pants 
1.30.4
yes i know it is an older version but we are stuck here for the time being
Copy code
20:46:12 00:00 [main]
               (To run a reporting server: ./pants server)
20:46:14 00:02   [setup]
20:46:14 00:02     [parse]
               Executing tasks in goals: gen -> pyprep -> run20:46:14 [DEBUG] pants.process.lock:pid=16680: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e05e6a0>

20:46:14 00:02   [gen]
20:46:14 00:02     [thrift-py]
20:46:14 00:02     [py-thrift-namespace-clash-check]20:46:14 [DEBUG] engine::scheduler: Launching 0 roots (poll=false).
20:46:14 [DEBUG] pants.engine.internals.scheduler:pid=16680: computed 0 nodes in 0.000226 seconds. there are 74 total nodes.

20:46:14 00:02   [pyprep]
20:46:14 00:02     [interpreter]
                   pants_backend_python_tasks_select_interpreter_SelectInterpreter will read from local artifact cache at /Users/distiller/talos/.pants.d/.pantscache/b56378cd3458bce8ae082d0ed362f8bd7ab9795b
20:46:14 00:02       [cache]
                   No cached artifacts for 2 targets.
                   Invalidated 2 targets.20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initializing Python interpreter cache matching filters `CPython>=3.6,<3.7` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/shims:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
20:46:14 [DEBUG] pants.process.lock:pid=16680: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2908>
20:46:14 [DEBUG] pants.process.lock:pid=16680: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2908>
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for CPython>=3.6,<3.7!
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Found no valid interpreters!
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initialized Python interpreter cache with
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initializing Python interpreter cache matching filters `CPython<3:CPython>=3.3:PyPy` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/shims:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
20:46:14 [DEBUG] pants.process.lock:pid=16680: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2400>
20:46:15 [DEBUG] pants.process.lock:pid=16680: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2400>
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for CPython<3!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for CPython>=3.3!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for PyPy!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Found no valid interpreters!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initialized Python interpreter cache with
20:46:15 [DEBUG] pants.process.lock:pid=16680: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e05e6a0>

FAILURE: Unable to detect a suitable interpreter for compatibilities: CPython>=3.6,<3.7 (Conflicting targets: apps/cli_clouddev/src:clouddev)

Pants detected these interpreter versions on your system:

Possible ways to fix this:
* Modify your Python interpreter constraints by following <https://www.pantsbuild.org/python_readme.html#configure-the-python-version>.
* Ensure the targeted Python version is installed and discoverable.
* Modify Pants' interpreter search paths via --python-setup-interpreter-search-paths.
h
Is it pyenv-installed as 
global
, 
local
or 
shell
?
s
global
@happy-kitchen-89482 its actually installed as local I believe now that im looking into it. 
/usr/local/bin/pyenv
This is a little unique case because its pyenv running in a mac osx executor for circleci. pyenv is installed with homebrew just like i have it on my local macbook.
h
Oh, I mean in pyenv you can activate an interpreter in one of three ways
s
install script:
Copy code
brew install pyenv
        echo 'export PYENV_ROOT="$HOME/.pyenv"' >> $BASH_ENV
        echo 'export PATH="$PYENV_ROOT/shims:$PATH"' >> $BASH_ENV

eval "$(pyenv init -)" &&
        pyenv install 3.6.12 &&
        pyenv global 3.6.12
oh yeah definately global
h
So it's on $PATH
s
Copy code
static:talos distiller$ echo $PATH
~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/shims:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
Here is my path and yes
h
Oh, dimly remembering, is the special string 
<PYENV>
on your interpreter search path?
s
yep.. that is 100% set in my pants.toml and I have tried overriding that with 
./pants --python-setup-interpreter-search-paths
and still same results
in the debug logs you can see it trying to find the binary at the correct location relative to what is set for search paths. even if i hard code it in there… same error. And in the logs you can see that the interpreter search reveals NONE installed on the system. even though 2.7 and 3.9 are installed at the system level
h
interesting and frustrating...
s
@happy-kitchen-89482 yeah ususally i wouldn’t bother anyone here with python path issues…. but I am out of ideas
h
What happens if you hard-code the real interpreter path (e.g., 
"$(pyenv root)/versions/3.6.6/bin/"
) onto your PATH?
I remember Pants being unable to detect the shims as python interpreters, since they aren't really
pyenv shims are a useful nightmare
s
@happy-kitchen-89482 first line of my path is the hardcoded pyenv version to 3.6.12. Its as 
~/.pyenv
i could try giving the absolute path one second
no change… this is so odd because on my local mac and in our centos devboxes all we use are pyenv shims and have never had an issue
h
Weird as hell
I'm trying to think of what could prevent interpreter discovery from seeing that as an interpreter
Does 
-ldebug
yield any insight?
s
let me get you the full logs one second
Copy code
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initializing Python interpreter cache matching filters `CPython>=3.6,<3.7` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/versions/3.6.12/bin/:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
22:34:39 [DEBUG] pants.process.lock:pid=16935: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5dd8>
22:34:39 [DEBUG] pants.process.lock:pid=16935: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5dd8>
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for CPython>=3.6,<3.7!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Found no valid interpreters!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initialized Python interpreter cache with
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initializing Python interpreter cache matching filters `CPython<3:CPython>=3.3:PyPy` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/versions/3.6.12/bin/:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
22:34:39 [DEBUG] pants.process.lock:pid=16935: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5eb8>
22:34:39 [DEBUG] pants.process.lock:pid=16935: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5eb8>
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for CPython<3!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for CPython>=3.3!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for PyPy!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Found no valid interpreters!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initialized Python interpreter cache with
22:34:39 [DEBUG] pants.process.lock:pid=16935: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104d54e10>

FAILURE: Unable to detect a suitable interpreter for compatibilities: CPython>=3.6,<3.7 (Conflicting targets: apps/cli_clouddev/src:clouddev)

Pants detected these interpreter versions on your system:

Possible ways to fix this:
* Modify your Python interpreter constraints by following <https://www.pantsbuild.org/python_readme.html#configure-the-python-version>.
* Ensure the targeted Python version is installed and discoverable.
* Modify Pants' interpreter search paths via --python-setup-interpreter-search-paths.
which python3
and 
which python
all lead to the pyenv shim for 3.6.12. I went as far as uninstalling python 3.9 which was brew installed. 3.8 comes preinstalled with this version of osx
when pants bootstraps it shows this:
Copy code
created virtual environment CPython3.6.12.final.0-64 in 824ms
h
Is this something you can reproduce in a minimal repo? I can take a look on my system
s
@happy-kitchen-89482 hmmm i may be able to get you access to the actual circleci box that this happens on via ssh. give me one second. let me make sure i am “allowed” to do this. I may take our code off and throw a dummy repo on there … it’ll 100% be reproducible either way
@happy-kitchen-89482 I can create a minimal repo that does what I need. Then i can give you access to said repo in our org. This should give you CircleCI access (i think) and you can SSH into the circleci executor. Would this work? basically giving you a VM in the state exactly where we have the problem.
h
It's a start!
And that might provide enough info to reproduce locally if I need to
👍 1
s
cool ill get to this at some point today. some high priority items came up that i have to address first
h
No rush, at-me when it's ready, so I know to pay attention 🙂
👍 1
s
@happy-kitchen-89482 I have provided you write access to a public repo that I created. This will give you proper access to look at our circleci builds via an ssh connection. I can walk you through this if you are unfamiliar with circleci and its ssh jobs to debug. https://github.com/NarrativeScience/pants-help.git Here is the repo access. Links to failed jobs and where the issue is coming shortly.
h
👍
I'm seeing a different error in that CI than the one you mentioned above?
s
ah one second… i forgot to put the BUILD file back to pants 1.30 syntax
if you click “rerun with ssh” and then let it rerun you can expand the ssh step to get the IP.
h
So, having trouble sshing
It times out on the connection, sees no response at all from the server
Hmm, I can't even get a route to the IP it gives me with 
traceroute
I'm not behind a VPN
huh, rerunning the CI job again made it work
OK, figured it out!
When running with 
PEX_VERBOSE=9
I see this error: 
Failed to spawn a job for /usr/bin/python3: OSError(24, 'Too many open files') while trying to execute ...
(the thing it's trying to execute is the script that determines the interpreter version it's running on)
And setting 
ulimit -n 10000
Fixes the issue
So the fix is to add 
ulimit -n 10000
somewhere in CI bootstrapping
In Pants 2.x this error would have been surfaced, in 1.30.x it is unfortunately not made explicit unless running with PEX_VERBOSE
Sorry for the trouble!
s
no worries thanks for the help!! sorry for late response I was on vacation
I will verify and fix this today
5 Views
Open in Slack
PreviousNext
Powered by