https://pantsbuild.org/ logo
#general
Title
# general
s

salmon-barista-63163

07/15/2021, 8:47 PM
Anyone ever come across this before? Python 3.6.12 install via pyenv and 100% available from the PATH. Pants finds it to bootstrap but when it comes to finding interpreters to resolve requirements and run the code it finds nothing… Running Pants
1.30.4
yes i know it is an older version but we are stuck here for the time being
Copy code
20:46:12 00:00 [main]
               (To run a reporting server: ./pants server)
20:46:14 00:02   [setup]
20:46:14 00:02     [parse]
               Executing tasks in goals: gen -> pyprep -> run20:46:14 [DEBUG] pants.process.lock:pid=16680: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e05e6a0>

20:46:14 00:02   [gen]
20:46:14 00:02     [thrift-py]
20:46:14 00:02     [py-thrift-namespace-clash-check]20:46:14 [DEBUG] engine::scheduler: Launching 0 roots (poll=false).
20:46:14 [DEBUG] pants.engine.internals.scheduler:pid=16680: computed 0 nodes in 0.000226 seconds. there are 74 total nodes.

20:46:14 00:02   [pyprep]
20:46:14 00:02     [interpreter]
                   pants_backend_python_tasks_select_interpreter_SelectInterpreter will read from local artifact cache at /Users/distiller/talos/.pants.d/.pantscache/b56378cd3458bce8ae082d0ed362f8bd7ab9795b
20:46:14 00:02       [cache]
                   No cached artifacts for 2 targets.
                   Invalidated 2 targets.20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initializing Python interpreter cache matching filters `CPython>=3.6,<3.7` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/shims:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
20:46:14 [DEBUG] pants.process.lock:pid=16680: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2908>
20:46:14 [DEBUG] pants.process.lock:pid=16680: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2908>
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for CPython>=3.6,<3.7!
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Found no valid interpreters!
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initialized Python interpreter cache with
20:46:14 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initializing Python interpreter cache matching filters `CPython<3:CPython>=3.3:PyPy` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/shims:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
20:46:14 [DEBUG] pants.process.lock:pid=16680: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2400>
20:46:15 [DEBUG] pants.process.lock:pid=16680: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e0c2400>
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for CPython<3!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for CPython>=3.3!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: No valid interpreters found for PyPy!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Found no valid interpreters!
20:46:15 [DEBUG] pants.backend.python.interpreter_cache:pid=16680: Initialized Python interpreter cache with
20:46:15 [DEBUG] pants.process.lock:pid=16680: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x10e05e6a0>

FAILURE: Unable to detect a suitable interpreter for compatibilities: CPython>=3.6,<3.7 (Conflicting targets: apps/cli_clouddev/src:clouddev)

Pants detected these interpreter versions on your system:

Possible ways to fix this:
* Modify your Python interpreter constraints by following <https://www.pantsbuild.org/python_readme.html#configure-the-python-version>.
* Ensure the targeted Python version is installed and discoverable.
* Modify Pants' interpreter search paths via --python-setup-interpreter-search-paths.
h

happy-kitchen-89482

07/15/2021, 9:28 PM
Is it pyenv-installed as
global
,
local
or
shell
?
s

salmon-barista-63163

07/15/2021, 9:47 PM
global
@happy-kitchen-89482 its actually installed as local I believe now that im looking into it.
/usr/local/bin/pyenv
This is a little unique case because its pyenv running in a mac osx executor for circleci. pyenv is installed with homebrew just like i have it on my local macbook.
h

happy-kitchen-89482

07/15/2021, 10:06 PM
Oh, I mean in pyenv you can activate an interpreter in one of three ways
s

salmon-barista-63163

07/15/2021, 10:07 PM
install script:
Copy code
brew install pyenv
        echo 'export PYENV_ROOT="$HOME/.pyenv"' >> $BASH_ENV
        echo 'export PATH="$PYENV_ROOT/shims:$PATH"' >> $BASH_ENV

eval "$(pyenv init -)" &&
        pyenv install 3.6.12 &&
        pyenv global 3.6.12
oh yeah definately global
h

happy-kitchen-89482

07/15/2021, 10:07 PM
So it's on $PATH
s

salmon-barista-63163

07/15/2021, 10:07 PM
Copy code
static:talos distiller$ echo $PATH
~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/shims:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
Here is my path and yes
h

happy-kitchen-89482

07/15/2021, 10:09 PM
Oh, dimly remembering, is the special string
<PYENV>
on your interpreter search path?
s

salmon-barista-63163

07/15/2021, 10:09 PM
yep.. that is 100% set in my pants.toml and I have tried overriding that with
./pants --python-setup-interpreter-search-paths
and still same results
in the debug logs you can see it trying to find the binary at the correct location relative to what is set for search paths. even if i hard code it in there… same error. And in the logs you can see that the interpreter search reveals NONE installed on the system. even though 2.7 and 3.9 are installed at the system level
h

happy-kitchen-89482

07/15/2021, 10:11 PM
interesting and frustrating...
s

salmon-barista-63163

07/15/2021, 10:12 PM
@happy-kitchen-89482 yeah ususally i wouldn’t bother anyone here with python path issues…. but I am out of ideas
h

happy-kitchen-89482

07/15/2021, 10:15 PM
What happens if you hard-code the real interpreter path (e.g.,
"$(pyenv root)/versions/3.6.6/bin/"
) onto your PATH?
I remember Pants being unable to detect the shims as python interpreters, since they aren't really
pyenv shims are a useful nightmare
s

salmon-barista-63163

07/15/2021, 10:18 PM
@happy-kitchen-89482 first line of my path is the hardcoded pyenv version to 3.6.12. Its as
~/.pyenv
i could try giving the absolute path one second
no change… this is so odd because on my local mac and in our centos devboxes all we use are pyenv shims and have never had an issue
h

happy-kitchen-89482

07/15/2021, 10:30 PM
Weird as hell
I'm trying to think of what could prevent interpreter discovery from seeing that as an interpreter
Does
-ldebug
yield any insight?
s

salmon-barista-63163

07/15/2021, 10:35 PM
let me get you the full logs one second
Copy code
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initializing Python interpreter cache matching filters `CPython>=3.6,<3.7` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/versions/3.6.12/bin/:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
22:34:39 [DEBUG] pants.process.lock:pid=16935: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5dd8>
22:34:39 [DEBUG] pants.process.lock:pid=16935: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5dd8>
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for CPython>=3.6,<3.7!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Found no valid interpreters!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initialized Python interpreter cache with
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initializing Python interpreter cache matching filters `CPython<3:CPython>=3.3:PyPy` from paths `/Users/distiller/.pyenv/versions/3.6.12/bin:~/.pyenv/versions/3.6.12/bin:/Users/distiller/.pyenv/shims:/Users/distiller/.pyenv/versions/3.6.12/bin/:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin`
22:34:39 [DEBUG] pants.process.lock:pid=16935: acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5eb8>
22:34:39 [DEBUG] pants.process.lock:pid=16935: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104dc5eb8>
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for CPython<3!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for CPython>=3.3!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: No valid interpreters found for PyPy!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Found no valid interpreters!
22:34:39 [DEBUG] pants.backend.python.interpreter_cache:pid=16935: Initialized Python interpreter cache with
22:34:39 [DEBUG] pants.process.lock:pid=16935: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104d54e10>

FAILURE: Unable to detect a suitable interpreter for compatibilities: CPython>=3.6,<3.7 (Conflicting targets: apps/cli_clouddev/src:clouddev)

Pants detected these interpreter versions on your system:

Possible ways to fix this:
* Modify your Python interpreter constraints by following <https://www.pantsbuild.org/python_readme.html#configure-the-python-version>.
* Ensure the targeted Python version is installed and discoverable.
* Modify Pants' interpreter search paths via --python-setup-interpreter-search-paths.
which python3
and
which python
all lead to the pyenv shim for 3.6.12. I went as far as uninstalling python 3.9 which was brew installed. 3.8 comes preinstalled with this version of osx
when pants bootstraps it shows this:
Copy code
created virtual environment CPython3.6.12.final.0-64 in 824ms
h

happy-kitchen-89482

07/16/2021, 1:43 AM
Is this something you can reproduce in a minimal repo? I can take a look on my system
s

salmon-barista-63163

07/16/2021, 4:13 PM
@happy-kitchen-89482 hmmm i may be able to get you access to the actual circleci box that this happens on via ssh. give me one second. let me make sure i am “allowed” to do this. I may take our code off and throw a dummy repo on there … it’ll 100% be reproducible either way
@happy-kitchen-89482 I can create a minimal repo that does what I need. Then i can give you access to said repo in our org. This should give you CircleCI access (i think) and you can SSH into the circleci executor. Would this work? basically giving you a VM in the state exactly where we have the problem.
h

happy-kitchen-89482

07/16/2021, 5:48 PM
It's a start!
And that might provide enough info to reproduce locally if I need to
👍 1
s

salmon-barista-63163

07/16/2021, 6:10 PM
cool ill get to this at some point today. some high priority items came up that i have to address first
h

happy-kitchen-89482

07/17/2021, 9:59 PM
No rush, at-me when it's ready, so I know to pay attention 🙂
👍 1
s

salmon-barista-63163

07/19/2021, 11:57 PM
@happy-kitchen-89482 I have provided you write access to a public repo that I created. This will give you proper access to look at our circleci builds via an ssh connection. I can walk you through this if you are unfamiliar with circleci and its ssh jobs to debug. https://github.com/NarrativeScience/pants-help.git Here is the repo access. Links to failed jobs and where the issue is coming shortly.
h

happy-kitchen-89482

07/20/2021, 1:00 AM
👍
I'm seeing a different error in that CI than the one you mentioned above?
s

salmon-barista-63163

07/20/2021, 2:01 AM
ah one second… i forgot to put the BUILD file back to pants 1.30 syntax
if you click “rerun with ssh” and then let it rerun you can expand the ssh step to get the IP.
h

happy-kitchen-89482

07/24/2021, 1:33 AM
So, having trouble sshing
It times out on the connection, sees no response at all from the server
Hmm, I can't even get a route to the IP it gives me with
traceroute
I'm not behind a VPN
huh, rerunning the CI job again made it work
OK, figured it out!
When running with
PEX_VERBOSE=9
I see this error:
Failed to spawn a job for /usr/bin/python3: OSError(24, 'Too many open files') while trying to execute ...
(the thing it's trying to execute is the script that determines the interpreter version it's running on)
And setting
ulimit -n 10000
Fixes the issue
So the fix is to add
ulimit -n 10000
somewhere in CI bootstrapping
In Pants 2.x this error would have been surfaced, in 1.30.x it is unfortunately not made explicit unless running with PEX_VERBOSE
Sorry for the trouble!
s

salmon-barista-63163

08/02/2021, 3:58 PM
no worries thanks for the help!! sorry for late response I was on vacation
I will verify and fix this today
5 Views