https://pantsbuild.org/ logo
Join Slack
Powered by
Hey Pants Team, our organization is adopting pants...
# development
r
Hey Pants Team, our organization is adopting pants for many good reasons (lint checks, project introspect, testing, packaging etc..). Right now we’re in the initial phase of lint checks with some custom plugins. Our biggest pain point of adopting pants is getting consistent runs of pants on all developer machines (we use MacOS). While we have documented setup guides using pyenv and venvs it’s not perfect. Developers still run into to errors that are not so obvious how to fix. An example:
Copy code
16:29:49.96 [INFO] Starting: Building isort.pex with 2 requirements: isort[pyproject]>=5.5.1,<5.6, setuptools
16:30:01.34 [INFO] Completed: Building isort.pex with 2 requirements: isort[pyproject]>=5.5.1,<5.6, setuptools
16:30:01.34 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/developer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.1rc3_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/developer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.1rc3_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/developer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.1rc3_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 190, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/developer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.1rc3_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/developer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.1rc3_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/developer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.1rc3_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 489, in _raise_on_error
    raise ExecutionError(
Exception message: 1 Exception encountered:
Engine traceback:
  in select
  in pants.core.goals.fmt.fmt
  in pants.backend.python.lint.python_fmt.format_python_target
  in pants.core.goals.fmt.enrich_fmt_result
  in custom_isort.rules.isort_fmt
  in custom_isort.rules.setup_isort
  in pants.backend.python.util_rules.pex.create_venv_pex (isort.pex)
  in pants.backend.python.util_rules.pex.build_pex (isort.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/developer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.1rc3_py38/lib/python3.8/site-packages/pants/engine/process.py", line 253, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building isort.pex with 2 requirements: isort[pyproject]>=5.5.1,<5.6, setuptools' failed with exit code 1.
stdout:
the solution for the above error was to upgrade pip but this is not obvious to most developers. My troubleshooting for pants errors usually goes like this: 1. what’s your python version 2. clear pants cache 3. try upgrading pip 4. try a virtualenv 5. if none above work just run lint checks directly to unblock (not ideal) Developers environments are widely different, most of the time it’s a battle of python versions and path issues. I’m not sure if there are any good solutions for this but I wanted to share some of experiences so far. Any suggestions to improve our consistency across environments would be greatly appreciated.
e
If just to underscore the difficulty and pain of all this: upgrading pip should have 0 to do with this!
➕ 1
r
I ask devs to enable debug and send over the logs. For this case this was the result: https://gist.github.com/Oshin159/cedba4fefc1bd67b9214f850efe47e8f What lead me to say upgrade pip was around these logs:
Copy code
File "/Users/oshin/.pyenv/versions/3.8.0/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pip._internal.cli.main'
c
Not the solution you want but to deal with this same problem I'm just getting devs to move their dev environment onto VM provisioned by Vagrant if they run into a problem like this.
✅ 1
r
so we do offer docker setup for developers but they stray away due to the resource usages on their laptops. For those who use docker they report no issues with pants which is expected.
c
I have found there are some fairly predictable errors when setting up pants on a dev box that are very difficult to debug.
For instance i've hit one where if 
/usr/bin/python
is not symlinked to python3 you get a totally indecipherable error
➕ 1
Also cmmon problem where packages don't properly specify setuptools as a dependency and never matters other than on pants because it's so ubiquitous
And caching behaviour can exacerbate the debugging
➕ 1
I have found starting from simplest to most complex can help isolate issues and make setup smoother:
Copy code
1. can you run `./pants` with no args
2. can you run `./pants test <very simple test with few deps>`
3. can you run `./pants test <complex test>
4. `./pants test ::`
r
that’s really good advice thank you. I will revise the troubleshooting guide for simplest -> most complex goal runs. We had a big increase of reported problem right after we migrated off python2.7 a month ago. For example:
Copy code
18:30:47.92 [INFO] Initialization options changed: reinitializing scheduler...
18:30:48.41 [INFO] Scheduler initialized.
18:30:54.36 [WARN] Completed: fmt - isort made changes.
Fixing app/payroll_filing/peo_filing/tests/test_filing_entity_resolver.py
18:30:55.42 [INFO] Starting: Building black.pex with 2 requirements: black==21.5b1, setuptools
18:30:57.44 [INFO] Completed: Building black.pex with 2 requirements: black==21.5b1, setuptools
18:30:57.45 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 190, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 489, in _raise_on_error
    raise ExecutionError(
Exception message: 1 Exception encountered:
Engine traceback:
  in select
  in pants.core.goals.fmt.fmt
  in pants.backend.python.lint.python_fmt.format_python_target
  in pants.core.goals.fmt.enrich_fmt_result
  in custom_black.rules.black_fmt
  in custom_black.rules.setup_black
  in pants.backend.python.util_rules.pex.create_venv_pex (black.pex)
  in pants.backend.python.util_rules.pex.build_pex (black.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/engine/process.py", line 253, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building black.pex with 2 requirements: black==21.5b1, setuptools' failed with exit code 1.
stdout:
stderr:
ERROR: Could not find a version that satisfies the requirement black==21.5b1
ERROR: No matching distribution found for black==21.5b1
pid 10938 -> /Users/sudhirkrishnakumar/.cache/pants/named_caches/pex_root/venvs/18b38500670558526e7bc85934f52e4c520a1fff/01df9bfe21d17113e3309daa4a1e72cea458c76f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-feature 2020-resolver --isolated -q --cache-dir /Users/sudhirkrishnakumar/.cache/pants/named_caches/pex_root --log /private/var/folders/y5/cdtnvhmj08gb0jsf0f1lcqnc0000gn/T/process-executionlUYnaV/.tmp/tmpvuvUyw/pip.log download --dest /private/var/folders/y5/cdtnvhmj08gb0jsf0f1lcqnc0000gn/T/process-executionlUYnaV/.tmp/tmpLZd0T4/usr.local.Cellar.python@2.2.7.17_1.Frameworks.Python.framework.Versions.2.7.bin.python2.7 black==21.5b1 setuptools --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
Traceback (most recent call last):
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 190, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 489, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
Engine traceback:
  in select
  in pants.core.goals.fmt.fmt
  in pants.backend.python.lint.python_fmt.format_python_target
  in pants.core.goals.fmt.enrich_fmt_result
  in custom_black.rules.black_fmt
  in custom_black.rules.setup_black
  in pants.backend.python.util_rules.pex.create_venv_pex (black.pex)
  in pants.backend.python.util_rules.pex.build_pex (black.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/sudhirkrishnakumar/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.wUoEMw/install/lib/python3.9/site-packages/pants/engine/process.py", line 253, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building black.pex with 2 requirements: black==21.5b1, setuptools' failed with exit code 1.
stdout:
stderr:
ERROR: Could not find a version that satisfies the requirement black==21.5b1
ERROR: No matching distribution found for black==21.5b1
pid 10938 -> /Users/sudhirkrishnakumar/.cache/pants/named_caches/pex_root/venvs/18b38500670558526e7bc85934f52e4c520a1fff/01df9bfe21d17113e3309daa4a1e72cea458c76f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-feature 2020-resolver --isolated -q --cache-dir /Users/sudhirkrishnakumar/.cache/pants/named_caches/pex_root --log /private/var/folders/y5/cdtnvhmj08gb0jsf0f1lcqnc0000gn/T/process-executionlUYnaV/.tmp/tmpvuvUyw/pip.log download --dest /private/var/folders/y5/cdtnvhmj08gb0jsf0f1lcqnc0000gn/T/process-executionlUYnaV/.tmp/tmpLZd0T4/usr.local.Cellar.python@2.2.7.17_1.Frameworks.Python.framework.Versions.2.7.bin.python2.7 black==21.5b1 setuptools --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
I agree caching seems to be a big pain point too, I always assume they have been running some goals before troubleshooting that put them in a weird state where we should just clear out the cache.
and 
/usr/bin/python
points to python2.7 on our macbooks and some had it installed via homebrew which I think some how contributed to that problem
c
I have also been caught out because we needed to both clear the cache and kill 
pantsd
processes.
👍 1
w
one thing that really helps lock down an org’s experience is to override the interpreter search path
for example: if you’d like all of your developers to be forced to use pyenv, you can set the following in your repository:
Copy code
[python-setup]
# Avoid system python.
interpreter_search_paths = ["<PYENV>"]
👀 1
note that rather than “adding” to the interpreter_search_paths, we’re overriding them entirely
that prevents lookups to 
/usr/bin/python
, macOS system python, your PATH, etc
as to the actual issue you are experiencing here with `pip`: i have seen it as well, and have been meaning to triage how to prevent it: more details in this thread: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1623776912133100
in short: it looks like when xcode (or perhaps macOS) are updated, the combination of symlinks/binaries that are used for system python shift in a way that PEX does not detect: that causes site-packages to be for the wrong version inside of the virtualenv that PEX creates, and the vendored 
pip
inside that venv to not be found.
i should have created an issue for this at the time (and have been snoozing further investigation via 
xcode-select
), but will create that issue now.
the workaround though (as you have detected) is to wipe 
~/.cache/pants/named_caches/pex_root
opened https://github.com/pantsbuild/pex/issues/1386 about this. but yea: see above re constraining your interpreters to only `pyenv`: that can go a long way.
✅ 1
r
Thank you very much @witty-crayon-22786! I’m going to try this out and will follow back here with the outcome. I have a good feeling this will help us
👍 2
Okay ran into a weird issue myself today:
Copy code
Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins (CPython==3.9.4)
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.find_interpreter (CPython==3.9.4)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.9ZYntu/install/lib/python3.9/site-packages/pants/engine/process.py", line 253, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython==3.9.4' failed with exit code 102.
stdout:

stderr:
Could not find a compatible interpreter.

Examined the following interpreters:
1.)          /Users/njgrisafi/.pyenv/versions/3.6.10/bin/python3.6 CPython==3.6.10
2.) /Users/njgrisafi/.pyenv/versions/3.7.10/envs/3.7/bin/python3.7 CPython==3.7.10
3.)           /Users/njgrisafi/.pyenv/versions/3.8.8/bin/python3.8 CPython==3.8.8
4.)           /Users/njgrisafi/.pyenv/versions/3.9.2/bin/python3.9 CPython==3.9.2

No interpreter compatible with the requested constraints was found:
  Version matches CPython==3.9.4
curious why 3.9.4? We don’t set any constraints.
(on version 2.5.0rc3 planning to upgrade soon)
w
Mm, that's probably https://github.com/pantsbuild/pants/issues/12361 ... the Pants script is choosing to run with 3.9.4 which it located somewhere on your PATH, and then failing to resolve plugins because it's constrained differently by python-setup.
👍 1
Sorry for the trouble... you can work around it by using the 
pants
script's options to select an interpreter. I'll see if we can fix that one in a cherry pick.
✅ 1
r
just to note this is with the suggestion of setting
Copy code
[python-setup]
interpreter_search_paths = [
  "<PYENV>"
]
so idk where else that could have came from
w
Yea, that's very likely to be related. It's still a good idea to constrain your interpreters
👍 1
r
cool thank you. Will follow that issue and look to constrain our interpreters
not sure if possible or already exists but we do use poetry would be great if we can infer interpreter constraints from 
pyproject.toml
okay I set the constraint but still see an error. Ran with debug level: https://gist.github.com/njgrisafi/79286581d8fd1e3c5c79619fb4753f4e
w
What constraint did you set for the 
pants
script? Basically, you'd need to have the 
pants
script also use pyenv
r
ahh sorry think I misunderstood for 
interpreter_constraints
in pants.toml. Would I update the 
determine_default_python_exe
function via https://raw.githubusercontent.com/pantsbuild/setup/gh-pages/pants?
w
yea, exactly: sorry for the confusion. will work on this this afternoon
you can also explicitly set 
PYTHON=…
to choose the interpreter used by the script
r
so I set this to 
PYTHON=$HOME/.pyenv/versions/3.9.2/bin/python
. Output looks a bit difference but I still see an error
Copy code
Examined the following interpreters:
1.)          /Users/njgrisafi/.pyenv/versions/3.6.10/bin/python3.6 CPython==3.6.10
2.) /Users/njgrisafi/.pyenv/versions/3.7.10/envs/3.7/bin/python3.7 CPython==3.7.10
3.)           /Users/njgrisafi/.pyenv/versions/3.8.8/bin/python3.8 CPython==3.8.8
4.)           /Users/njgrisafi/.pyenv/versions/3.9.2/bin/python3.9 CPython==3.9.2

No interpreter compatible with the requested constraints was found:
  Version matches CPython==3.9.4



(Use --no-process-execution-local-cleanup to inspect chroots and/or -ldebug for more logs. See <https://www.pantsbuild.org/v2.6/docs/troubleshooting> for common issues. Consider reaching out for help: <https://www.pantsbuild.org/v2.6/docs/getting-help.>)
w
argh. yea, this is a super frustrating issue because you also have to wipe 
~/.cache/pants/setup
, since it’s only a patch version change in the interpreter from 3.9.2 to 3.9.4
✅ 1
updated the ticket description with the full workaround. sorry about that.
hope to have a fix out this afternoon.
r
clearing the cache did the trick! Thank you so much for the help on this I learned a lot here
will commit this temporary and will try after the fix is merged so we can confirm
w
sorry that you needed to! in the medium term i hope that we can be either using the exact same code in the bootstrap script as in pants itself, or remove the need for a bootstrap script
💯 1
but in the short term, will get a fix out and close the issue
👍 1
r
no worries at all thank you for taking the time on this.
Removed the workaround in pants script and tested out the new changes everything seems to be resolved now!
🙌 1
w
thanks for checking!
Open in Slack
PreviousNext
Powered by