question about how

[python-bootstrap].search_path

works: it doesn’t seem to care about the order in which arguments are passed? we support both python 3.8 and 3.9 on our monorepo, and currently explicitly declare these in `pants.toml`:

[python]
interpreter_constraints = [">=3.8,<3.10"]

[python-bootstrap]
search_path = ["<PYENV_LOCAL>", "<PYENV>", "<PATH>"]
names = ["python3.9", "python3.8", "python3", "python"]

m1 macs only work with python 3.9 because of incompatibility issues, but i would expect this not to be a problem since we use pyenv to set the local (and global) python version to be 3.9. however, since python 3.8 is also installed on my machine as part of Xcode’s command line tools, it seems like pants always uses 3.8 from

<PATH>

regardless of the order in

search_path

(i guess it’s using the paths in alphabetical order?)

pex.environment.ResolveError: A distribution for sqlalchemy could not be resolved in this environment.Found 1 distribution for sqlalchemy that do not apply:
1.) The wheel tags for SQLAlchemy 1.4.31 are cp38-cp38-macosx_10_14_arm64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8', PythonIdentity('/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8', 'cp38', 'cp38', 'macosx_12_0_arm64', (3, 8, 9)))):

the only way i can get pants to use the correct python version is if I set

search_path

to not include

<PATH>

at all. I would expect that the paths and names are searched in the order they are provided, i.e. first check

<PYENV_LOCAL>

for the first of

["python3.9", "python3.8", "python3", "python"]

that exists, next check

<PYENV>

, and so on

I’ll defer a definite answer, but provide my hunch in the mean time. And that is that Pants locates all available Pythons found on the provided search path, then picks one that satisifies the provided constraints, and in doing so, goes with the oldest version available (that is, 3.8 if there is one in your case). This may be completely off, but that is my feeling from what I’ve experienced…

hmm, removing

python3.8

from

names

still results in the same error 🤔

Ah, yes, so the online help for

python-bootstrap

clearly states that the order of the path entries does not matter:

$ ./pants help-advanced python-bootstrap
`python-bootstrap` subsystem advanced options
---------------------------------------------
 
  --python-bootstrap-search-path="[<binary-paths>, <binary-paths>, ...]"
  PANTS_PYTHON_BOOTSTRAP_SEARCH_PATH
  search_path
      default: [
          "<PYENV>",
          "<PATH>"
      ]
      current value: [
          "<PYENV>",
          "<PATH>"
      ]
      A list of paths to search for Python interpreters.
      
      Which interpeters are actually used from these paths is context-specific: the Python backend selects interpreters using options on the `python` subsystem, in particular, the
      `[python].interpreter_constraints` option.
      
      You can specify absolute paths to interpreter binaries and/or to directories containing interpreter binaries. The order of entries does not matter.
      
      The following special strings are supported:
      
      * `<PATH>`, the contents of the PATH env var
      * `<ASDF>`, all Python versions currently configured by ASDF `(asdf shell, ${HOME}/.tool-versions)`, with a fallback to all installed versions
      * `<ASDF_LOCAL>`, the ASDF interpreter with the version in BUILD_ROOT/.tool-versions
      * `<PYENV>`, all Python versions under $(pyenv root)/versions
      * `<PYENV_LOCAL>`, the Pyenv interpreter with the version in BUILD_ROOT/.python-version
      * `<PEXRC>`, paths in the PEX_PYTHON_PATH variable in /etc/pexrc or ~/.pexrc

  --python-bootstrap-names="[<python-binary-names>, <python-binary-names>, ...]"
  PANTS_PYTHON_BOOTSTRAP_NAMES
  names
      default: [
          "python",
          "python3"
      ]
      current value: [
          "python",
          "python3"
      ]
      The names of Python binaries to search for. See the `--search-path` option to influence where interpreters are searched for.
      
      This does not impact which Python interpreter is used to run your code, only what is used to run internal tools.

So

[python-boostrap].names

is solely for choosing the Python interpreter to run PEX itself with. Doesn't influence which interpreters PEX then chooses to run your own code Instead, what I recommend is using a pants.rc file for your M1 users: https://www.pantsbuild.org/v2.10/docs/options#pantsrc-file Note that only 2.10 recognizes

<build root>/.pants.rc

by default. If you're not able to use 2.10 yet, you could instead use one of those two global paths, or use an alternative approach like using a

.env

file and

direnv

so my workaround has been to use the pants.rc file to force pants to use pyenv

[python-bootstrap]
search_path = ["<PYENV>"]

, but I don’t understand why it wouldn’t default to pyenv anyway, since that’s also in my

PATH

martim@ip-192-168-1-227 allpdl % echo $PATH
/Users/martim/.pyenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/martim/.poetry/bin:/Users/martim/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki:/Users/martim/.cargo/bin

It does use <PYENV> because it's in your PATH, but it also uses all other entries in your <PATH> like Xcode Python. Interpreter selection first identifies all candidate interpreters on the machine, then it checks which are compatible with your interpreter constraints and goes with the minimum compatible

hmm

No https://pantsbuild.slack.com/archives/C046T6T9U/p1644591537567709?thread_ts=1644588604.171889&amp;cid=C046T6T9U

is there any way to prefer a certain version over another? instead of going with the minimum compatible?

There is not atm. What would that look like? As in, what criteria are you using to declare preference?

my idea would be to use the order that the search paths are provided, intuitively I would expect there to be a difference between

["<PYENV>", "<PATH>"]

and

["<PATH>", "<PYENV>"]

Hey colleague of @high-energy-55500 here 🙂 Not sure what an ideal way to select preference would be, but if it helps here’s more info on our goals and current config: Currently we want to support the following platforms: • Intel Mac (3.8 or 3.9) • M1 Mac (3.9) • Linux (3.8 or 3.9) And we don’t want to make certain users configure a pantsrc if they are on a specific platform if possible. So we did this in our

pants.toml

, with the assumption that the order for

--python-bootstrap-names

was significant and it just selected the first one that works in the constraints:

[python-bootstrap]
search_path = ["<PATH>"]
names = ["python3.9", "python3.8", "python3", "python"]

So i.e. even if you are on a system that only works with 3.9, as long as

python3.9

is on your PATH it wouldn’t matter.

even within the same path, I would expect it to use the first compatible python version. so since I have pyenv at the top of my PATH, I would still expect it to use that version instead of some other version that comes later

That ordering generally makes sense to me for Pants's perspective. I like it because it doesn't require inventing some new mechanism to declare your preference. Although not sure it makes sense for PEX. It's more nuanced than PEX simply choosing one interpreter. When you say

pex --interpreter-constraint='CPython>=3.5,<3.10 Django

, it interprets that as you asking to build a PEX that works with all of 3.5-3.10. You're not just matching one interpreter. Currently, the algorithm is for each minor version (3.6 etc), choose the version with the highest patch, so prefer 3.6.9 over 3.6.2. Instead, the impact of this change would be not caring about patch version and going with what's first on the

--python-path

. I'm not certain that will make sense for Pex.

Got it @hundreds-father-404, appreciate your insight here. It seems we thought `python-bootstrap-names was something that it actually wasn’t 😛 Being able to configure pex to choose the max version rather than the max patch of the min version when building PEXes would be nice, but I think there are other areas we would actually prefer the other behavior (for example we want linting CI to use the min version to flag 3.9+ syntax) so we would have to reconcile those goals there, adding some more complexity.

but I think there are other areas we would actually prefer the other behavior (for example we want linting CI to use the min version to flag 3.9+ syntax) so we would have to reconcile those goals there, adding some more complexity.

That is precisely why I think minimum is the better default and why I chose to not focus on https://github.com/pantsbuild/pex/issues/430 when writing https://github.com/pantsbuild/pex/pull/1088

So from your suggestions, I think the course of action we may take is to: 1. Move to 3.9.* as soon as we can. We have a good amount of infra using 3.8 today which will take some time to move over, but we’ll get there. 2. Until 3.8 is out of the picture, have users on M1 macbooks use a

.pants.rc

file to override interpreter_constraints to

==3.9.*

Ahhh a third solution: 3. Stop using

<PATH>

in

[python-bootstrap].search_paths

. Only use

<PYENV>

and possibly some blessed paths like

/opt/python

. See this great soon-to-be-published blog from @fresh-cat-90827 about this topic https://docs.google.com/document/d/1IQk3pM3YKLypwvw0Y8vMq8jRAJ9J0n-0xZ-GhbuFPE4/edit?usp=sharing THat way you don't use the Xcode python

@faint-businessperson-86903 A feature request for platform-specific config would be great!

(That's what 7735 already is)