https://pantsbuild.org/ logo
#general
Title
# general
h

high-energy-55500

02/11/2022, 2:10 PM
question about how
[python-bootstrap].search_path
works: it doesn’t seem to care about the order in which arguments are passed? we support both python 3.8 and 3.9 on our monorepo, and currently explicitly declare these in `pants.toml`:
Copy code
[python]
interpreter_constraints = [">=3.8,<3.10"]

[python-bootstrap]
search_path = ["<PYENV_LOCAL>", "<PYENV>", "<PATH>"]
names = ["python3.9", "python3.8", "python3", "python"]
m1 macs only work with python 3.9 because of incompatibility issues, but i would expect this not to be a problem since we use pyenv to set the local (and global) python version to be 3.9. however, since python 3.8 is also installed on my machine as part of Xcode’s command line tools, it seems like pants always uses 3.8 from
<PATH>
regardless of the order in
search_path
(i guess it’s using the paths in alphabetical order?)
Copy code
pex.environment.ResolveError: A distribution for sqlalchemy could not be resolved in this environment.Found 1 distribution for sqlalchemy that do not apply:
1.) The wheel tags for SQLAlchemy 1.4.31 are cp38-cp38-macosx_10_14_arm64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8', PythonIdentity('/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8', 'cp38', 'cp38', 'macosx_12_0_arm64', (3, 8, 9)))):
the only way i can get pants to use the correct python version is if I set
search_path
to not include
<PATH>
at all. I would expect that the paths and names are searched in the order they are provided, i.e. first check
<PYENV_LOCAL>
for the first of
["python3.9", "python3.8", "python3", "python"]
that exists, next check
<PYENV>
, and so on
c

curved-television-6568

02/11/2022, 2:19 PM
I’ll defer a definite answer, but provide my hunch in the mean time. And that is that Pants locates all available Pythons found on the provided search path, then picks one that satisifies the provided constraints, and in doing so, goes with the oldest version available (that is, 3.8 if there is one in your case). This may be completely off, but that is my feeling from what I’ve experienced…
👍 1
👆 1
h

high-energy-55500

02/11/2022, 2:24 PM
hmm, removing
python3.8
from
names
still results in the same error 🤔
i guess
python
would still point to
python3.8
in that case, if i understand what you’re saying, so removing it wouldn’t change anything
c

curved-television-6568

02/11/2022, 2:58 PM
Ah, yes, so the online help for
python-bootstrap
clearly states that the order of the path entries does not matter:
Copy code
$ ./pants help-advanced python-bootstrap
`python-bootstrap` subsystem advanced options
---------------------------------------------
 
  --python-bootstrap-search-path="[<binary-paths>, <binary-paths>, ...]"
  PANTS_PYTHON_BOOTSTRAP_SEARCH_PATH
  search_path
      default: [
          "<PYENV>",
          "<PATH>"
      ]
      current value: [
          "<PYENV>",
          "<PATH>"
      ]
      A list of paths to search for Python interpreters.
      
      Which interpeters are actually used from these paths is context-specific: the Python backend selects interpreters using options on the `python` subsystem, in particular, the
      `[python].interpreter_constraints` option.
      
      You can specify absolute paths to interpreter binaries and/or to directories containing interpreter binaries. The order of entries does not matter.
      
      The following special strings are supported:
      
      * `<PATH>`, the contents of the PATH env var
      * `<ASDF>`, all Python versions currently configured by ASDF `(asdf shell, ${HOME}/.tool-versions)`, with a fallback to all installed versions
      * `<ASDF_LOCAL>`, the ASDF interpreter with the version in BUILD_ROOT/.tool-versions
      * `<PYENV>`, all Python versions under $(pyenv root)/versions
      * `<PYENV_LOCAL>`, the Pyenv interpreter with the version in BUILD_ROOT/.python-version
      * `<PEXRC>`, paths in the PEX_PYTHON_PATH variable in /etc/pexrc or ~/.pexrc

  --python-bootstrap-names="[<python-binary-names>, <python-binary-names>, ...]"
  PANTS_PYTHON_BOOTSTRAP_NAMES
  names
      default: [
          "python",
          "python3"
      ]
      current value: [
          "python",
          "python3"
      ]
      The names of Python binaries to search for. See the `--search-path` option to influence where interpreters are searched for.
      
      This does not impact which Python interpreter is used to run your code, only what is used to run internal tools.
but maybe the second option there would be interesting, in case you set
[python-bootstrap].names=["python3.9"]
perhaps.. as a work around?
h

hundreds-father-404

02/11/2022, 4:48 PM
So
[python-boostrap].names
is solely for choosing the Python interpreter to run PEX itself with. Doesn't influence which interpreters PEX then chooses to run your own code Instead, what I recommend is using a pants.rc file for your M1 users: https://www.pantsbuild.org/v2.10/docs/options#pantsrc-file Note that only 2.10 recognizes
<build root>/.pants.rc
by default. If you're not able to use 2.10 yet, you could instead use one of those two global paths, or use an alternative approach like using a
.env
file and
direnv
h

high-energy-55500

02/11/2022, 5:04 PM
so my workaround has been to use the pants.rc file to force pants to use pyenv
Copy code
[python-bootstrap]
search_path = ["<PYENV>"]
, but I don’t understand why it wouldn’t default to pyenv anyway, since that’s also in my
PATH
Copy code
martim@ip-192-168-1-227 allpdl % echo $PATH
/Users/martim/.pyenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/martim/.poetry/bin:/Users/martim/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki:/Users/martim/.cargo/bin
h

hundreds-father-404

02/11/2022, 5:06 PM
It does use <PYENV> because it's in your PATH, but it also uses all other entries in your <PATH> like Xcode Python. Interpreter selection first identifies all candidate interpreters on the machine, then it checks which are compatible with your interpreter constraints and goes with the minimum compatible
h

high-energy-55500

02/11/2022, 5:07 PM
hmm
so it doesn’t use the first one that’s available?
h

high-energy-55500

02/11/2022, 5:09 PM
is there any way to prefer a certain version over another? instead of going with the minimum compatible?
h

hundreds-father-404

02/11/2022, 5:10 PM
There is not atm. What would that look like? As in, what criteria are you using to declare preference?
h

high-energy-55500

02/11/2022, 5:12 PM
my idea would be to use the order that the search paths are provided, intuitively I would expect there to be a difference between
["<PYENV>", "<PATH>"]
and
["<PATH>", "<PYENV>"]
but there’s probably some reason that’s not done already
f

faint-businessperson-86903

02/11/2022, 5:19 PM
Hey colleague of @high-energy-55500 here 🙂 Not sure what an ideal way to select preference would be, but if it helps here’s more info on our goals and current config: Currently we want to support the following platforms: • Intel Mac (3.8 or 3.9) • M1 Mac (3.9) • Linux (3.8 or 3.9) And we don’t want to make certain users configure a pantsrc if they are on a specific platform if possible. So we did this in our
pants.toml
, with the assumption that the order for
--python-bootstrap-names
was significant and it just selected the first one that works in the constraints:
Copy code
[python-bootstrap]
search_path = ["<PATH>"]
names = ["python3.9", "python3.8", "python3", "python"]
So i.e. even if you are on a system that only works with 3.9, as long as
python3.9
is on your PATH it wouldn’t matter.
h

high-energy-55500

02/11/2022, 5:19 PM
even within the same path, I would expect it to use the first compatible python version. so since I have pyenv at the top of my PATH, I would still expect it to use that version instead of some other version that comes later
h

hundreds-father-404

02/11/2022, 5:23 PM
That ordering generally makes sense to me for Pants's perspective. I like it because it doesn't require inventing some new mechanism to declare your preference. Although not sure it makes sense for PEX. It's more nuanced than PEX simply choosing one interpreter. When you say
pex --interpreter-constraint='CPython>=3.5,<3.10 Django
, it interprets that as you asking to build a PEX that works with all of 3.5-3.10. You're not just matching one interpreter. Currently, the algorithm is for each minor version (3.6 etc), choose the version with the highest patch, so prefer 3.6.9 over 3.6.2. Instead, the impact of this change would be not caring about patch version and going with what's first on the
--python-path
. I'm not certain that will make sense for Pex.
Hey @faint-businessperson-86903, yeah,
python-bootstrap-names
is completely irrelevant to this. Your options are: 1. Use
.pants.rc
or an alternative like
.env
and
direnv
2. Update your repo to use
==3.9
The second option is more sound, as it means that all your engineers will be using the same Python version. Generally, when possible, we encourage organizations to pin their interpreter constraints to an exact major/minor version like
==3.9.*
.
Currently, the algorithm is for each minor version (3.6 etc), choose the version with the highest patch
We would be reversing https://github.com/pantsbuild/pex/pull/1088, which was a big improvement from before and avoided lots of issues. I'm skeptical we will want to give up that win
Ah, rather than doing this based on
[python-bootstrap].search_path
order (which becomes PEX's
--python-path
), https://github.com/pantsbuild/pex/issues/430 would let you work around this quite nicely.
f

faint-businessperson-86903

02/11/2022, 5:35 PM
Got it @hundreds-father-404, appreciate your insight here. It seems we thought `python-bootstrap-names was something that it actually wasn’t 😛 Being able to configure pex to choose the max version rather than the max patch of the min version when building PEXes would be nice, but I think there are other areas we would actually prefer the other behavior (for example we want linting CI to use the min version to flag 3.9+ syntax) so we would have to reconcile those goals there, adding some more complexity.
h

hundreds-father-404

02/11/2022, 5:36 PM
but I think there are other areas we would actually prefer the other behavior (for example we want linting CI to use the min version to flag 3.9+ syntax) so we would have to reconcile those goals there, adding some more complexity.
That is precisely why I think minimum is the better default and why I chose to not focus on https://github.com/pantsbuild/pex/issues/430 when writing https://github.com/pantsbuild/pex/pull/1088
f

faint-businessperson-86903

02/11/2022, 5:37 PM
So from your suggestions, I think the course of action we may take is to: 1. Move to 3.9.* as soon as we can. We have a good amount of infra using 3.8 today which will take some time to move over, but we’ll get there. 2. Until 3.8 is out of the picture, have users on M1 macbooks use a
.pants.rc
file to override interpreter_constraints to
==3.9.*
👍 2
As an aside, if it were possible to set configs in pants that would only load if a certain platform is being used, that would also be helpful for this and possibly other platform-specific issues that may come up in the future
Maybe I should create a feature request in pants for that one? 🙂
h

hundreds-father-404

02/11/2022, 5:41 PM
Ahhh a third solution: 3. Stop using
<PATH>
in
[python-bootstrap].search_paths
. Only use
<PYENV>
and possibly some blessed paths like
/opt/python
. See this great soon-to-be-published blog from @fresh-cat-90827 about this topic https://docs.google.com/document/d/1IQk3pM3YKLypwvw0Y8vMq8jRAJ9J0n-0xZ-GhbuFPE4/edit?usp=sharing THat way you don't use the Xcode python
Maybe I should create a feature request in pants for that one?
It'd be helpful if you can comment on https://github.com/pantsbuild/pants/issues/7735 explaining the problem you have with M1s at your org and how this feature would have helped you
👍 1
h

happy-kitchen-89482

02/11/2022, 9:01 PM
@faint-businessperson-86903 A feature request for platform-specific config would be great!
h

hundreds-father-404

02/11/2022, 9:02 PM
(That's what 7735 already is)