Hi all, sorry for some questions on using pants - ...
# general
Hi all, sorry for some questions on using pants - I just started trying to use it for a monorepo with existing projects. While I've hit a few issues, I'm currently trying to understand this "least astonishment" issue for pants with python. I suspected that pants would use either my currently activated virtual environment (default to
in the
) or otherwise use my system python from MacOS. however, it seems to behave in a completely unexpected way. In the output (see thread), you can see that I'm running in a
venv, pants uses a seemingly random
binary from my pyenv shims, and then when the goal is run, it uses a
binary (maybe the original system binary?).
Copy code
% python --version           
Python 3.10.3

% pyenv version              
global-3.10.3 (set by PYENV_VERSION environment variable)

% ./pants generate-lockfiles 
Bootstrapping Pants using /Users/mike/.pyenv/shims/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex> verified.
Installing pantsbuild.pants==2.11.0rc3 into a virtual environment at /Users/mike/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0rc3_py39
New virtual environment successfully created at /Users/mike/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0rc3_py39.
09:21:04.25 [INFO] Completed: Generate lockfile for python-default
09:21:04.26 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
The currently activated Python version 3.7.12 is not supported by the project (>=3.10).
Trying to find and use a compatible version. 

I also have this set in my
file making it even more confusing:
Copy code
interpreter_constraints = ["CPython>=3.10"]
I can't directly speak to the final problem, Pants should be respecting the
you configured when trying to generate a lock for the
resolve. As to the Python 3.9 - Pants itself needs Python to run. This is true even if you only use Pants to work with Go or the JVM. As such it has its own interpreter constraints of
; i.e. it runs on Python 3.{7,8,9}. So you see it picking a Python 3.9 interpreter to run on from your machine. That should generally be of no concern to you; except here it caused understandable confusion when downstream errors made you pause to read the logs more fully. After picking the interpreter it needs, Pants then (generally!) picks the interpreter your code needs if you're using Pants to work with Python code like you are.
@ripe-analyst-17330 do you have custom configuration for
[python-bootstrap] search_path
?: https://www.pantsbuild.org/docs/reference-python-bootstrap#section-search-path
that makes sense for pants itself to use
, thanks. It does seem like pants eventually runs a different venv. I've tried both with and without the following with the same result:
Copy code
search_path = ["<PYENV>"]
What do you have configured for
[python] lockfile_generator
? I'm on a little firmer footing talking about the Pex locker. This may be using the Poetry locker though under the covers.
yes, I'm using poetry. Currently, all our projects have their own folders and individual poetry lock files. I was just trying to get to the point I could use those, and ran across this issue.
it does seem to work if I explicitly set
lockfile_generator = 'pex'
after looking at the code, I determined this was the cause: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/subsystems/poetry.py#L30 Setting this explicitly in the poetry config got me past it:
Copy code
version = "poetry==1.1.13"
interpreter_constraints = ["CPython>=3.10,<4"]
Is there a per-project poetry config that can be set for this as it seems it would break a monorepo with different version constraints per project? We have this issue in our repo I just don't know if its enough to cause problems here (most are 3.9+ compatible)
Ah, yeah - and that's used here with no regard to the
ICs: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/goals/lockfile.py#L208-L214 Just above though, using the Pex locker, the right thing is done: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/goals/lockfile.py#L159-L178 So, it looks like you've found a bug for locking using the Poetry locker. I'll check to see if that's filed / known. You might try switching to the Pex locker if you need per-project support. We're headed to that being the default anyhow and likely will strip support for Poetry being the locker since it simply can't handle some permutations of the inputs - like custom find links repos.
Hrm, I could be wrong. Do you have
[python] enable-resolves
configured to
? (It may be useful to just dump the `[python]`section of your
to prevent a bit of back-forth): https://www.pantsbuild.org/v2.11/docs/reference-python#section-enable-resolves
@hundreds-father-404 will certainly better be able to guide you through all this more efficiently. I've been working on the Pex support for generating and consuming locks and have a lot less familiarity with the Pants side of this.
Aha, the custom ICs do get threaded through for Poetry into a generated
here: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/subsystems/poetry.py#L68-L83 I think the problem is when The Poetry tool (PEX) is installed using different ICs than the ICs fed to it in
. IOW we use 1 global ~venv for Poetry and we then try to feed it many different generated
, each with custom ICs. If the ICs conflict - boom.
@hundreds-father-404 if you can sanity check my guess immediately above, that seems like an un-filed bug.
Ok, thanks @ripe-analyst-17330 for digging into the code and fiddling with your
config. That was very helpful. I'm pretty sure my analysis is correct so I went ahead and filed: https://github.com/pantsbuild/pants/issues/15171 Hopefully you can use the workaround you found or else switch to the "pex" lockfile generator in the meantime.
I'm glad you called that out - my next step is how to do multiple resolves as we currently have poetry lock files on a per-project basis. internal code will basically all use
but with 3rd party deps, we sometimes use different versions in different projects. But it sounds like this breaks the multiple resolves as well, which I hadn't yet figured out
Although you can mix Poetry projects with the Pex locker, that is probably not a great idea since they could produce different locks. That would then lead to
(~/dev/repo) $ ./pants foo poetry_project1/...
having different results than
(~/dev/repo/poetry_project1) $ poetry foo ...
I could probably make a global lock work. I would just need to figure out if there is any other work that needs to be done around the poetry config in
- but I'm guessing we would just need per-project
poetry build
commands. Sorry I'm just starting to look at pants, so I'm not too familiar with the build steps yet. I was building CI matrixes with scripts when this was suggested.
Hey @ripe-analyst-17330, my apologies for not getting back to you earlier. I was moving from AZ to Mexico City when this happened Where are you at now with things? Anything we can help with?