https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all, sorry for some questions on using pants - ...
# general
r
Hi all, sorry for some questions on using pants - I just started trying to use it for a monorepo with existing projects. While I've hit a few issues, I'm currently trying to understand this "least astonishment" issue for pants with python. I suspected that pants would use either my currently activated virtual environment (default to 
python
in the 
PATH
) or otherwise use my system python from MacOS. however, it seems to behave in a completely unexpected way. In the output (see thread), you can see that I'm running in a 
3.10
venv, pants uses a seemingly random 
3.9
binary from my pyenv shims, and then when the goal is run, it uses a 
3.7
binary (maybe the original system binary?).
Copy code
% python --version           
Python 3.10.3

% pyenv version              
global-3.10.3 (set by PYENV_VERSION environment variable)

% ./pants generate-lockfiles 
Bootstrapping Pants using /Users/mike/.pyenv/shims/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex> verified.
Installing pantsbuild.pants==2.11.0rc3 into a virtual environment at /Users/mike/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0rc3_py39
New virtual environment successfully created at /Users/mike/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0rc3_py39.
09:21:04.25 [INFO] Completed: Generate lockfile for python-default
09:21:04.26 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:
The currently activated Python version 3.7.12 is not supported by the project (>=3.10).
Trying to find and use a compatible version. 

  NoCompatiblePythonVersionFound
I also have this set in my 
pants.toml
file making it even more confusing:
Copy code
[python]
interpreter_constraints = ["CPython>=3.10"]
e
I can't directly speak to the final problem, Pants should be respecting the 
interpreter_constraints
you configured when trying to generate a lock for the 
python-default
resolve. As to the Python 3.9 - Pants itself needs Python to run. This is true even if you only use Pants to work with Go or the JVM. As such it has its own interpreter constraints of 
>=3.7,<3.10
; i.e. it runs on Python 3.{7,8,9}. So you see it picking a Python 3.9 interpreter to run on from your machine. That should generally be of no concern to you; except here it caused understandable confusion when downstream errors made you pause to read the logs more fully. After picking the interpreter it needs, Pants then (generally!) picks the interpreter your code needs if you're using Pants to work with Python code like you are.
@ripe-analyst-17330 do you have custom configuration for 
[python-bootstrap] search_path
?: https://www.pantsbuild.org/docs/reference-python-bootstrap#section-search-path
r
that makes sense for pants itself to use 
3.9
, thanks. It does seem like pants eventually runs a different venv. I've tried both with and without the following with the same result:
Copy code
[python-bootstrap]
search_path = ["<PYENV>"]
e
What do you have configured for 
[python] lockfile_generator
? I'm on a little firmer footing talking about the Pex locker. This may be using the Poetry locker though under the covers.
r
yes, I'm using poetry. Currently, all our projects have their own folders and individual poetry lock files. I was just trying to get to the point I could use those, and ran across this issue.
it does seem to work if I explicitly set 
lockfile_generator = 'pex'
after looking at the code, I determined this was the cause: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/subsystems/poetry.py#L30 Setting this explicitly in the poetry config got me past it:
Copy code
[poetry]
version = "poetry==1.1.13"
interpreter_constraints = ["CPython>=3.10,<4"]
Is there a per-project poetry config that can be set for this as it seems it would break a monorepo with different version constraints per project? We have this issue in our repo I just don't know if its enough to cause problems here (most are 3.9+ compatible)
e
Ah, yeah - and that's used here with no regard to the 
req
ICs: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/goals/lockfile.py#L208-L214 Just above though, using the Pex locker, the right thing is done: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/goals/lockfile.py#L159-L178 So, it looks like you've found a bug for locking using the Poetry locker. I'll check to see if that's filed / known. You might try switching to the Pex locker if you need per-project support. We're headed to that being the default anyhow and likely will strip support for Poetry being the locker since it simply can't handle some permutations of the inputs - like custom find links repos.
Hrm, I could be wrong. Do you have 
[python] enable-resolves
configured to 
true
? (It may be useful to just dump the `[python]`section of your 
pants.toml
to prevent a bit of back-forth): https://www.pantsbuild.org/v2.11/docs/reference-python#section-enable-resolves
@hundreds-father-404 will certainly better be able to guide you through all this more efficiently. I've been working on the Pex support for generating and consuming locks and have a lot less familiarity with the Pants side of this.
Aha, the custom ICs do get threaded through for Poetry into a generated 
pyproject.toml
here: https://github.com/pantsbuild/pants/blob/2d2e287193e5d2ee68058c0b016a184d4f02489d/src/python/pants/backend/python/subsystems/poetry.py#L68-L83 I think the problem is when The Poetry tool (PEX) is installed using different ICs than the ICs fed to it in 
pyproject.toml
. IOW we use 1 global ~venv for Poetry and we then try to feed it many different generated 
pyproject.toml
, each with custom ICs. If the ICs conflict - boom.
@hundreds-father-404 if you can sanity check my guess immediately above, that seems like an un-filed bug.
Ok, thanks @ripe-analyst-17330 for digging into the code and fiddling with your 
pants.toml
config. That was very helpful. I'm pretty sure my analysis is correct so I went ahead and filed: https://github.com/pantsbuild/pants/issues/15171 Hopefully you can use the workaround you found or else switch to the "pex" lockfile generator in the meantime.
r
I'm glad you called that out - my next step is how to do multiple resolves as we currently have poetry lock files on a per-project basis. internal code will basically all use 
HEAD
but with 3rd party deps, we sometimes use different versions in different projects. But it sounds like this breaks the multiple resolves as well, which I hadn't yet figured out
e
Although you can mix Poetry projects with the Pex locker, that is probably not a great idea since they could produce different locks. That would then lead to 
(~/dev/repo) $ ./pants foo poetry_project1/...
having different results than 
(~/dev/repo/poetry_project1) $ poetry foo ...
.
r
I could probably make a global lock work. I would just need to figure out if there is any other work that needs to be done around the poetry config in 
[tool.poetry.scripts]
- but I'm guessing we would just need per-project 
poetry build
commands. Sorry I'm just starting to look at pants, so I'm not too familiar with the build steps yet. I was building CI matrixes with scripts when this was suggested.
h
Hey @ripe-analyst-17330, my apologies for not getting back to you earlier. I was moving from AZ to Mexico City when this happened Where are you at now with things? Anything we can help with?
Open in Slack
PreviousNext
Powered by