https://pantsbuild.org/ logo
Join Slack
Powered by
hi all. hopefully this is a good place to ask. i'm...
# general
g
hi all. hopefully this is a good place to ask. i'm having an issue using 
snowflake-snowpark-python
in pants when generating a lock file. i've created this repo that can be used to reproduce. i added more info to the readme in that repo, but the jist is that if i try and use `snowflake-snowpark-python`(i haven't found any others that fail) and then do 
pants generate-lockfiles --resolve=python-default
i get a 
No matching distribution found for snowflake-snowpark-python
error. But I expect it to work, I suppose, because of some other steps like copying the errored command from the output and trying it with 
pip install
in the "same" environment.
e
So, quick check says you did not see this 
Requires: Python >=3.8, <3.11
for https://pypi.org/project/snowflake-snowpark-python/1.6.1/ (I did not check older versions). Combine that with your 
interpreter_constraints = [">=3.10.*"]
and totally expected since you don't upper bound and both 3.11 and 3.12 violate `snowflake-snowpark-python`'s Python requirement and so ... the lock is impossible.
For example, this works fine:
Copy code
$ git diff
diff --git a/pants.toml b/pants.toml
index d045243..6cc4502 100644
--- a/pants.toml
+++ b/pants.toml
@@ -7,5 +7,5 @@ backend_packages = ["pants.backend.python"]
 enabled = false

 [python]
-interpreter_constraints = [">=3.10.*"]
+interpreter_constraints = ["==3.10.*"]
 enable_resolves = true
So you actually need to support `>=3.10`* (aka 3.11)? ā€¢ N.B.: 
>=3.10.*
doesn't make much sense, that's just 
>=3.10
.
Pex will keep you honest.
l
Uff, but def an opportunity for us to improve the error message. I've been wanting to learn a bit more about pex internals so I'll try to see if what it would be take to provide something more descriptive as to why no satisfactory version could be found.
e
There's not much to be done except recapitulate what the user asked for as a preamble afaict without violating Pex's explicit decision to get out of the resolver business in ~2018 by ditching it's bespoke resolver then for Pip. All that really can be said is something like "you asked for <reqs> --interpreter-constraint <ic> ..." as a pre-amble to the error, but when using the Pex CLI directly, that's a bit redundant.
I think the primary issue here is the user isn't using Pex, Pants is on their behalf.
IOW it's Pants that creates and hides the Pex command line.
I won't object to whatever Pants wants to do here but I will have opinions on Pex changes wrt to error messages and the coddling to cryptic message spectrum. Best to file a tracking issue in Pex if you want to attempt a fix there where ideas can be discussed to make sure they make sense given everything else.
šŸ‘ 2
l
+1 or in this case would you advise to warn in pants when interpreter constraints are set to something likely unintended like 
">=3.10.*"
e
I think my opinions may not match Pants maintainer opinions. I personally prefer tools that don't nanny / only give advice they ~know to be true. For example, I don't like Pants' generic error message advice about the OOMKiller, it's generally wrong.
If you tell me not to use 
">=3.10"
, but that's exactly what I intend, I won't be happy.
So @green-match-60388, to circle back. Your experiment was a great idea here, it just wasn't faithful. To be faithful, you'd need to proceed past your initial Python 3.10 venv 
pip install
success to try said same in a Python 3.11 venv. That would have failed and maybe made it more clear why the lock is impossible.
g
That definitely works. Thanks so much for the rapid response.
And it looks like I definitely don't understand what 
pex
is doing. So, 
pex
uses 
pip
under the hood(makes sense why the arguments to 
python pex
look like 
pip
args). and when i copy the same 
python pex
command that is in the "error" output and run it successfully, that is an invalid test because i'm in 3.10?
do you think there could possible be an option that iterates over a few different 
constraints
or can we list multiple with 
parametrize
or something and have them all tried until successful or exhausted? my guess is this brute force approach is likely super time consuming and still might not be super helpful.
e
So, pex uses pip under the hood(makes sense why the arguments to python pex look like pip args). and when i copy the same python pex command that is in the "error" output and run it successfully, that is an invalid test because i'm in 3.10?
So Pex uses Pip but it does not do the same thing as Pip. When you issue 
pip install
it resolves distributions for the current Python on the current computer and installs those. When you ask Pex to create a lock file (which is what Pants is doing here), you're asking it to resolve distributions for every possible machine and interpreter you specify. Now Pants doesn't give you control over which machines (it tells Pex 
--target-system linux --target-system mac
unconditionall currently - and so it limits to linux and mac (i.e.: the lock won't work for Windows Python interpreters)), but it does give you control over which interpreters via 
interpreter_constraints
. Since your 
interpreter_constraints
specified the open range 
>=3.10
, Pex was told to solve a lock that works for both linux and mac machines and every Python interpreter 
>=3.10
. So 3.10.0, 3.10.1, ..., 3.11.0, 3.11.1, ..., etc.
do you think there could possible be an option that iterates over a few different constraints or can we list multiple with parametrize or something and have them all tried until successful or exhausted? my guess is this brute force approach is likely super time consuming and still might not be super helpful. (edited)
I really have no clue what you're getting at here. So do you actually need 
>=3.10
, i.e. 3.11 too and thus you've moved on in this paragraph to try to solve that problem?
I guess a tacit assumption here is Pants assumes you know what a lock file does / looks like / etc from either other Python tools like Poetry or else other ecosystems with similar facilities (Rust+Cargo.lock, etc.).
@green-match-60388 does https://pantsbuild.slack.com/archives/C046T6T9U/p1692724927438779?thread_ts=1692667074.554899&amp;cid=C046T6T9U make sense to you? Do you have familiarity with lock files or is this your 1st exposure to multi-platform / multi-interpreter lock files?
g
lol, i thought i was more familiar with the concept of a lock file. i've used cargo and poetry for a few years now. i guess i put the constraint as 
>=3.10
because we use python 
match
statements and i believe those are support in 3.10 and greater. so, it made sense to me to do that. i also believe this was working in `poetry`(which i'm migrating from), but now that i check we had the upper bound set to 
<3.11
.
e
What is a python 
match
statement? you mean Poetry's version of an interpreter_constraint? If so, OK. That makes more sense. You had a transcription error and wrote 
>=3.10.*
instead of transcribing 
>=3.10,<3.11
from the Poetry setup.
Drop the upper bound from the Poetry config and hopefully it also fails to lock.
g
yeah, it does fail to lock in poetry, and embarrassingly, i think i went through this with 
poetry
like 6 months ago... and when i mentioned the python 
match
, i mean that structural pattern matching that was introduced in python 3.10.
e
Aha - gotcha.
g
thanks for taking the time to walk me through this. very helpful.
Open in Slack
PreviousNext
Powered by