https://pantsbuild.org/ logo
Join Slack
Powered by
<@U051221NF>, <@UB2J9BQA0>, <@U021C96KUGJ>, <@U04S...
# development
w
@happy-kitchen-89482, @hundreds-father-404, @ancient-vegetable-10556, @enough-analyst-54434: i think that last time https://github.com/pantsbuild/pants/pull/14050 occurred (plugin with an incompatible requirement), i had hoped that moving to releasing a PEX (via #12397) would prevent the issue from recurring. but thinking about it more, anything that doesn’t do an on-the fly resolve at use-time (PEX, a lockfile, a rust binary, etc) would all potentially still suffer from this issue
…because there is one resolve which has already occurred, and then another resolve is happening for the plugin
➕ 1
h
Yeah, we need a lockfile for plugins to resolve this iiuc
w
so: changing tacks (and trying to tackle this more aggressively this time… “fool me once”): it occurred to me that we might want to skip using the code paths in `pkg_resources`/`setuptools` that actually check for version conflicts, and add to the 
sys.path
regardless of conflicts
@hundreds-father-404: i don’t think that that would do it: it’s still two separate resolves
👍 1
a
@hundreds-father-404 Moreso than that, it’s probably-invalid for a plugin to specify a dependency that’s in Pants’ own transitive dependencies
w
@ancient-vegetable-10556: it’s ok as long as they’re compatible… the dependency that’s often in question is 
requests
a
@witty-crayon-22786 Compatibility means that plugins need to move in lockstep with mainline, so that counts to me as “eventually invalid”, and should be discouraged for that reason
w
“should be discouraged”, sure. but it’s awkward for either party to need to avoid the most common packages in the ecosystem
➕ 1
a
The alternative is that the resolve for a given dependency of Pants is actually part of Pants’ API, and we make sure that the plugin itself is invalid for a given version if it conflicts with Pants’ API
(which is to say, yes, we need to check for version conflicts, and make better guarantees on Pants’ side)
@witty-crayon-22786 I’m not saying that we discourage use of 
requests
, but rather that we say 
requests
is provided by Pants (and we make guarantees about versions). Either way, plugins need to move in lockstep with Pants’ dependencies, but making the requirements for a given version of Pants explicit seems wise.
w
@ancient-vegetable-10556: range/floating resolves do actually mostly work, except for the two resolves problem, which is the main thrust of this thread
i.e., if both the plugin and pants itself are flexible in which version of 
requests
they accept, things work from a “there is a compatible version for both of them” perspective. it could be better formalized, sure.
but two resolves mean that each range can land somewhere different.
PEX or a rust binary don’t allow for a single resolve, since the whole point is that things have been pre-resolved, and we don’t know about the plugins until runtime. generating a single lockfile for pants and the plugins would also work, because doing that requires a single resolve
➕ 1
so my thinking in https://pantsbuild.slack.com/archives/C0D7TNJHL/p1641235877010300?thread_ts=1641235771.009700&amp;cid=C0D7TNJHL is to remove checks from plugin loading, or otherwise harmonize the resolves. we’re likely to need to switch to distributing a PEX soon anyway (for PyPI storage reasons), so the “single lockfile for pants and plugins” approach isn’t viable
a
As long as there isn’t a plugin that specifies an incompatible version of a dependency, sure
w
we either fail fast (as we did this morning) or slow (with an actual API incompatiblility) in that case
➕ 1
but yes
h
slow failure seems less likely to happen, these are stable libraries like requests
w
e
We should probably keep the result of pip freeze from the Pants venv as a constraints input for plugin resolves. That would both be truthful and allow plugin resolves to solve correctly.
Or ... we have the constraints / lock already I guess - perhaps in the dist?
h
Longer term, perhaps we shade and vendor Pants's requirements?
e
Yeah, or isolate them in a separate Python only used for rules. Right now the Pants deps are mainly outside of rules IIUC and in boot-time / tear-down-time code paths. They all happen to share a sys.path today but maybe they don't need to. Could be complex though.
w
We should probably keep the result of pip freeze from the Pants venv as a constraints input for plugin resolves.
ah: yes… this seems like an exact match for what we want to do. basically, take the current working set, and feed it to PEX while we’re resolving plugins!
➕ 1
💡 1
totally blanked on constraints… and also the fact that PEX is the thing doing the resolve, rather than 
pkg_resources
.
thanks.
and yea @happy-kitchen-89482: shading would make sense too.
Open in Slack
PreviousNext
Powered by