is our expected course right now that we can do a single shared resolve for all relevant interpreters?

if so, can probably remove the

local_only

mode.

but yea… i suppose consuming the lockfile will either result in all of the wheels necessary for all interpreters, or not. so that portion might remain independent.

write, read, then meet

Proposal for how to robustly handle `--platform`: https://github.com/pantsbuild/pants/issues/12557 Credit to @ancient-vegetable-10556 for the breakthrough with post-processing! (And John for implementing that idea a few weeks ago)

i’ll book something.

Here's what I've come up with by looking at PythonRepos / PythonSetup / target options involved in resolves and env-var back doors:

* Require Solution

PythonRepos:
+ --repos: No --find-links support in Poetry.

PythonSetup:
+ --requirement-constraints: Is there a way to map this concept all of the time or some of the time?

Support pip options for requirements <https://github.com/pantsbuild/pants/issues/12090>: Is the answer here that folks using pip options just don't get to use Pants generated lock files?

* Require care

There are some Pex options that affect resolves that Pants doesn't currently use (like --pre). If Pants decides to use them, care must be taken to map the Pex option to a Poetry equivalent when setting up a lock. In the `--pre` case there is a Poetry mapping of the concept that requires suffixing version with * and adding `allow-prereleases = true`.

* Solved

PexPlatformsField: We will produce locks that will cause Pex to needlessly error when building a PEX file whenever the latest compatible version of a dist is available as sdist only, but earlier compatible versions have compatible published wheels. 

* Checked

SubprocessEnvironment (Pex / Pip / requests) knobs:
+ It looks like there are no official docs calling out use of PEX_ or PIP_ to solve any requirement resolution problems.
+ Poetry uses requests so these (HTTP_PROXY, etc) should pass through.

--ca-certs-path: Can map via `poetry config --local` (i.e.: poetry.toml) or POETRY_ env var.

PANTS_ knobs: Configuring indexes with auth in url and not wanting this written in a file can be achieved with POETRY_ env vars.

That gives you (optionally) platform independent locks, pip legacy and pip 2020 support, --find-links support, constraints support, pip args support, --platform support without crying wolf and tests locking pantsbuild.pants show slightly fatser (~10.4s lock vs ~11s lock with Poetry).

Let me know what further you want me to do if anything. To clean up my weekend hacks and do more exhaustive testing to get this added as a feature to Pex will probably be a few weeks of work.

The interesting cases I hadn't considered and that came up in my hacking were: 1. sdist resolution: Pip runs

python setup.py egg_info

(or PEP-517) to get the metadata to trace dependencies and you can't do this when you want to do a platform agnostic lock since you may not have an appropriate interpreter to run that setup.py with (some setup.py actively check sys.executable and fail fast if its version is undersired - functools32 is an exapmple). Poetry punts here and simply expects the sdist has a PKG-INFO file in it already. I expect this is ok for any modern sdist and it just dies on old ones. I had to hack Pip to do similar and use PKG-INFO if present. 2. You can't simply turn off tag evaulation and marker evaluation. For marker evaluation you must handle the

extras

marker and you must handle

python_version

and

python_full_version

- but just those three and no others.

@enough-analyst-54434: and given that 1) this is the long pole on having lockfile support actually be stable, and 2) we’d love to get more folks involved in PEX… if you’re able to identify any portions of that work that would be delegatable to @ancient-vegetable-10556 or @hundreds-father-404 , that would be awesome too

I can probably check that here in a few minutes. I just wanted to see if that check had already been done.

Yeah so they do use resolvelib.

I think thats +1 to my concern about the whole feature <cross-platform> since it appears Poetry is the last man standing on that one

Yeah, I do think it's important (if we can robustly support it). We know many Pants users use macOS for desktop and Linux for CI / some desktop users When updating lockfiles, ideally Linux users don't need to bother a macOS dev to also regenerate the lockfile - imagine if you had to do that whenever changing pantsbuild/pants

The fact we all look past ~daily with large lockfiles is no one reads them anyway. They're collapsed in GH views for exmaple. I don't read them (Cargo.lock) anyhow! We only care tests still pass.

I won't battle to hard on this. I think dynamic dependency resolution is a lie in most ecosystems - the jvm and python for sure - but that is a belief held by almost no-one afaict and a lockfile covering all platforms is just one falsehood aspect. Clearly lots of folks like this and are fine with it.

Alright - I'll spin up some Pex issues to write down / track what's needed here.

Ok, and the major work is now all linked back to #1400 and in https://github.com/pantsbuild/pex/projects/1 Everything is blocked on #1401 but once that is complete (platform specific locks), then some parallel paths open up if others want to get involved.

yea, once #1401 is done, let’s talk about how Eric and Chris can get involved