That’s unnecessary invalidation and scanning of the repo, done every time you use Flake8 because of the upcoming lockfile invalidation

this goes back to the actual ticket, and so i can comment there. but isn’t it only necessary to invalidate if your IC is not a member of the lockfile? it’s not relevant for that run whether some other IC is missing from the lockfile.

Good question, interesting! If we made our invalidation logic much more nuanced, then that could work. R/n it's a hash that combines all the input requirements + the ICs used to generate the lockfile for the whole repo. Concretely, the call sites use

await Get

to get the

PythonLockfileRequest

, access its

invalidation_digest

, and compare to what is written on disk in the lockfile (https://github.com/pantsbuild/pants/pull/12448) We could instead store what the ICs are directly in the lockfile. Then the only thing we are missing afaict is being able to statically determine if two IC ranges are compatible are not which gets to the comment I punted on above. Every time I have attempted to do clever things w/ ICs—like simplifying

['==3.7,==2.7', '==2.7']

to be

['==2.7']

b/c the first is impossible to satisfy—I have not been able to figure it out. All our

interpreter_constraints.py

code r/n that checks for "validitity" works by iterating over every anticipated version in our Python "universe". It feels like it shouldn't be hard: as a human, I can very quickly deduce it's unsatisfiable. But I haven't been able to figure out how to automate the static analysis

Thanks for talking this all through, Stu! There are 3 proposals based on this convo, and I don't think any of them are competing actually: https://github.com/pantsbuild/pants/pull/12491 I'm going to focus on finishing that PR today to provide a common denominator for which of the 3 (if any) we want to follow up on.

Huh, turns out Poetry's lockfile support already handles the edge cases with env markers that we've been worrying about, including properly handling platform-specific deps? 🧵 We already decided to stop using pip-compile, so had been planning on teaching Pex to generate lockfiles, hopefully w/ PEP 665. But there's huge risk there, including pushing back lockfile project's timeline >1 month. Instead, we can use something that's already battle tested and handles everything we need fwict. So far, only downsides I can see are switching up the resolver from new-pip to Poetry, and not sharing the caches between Pex/Poetry

Edge case 1: interpreter constraint env markers. `Black`'s lockfile varies if it was generated w/ Py36 vs Py39, e.g. the

typing-extensions

dep Poetry handles this, and specifically based on what you set the

python

value to. If you set it to

==3.9

, the lockfile only works w/ 3.9. If you set to

>=3.7,<3.10

, it works with all those intepreters, even tho you only run it w/ one interpreter! Amazing

Edge case 2: platform env markers.

ipython

depends on

appnode

, but only on macOS. Regardless of if you generate lockfile on macOS vs Linux, it handles that! Amazing

Adding Poetry support was a really good decision.

Developer joy!

PEP 655 thoughts: it sounds like it won't be accepted until end of the year, and even with that, the rest of the ecosystem probably won't be fully migrated, e.g. IDE support We do want to be an early adopter and love the standardization, but also should be realistic about the state of the world imo

Adding Poetry support was a really good decision.

Agreed! To clarify, this is a bit different. Liam's project was so that Pants can consume your

pyproject.toml

, where the user says their direct deps they care about Here, we're talking about switching the backend that generates lockfiles for you. We were using pip-compile, and now would be using Poetry. That should be relatively agnostic to the user

not sharing the caches between Pex/Poetry

This is a bummer, but also possibly something we could optimize? It seems theoretically possible for pex to learn how to consume the PyPI artifacts Poetry downloads

including pushing back lockfile project's timeline >1 month.

If we switch to Poetry, I think it becomes feasible for Pants 2.7 to finish the lockfile project this month! At least for tool lockfiles and a single user lockfile

Poetry is well-loved, and using it to save ourselves time is a win! We can continue to engage with that PEP.

another thought here: it’s possible that it would be both more ergonomic and a bit more performant if we deprecated

interpreter_constraints

on targets as defining an actual list, and replaced it with a reference to a name/slug defined globally:

$ cat BUILD
python_library(
  interpreters='default',
)

python_library(
  interpreters='unusual',
)
$ cat pants.toml
..
[python-setup]
interpreters = """{
    'default': '[">=3.7,<3.10"]',
    'unusual': '["==2.7.*", "==3.5.*"]',
}"""

…because then you’d have a closed universe of interpreters

(…having said that, it wouldn’t help with the mypy case.)

I'm interested in that! It helps w/ part of my motivation for making

interpreter_constraints

opt-in: imo, it's too easy for every day developers to accidentally use a feature they shouldn't be (To be clear, I do still want it to be easy to use when you know that's what you want, i.e. the migration discussion yesterday. Only make it something more intentional, but still ergonomic)

Gr, somewhat regretting that we leaned into

interpreter_constraints

using a list to OR constraints, like

['==2.7.*', '>3.6']

. Poetry needs a single constraint, and I don't know of a way to programmatically convert the list into a single value

do we currently intentionally not put

experimental

backends on the docsite?

I had a PR time out while building Rust crates. Is any one else seeing slow PR build times? (context is https://github.com/pantsbuild/pants/pull/12522/checks?check_run_id=3284242628)

….mmmm. with a lockfile/constraints.txt, we’re still limiting PEX to

--jobs=1

.

Not sure if I should ask here or general... Is this the most recent/correct docs for building pants for a manual integration test? https://v1.pantsbuild.org/howto_develop.html

After a series of PRs, the

Optionable

class is no more. Everything has been folded into

Subsystem

, and greatly simplified.

two bits of good news: 1. i made some substantial progress on https://github.com/pantsbuild/pants/issues/11269 this weekend 2. i realized yesterday that #11269 isn’t actually necessary to re-add structure sharing: https://github.com/pantsbuild/pants/pull/12539

I’m experiencing a flaky test in

fs_test.py

(see https://github.com/pantsbuild/pants/pull/12448/checks?check_run_id=3292684957). Has anyone seen this before? (ping @witty-crayon-22786)

a thing that might already be on the minds of lockfile folks is that we are currently in a situation where tests use one

local_only=True

lockfile resolve, and binaries use a

local_only=False

lockfile resolve.

FYI @proud-dentist-22844

ooh.

One word of warning: Poetry does not handle all packages well. If any package relies on custom logic in setup.py to generate its requirements, poetry will fail to resolve that. (I still like poetry, even with the gotcha). Poetry cannot, for example, understand the requirements for

ansible

(prob

ansible-core

at this point, I ran into this years ago). If someone relies on

ansible

, they need to record

ansible

's dependencies as well as

ansible

itself.

I don't know if that gotcha applies to lockfile generation. ¯\_(ツ)_/¯