Yeah, note that we say this in our docs:

This means that every dependency of a target must also be compatible with its interpreter constraints. Generally, you will want to be careful that your common python_library targets are compatible with multiple Python versions because they may be depended upon by other targets. Meanwhile, pex_binary and python_tests targets can have specific constraints because they are (conventionally) never dependencies for other targets.

But we say "can have specific constraints", not "must". I'm not sure how we'd enforce it, and it would also add new boilerplate in some cases not necessary. Like, my above example already would end up being ==3.6 b/c the transitive dep, and now you have to explicitly say that are you envisioning this change because it avoids these perf hits of looking at transitive targets? or other motivation?

are you envisioning this change because it avoids these perf hits of looking at transitive targets?

Note that I think this discussion would only help w/ Pytest fwict? MyPy and Pylint could run purely over

python_library

targets, and we'd still need to look at transitive deps there

are you envisioning this change because it avoids these perf hits of looking at transitive targets? or other motivation?

both that it makes your graph more accurate, and that it assists in this case, yea.

MyPy and Pylint could run purely over 

python_library

 targets, and we’d still need to look at transitive deps there

i don’t think so. i’ve been giving the example of libraries vs binaries, but this applies throughout the graph: even when a library depends on a library.

Hm if we do go with PEP 665 for Pants's lockfile project, I think we need an intermediate

export

command to convert to

requirements.txt

? For now, nothing like IDEs understand PEP 665

i don’t see anything else bound for 2.6.x today: i can cut a new rc?

i’ve been giving the example of libraries vs binaries, but this applies throughout the graph: even when a library depends on a library.

To be clear, that enhancement only works if we make a breaking change to how ICs work: they must include the ICs of all transitive deps, not just the source files themselves. Is that right? With binaries and tests, you mentioned that seems more reasonable to ask. I do think we'd need the same restriction on all

python_library

targets too, given how MyPy and Pylint operate on them

To be clear, that enhancement only works if we make a breaking change to how ICs work: they must include the ICs of all transitive deps, not just the source files themselves. Is that right?

yea, i think that this requires a deprecation cycle. but it’s likely that the optimization could be applied even before the deprecation cycle has completed (because the check for whether the graph is valid is sufficient to either warn or enable the optimization).

With binaries and tests, you mentioned that seems more reasonable to ask. I do think we’d need the same restriction on all 

python_library

 targets too, given how MyPy and Pylint operate on them

yes, but i think that it is a valid constraint in all cases. it makes the constraints match reality: if a target’s dependencies cannot be used with X, then it also cannot be used with X.

Okay, glad we're on the same page with what this change would entail. I feel strongly from my experience w/ Py3 migrations that it's imperative you are able to set your ICs only based on the code itself's compatibility, rather than needing to consider all transitive deps—and then Pants will ensure the result is valid by merging the ICs From the Py3 blog:

A key ingredient to a large-scale, incremental migration is maintaining a real-time status of your project. Ideally, this information would be:

- decentralized to change, so that teams can work independently

- readable through a centralized view, so that migration leaders can track progress

- aware of how dependencies impact compatibility

- used at runtime to avoid becoming stale

- as precise as a file, which lowers the activation energy needed to port code

I've found there's lots of value in empowering individual contributors/teams to make progress on their portion of the code w/o being blocked by others Concrete example: • your org has a common util that is still Py2 and is blocked upgrading • your team's code is currently Py2 only, but you'd love to update it • with the current setup, you can migrate your code to be Py2 OR Py3, and then Pants will see the dep and merge to be Py2 at runtime • with the proposed setup, saying your own code is Py2 or Py3 is a lie, b/c a dep is still Py2 only

you can migrate your code

Of course you could still migrate your code and not update the ICs, but then you lose the abilitity for Pants to do fine-grained tracking of your migration -- FYI that entire idea of "fine grained tracking" and the blog's wishlist comes from the approach I took for Foursquare, Pants, and parts of Twitter: I used a Google sheet for file level tracking I realized how much more powerful it would be for Pants to do that

I feel strongly from my experience w/ Py3 migrations that it’s imperative you are able to set your ICs only based on the code itself’s compatibility, rather than needing to consider all transitive deps—and then Pants will ensure the result is valid by merging the ICs

i disagree. if you set a constraint based only on the code, then it’s a lie. the code cannot be used without its dependencies. it’s not useful to anyone to know what the constraints of the code would be if it didn’t have any deps.

what is the concrete usecase for knowing what the constraints would be if the code didn’t have deps? afaict, all of the usecases you mention require that the deps actually be upgraded first… and you can’t actually know that some code is actually compatible with a newer version until you run it (which you can’t do until the deps are updated)

…unfortunately i’ve got to head to the dentist. back afterward

You're shifting the burden of tracking from the dev to the build system. Pants ensures deps are congruent, rather than forcing you to do it when deciding what IC to set

it’s not useful to anyone to know what the constraints of the code would be if it didn’t have any deps.

Based on what I found doing migrations, I disagree here. There's value in knowing "what is this code itself compatible with", along with also "okay, now include deps: what is it compatible with?" Note that

py-constraints --summary

will show you both that code's own constraints and the transitive constraints. See the CSV in https://blog.pantsbuild.org/python-3-migrations/

that the deps actually be upgraded first…

Nack, you can migrate Py2-only code to become Py2 or Py3, even if deps are still Py2 only You are correct that you cannot migrate to be Py3-only until all deps are at a minimum Py2 or Py3

I think that one way to think about this is with the difference between "source" vs "direct" vs "transitive" constraints. Currently these are being used as source constraints, but I think that they are more useful as transit constraints.

Agreed on source vs. direct vs. transitive constraints To reiterate what I'm trying to get at: right now, ICs are modeled as source constraints. At runtime, Pants uses either source, direct, or transitive constraints depending on the goal (Flake8 vs. Pylint vs. MyPy/Pytest/package/run). It also exposes that to you via the

py-constraints

goal for migration planning I think there's huge benefit to keeping source constraints. Iiuc, if we change ICs to now alway be transitive constraints, we lose the notion of source constraints and are back in the territory of using a manually maintained Google Sheet like I had to do w/ the 3 migrations

I think that it would likely be incorrect to ever use the source constraints with a tool. An example: if you have a formatter or linter that checks for a Python 3 specific condition and you run it on code whose transitive constraints are actually only valid for Python 2, you will break that code.

AFAICT, source constraints are only useful for human consumption. But also, I don't think it's accurate to say that changing what interpreter constraints mean would make the py-constraints goal noticeably less useful: it just means that edits in the middle of the graph rather than the bottom can't be encoded in the interpreter constraints. But since that code can't ever be checked to actually be compatible by running it, source constraints are only ever a human marker anyway... so could just as well be a tag.

If anything, we could add a source_interpreter_constraints field only in the mixed_interpreter_constraints backend...

I think that it would likely be incorrect to ever use the source constraints with a tool. An example: if you have a formatter or linter that checks for a Python 3 specific condition and you run it on code whose transitive constraints are actually only valid for Python 2, you will break that code.

Hm. Actually, I think that this applies regardless of source/transitive... the tool would need to be aware that the code was trying to be compatible with both interpreters, regardless.

Yeah that part is awkward. If you claim source code is Py2 and Py3 compatible (regardless of deps), the correct thing to do is run Flake8 w/ both constraints. Rn, only way to do that is create two targets over the same source file but w/ different constraints. In Twitter, this was the

python23_library

macro. It works, but breaks dep inference due to ambiguity In contrast, w/ tests—which do care about transitive deps—the whole transitive closure must be Py2 and Py3 compatible if you want to test w/ both interpreters

how would you feel about adding a source_interpreter_constraints field in the mixed_interpreter_constraints backend, and migrating the existing interpreter_constraints field to transitive behavior? i really think that that behavior is the most useful to have in core.

Taking a step back to the beginning of this thread..let's say we do make this change so that ICs now represent transitive constraints, not source constraints. And we validate that the transitive constraints are congruent If you are one of the many repos that only uses one single constraint for the whole project, Pants would still be doing unnecessary work. Perhaps not as painful as status quo, but unnecessary. Does that make sense? (I get that there's a tradeoff of if it's worth making this change. That's fine, rn I'm only trying to make sure we're on the same page here w/ the idea of unnecessary work)

you’re trading a scan through the root targets for a backend that you need to enable to use multiple versions at once

the question isn’t whether it is “unnecessary”… rather, is the cost noticeable/worth it

because there is no end to the stuff that we could classify as “unnecessary” and put behind options. but whether the feature is 1) important enough, 2) cheap enough to enable by default is important.

you’re trading a scan through the root targets

And validation of the transitive constraints, right? Even for Flake8 where it doesn't consider deps, we need to validate your ICs were not mucked up when considering deps, iiuc?

a backend that you need to enable to use multiple versions at once

I am not seeing the big deal of having to do this. We already had the backend, and if you're using mixed ICs you really should be activating it so that you have the

py-constraints

goal. I suspect the majority of (at least new) users already have it activated. If you don't already, it's a single line change done one time by a codebase admin There's also benefits beyond performance: you don't clutter

./pants help

with a field that you don't want your teammates using because your repo should be using uniform ICs Ack that the deprecation cycle is awkward w/ having to have the

python-setup

option, but that's temporary

because there is no end to the stuff that we could classify as “unnecessary” and put behind options.

this might be slippery slope fallacy

And validation of the transitive constraints, right? Even for Flake8 where it doesn’t consider deps, we need to validate your ICs were not mucked up when considering deps, iiuc?

i think that it’s sufficient to only validate the portion of the graph that you actually expand. so pytest/mypy would validate the full graph that they were about to invoke, other tools would only validate the portion that they loaded. for example, after partitioning, mypy would compute transitive (which it needs to do to run), and then explode if it found something incompatible.

I am not seeing the big deal of having to do this. We already had the backend, and if you’re using mixed ICs you really should be activating it so that you have the 

py-constraints

 goal. I suspect the majority of (at least new) users already have it activated.

it’s not enabled in example-python. and i don’t know why new users would enable it… it’s not required for usage currently.

There’s also benefits beyond performance: you don’t clutter 

./pants help

 with a field that you don’t want your teammates using because your repo should be using uniform ICs

but you are paying the cost of an administrator needing to know the trick to actually enable that field, and for new users to have to discover in order to gain one of the benefits of pants. as an example: would we need a helper in core to suggest enabling that field when someone uses it and is confused about why it doesn’t exist?

because there is no end to the stuff that we could classify as “unnecessary” and put behind options.

…

this might be slippery slope fallacy

i’m instead pointing out that you’re saying it’s black and white and that “unnecessary” features should be behind flags, regardless of their relative cost. i’m suggesting instead that it is a spectrum: each feature has a cost to weigh

it’s not enabled in example-python

Because example-python uses a single constraint for the whole repo. It used to show Py2 vs Py3 partitioning, but we decided it was too niche of a feature and was distracting w/ how much we already have in the repo

and i don’t know why new users would enable it…

Because they read https://www.pantsbuild.org/docs/python-interpreter-compatibility, saw the section "Using multiple Python versions in the same project" and decided it's relevant to them, so followed the instructions to activate the backend That flow feels pretty similar to how one decides to activate Protobuf, a particular linter, etc. It sounds like those backends feel different than this functionality though. I'm trying to understand your thinking - what makes mixed ICs (e.g. partitioning) seem more core than these other things?

it’s not required for usage currently.

Right, but my points are a) most people don't use or need it, and b) this is why we have the deprecation

but you are paying the cost of an administrator needing to know the trick to actually enable that field, and for new users to have to discover in order to gain one of the benefits of pants.

I don't totally follow that: this is how all of Pants v2 is. Everything is opt-in minus project introspection and

count-loc

(Ack that we should continue to improve discovery of backends, like a

./pants help backends

, for sure)

i’m instead pointing out that you’re saying it’s black and white and that “unnecessary” features should be behind flags

Ah, to try to clarify: by "unnecessary" I am strictly talking about the performance impact being unnecessary, regardless of how much we optimize. Ik I brought up downsides of

./pants help

being cluttered: I do think that's true and valuable, but yeah, probably that alone is not sufficient to justify the change We have a lot of features that folks might not use thanks to sensible defaults, like pex's

zip_safe

field. Of course, I agree that should not be feature gated. But also evaluating that field is so trivial performance wise it's irrelevant In contrast, mixed ICs will always have some fundamental perf cost. I mostly buy your idea about how we can make partitioning a lot faster by only looking at roots. (I have questions on how to perform the validation statically and that perf hit, but can punt.) Perhaps it optimizes to the point that the perf hit is close to negligible Even if we solve partitioning enough, there still is a fundamental perf hit to determining the ICs for tool lockfiles like Flake8. Rather than looking at as single option, we must consult the entire repo to look at ICs. That's unnecessary invalidation and scanning of the repo, done every time you use Flake8 because of the upcoming lockfile invalidation