hundreds-father-40408/03/2021, 5:51 PM
This means that every dependency of a target must also be compatible with its interpreter constraints. Generally, you will want to be careful that your common python_library targets are compatible with multiple Python versions because they may be depended upon by other targets. Meanwhile, pex_binary and python_tests targets can have specific constraints because they are (conventionally) never dependencies for other targets.But we say "can have specific constraints", not "must". I'm not sure how we'd enforce it, and it would also add new boilerplate in some cases not necessary. Like, my above example already would end up being ==3.6 b/c the transitive dep, and now you have to explicitly say that are you envisioning this change because it avoids these perf hits of looking at transitive targets? or other motivation?
hundreds-father-40408/03/2021, 5:52 PM
are you envisioning this change because it avoids these perf hits of looking at transitive targets?Note that I think this discussion would only help w/ Pytest fwict? MyPy and Pylint could run purely over
targets, and we'd still need to look at transitive deps there
witty-crayon-2278608/03/2021, 6:07 PM
are you envisioning this change because it avoids these perf hits of looking at transitive targets? or other motivation?both that it makes your graph more accurate, and that it assists in this case, yea.
witty-crayon-2278608/03/2021, 6:08 PM
MyPy and Pylint could run purely overi don’t think so. i’ve been giving the example of libraries vs binaries, but this applies throughout the graph: even when a library depends on a library.targets, and we’d still need to look at transitive deps there
hundreds-father-40408/04/2021, 6:00 PM
command to convert to
? For now, nothing like IDEs understand PEP 665
witty-crayon-2278608/05/2021, 5:51 PM
hundreds-father-40408/05/2021, 6:09 PM
i’ve been giving the example of libraries vs binaries, but this applies throughout the graph: even when a library depends on a library.To be clear, that enhancement only works if we make a breaking change to how ICs work: they must include the ICs of all transitive deps, not just the source files themselves. Is that right? With binaries and tests, you mentioned that seems more reasonable to ask. I do think we'd need the same restriction on all
targets too, given how MyPy and Pylint operate on them
witty-crayon-2278608/05/2021, 6:21 PM
To be clear, that enhancement only works if we make a breaking change to how ICs work: they must include the ICs of all transitive deps, not just the source files themselves. Is that right?yea, i think that this requires a deprecation cycle. but it’s likely that the optimization could be applied even before the deprecation cycle has completed (because the check for whether the graph is valid is sufficient to either warn or enable the optimization).
With binaries and tests, you mentioned that seems more reasonable to ask. I do think we’d need the same restriction on allyes, but i think that it is a valid constraint in all cases. it makes the constraints match reality: if a target’s dependencies cannot be used with X, then it also cannot be used with X.targets too, given how MyPy and Pylint operate on them
hundreds-father-40408/05/2021, 6:31 PM
A key ingredient to a large-scale, incremental migration is maintaining a real-time status of your project. Ideally, this information would be:
- decentralized to change, so that teams can work independently
- readable through a centralized view, so that migration leaders can track progress
- aware of how dependencies impact compatibility
- used at runtime to avoid becoming stale
- as precise as a file, which lowers the activation energy needed to port codeI've found there's lots of value in empowering individual contributors/teams to make progress on their portion of the code w/o being blocked by others Concrete example: • your org has a common util that is still Py2 and is blocked upgrading • your team's code is currently Py2 only, but you'd love to update it • with the current setup, you can migrate your code to be Py2 OR Py3, and then Pants will see the dep and merge to be Py2 at runtime • with the proposed setup, saying your own code is Py2 or Py3 is a lie, b/c a dep is still Py2 only
you can migrate your codeOf course you could still migrate your code and not update the ICs, but then you lose the abilitity for Pants to do fine-grained tracking of your migration -- FYI that entire idea of "fine grained tracking" and the blog's wishlist comes from the approach I took for Foursquare, Pants, and parts of Twitter: I used a Google sheet for file level tracking I realized how much more powerful it would be for Pants to do that
witty-crayon-2278608/05/2021, 6:33 PM
I feel strongly from my experience w/ Py3 migrations that it’s imperative you are able to set your ICs only based on the code itself’s compatibility, rather than needing to consider all transitive deps—and then Pants will ensure the result is valid by merging the ICsi disagree. if you set a constraint based only on the code, then it’s a lie. the code cannot be used without its dependencies. it’s not useful to anyone to know what the constraints of the code would be if it didn’t have any deps.
witty-crayon-2278608/05/2021, 6:35 PM
witty-crayon-2278608/05/2021, 6:36 PM
hundreds-father-40408/05/2021, 6:36 PM
it’s not useful to anyone to know what the constraints of the code would be if it didn’t have any deps.Based on what I found doing migrations, I disagree here. There's value in knowing "what is this code itself compatible with", along with also "okay, now include deps: what is it compatible with?" Note that
will show you both that code's own constraints and the transitive constraints. See the CSV in https://blog.pantsbuild.org/python-3-migrations/
hundreds-father-40408/05/2021, 6:38 PM
that the deps actually be upgraded first…Nack, you can migrate Py2-only code to become Py2 or Py3, even if deps are still Py2 only You are correct that you cannot migrate to be Py3-only until all deps are at a minimum Py2 or Py3
witty-crayon-2278608/05/2021, 7:03 PM
hundreds-father-40408/05/2021, 7:06 PM
goal for migration planning I think there's huge benefit to keeping source constraints. Iiuc, if we change ICs to now alway be transitive constraints, we lose the notion of source constraints and are back in the territory of using a manually maintained Google Sheet like I had to do w/ the 3 migrations
witty-crayon-2278608/05/2021, 8:22 PM
witty-crayon-2278608/05/2021, 8:25 PM
witty-crayon-2278608/05/2021, 8:27 PM
witty-crayon-2278608/05/2021, 8:47 PM
I think that it would likely be incorrect to ever use the source constraints with a tool. An example: if you have a formatter or linter that checks for a Python 3 specific condition and you run it on code whose transitive constraints are actually only valid for Python 2, you will break that code.
Hm. Actually, I think that this applies regardless of source/transitive... the tool would need to be aware that the code was trying to be compatible with both interpreters, regardless.
hundreds-father-40408/05/2021, 8:51 PM
macro. It works, but breaks dep inference due to ambiguity In contrast, w/ tests—which do care about transitive deps—the whole transitive closure must be Py2 and Py3 compatible if you want to test w/ both interpreters
witty-crayon-2278608/05/2021, 9:23 PM
hundreds-father-40408/05/2021, 9:26 PM
witty-crayon-2278608/05/2021, 9:29 PM
witty-crayon-2278608/05/2021, 9:29 PM
witty-crayon-2278608/05/2021, 9:30 PM
hundreds-father-40408/05/2021, 9:36 PM
you’re trading a scan through the root targetsAnd validation of the transitive constraints, right? Even for Flake8 where it doesn't consider deps, we need to validate your ICs were not mucked up when considering deps, iiuc?
a backend that you need to enable to use multiple versions at onceI am not seeing the big deal of having to do this. We already had the backend, and if you're using mixed ICs you really should be activating it so that you have the
goal. I suspect the majority of (at least new) users already have it activated. If you don't already, it's a single line change done one time by a codebase admin There's also benefits beyond performance: you don't clutter
with a field that you don't want your teammates using because your repo should be using uniform ICs Ack that the deprecation cycle is awkward w/ having to have the
option, but that's temporary
hundreds-father-40408/05/2021, 9:36 PM
because there is no end to the stuff that we could classify as “unnecessary” and put behind options.this might be slippery slope fallacy
witty-crayon-2278608/05/2021, 9:43 PM
And validation of the transitive constraints, right? Even for Flake8 where it doesn’t consider deps, we need to validate your ICs were not mucked up when considering deps, iiuc?i think that it’s sufficient to only validate the portion of the graph that you actually expand. so pytest/mypy would validate the full graph that they were about to invoke, other tools would only validate the portion that they loaded. for example, after partitioning, mypy would compute transitive (which it needs to do to run), and then explode if it found something incompatible.
I am not seeing the big deal of having to do this. We already had the backend, and if you’re using mixed ICs you really should be activating it so that you have theit’s not enabled in example-python. and i don’t know why new users would enable it… it’s not required for usage currently.goal. I suspect the majority of (at least new) users already have it activated.
There’s also benefits beyond performance: you don’t clutterbut you are paying the cost of an administrator needing to know the trick to actually enable that field, and for new users to have to discover in order to gain one of the benefits of pants. as an example: would we need a helper in core to suggest enabling that field when someone uses it and is confused about why it doesn’t exist?with a field that you don’t want your teammates using because your repo should be using uniform ICs
because there is no end to the stuff that we could classify as “unnecessary” and put behind options.
this might be slippery slope fallacyi’m instead pointing out that you’re saying it’s black and white and that “unnecessary” features should be behind flags, regardless of their relative cost. i’m suggesting instead that it is a spectrum: each feature has a cost to weigh
hundreds-father-40408/05/2021, 10:02 PM
it’s not enabled in example-pythonBecause example-python uses a single constraint for the whole repo. It used to show Py2 vs Py3 partitioning, but we decided it was too niche of a feature and was distracting w/ how much we already have in the repo
and i don’t know why new users would enable it…Because they read https://www.pantsbuild.org/docs/python-interpreter-compatibility, saw the section "Using multiple Python versions in the same project" and decided it's relevant to them, so followed the instructions to activate the backend That flow feels pretty similar to how one decides to activate Protobuf, a particular linter, etc. It sounds like those backends feel different than this functionality though. I'm trying to understand your thinking - what makes mixed ICs (e.g. partitioning) seem more core than these other things?
it’s not required for usage currently.Right, but my points are a) most people don't use or need it, and b) this is why we have the deprecation
but you are paying the cost of an administrator needing to know the trick to actually enable that field, and for new users to have to discover in order to gain one of the benefits of pants.I don't totally follow that: this is how all of Pants v2 is. Everything is opt-in minus project introspection and
(Ack that we should continue to improve discovery of backends, like a
, for sure)
./pants help backends
i’m instead pointing out that you’re saying it’s black and white and that “unnecessary” features should be behind flagsAh, to try to clarify: by "unnecessary" I am strictly talking about the performance impact being unnecessary, regardless of how much we optimize. Ik I brought up downsides of
being cluttered: I do think that's true and valuable, but yeah, probably that alone is not sufficient to justify the change We have a lot of features that folks might not use thanks to sensible defaults, like pex's
field. Of course, I agree that should not be feature gated. But also evaluating that field is so trivial performance wise it's irrelevant In contrast, mixed ICs will always have some fundamental perf cost. I mostly buy your idea about how we can make partitioning a lot faster by only looking at roots. (I have questions on how to perform the validation statically and that perf hit, but can punt.) Perhaps it optimizes to the point that the perf hit is close to negligible Even if we solve partitioning enough, there still is a fundamental perf hit to determining the ICs for tool lockfiles like Flake8. Rather than looking at as single option, we must consult the entire repo to look at ICs. That's unnecessary invalidation and scanning of the repo, done every time you use Flake8 because of the upcoming lockfile invalidation