I was going to follow up today on the discussion two weeks ago about Pants being released with the last 3 released Python versions (3.8-3.10 for now). But now I'm not sure I should push that with @ancient-vegetable-10556's exciting Pyoxidizer progress?

A few points: 1. Even with PyO…r available as a distribution method, we’ll still need to specify which Python version that will run under the hood

Although! Now that Pants has multiple resolves and good support for conflicting interpreter constraints, I'm not sure we need to support multiple Python versions for plugin authors? For example, Toolchain uses Python 3.9 for all its code, but 3.7-3.9 for its Pants plugin. Works well What if we said a Pants plugin could only be Python 3.10, for example? (No impact on your own non-plugin code)

because: 2. Despite PyO…r progress, we still haven’t decided whether we’re going to abandon our Python distributions. Obviously we’ll be limited to versions that PyO…r supports, but I think there’ll still be a number of releases before we decide to abandon Python distributions

How does the new world look? "Use this standalone thing with Python baked in, or bring your own if you need it?"

That is exactly the question

Is the only reason to bring your own plugin authors?

The PyO…r world will give you a standalone binary (which the

./pants

script would fetch and keep updated for you). So we can either switch entirely to that; we can provide that as an option alongside the “use the Python Wheel” option we currently provide; or we can decide that PyO…r is a bad idea

So maybe put on pause this discussion to see what happens? I don't want to invest Pants's political capital + our time working on CI infrastructure if this will change soon™️

I think it’ll be useful to understand what our options are, yeah

we can provide that as an option alongside the “use the Python Wheel” option we currently provide

We may need the wheel for the sake of plugin authors, so that things like MyPy works. But, we can decide to only release for one Python version We have a lot of incentive for that -- we consume a lot of resources to release for 3 Python versions x 2.5 OSes. CI resources, PyPI, etc

If the decision is to move to a single Python for plugin developers, how would this ecosystem look re: in-repo plugins? Develop in one version, release in another?

Develop in one version, release in another?

What do you mean? Release your plugin, or release your own code? -- The key insight from https://pantsbuild.slack.com/archives/C0D7TNJHL/p1659539778291099?thread_ts=1659539697.973779&cid=C0D7TNJHL is that the Python version you use for plugin development can be 100% decoupled from what you use for your own code. Only now possible thanks to multiple resolves

I think I was mostly trying to understand this line, re: existing plugins:

What if we said a Pants plugin could only be Python 3.10

I'm not sure what the implications would be. Mainline repo plugins are one thing - but lets say I have 5 custom, private plugins, would I then be required to update those plugins ensure they work only on (e.g.) 3.10? Doesnt' affect me, because I only use the most recent 1-2 Pythons, so 🤷

Yeah, your custom plugins would need to use whatever Python version the corresponding Pants versions are using. But your non-plugin code can keep using whatever cc @polite-garden-50641

Okay, excellent, just confirming I understood that comment

As someone who writes lots of plugins, @wide-midnight-78598 what are you thoughts on us expecting you to use a particular Python version for plugins?

there is one other consideration here which was discussed in the context of shipping only a PEX (which is clearly not going to be the path given the pyoxidizer progress): plugin code also needs an actual distribution to develop against. https://github.com/pantsbuild/pants/pull/13234#issuecomment-942974822 as mentioned there though, those distributions could be in S3 or github fairly easily

As someone who writes lots of plugins, @wide-midnight-78598 what are you thoughts on us expecting you to use a particular Python version for plugins?

So, (un)fortunately, I literally don't care... ... so long as it's a recent-ish Python (3.9, 3.10, 3.11) - I don't want to be limited, but I keep my code on recent Pythons and my test coverage covers any potential migration concerns (and Python is pretty solid for compatibility). However, for companies with their own in-repo plugins, I don't know what the appetite is for upgrading or testing. Since this is dev tooling though and not production code, I think putting these kind of constraints on is perfectly reasonable. Production code is where these kinda changes/limitations can really hurt.

there is one other consideration here which was discussed in the context of shipping only a PEX (which is clearly not going to be the path given the pyoxidizer progress): plugin code also needs an actual distribution to develop against.

Ack, that's specifically what I'm discussing. I think likely we want to distribute a) one binary for normal users, and b) one wheel for plugin authors. Both those only support the newest stable Python version

Just to be clear though, none of this is retroactive, it would only happen for new Pants releases, correct?

Correct

Why a wheel for plugin authors? Did you say for mypy? Why not distribute a types package with pyi files?

So that they can do things like run Pytest, Pylint, and MyPy on the code. MyPy could theoretically just use stubs, but I think tests need the actual library

Ah right, need it as a library 🤔 makes sense with multiple resolves though! Supporting just one version trims our package size still, right?

Exactly, we only will need to release 3-4 wheels {linux,macos} x {x86,arm}, vs up to 12 wheels (x3 for 3 Py versions)