https://pantsbuild.org/ logo
#development
Title
# development
h

hundreds-father-404

08/03/2022, 3:14 PM
I was going to follow up today on the discussion two weeks ago about Pants being released with the last 3 released Python versions (3.8-3.10 for now). But now I'm not sure I should push that with @ancient-vegetable-10556's exciting Pyoxidizer progress?
I think John pointed out the policy would still probably be relevant because plugin authors?
a

ancient-vegetable-10556

08/03/2022, 3:16 PM
A few points: 1. Even with PyO…r available as a distribution method, we’ll still need to specify which Python version that will run under the hood
h

hundreds-father-404

08/03/2022, 3:16 PM
Although! Now that Pants has multiple resolves and good support for conflicting interpreter constraints, I'm not sure we need to support multiple Python versions for plugin authors? For example, Toolchain uses Python 3.9 for all its code, but 3.7-3.9 for its Pants plugin. Works well What if we said a Pants plugin could only be Python 3.10, for example? (No impact on your own non-plugin code)
we’ll still need to specify which Python version that will run under the hood
Right, but why release under multiple Python versions? If we only release with Python 3.10, then we get to use all its new benefit
a

ancient-vegetable-10556

08/03/2022, 3:18 PM
because: 2. Despite PyO…r progress, we still haven’t decided whether we’re going to abandon our Python distributions. Obviously we’ll be limited to versions that PyO…r supports, but I think there’ll still be a number of releases before we decide to abandon Python distributions
That said, I think it’s a good idea to only support versions of Python which themselves are maintained by the Python release process, and that we should be dropping support for older versions well before they EOL in the Python process
(we shouldn’t lag Python, because our users will certainly lag us, and it’s poor form to actively give people a reason to stay on an version of Python)
👆 1
b

bitter-ability-32190

08/03/2022, 3:47 PM
How does the new world look? "Use this standalone thing with Python baked in, or bring your own if you need it?"
a

ancient-vegetable-10556

08/03/2022, 3:47 PM
That is exactly the question
b

bitter-ability-32190

08/03/2022, 3:48 PM
Is the only reason to bring your own plugin authors?
a

ancient-vegetable-10556

08/03/2022, 3:48 PM
The PyO…r world will give you a standalone binary (which the
./pants
script would fetch and keep updated for you). So we can either switch entirely to that; we can provide that as an option alongside the “use the Python Wheel” option we currently provide; or we can decide that PyO…r is a bad idea
h

hundreds-father-404

08/03/2022, 3:48 PM
So maybe put on pause this discussion to see what happens? I don't want to invest Pants's political capital + our time working on CI infrastructure if this will change soon™️
a

ancient-vegetable-10556

08/03/2022, 3:49 PM
I think it’ll be useful to understand what our options are, yeah
h

hundreds-father-404

08/03/2022, 3:50 PM
we can provide that as an option alongside the “use the Python Wheel” option we currently provide
We may need the wheel for the sake of plugin authors, so that things like MyPy works. But, we can decide to only release for one Python version We have a lot of incentive for that -- we consume a lot of resources to release for 3 Python versions x 2.5 OSes. CI resources, PyPI, etc
💰 1
w

wide-midnight-78598

08/03/2022, 3:56 PM
If the decision is to move to a single Python for plugin developers, how would this ecosystem look re: in-repo plugins? Develop in one version, release in another?
h

hundreds-father-404

08/03/2022, 3:58 PM
Develop in one version, release in another?
What do you mean? Release your plugin, or release your own code? -- The key insight from https://pantsbuild.slack.com/archives/C0D7TNJHL/p1659539778291099?thread_ts=1659539697.973779&cid=C0D7TNJHL is that the Python version you use for plugin development can be 100% decoupled from what you use for your own code. Only now possible thanks to multiple resolves
w

wide-midnight-78598

08/03/2022, 4:03 PM
I think I was mostly trying to understand this line, re: existing plugins:
What if we said a Pants plugin could only be Python 3.10
I'm not sure what the implications would be. Mainline repo plugins are one thing - but lets say I have 5 custom, private plugins, would I then be required to update those plugins ensure they work only on (e.g.) 3.10? Doesnt' affect me, because I only use the most recent 1-2 Pythons, so 🤷
h

hundreds-father-404

08/03/2022, 4:19 PM
Yeah, your custom plugins would need to use whatever Python version the corresponding Pants versions are using. But your non-plugin code can keep using whatever cc @polite-garden-50641
👍 1
w

wide-midnight-78598

08/03/2022, 4:52 PM
Okay, excellent, just confirming I understood that comment
h

hundreds-father-404

08/03/2022, 4:53 PM
As someone who writes lots of plugins, @wide-midnight-78598 what are you thoughts on us expecting you to use a particular Python version for plugins?
w

witty-crayon-22786

08/03/2022, 4:53 PM
there is one other consideration here which was discussed in the context of shipping only a PEX (which is clearly not going to be the path given the pyoxidizer progress): plugin code also needs an actual distribution to develop against. https://github.com/pantsbuild/pants/pull/13234#issuecomment-942974822 as mentioned there though, those distributions could be in S3 or github fairly easily
w

wide-midnight-78598

08/03/2022, 4:58 PM
As someone who writes lots of plugins, @wide-midnight-78598 what are you thoughts on us expecting you to use a particular Python version for plugins?
So, (un)fortunately, I literally don't care... ... so long as it's a recent-ish Python (3.9, 3.10, 3.11) - I don't want to be limited, but I keep my code on recent Pythons and my test coverage covers any potential migration concerns (and Python is pretty solid for compatibility). However, for companies with their own in-repo plugins, I don't know what the appetite is for upgrading or testing. Since this is dev tooling though and not production code, I think putting these kind of constraints on is perfectly reasonable. Production code is where these kinda changes/limitations can really hurt.
1
This could/would be one of those applications of an official docker image that could be used as a builder/local test runner to keep teams in sync though (or a VS Code Workspace).
h

hundreds-father-404

08/03/2022, 5:01 PM
there is one other consideration here which was discussed in the context of shipping only a PEX (which is clearly not going to be the path given the pyoxidizer progress): plugin code also needs an actual distribution to develop against.
Ack, that's specifically what I'm discussing. I think likely we want to distribute a) one binary for normal users, and b) one wheel for plugin authors. Both those only support the newest stable Python version
w

wide-midnight-78598

08/03/2022, 5:04 PM
Just to be clear though, none of this is retroactive, it would only happen for new Pants releases, correct?
h

hundreds-father-404

08/03/2022, 5:08 PM
Correct
1
b

bitter-ability-32190

08/03/2022, 6:18 PM
Why a wheel for plugin authors? Did you say for mypy? Why not distribute a types package with pyi files?
h

hundreds-father-404

08/03/2022, 7:36 PM
So that they can do things like run Pytest, Pylint, and MyPy on the code. MyPy could theoretically just use stubs, but I think tests need the actual library
b

bitter-ability-32190

08/03/2022, 7:40 PM
Ah right, need it as a library 🤔 makes sense with multiple resolves though! Supporting just one version trims our package size still, right?
👍 1
h

hundreds-father-404

08/03/2022, 7:42 PM
Exactly, we only will need to release 3-4 wheels {linux,macos} x {x86,arm}, vs up to 12 wheels (x3 for 3 Py versions)
2