https://pantsbuild.org/ logo
#development
Title
# development
w

witty-crayon-22786

04/24/2020, 5:29 PM
@hundreds-father-404: so… i’m deep on this cpython crate stuff. am pretty close to the finish line, so not going to give up today. but beyond the linker error i’m looking at (working on reproing locally in docker), i just also noticed that the crate doesn’t use PEP-384 (although it has nascent support for it)
how big a deal is PEP-384 in this case? would not using it mean requiring Python 3.6 to use the wheels? and is that… the end of the world?
h

hundreds-father-404

04/24/2020, 5:34 PM
PEP 384 means that we only need to build one single pantsbuild.pants wheel that can be run with any Python interpreter >= 3.6+. Otherwise, we need to have a 3.6 wheel, 3.7 wheel, 3.8 wheel, etc. The main downside is that it’s harder to maintain for us. Now we need multiple wheel building shards. We also need to track new Python releases; Pants 1.28.0 would never work for Python 3.9 ever (which is maybe fine) -- This is all a tradeoff, of course. It would be great to keep ABI3, but if it blocks dropping this really complex CFFI code, we should consider no longer using a stable ABI
w

witty-crayon-22786

04/24/2020, 5:36 PM
it’s not insurmountable i don’t think… certainly i should be able to do it in a followup.
but in the meantime, i’m wondering what it would look like to publish only a 3.6 wheel… it will be rare to have exactly 3.6 available right now?
h

hundreds-father-404

04/24/2020, 5:37 PM
Be able to get ABI3 working in a followup?
w

witty-crayon-22786

04/24/2020, 5:37 PM
yea.
h

hundreds-father-404

04/24/2020, 5:38 PM
At a minimum, we would need to release 3.6, 3.7, and 3.8 wheels. NB that Toolchain uses 3.8 It would be good, but not necessary, to release a 3.9.a5 (whatever they’re on) wheel as well
w

witty-crayon-22786

04/24/2020, 5:39 PM
ew. alright.
h

hundreds-father-404

04/24/2020, 5:39 PM
We would also possibly need to rewrite
./pants
to handle the change, I think. Which is hard because that doesn’t get updated unless users manually curl it
w

witty-crayon-22786

04/24/2020, 5:40 PM
so i might need to dive in on the ABI3 thing. i’ll probably put the rest of the change down while that’s changing upstream.
h

hundreds-father-404

04/24/2020, 5:40 PM
If we want to stick with ABI3, then, yes, it would be great to get it fixed before this change, unfortunately. If we decide that the tradeoff isn’t worth it and we should give up on abi3, then we could land the change now.
w

witty-crayon-22786

04/24/2020, 5:56 PM
and concretely, not using abi3 would mean ~three times more python installs and wheels built on the wheel building shards?
blargh.
alright. i might need to drop this to the background then, and spend a small amount of time on PEP-384 upstream
h

hundreds-father-404

04/24/2020, 6:00 PM
Yes, and 4x when 3.9 is released in a few months. Although, I do really want us to drop 3.6 soon
w

witty-crayon-22786

04/24/2020, 6:48 PM
ooof. PEP-384-ifying this library was looking ok until i noticed that
PyTuple
is not available. and the
call
syntax in the library uses
PyTuple
.
damn.
so it would either be “publish 4x the wheels” or “switch our distribution model to rust embeds python”
h

hundreds-father-404

04/24/2020, 6:52 PM
“switch our distribution model to rust embeds python”
How trivial is this in terms of order-of-magnitude? Sounds pretty complex. The third option is do nothing, but I think that the win of replacing cffi >>> the cost of 4 wheels (well, 8, because macOS)
w

witty-crayon-22786

04/24/2020, 6:57 PM
i had hoped that switching to
cpython
would be a step toward “rust embeds python”, because pyoxidizer uses
cpython
👍 1
so the change to the cpython crate removed a roadblock there.
h

hundreds-father-404

04/24/2020, 6:58 PM
And it still can remove that roadblock. Building 8 wheels sucks, but it means we can unblock the eventual upgrade to PyOxidizer. And the pain of building 8 wheels is more motivation
w

witty-crayon-22786

04/24/2020, 6:58 PM
so… maybe it is worth an intermediate step where we build more wheels, if it gets us closer to embedding python…? but there are uncertainties around embedding i’d need to look at. in particular: confirming that plugin loading works.
👍 1
alright.
i’m going to dump all of this somewhere and put this down before eod.
h

hundreds-father-404

04/24/2020, 7:00 PM
Let’s say that we never end up using PyOxidizer. Do you think removing CFFI would still be worth the cost of having to build 8 wheels? It sounds like yes, given that it will make Rust much easier to maintain, unblock Pantsd work, and make for nicer error messages
w

witty-crayon-22786

04/24/2020, 7:11 PM
possibly. the performance benefit looks like 8-10%
and it’s smaller
and the errors are better
etc
h

hundreds-father-404

04/24/2020, 7:12 PM
Performance benefit of 8-10% sounds very very substantial. Is that no-op performance,
list ::
performance, etc?
w

witty-crayon-22786

04/24/2020, 7:12 PM
for
list ::
h

hundreds-father-404

04/24/2020, 7:15 PM
Awesome. Then, yes, CPython seems worth it to me, regardless of PyOxidizer or not. We can work around the wheels thing. For example, as soon as we drop Centos6, then we can now start building Py36, Py37, and Py38 all on the same shard; right now, we need a Centos6 shard and a Centos7 shard. At that point, the Rust compilation can be used across all 3 Python versions. I don’t think the wheel building is too expensive - it’s the Rust compilation. So, we could build Py36, Py37, and Py38 in the same job, and I think it would only add maybe 5-10 minutes? (This is why I’ve wanted to drop Centos6 for the past 1.5 years, hehe)
w

witty-crayon-22786

04/24/2020, 8:12 PM
ok. i’ll try to make background progress on this, and dump out an explanation of what i’m waiting for on the PR.
h

hundreds-father-404

04/24/2020, 8:14 PM
i’ll try to make background progress on this,
On what in particular? On still trying to get CPython crate to work with abi3, on investigating PyOxidizer, or on tweaking our infrastructure to build multiple wheels
w

witty-crayon-22786

04/24/2020, 8:19 PM
1) there is still one linking error for the linux rust tests shard, 2) would need to publish wheels, 3) want (if not need) to confirm that PyOxidizer is worth pursuing
added this to the PR.