@hundreds-father-404: so… i’m deep on this cpython crate stuff. am pretty close to the finish line, so not going to give up today. but beyond the linker error i’m looking at (working on reproing locally in docker), i just also noticed that the crate doesn’t use PEP-384 (although it has nascent support for it)

PEP 384 means that we only need to build one single pantsbuild.pants wheel that can be run with any Python interpreter >= 3.6+. Otherwise, we need to have a 3.6 wheel, 3.7 wheel, 3.8 wheel, etc. The main downside is that it’s harder to maintain for us. Now we need multiple wheel building shards. We also need to track new Python releases; Pants 1.28.0 would never work for Python 3.9 ever (which is maybe fine) -- This is all a tradeoff, of course. It would be great to keep ABI3, but if it blocks dropping this really complex CFFI code, we should consider no longer using a stable ABI

it’s not insurmountable i don’t think… certainly i should be able to do it in a followup.

Be able to get ABI3 working in a followup?

yea.

At a minimum, we would need to release 3.6, 3.7, and 3.8 wheels. NB that Toolchain uses 3.8 It would be good, but not necessary, to release a 3.9.a5 (whatever they’re on) wheel as well

ew. alright.

We would also possibly need to rewrite

./pants

to handle the change, I think. Which is hard because that doesn’t get updated unless users manually curl it

so i might need to dive in on the ABI3 thing. i’ll probably put the rest of the change down while that’s changing upstream.

If we want to stick with ABI3, then, yes, it would be great to get it fixed before this change, unfortunately. If we decide that the tradeoff isn’t worth it and we should give up on abi3, then we could land the change now.

and concretely, not using abi3 would mean ~three times more python installs and wheels built on the wheel building shards?

Yes, and 4x when 3.9 is released in a few months. Although, I do really want us to drop 3.6 soon

ooof. PEP-384-ifying this library was looking ok until i noticed that

PyTuple

is not available. and the

call

syntax in the library uses

PyTuple

.

“switch our distribution model to rust embeds python”

How trivial is this in terms of order-of-magnitude? Sounds pretty complex. The third option is do nothing, but I think that the win of replacing cffi >>> the cost of 4 wheels (well, 8, because macOS)

i had hoped that switching to

cpython

would be a step toward “rust embeds python”, because pyoxidizer uses

cpython

And it still can remove that roadblock. Building 8 wheels sucks, but it means we can unblock the eventual upgrade to PyOxidizer. And the pain of building 8 wheels is more motivation

so… maybe it is worth an intermediate step where we build more wheels, if it gets us closer to embedding python…? but there are uncertainties around embedding i’d need to look at. in particular: confirming that plugin loading works.

Let’s say that we never end up using PyOxidizer. Do you think removing CFFI would still be worth the cost of having to build 8 wheels? It sounds like yes, given that it will make Rust much easier to maintain, unblock Pantsd work, and make for nicer error messages

possibly. the performance benefit looks like 8-10%

Performance benefit of 8-10% sounds very very substantial. Is that no-op performance,

list ::

performance, etc?

for

list ::

Awesome. Then, yes, CPython seems worth it to me, regardless of PyOxidizer or not. We can work around the wheels thing. For example, as soon as we drop Centos6, then we can now start building Py36, Py37, and Py38 all on the same shard; right now, we need a Centos6 shard and a Centos7 shard. At that point, the Rust compilation can be used across all 3 Python versions. I don’t think the wheel building is too expensive - it’s the Rust compilation. So, we could build Py36, Py37, and Py38 in the same job, and I think it would only add maybe 5-10 minutes? (This is why I’ve wanted to drop Centos6 for the past 1.5 years, hehe)

ok. i’ll try to make background progress on this, and dump out an explanation of what i’m waiting for on the PR.

i’ll try to make background progress on this,

On what in particular? On still trying to get CPython crate to work with abi3, on investigating PyOxidizer, or on tweaking our infrastructure to build multiple wheels

1) there is still one linking error for the linux rust tests shard, 2) would need to publish wheels, 3) want (if not need) to confirm that PyOxidizer is worth pursuing