So wait, why don't we stop releasing to PyPI today (on

main

)? AFAICT, we could release the wheels as GitHub release artifacts, which is super easy to automate (especially since we're already using GHA 😉) As best I can tell, we should be able to install from a URL here which would be the wheel. All it takes is attaching the wheel to the release

How would plugin developers depend on Pants in their codebase?

The pants wheel can still come from the URL for plugins. I'm not sure I understand the issue. Can you lay it out?

Oh I see, still resolve the same way, just use github as a wheelhouse?

Plopping wheels on GH releases seems like a pretty good idea to me, although I have two questions/concerns: • I think it'd require constructing the appropriate platform-specific URL for which of the 4 wheels to install, or use

--extra-index-url

somehow? I assume that's predictable/computable from inside Python fairly easily, but it'd be good to confirm • I assume plugins can only depend on other PyPI packages, not URLs (git or direct wheel ones)? So a plugin published to PyPI must not specify a dependency on

pantsbuild.pants

... I suspect that's okay functionally (since it'll be pants managing installing it, into an appropriate venv), but may be surprising to plugin authors, and means there's not a 'obvious' way to specify that a particular plugin version is only compatible with particular version(s) of pants? Potential path forward? 1. Upload wheels to GH releases 2. Optional: Have a one off script that backfills wheels to GH releases for old releases 3. Adjust scie-pants to read from GH releases (either unconditionally, or fallback to PyPI, for too-old versions, depending on 2) 4. Communicate the deprecation/removal of installing from PyPI in some way

So for 1. Yes we'd upload all the wheels and either: • construct the URL ourselves (in rust in Scie-Pants) • Or if URL+platform marker works, just spell out all 4 urls with appropriate marker and let pip decide (I think thisd work)

For 2: why do you assume that? A URL is a valid pip requirement specifier, no?

PyPI doesn't have to accept all pip requirement specifiers. I know for sure that Rust's crates.io requires that all dependencies exist on crates.io, as an ecosystem assurance thing: reducing the risk of bitrot, when external URLs die (especially bad if some highly-depended-upon dep accidentally points to an external URL, kinda like left-pad). I don't know for sure for PyPI, but there's some hints it's similar, e.g. https://stackoverflow.com/a/54894359/1256624 and https://github.com/pypi/warehouse/blob/8d01d4bbe84d2235b2761ef5316a561e221cbc34/warehouse/forklift/legacy.py#L272-L275

Hmm, so bummer that

--find-links

pointing to the releases page for a particular release doesn't "just work" (seems GitHub doesn't serve that page up)

Yes definitely don't want to use a universal PEX for that

Hmm, so bummer that

--find-links

pointing to the releases page for a particular release doesn't "just work" (seems GitHub doesn't serve that page up)

We could, of course, host a little

index.html

redirector

I wondered if we could attach that as a release artifact too, e.g.

--find-links <https://github.com/pantsbuild/pants/releases/download/release_VERSION/wheel-links.html>

that contains the appropriate

*.whl

links... the answer is no: github serves that with

Content-Type: application/octet-stream

but

pip

doesn't like this:

WARNING: Skipping page https://github.com/huonw/pants/releases/download/testing-pip-find-wheels-release-artifact/wheel-links.html because the GET request got Content-Type: application/octet-stream. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html

(Experiment at https://github.com/huonw/pants/releases/tag/testing-pip-find-wheels-release-artifact) The content-type cannot be changed, AFAICT, so, if we're going the

--find-links

route, we'd have to upload the HTML file to S3, as you mention. This seems quite achievable, given pants already uploads to S3 as part of the release process? (Alternatively, the set of supported platforms is pretty narrow, so deciding on the platform tags is probably not actually that hard. If we go with a non-universal PEX, we'd still have to decide on a per-platform tagging convention for scie-pants and pants to sync on... presumably much easier than reconstructing the wheel tags, though.)

Oh that's hilarious

Soooo, turns out scie-pants already synthesises a file for use with

--find-links

, that actually points pip to

<http://binaries.pantsbuild.org|binaries.pantsbuild.org>

for all the third-party wheels, having queried a listing of wheels from S3 directly: https://github.com/pantsbuild/scie-pants/blob/622df25b71c054fc0a7c58613d596e7e02bfcb88/tools/src/scie_pants/pants_version.py#L46 One could imagine adapting that process to read a pre-formed file from GH releases instead of

<http://binaries.pantsbuild.org|binaries.pantsbuild.org>

One could imagine it 🤔