I wasn't following along, but it appears that between 2.11.0.dev3 and 2.11.0rc0 something changed regarding the 3rdparty wheels we store in s3. Before that change we have, e.g.,

$ aws s3 ls <s3://binaries.pantsbuild.org/wheels/3rdparty/db4e861e4613aab9b0b2cb77a6cdd6496e61dca6/2.11.0.dev3+gitdb4e861e/> | grep psutil
2022-03-04 16:51:12     238334 psutil-5.9.0-cp37-cp37m-macosx_10_9_x86_64.whl
2022-03-04 16:39:56     280736 psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2022-03-04 16:51:12     238620 psutil-5.9.0-cp38-cp38-macosx_10_9_x86_64.whl
2022-03-04 16:39:56     283823 psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2022-03-04 16:51:12     238613 psutil-5.9.0-cp39-cp39-macosx_10_9_x86_64.whl
2022-03-04 16:39:56     280415 psutil-5.9.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl

and after:

$ aws s3 ls <s3://binaries.pantsbuild.org/wheels/3rdparty/15da8db4f25d54b30d50ca4e00c066ae642a099c/2.11.0rc0+git15da8db4/> | grep psutil
2022-03-17 15:46:26     238334 psutil-5.9.0-cp37-cp37m-macosx_10_9_x86_64.whl
2022-03-17 15:19:43     280736 psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl

And this has broken the pants release process, when it attempts to build a release pex from the wheels stored in s3

(I am encountering this with 2.12.0.dev0 but I notice that 2.11.0rc0 has no release pex, so @hundreds-father-404 perhaps you encountered this too when you did that release?)

The pex command that fails (presumably due to missing wheels for python 3.8 and 3.9) is

pex -o dist/pants.2.12.0.dev0+git2229d781.pex --no-build --no-pypi --disable-cache -f /Users/benjyw/src/pants/dist/deploy/wheels/pantsbuild.pants/2229d781a02ca2e3891c3595eee22c11d5c37113/2.12.0.dev0+git2229d781 -f /Users/benjyw/src/pants/dist/deploy/wheels/3rdparty/2229d781a02ca2e3891c3595eee22c11d5c37113/2.12.0.dev0+git2229d781 --no-strip-pex-env --console-script=pants --python-shebang "/usr/bin/env python" --interpreter-constraint 'CPython>=3.7,<3.10' --platform=linux_x86_64-cp-37-m --platform=linux_x86_64-cp-38-cp38 --platform=linux_x86_64-cp-39-cp39 --platform=macosx_10.15_x86_64-cp-37-m --platform=macosx_10.15_x86_64-cp-38-cp38 --platform=macosx_10.15_x86_64-cp-39-cp39 "pantsbuild.pants==2.12.0.dev0+git2229d781" --venv

Where the release script has downloaded those wheels to

/Users/benjyw/src/pants/dist/deploy/wheels

I should note that I still get failures when running that command manually with just

--platform=linux_x86_64-cp-37-m --platform=macosx_10.15_x86_64-cp-37-m

Is this a known thing? How much digging should I do?

Is it perhaps time to simplify this, deploy regularly-built pexes without the S3 intermediation and trust the lockfiles?

Also, surprised that it doesn't work since we use prebuilt wheels and set the platforms explicitly. Where does it fail?