https://pantsbuild.org/ logo
#development
Title
# development
h

happy-kitchen-89482

03/26/2022, 5:02 AM
But before that there was, e.g.,
Fetching <https://binaries.pantsbuild.org/wheels%2F3rdparty%2F2229d781a02ca2e3891c3595eee22c11d5c37113%2F2.12.0.dev0%2Bgit2229d781%2Ftypes_PyYAML-6.0.3-py3-none-any.whl>
I wasn't following along, but it appears that between 2.11.0.dev3 and 2.11.0rc0 something changed regarding the 3rdparty wheels we store in s3. Before that change we have, e.g.,
Copy code
$ aws s3 ls <s3://binaries.pantsbuild.org/wheels/3rdparty/db4e861e4613aab9b0b2cb77a6cdd6496e61dca6/2.11.0.dev3+gitdb4e861e/> | grep psutil
2022-03-04 16:51:12     238334 psutil-5.9.0-cp37-cp37m-macosx_10_9_x86_64.whl
2022-03-04 16:39:56     280736 psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2022-03-04 16:51:12     238620 psutil-5.9.0-cp38-cp38-macosx_10_9_x86_64.whl
2022-03-04 16:39:56     283823 psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2022-03-04 16:51:12     238613 psutil-5.9.0-cp39-cp39-macosx_10_9_x86_64.whl
2022-03-04 16:39:56     280415 psutil-5.9.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
and after:
Copy code
$ aws s3 ls <s3://binaries.pantsbuild.org/wheels/3rdparty/15da8db4f25d54b30d50ca4e00c066ae642a099c/2.11.0rc0+git15da8db4/> | grep psutil
2022-03-17 15:46:26     238334 psutil-5.9.0-cp37-cp37m-macosx_10_9_x86_64.whl
2022-03-17 15:19:43     280736 psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
And this has broken the pants release process, when it attempts to build a release pex from the wheels stored in s3
(I am encountering this with 2.12.0.dev0 but I notice that 2.11.0rc0 has no release pex, so @hundreds-father-404 perhaps you encountered this too when you did that release?)
The pex command that fails (presumably due to missing wheels for python 3.8 and 3.9) is
Copy code
pex -o dist/pants.2.12.0.dev0+git2229d781.pex --no-build --no-pypi --disable-cache -f /Users/benjyw/src/pants/dist/deploy/wheels/pantsbuild.pants/2229d781a02ca2e3891c3595eee22c11d5c37113/2.12.0.dev0+git2229d781 -f /Users/benjyw/src/pants/dist/deploy/wheels/3rdparty/2229d781a02ca2e3891c3595eee22c11d5c37113/2.12.0.dev0+git2229d781 --no-strip-pex-env --console-script=pants --python-shebang "/usr/bin/env python" --interpreter-constraint 'CPython>=3.7,<3.10' --platform=linux_x86_64-cp-37-m --platform=linux_x86_64-cp-38-cp38 --platform=linux_x86_64-cp-39-cp39 --platform=macosx_10.15_x86_64-cp-37-m --platform=macosx_10.15_x86_64-cp-38-cp38 --platform=macosx_10.15_x86_64-cp-39-cp39 "pantsbuild.pants==2.12.0.dev0+git2229d781" --venv
Where the release script has downloaded those wheels to
/Users/benjyw/src/pants/dist/deploy/wheels
I should note that I still get failures when running that command manually with just
--platform=linux_x86_64-cp-37-m --platform=macosx_10.15_x86_64-cp-37-m
Is this a known thing? How much digging should I do?
Is it perhaps time to simplify this, deploy regularly-built pexes without the S3 intermediation and trust the lockfiles?
h

hundreds-father-404

03/27/2022, 1:41 AM
I can't do release PEXes on my M1
h

happy-kitchen-89482

03/27/2022, 1:56 AM
So when you do releases do we just go without those pexes?
h

hundreds-father-404

03/28/2022, 2:33 PM
yep
h

happy-kitchen-89482

03/28/2022, 3:50 PM
That seems... off? Either we need those pexes or we don't?
Also, surprised that it doesn't work since we use prebuilt wheels and set the platforms explicitly. Where does it fail?
h

hundreds-father-404

03/28/2022, 5:58 PM
Definitely not ideal: I raised it a few weeks ago https://pantsbuild.slack.com/archives/G0107QFGGAW/p1644293018693599