From <https pantsbuild slack com archives C046T6T9U p169399 Pants #development

(From <https://pantsbuild.slack.com/archives/C046T...

bitter-ability-32190

09/07/2023, 1:26 AM

(From https://pantsbuild.slack.com/archives/C046T6T9U/p1693997543934359) I think for us to completely be off our old S3-backed cheeseshop (of other peoples code), we likely want to backport the per-platform PEXs of Pants to every release (in v2 at least). This simplifies not only our distribution model (going backwards) but also

scie-pants

as well. Doing so would be simple enough, and I can (ab)use GitHub runners for this, like I did the other forms of backporting as well. Anyone opposed?

broad-processor-92400

09/07/2023, 1:39 AM

Not opposed. How does this work with

pants_requirements

and its find_links? Iirc that doesn’t use the pex?

bitter-ability-32190

09/07/2023, 1:42 AM

It'd operate in the same way as today. So, you'd get a set of dependencies which should be compatible with those in the PEX. https://github.com/pantsbuild/pants/issues/19619 is filed to make the wheel pin exact matches.

bitter-ability-32190

09/07/2023, 1:43 AM

Oh... you mean for thirdparty wheels? That didn't provide find links for the old S3 cheeseshop, so I think what used to work still does (until the S3 bucket is turned off, if ever).

👍 1

bitter-ability-32190

09/07/2023, 7:00 AM

(which really only tells me that people aren't doing much plugin development and typechecking/testing their plugins on the machines that fail for the wheel reasons...)

bitter-ability-32190

09/08/2023, 2:30 PM

I have a script ready to fire. We'l discuss it on Mondya then I'll pull the trigger

bitter-ability-32190

09/11/2023, 8:44 PM

OK as discussed in the meeting, I'm going to kick this off. For provenance, here' the Pex invocation:

Copy code

subprocess.run(
            [
                "pex",
                "--disable-cache",
                "--python-shebang",
                "/usr/bin/env python",
                "-o",
                pex_name,
                "-f",
                "<https://wheels.pantsbuild.org/simple>",
                "-f",
                "links.html",
                f"pantsbuild.pants=={version}",
                "--no-build",
                "--no-pypi",
                "--disable-cache",
                "--no-strip-pex-env",
                "--console-script=pants",
                "--venv",
                f"--platform={platform}-cp-{pyver[2:]}-{pyver}",
            ],
            check=True
        )

Note that

links.html

is massaged from

requests.get(f"<https://binaries.pantsbuild.org?prefix=wheels/3rdparty/{commit_sha[:8]}>").content

which is code loosely stolen from scie-pants. I've verified that all the way back to

2.0.0.dev0

the

console-script

is valid

❤️ 1

bitter-ability-32190

09/11/2023, 8:45 PM

The inflection point for Python 3.9 is Pants 2.5. E.g. All releases of 2.5.x and up are 3.9

bitter-ability-32190

09/11/2023, 9:10 PM

The deployment fairies have been released

bitter-ability-32190

09/12/2023, 2:01 AM

So it turns out we're missing 3rdparty wheels for some packages. Looks like Mac Arm wheels, in particular. Will look further tomorrow

bitter-ability-32190

09/12/2023, 8:44 PM

OK, almost done. Had to ask PyPI for a few binaries. Now we're only lacking for about 50 PEXs almost all apple. good split on x84_64 and aarch64

bitter-ability-32190

09/12/2023, 8:46 PM

It's really weird to me that we'd be missing certain wheels 😕

bitter-ability-32190

09/12/2023, 8:52 PM

Copy code

release_2.17.0.dev2 is missing ['pants.2.17.0.dev2-cp39-darwin_arm64.pex', 'pants.2.17.0.dev2-cp39-linux_aarch64.pex']
release_2.17.0.dev1 is missing ['pants.2.17.0.dev1-cp39-darwin_arm64.pex', 'pants.2.17.0.dev1-cp39-linux_aarch64.pex']
release_2.17.0.dev0 is missing ['pants.2.17.0.dev0-cp39-darwin_arm64.pex', 'pants.2.17.0.dev0-cp39-linux_aarch64.pex']
release_2.12.1rc0 is missing ['pants.2.12.1rc0-cp39-darwin_arm64.pex']
release_2.12.0 is missing ['pants.2.12.0-cp39-darwin_arm64.pex']
release_2.11.1 is missing ['pants.2.11.1-cp39-darwin_arm64.pex']
release_2.12.0rc3 is missing ['pants.2.12.0rc3-cp39-darwin_arm64.pex']
release_2.11.1rc3 is missing ['pants.2.11.1rc3-cp39-darwin_arm64.pex']
release_2.12.0rc2 is missing ['pants.2.12.0rc2-cp39-darwin_arm64.pex']
release_2.11.1rc2 is missing ['pants.2.11.1rc2-cp39-darwin_arm64.pex']
release_2.9.2 is missing ['pants.2.9.2-cp39-darwin_arm64.pex']
release_2.8.1 is missing ['pants.2.8.1-cp39-darwin_arm64.pex']
release_2.10.1 is missing ['pants.2.10.1-cp39-darwin_arm64.pex']
release_2.11.1rc0 is missing ['pants.2.11.1rc0-cp39-darwin_arm64.pex']
release_2.11.0 is missing ['pants.2.11.0-cp39-darwin_arm64.pex']
release_2.11.0rc6 is missing ['pants.2.11.0rc6-cp39-darwin_arm64.pex']
release_2.12.0a0 is missing ['pants.2.12.0a0-cp39-darwin_arm64.pex']
release_2.11.0rc3 is missing ['pants.2.11.0rc3-cp39-darwin_arm64.pex']
release_2.11.0rc2 is missing ['pants.2.11.0rc2-cp39-darwin_arm64.pex', 'pants.2.11.0rc2-cp39-darwin_x86_64.pex']
release_2.9.2rc0 is missing ['pants.2.9.2rc0-cp39-darwin_arm64.pex']
release_2.10.1rc0 is missing ['pants.2.10.1rc0-cp39-darwin_arm64.pex']
release_2.11.0rc1 is missing ['pants.2.11.0rc1-cp39-darwin_x86_64.pex']
release_2.12.0.dev0 is missing ['pants.2.12.0.dev0-cp39-darwin_x86_64.pex']
release_2.10.0 is missing ['pants.2.10.0-cp39-darwin_arm64.pex']
release_2.11.0rc0 is missing ['pants.2.11.0rc0-cp39-darwin_x86_64.pex']
release_2.10.0rc2 is missing ['pants.2.10.0rc2-cp39-darwin_arm64.pex']
release_2.9.1rc0 is missing ['pants.2.9.1rc0-cp39-darwin_arm64.pex']
release_2.9.0 is missing ['pants.2.9.0-cp39-darwin_arm64.pex']
release_2.9.0rc6 is missing ['pants.2.9.0rc6-cp39-darwin_arm64.pex']
release_2.9.0rc5 is missing ['pants.2.9.0rc5-cp39-darwin_arm64.pex']
release_2.9.0rc4 is missing ['pants.2.9.0rc4-cp39-darwin_arm64.pex']
release_2.9.0rc3 is missing ['pants.2.9.0rc3-cp39-darwin_arm64.pex']
release_2.9.0rc2 is missing ['pants.2.9.0rc2-cp39-darwin_arm64.pex']
release_2.9.0rc0 is missing ['pants.2.9.0rc0-cp39-darwin_arm64.pex']
release_2.8.1rc1 is missing ['pants.2.8.1rc1-cp39-darwin_arm64.pex']
release_2.8.1rc0 is missing ['pants.2.8.1rc0-cp39-darwin_arm64.pex']
release_2.8.0 is missing ['pants.2.8.0-cp39-darwin_arm64.pex']
release_2.8.0rc5 is missing ['pants.2.8.0rc5-cp39-darwin_arm64.pex']
release_2.7.2rc2 is missing ['pants.2.7.2rc2-cp39-darwin_arm64.pex']
release_2.8.0rc2 is missing ['pants.2.8.0rc2-cp39-darwin_arm64.pex']
release_2.7.2rc1 is missing ['pants.2.7.2rc1-cp39-darwin_arm64.pex']
release_2.7.1rc1 is missing ['pants.2.7.1rc1-cp39-darwin_arm64.pex']
release_2.7.1rc0 is missing ['pants.2.7.1rc0-cp39-darwin_arm64.pex']
release_2.7.0 is missing ['pants.2.7.0-cp39-darwin_arm64.pex']
release_2.7.0rc5 is missing ['pants.2.7.0rc5-cp39-darwin_arm64.pex']
release_2.6.1 is missing ['pants.2.6.1-cp39-darwin_arm64.pex']
release_2.7.0rc3 is missing ['pants.2.7.0rc3-cp39-darwin_arm64.pex']
release_2.7.0rc2 is missing ['pants.2.7.0rc2-cp39-darwin_arm64.pex']
release_2.7.0rc1 is missing ['pants.2.7.0rc1-cp39-darwin_arm64.pex']

bitter-ability-32190

09/12/2023, 8:53 PM

Unfortunately we can't just build these on GitHub, as the lowest version they provide is

osx_11_0

happy-kitchen-89482

09/13/2023, 4:57 AM

Missing pexes, not wheels, ya?

bitter-ability-32190

09/13/2023, 11:31 AM

It's missing PEXs because I can't build them due to missing wheels

bitter-ability-32190

09/13/2023, 11:32 AM

A common one is

ijson==3.1.4

somehow we have Mac wheels on arm for pants, but I've never seen that wheel in our S3 bucket

broad-processor-92400

09/13/2023, 12:11 PM

I cannot install, for instance, 2.7.0 on my arm mac using

scie-pants

(error while attempting to build a PyYAML wheel). I can install it using the old

./pants

script, though. I don't know if that's relevant.

bitter-ability-32190

09/13/2023, 1:13 PM

That's very relevant 🤔

bitter-ability-32190

09/13/2023, 1:13 PM

the

scie-pants

error would be understandable if you're not using the version with the find-links fix

bitter-ability-32190

09/13/2023, 1:15 PM

I can install it using the old
./pants
script, though.

Can we dive into this?

bitter-ability-32190

09/13/2023, 1:29 PM

OK from everything I'm seeing, it looks like the difference is likely just PythonBuildStandlone vs System Python. I suspect if you do

python3.9 -m pip install ijson==3.1.4

(in a venv) it'll succeed

bitter-ability-32190

09/13/2023, 1:31 PM

And @witty-crayon-22786 for posterity, I suspect that's why you thought we didn't use the 3rdparty wheels for normal Pants installation. We didn't used to: https://github.com/pantsbuild/setup/blob/gh-pages/pants But then, with

scie-pants

and PBS, we needed a way for people to get wheels of packages not on PyPI knowing they can no longer build them themselves.

witty-crayon-22786

09/13/2023, 9:44 PM

yea, makes sense.

bitter-ability-32190

09/14/2023, 4:46 PM

SO I still don't understand where these wheels came from. If you take

2.10.0

for instance. On PyPI the wheel was uploaded on the same date. If you look at the wheels workflow, there obviously isn't any ARM building: https://github.com/pantsbuild/pants/actions/runs/2210245521/workflow ...Does anyone remember how we built these? The docs don't help: https://www.pantsbuild.org/v2.10/docs/release-process

witty-crayon-22786

09/14/2023, 4:50 PM

i don’t fully understand which wheels you mean, but: back in the day, macOS ARM was built manually on macOS: https://blog.pantsbuild.org/how-we-added-apple-silicon-support-to-pants/

bitter-ability-32190

09/14/2023, 5:00 PM

Oh... that explains it. Whoever built these wheels naturally didn't upload the relevant 3rdparty wheels to the S3 bucket 😕

bitter-ability-32190

09/14/2023, 5:00 PM

Anyone got some Apple silicon with OS X 11 lying around?

bitter-ability-32190

09/14/2023, 5:10 PM

Oooh looks like http://binaries.pantsbuild.org/?prefix=wheels/3rdparty/852f420 has what we need!

curved-television-6568

09/14/2023, 5:20 PM

I think it was mostly Eric doing the ARM builds manually..

bitter-ability-32190

09/14/2023, 5:21 PM

Yeah, I found the first instance of us building arm on CI, and I'm going to hardcode those 3rdparty wheels in addition to the normal uploaded ones. Wish me luck!

🍀 1

bitter-ability-32190

09/14/2023, 5:31 PM

ITS WORKING! 😂

bitter-ability-32190

09/14/2023, 5:31 PM

New list is shorter:

Copy code

release_2.17.0.dev2 is missing ['pants.2.17.0.dev2-cp39-darwin_arm64.pex', 'pants.2.17.0.dev2-cp39-linux_aarch64.pex']
release_2.17.0.dev1 is missing ['pants.2.17.0.dev1-cp39-darwin_arm64.pex', 'pants.2.17.0.dev1-cp39-linux_aarch64.pex']
release_2.17.0.dev0 is missing ['pants.2.17.0.dev0-cp39-darwin_arm64.pex', 'pants.2.17.0.dev0-cp39-linux_aarch64.pex']
release_2.8.1 is missing ['pants.2.8.1-cp39-darwin_arm64.pex']
release_2.12.0.dev0 is missing ['pants.2.12.0.dev0-cp39-darwin_x86_64.pex']
release_2.9.0 is missing ['pants.2.9.0-cp39-darwin_arm64.pex']
release_2.9.0rc6 is missing ['pants.2.9.0rc6-cp39-darwin_arm64.pex']
release_2.9.0rc5 is missing ['pants.2.9.0rc5-cp39-darwin_arm64.pex']
release_2.9.0rc4 is missing ['pants.2.9.0rc4-cp39-darwin_arm64.pex']
release_2.9.0rc3 is missing ['pants.2.9.0rc3-cp39-darwin_arm64.pex']
release_2.9.0rc2 is missing ['pants.2.9.0rc2-cp39-darwin_arm64.pex']
release_2.9.0rc0 is missing ['pants.2.9.0rc0-cp39-darwin_arm64.pex']
release_2.8.1rc1 is missing ['pants.2.8.1rc1-cp39-darwin_arm64.pex']
release_2.8.1rc0 is missing ['pants.2.8.1rc0-cp39-darwin_arm64.pex']
release_2.8.0 is missing ['pants.2.8.0-cp39-darwin_arm64.pex']
release_2.8.0rc5 is missing ['pants.2.8.0rc5-cp39-darwin_arm64.pex']
release_2.7.2rc2 is missing ['pants.2.7.2rc2-cp39-darwin_arm64.pex']
release_2.8.0rc2 is missing ['pants.2.8.0rc2-cp39-darwin_arm64.pex']
release_2.7.2rc1 is missing ['pants.2.7.2rc1-cp39-darwin_arm64.pex']
release_2.7.1rc1 is missing ['pants.2.7.1rc1-cp39-darwin_arm64.pex']
release_2.7.1rc0 is missing ['pants.2.7.1rc0-cp39-darwin_arm64.pex']
release_2.7.0 is missing ['pants.2.7.0-cp39-darwin_arm64.pex']
release_2.7.0rc5 is missing ['pants.2.7.0rc5-cp39-darwin_arm64.pex']
release_2.6.1 is missing ['pants.2.6.1-cp39-darwin_arm64.pex']
release_2.7.0rc3 is missing ['pants.2.7.0rc3-cp39-darwin_arm64.pex']
release_2.7.0rc2 is missing ['pants.2.7.0rc2-cp39-darwin_arm64.pex']
release_2.7.0rc1 is missing ['pants.2.7.0rc1-cp39-darwin_arm64.pex']

bitter-ability-32190

09/14/2023, 5:32 PM

So I think we bumped deps at some point, so the earlier builds can't be patched with later uploaded wheels 😕

bitter-ability-32190

09/14/2023, 5:33 PM

So I still need some Apple silicone running 11.0 OS X

witty-crayon-22786

09/14/2023, 5:34 PM

Um, we do actually have GitHub actions workers for macOS arm now, I think?

witty-crayon-22786

09/14/2023, 5:35 PM

Benjy got that set up at some point to replace the manual process

bitter-ability-32190

09/14/2023, 5:35 PM

......Can I borrow it? 😛

bitter-ability-32190

09/14/2023, 5:36 PM

I need ot build some wheels 😛

witty-crayon-22786

09/14/2023, 5:36 PM

https://blog.pantsbuild.org/m1-support-self-hosted/

bitter-ability-32190

09/14/2023, 5:36 PM

lol you got a blogpost for everything lol

🤣 1

witty-crayon-22786

09/14/2023, 5:37 PM

https://github.com/pantsbuild/pants/settings/actions/runners

bitter-ability-32190

09/14/2023, 5:37 PM

I can run GItHub Actions from a branch. Sooooo brb 😉

bitter-ability-32190

09/14/2023, 8:58 PM

That did it!

🎉 2

bitter-ability-32190

09/14/2023, 9:00 PM

Just some stragglers left. Will attempt next week

bitter-ability-32190

09/14/2023, 9:36 PM

No missing PEXs 🙂

happy-kitchen-89482

09/15/2023, 10:24 AM

Yep, we have a macstadium M1 running OSX11

Open in Slack

Previous Next