https://pantsbuild.org/ logo
Join Slack
Powered by
Where did ARM-on-linux support land on the priorit...
# development
f
Where did ARM-on-linux support land on the priority scale? And how can I help with that? Is it mostly just a matter of getting a builder instance in CI that can run the build?
p
I don't think GHA supports it natively, so it would require having our own runners (self hosted runnings). Just out of curiosity... what is the benefit of having pants support it? what is the use case for CI workloads on this architecture?
f
To build ARM manylinux wheels
I suppose it might be possible to cross-compile, but I've never tried cross-compiling either Rust or python wheels
p
ah... makes sense. thanks for clarifying!
f
We're moving to use ARM on linux more agressively in our org, and not having ARM support will become a blocker for us at some point
I'm trying to get ahead of the curve and I'm willing to contribute in this area if someone can point me in the right direction
h
Thanks @flat-zoo-31952 ! Yeah I think self-hosted runner on a machine of that arch would be the way to go.
Same problem as MacOS on M1, but GHA doesn't support self-hosted runners for those, and it does for linux on ARM
p
f
you're publishing wheels for macos arm though: https://pypi.org/project/pantsbuild.pants/#files
p
🤷‍♂️
h
Wheels for macOS ARM are because I have an M1 so am publishing from my laptop. Not very sustainable, but we're trying to be pragmatic https://blog.pantsbuild.org/how-we-added-apple-silicon-support-to-pants/
👀 1
🙈 1
😅 1
you can see the survery results here: https://docs.google.com/forms/d/e/1FAIpQLSd5HBUDNKQCxmPORcdwAOYoU9fHjrmRCKyM1R3JZvtig0Y5HQ/viewanalytics. The entry is called "First-class support for Linux ARM" I personally am very pro-prioritizing ARM support because it unblocks people from using Pants. Vs making Pants better for people already using it. Unblocking is more important in my book
f
hmmm, i've been debating moving to an ARM machine myself
if you're willing to accept hacky unsustainable results I'm sure we could cobble together a VM setup that would do this on a dev machine
p
Toolchain can probably sponsor an ec2 instance running arm (aka https://aws.amazon.com/ec2/graviton/) with a github runner on it. Happy to help with this if this is important... @happy-kitchen-89482
publishing releases from people personal laptops is probably a bad idea long term... I like how pex does it, from a GHA workflow.
➕ 1
f
i briefly saw some solutions to do this via qemu on a standard linux gha runner, but yeah an ec2 instance would be more straightforward, especially if you could make it on-demand or get away with spot instance pricing
p
I think there is a GH Action that can provision ec2 instances so we only spin up a runner when we need one. so in terms of $$ this is very cheap.
it would cost more (in terms of eng time) to set this up and and maintain it. so that is the actual concern here not the cost that would be payed to AWS.
f
I can look into this. My concern however is that at some point I'll need someone to put a cred in that gives the actions access to an AWS account that someone is paying for.
Is there a GH issue or set of issues about this?
p
I don't think there an existing issue about this.
h
p
ah.. I was looking for 
arm64
in title...
➕ 1
If all it takes on to toolchain side is to provide the AWS creds for an IAM account with the needed IAM permissions then this makes it a no-brainer and it would probably take me less then an hour to set up in an isolated and secure manner (i.e. setting up a separate AWS organization under toolchain and setting up the IAM user). @happy-kitchen-89482
h
I was looking for arm64 in title
good point, I updated the title
p
or we can use the existing pants AWS account for that (I know there is one... because of the usage of s3, but I don't have access to it)
f
Yeah just an access token for a very locked down IAM role would be perfect
I may however try playing around with the QEMU stuff, since based on that issue thread, it looks like there can be bumps to getting it to build from time to time, and it would be nice to be able to debug that locally
p
t4g.medium (https://aws.amazon.com/ec2/instance-types/t4/) will cost about 25$/month (https://aws.amazon.com/ec2/pricing/on-demand ) to run for an entire month which in terms of cost, I consider free (compared to how much an engineer time is worth/costs). so given that the eng work is done to use that. the AWS cost is not a concern.
➕ 1
💯 1
f
Oh that really isn't that bad at all... and if it stays running then the dev cycle would be short enough to not need to worry about having it locally
h
yeah pantsbuild org should be able to afford that fwict, esp since GHA is free to us whereas we were paying a lot of money for Travis
f
Cool, I can get started on this soonish, like maybe this weekend
h
Great!
I can get the instance set up on the pantsbuild AWS org
p
I think it is better to have one created on demand via this action: https://github.com/machulav/ec2-github-runner since we will probably want it only for the nightly build (at least initially) and also, not having an instance around running all the time is probably better from a security stance.
so what needs to be provisioned is an IAM role/user with limited set of permissions to provision this instance (there is a way to also limit the instance type that an IAM user can provision, so. it can't provision other instance types)
f
I think I might need permission to access the instance and build an AMI as well, unless y'all already know what needs to go in the AMI
p
I don't. but you can probably use packer to create an AMI, then we can add that packer file to the pants repo and create the AMI using that. https://www.packer.io/ I also found this: https://github.com/jonico/github-runner-1
👀 1
f
I’d be very to help with this! Either accessing the cloud machine and testing/hacking or running local experiments on demand. I have NVIDIA Jetson arm64 machine at my disposal 🙂 running Pants on this would also become relevant at some point, so it’s great to sort out the Pants distribution for that moment 🙂
✔️ 1
f
Looks like I'll have some time freed up to look at this weekend to look at this again. I'll be trying with the ec2-github-runner action, plus using Packer to create an AMI. Has the IAM role been created yet? I probably have enough work to do before that but I'll run into a wall there eventually if I can't create an instance.
h
Nope, I think we should do this in the pantsbuild AWS account though, not the toolchain one?
👍 1
@polite-garden-50641 thoughts on that choice?
p
I agree this is should be done with the pants aws account. if we were to do it on the toolchain account, I would have setup a different aws org and vpc for this. so it is less work to do it on the pants account (when initially talking about this, I didn't remember that pants had an aws account)
h
OK, so I can give you some permissions there?
p
Sure… I can help with setting up the iam role and get things going
I am working on some in my fork since I have more flexibility there to push to a branch, access repo secrets, etc...I made some initial AWS config needed by this action and I am iterating on this.
🙌 1
👀 1
f
Is that AMI you're referencsing something you've built directly on the console or did you have a packer recipe for tha?
p
I didn't use packer, I created an instance installed stuff based on the GH instructions:
then stopped the instance and create an AMI from it.
I think it is safe to punt on the packer stuff until we get stuff working in terms of the GH workflow itself.
the packer stuff is something that should and could be done later after the thing is actually working.
👍🏻 1
I have the AWS ec2 runner working as part of the GH workflow and I can run code on that machine from the workflow: see: https://github.com/asherf/pants/runs/5576190005?check_suite_focus=true#step:2:6 https://github.com/asherf/pants/blob/jerry/.github/workflows/ec2-test.yaml#L43
so the next step is figuring out what code/script we want to run on this runner... this is up to @flat-zoo-31952 @hundreds-father-404 you can open a PR on my fork and I can merge it into this branch or just share the code via a gist and I can get it there, totally up to you.
❤️ 1
h
Josh do you know where to go from here with the 
_release_helper.py
logic? This is probably stale, but https://github.com/pantsbuild/pants/issues/12183#issuecomment-857844133
f
I can look later, should have some time in a day or two
❤️ 1
took a brief look there today... gonna have to come back in a bit when I have the space to load this all in my head; doubt I'll get to this soon now 😩
😞 1
h
Would you be able to summarize the work so far so someone else can pick up where you leave off?
f
I didn't accomplish anything noteworthy here, didn't even really get a chance to load 
_release_helper
into my head
h
No worries
f
Yeah I thought I would be able to get to this, but between work and an impending baby, there's too much going on
Also I'm surprised at how much custom stuff there is in 
build-support
... guess that's always going to be custom depending on the requirements of a project and where it needs to be delivered to
h
Yeah, that's unfortunate. Part of it is that it's been around a long time, since Pants was less capable. We would like to fold more of that sort of thing into core Pants, for sure.
f
We'll see how much time I have to look at this later, I'm just concerned that for the next several weeks I won't be doing much in the way of opensource work or even getting much sleep 😂
h
Definitely - and congrats on the baby to be!
🙏🏻 1
f
took a bit of a look again at this; I think I'll try to keep poking at it. I'm just curious how I'm going to test this? Any tips there?
p
@abundant-finland-56154 as you can see I already have the logic needed the provision and run stuff on arm. the follow up step is to put actual logic into the GH workflow to build the pants wheels for arm. https://github.com/pantsbuild/pants/issues/12183#issuecomment-1069606467
a
(Congrats on the baby, Josh 👶 🎉 ) Asher, any way we can help? As mentioned elsewhere, it would simplify things for us.
p
@hundreds-father-404 can probably provide context here on which scripts needs to be modified. You simply need to look at the logic used to build wheels and add that kind of logic to the the temp workflow file I created, to see if it works on arm.
and there you will need to tweak the various scripts that build step uses in order to make them work on arm.
a
@steep-waitress-53641 cc'ing Thomas on this, as he has more context on our end.
s
@abundant-finland-56154 i can also take a look
a
👍
s
Yep that was the thread I was following. @polite-garden-50641 when you need alpha/beta testers let us know and we can take it for a spin
Also funnily enough pantsv1 was actually building arm instances at one point.
a
Yeah, I'm not sure if there's anything we can do in the meantime. We could run the fork in GHA, maybe? I'm not sure what the best way for us to help here is.
f
Has there been any progress on this? I might be able to dedicate some time to this in the coming weeks, and I want to get up-to-date on where it's at
h
I think @numerous-waitress-55735 has been looking at it some?
n
I was. Got arm from oracle cloud finally, butlast weekend was hectic. Will try on Sunday
f
in case that’s helpful to anyone — https://github.com/pantsbuild/pants/pull/15536 am able to build Pants on an Nvidia ARM machine and run tests in a custom repo (running Pants from sources)
🙌🏻 1
🙌 3
💯 1
FWIW I am able to produce wheels on the machine with ARM. Updated the PR body as well. Building a wheel:
Copy code
$ USE_PY38=true build-support/bin/release.sh build-wheels
Installing a wheel:
Copy code
$ python3.8 -m venv pantsenv
$ source pantsenv/bin/activate
$ pip install wheel pip --upgrade
$ pip install pants/dist/pantsbuild.pants-2.13.0.dev2+gitc0386feb-cp38-cp38-linux_aarch64.whl
$ which pants
/home/user/directory/pantsenv/bin/pants

$ export PANTS_LOCAL_STORE_FILES_MAX_SIZE_BYTES=25000000000
$ pants version
17:08:41.68 [INFO] Initializing scheduler...
17:08:42.09 [INFO] Scheduler initialized.
2.13.0.dev2+gitc0386feb

$ cd my-project
$ pants filter --target-type=pex_binary ::
# getting list of pex_binary in a custom repository
🎉 1
f
awesome!
f
I filed a ticket https://github.com/pantsbuild/pants/issues/15569 — we need to figure out what kind of settings people need to set (or we need to set when running Pants projects tests) on ARM devices (we can start with mine 😛 ). There is something peculiar about RAM management on ARM.
👍 1
back to ARM wheels. What do you folks think would be reasonable approach to publish ARM wheels of Pants? Will we just call 
build-support/bin/release.sh build-wheels
in a GH Action step e.g. https://github.com/asherf/pants/blob/jerry/.github/workflows/ec2-test.yaml (perhaps in an EC2 instance) and upload the wheel to PyPI? Or do we want to have tests (or a portion of them) passing on an ARM machine before building and publishing the wheel?
👍 1
n
Updating that my life is a bit hectic right now and I don't know when it will settle down, so I can't commit to working on it, as much as I want to :(
💜 2
h
@fresh-cat-90827 we have some tests marked 
@platform_specific_behavior
. We want to run those if possible on Linux ARM. They check things like "Can we actually install Protoc?". See how we do CI for macOS in 
generate_github_workflows.py
But to save time & resources, we generally don't run the full test suite on other platforms. We trust that things like 
strutil.py
are going to work the same
👀 1
f
okay, this is great @hundreds-father-404. The reason I’ve asked is that I continue on the ARM support on Linux and filed issue https://github.com/pantsbuild/pants/issues/15569 — I’d love to run those tests you mention, but am drowning in test failures caused by cache/memory related issue 😞
❤️ 1
👀 1
Open in Slack
PreviousNext
Powered by