https://pantsbuild.org/ logo
Join Slack
Powered by
It looks like our Linux ARM64 runners aren't picki...
# development
b
It looks like our Linux ARM64 runners aren't picking up jobs at the moment, e.g. release workflow waiting for 2 hours: https://github.com/pantsbuild/pants/actions/runs/14071173349/job/39405495803 @happy-kitchen-89482 I think you have the best access to our RunsOn stack at the moment, is this something you can look at?
Alternatively, grant me read access to the relevant info (I think it'll require some CFN and some CloudWatch read perms):
Copy code
$ AWS_PROFILE=pants roc logs '<https://github.com/pantsbuild/pants/actions/runs/14071173349/job/39405495803>'
...
failed to describe stack: operation error CloudFormation: DescribeStacks, https response error StatusCode: 403, RequestID: 827da8b0-20b1-40b0-8598-96a2619a5811, api error AccessDenied: User: arn:aws:iam::4...5:user/huonwilson is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-east-1:4...5:stack/runs-on/9e1...a71 because no identity-based policy allows the cloudformation:DescribeStacks action
h
Runner version v2.321.0 is deprecated and cannot receive messages.
Sigh
This happens from time to time. I will manually rebuild the AMI via https://github.com/pantsbuild/pants/blob/main/build-support/packer/runson/runson.pkr.hcl
But I guess we should automate this
b
Thanks! Thinking sideways: would it be possible to get rid of the custom AMI entirely? Potentially at the cost of spending more time doing init during the build, but maybe (much) easier maintenance than more automation.
h
It’s a lot of init. The AMI takes ~7 minutes to build
But I suspect it’s pretty straightforward to rebuild the AMI once a month in an action
b
Oh, 7 minutes... seems like a surprisingly long time to install 7 versions of Python. Maybe there's overhead there that wouldn't apply we were installing just for the current job rather than building an AMI? (For reference, it seems setting up pythons for the mac runners takes ~15s per version that needs to be downloaded e.g. https://github.com/pantsbuild/pants/actions/runs/14071328460/job/39406029677#step:3:22 installs 3.9 and 3.10 in 32s, with 3.11, 3.12, 3.13 cached)
That said, if we can get sufficiently automated AMI rebuild + PR (cherry picked to all branches?) working, that seems fine too.
h
There’s no need for cherrypicking, fortunately. It always takes the config from 
main
.
I’ll look into what exactly is happening during those 7 minutes…
b
Ah cool. Just confirming that it takes the config from 
main
even when doing a 
push
-triggered build, instead of 
pull_request
one?
h
I believe so, will double check
👍 1
2 Views
Open in Slack
PreviousNext
Powered by