FYI since mid-Jan our AWS bill has been about $60/...
# development
h
FYI since mid-Jan our AWS bill has been about $60/month higher than the previous steady state. Almost all of this is attributed to
EBS:VolumeP-Throughput.gp3
. Anyone have any thoughts on what changed in mid-Jan? Note that this isn’t the switch to RunsOn, which happened last August, but it may be due to some RunsOn-related change.
It shot up on precisely Jan 14
I did create a new RunsOn AMI on Jan 13…
But the AMI doesn’t itself provision EBS volume capacity
f
Are we paying for some sort of iops guaranteed throughput?
maybe that setting changed somewhere?
h
RunsOn requests instances with some default size and EBS provisioning. Maybe they changed that provisioning (although I see nothing on their blog or release notes)
We can override that however
b
Acknowledging your comment about release notes... but do you remember if we did a RunsOn upgrade around that time?
h
We pushed a new AMI (see above) but I’m not sure what you’re referring to by “RunsOn upgrade”?
b
I assume there's a CloudFormation stack that was deployed from a particular versioned RunsOn template, e.g.
<https://runs-on.s3.eu-west-1.amazonaws.com/cloudformation/template-v2.7.1.yaml>
. I believe one installs new versions of the RunsOn orchestrator by updating that stack with a new template. I think the CloudFormation console will show a timeline of events that'll indicate whether changes were made to the stack around this time period (although I don't think they'll indicate the before/after versions if there was an update).
Ah, I can answer my own question from the public logs: we're using RunsOn v2.3.2 (from 2024-06-21) ( https://github.com/pantsbuild/pants/actions/runs/14230483226/job/39879955488#step:1:15 ) so seems unlikely there was an upgrade in mid-2025-01. Thanks for listening.
h
FWIW our AWS spend in April was down slightly from March (due to slightly lower S3 usage). Other than that the overall cost allocation per service was almost identical across Feb-Mar-Apr. So it looks like we’re in a new steady state rather than a runaway situation.
b
Thanks for following up