We're running into issues with premature cache eviction on Github due to its 10GB limit and the size of our repo. I'm trying to set up S3 as a cache instead and doing this:

- uses: tespkg/actions-cache@v1
  env:
    AWS_REGION: "us-east-1"
    AWS_ACCESS_KEY_ID: ${{ inputs.aws_access_key_id }}
    AWS_SECRET_ACCESS_KEY: ${{ inputs.aws_secret_access_key }}
  with:
    bucket: my-bucket
    use-fallback: false
    key: ${{ hashFiles('locks/*.lock.jsonc') }}
    # cache directories based on the docs here: <https://www.pantsbuild.org/prerelease/docs/using-pants/using-pants-in-ci#directories-to-cache>
    path: |
      ~/.cache/nce
      ~/.cache/pants/named_caches
      ~/.cache/pants/lmdb_store
# <https://github.com/pantsbuild/actions/tree/main/init-pants/>
- uses: pantsbuild/actions/init-pants@v10

There's a couple issues here: 1. The init-pants action is still going to use the GitHub cache to cache some things (thereby making this hard to test) 2. The tespkg/actions-cache wrote something to S3 bucket after an initial run of this. But on a subsequent run (where I changed a test file to trigger the test to run), it built the PEX from scratch and ran the test, but afterwards, didn't upload anything back to the bucket. This makes me think something about the paths I'm providing is wrong. Does anyone have any suggestions?

I'd suggest using remote caching with a backend like bazel-remote that is using S3 as the storage backend.

I was looking at remote caching as the next step for all of this. Does it require us to run a service on our own infrastructure? Or does everything run inside of the github action with S3 as the storage layer?

I am in the same boat, having a bit too much trouble with github actions. It looks like you would need to spin up an instance of

bazel-remote-cache

on your own infra (somewhere that your github runners can communicate to). You'll supply your own backing storage to it, since its just a frontend. (It can just use its local filesystem as the storage, but then it gets lost if you need to restart the container or something). You can do a trial run of this by simply running

bazel-remote-cache

locally using docker on your own system, and then setting your local dev checkout of your repo to have pants do "remote caching" where the url is

localhost:<whatever-port>

Would we run bazel-remote inside of our github action, or would I need to spin it up as a service on our own infra?

You would host bazel-remote on your infra but with a storage backend like S3

Looks like Bazel supports just using GCS directly without any additional infra. Does Pants support this as well?

From that link, it looks like it works because GCS supports the API bazel expects (ie. its NOT because of bazel doing something special because it is a google thing and recognizes GCS). Its probably reasonable to assume its using the standard REAPI . This is what pants also uses, so it might just work out of the box. Its probably something that hasn't been tested before, so YMMV, but I think you could give it a try if you've already got GCS setup for some other reason

Looks like Bazel supports just using GCS directly without any additional infra. Does Pants support this as well?

Yes pants support REAPI, so the bazel-remote backend support is unrelated to pants, and yes it support AWS, GCS etc out of the box

Got it. I understand that if bazel can do something thanks to REAPI, then Pants can do it too. I'm specifically curious about interacting directly with GCS from Pants and not setting up bazel-remote at all. Based on the docs page I linked earlier, it looked like Bazel needs a "google_credentials" flag passed to it. I didn't immediately see an equivalent option for this in the Pants docs. Would I just set up those credentials locally and pass in the relevant information as env vars to the Pants process? What you've mentioned about running everything on our org's infra for cost reasons totally makes sense and I think we will go that route in the long run. I'm just trying to quickly side step the issue of us blowing past the 10GB GitHub action cache limit on a daily basis

Pants doesn't support any caching directly with GCS that I know about other than via REAPI and what I've mentioned above

Updates on this: • Using a separate GitHub action step for caching things like the lmdb_store doesn't do any good because coming up with a re-usable, deterministic hash key is hard • As a prototype, I'm installing

bazel-remote

into the GitHub action runner. It's a pre-built binary, so it takes less than ten seconds to download the binary and start

bazel-remote

. We're using S3 as the backing data store currently • The cache hit experience is much improved now. However, for situations where we run a large number of tests, we end up spending a decent chunk of time waiting for

bazel-remote

to download the results from S3. • Long term, we're going to set up

bazel-remote

on our own infra as a long running service so it can cache things to disk instead of hitting S3 all of the time • Longer term, we're going to move the GitHub action runners into our own infra for a variety of reasons

@fierce-truck-19259 What are you using to pull this off? Seems like there's a few different options available

Host runners on the infra https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners Host bazel-remote on the same infra Use a backing store for bazel-remote that is also on that same infra