also a question for remote-caching/execution: for buildling/pushing such large images, we're currently planning to use github actions runners to handle this. How much pain will we have if we don't go with a remote-caching / remote execution solution upfront? Is caching-only (with the recommended

bazel-remote-cache

) sufficient to mitigate most of the pain?

We run

bazel-remote-cache

happily from github action runners. Caching on GitHub can be problematic depending on your workspace size, it tends to get rate limited fairly easily.

Thanks! I was thinking we go bazel-remote (self-hosted) and GH hosted-runners, are you saying we run the risk of the runners getting rate-limited unless we self-host runners?

No it's only if you cache on github that it is rate limited, if you host bazel-remote you are good

thanks for clarifying!

Yeah that sounds extremely big, but depends on what's in it of course.

This bazel-remote-cache was pretty straightforward to set up, but unfortunately the github actions runners are running out-of-disk space during

pants package

. Is my only option at this point to either a) self-host GH runners, or b) set up

BuildBarn/Buildfarm/BuildGrid

? Which of the 3 Build* options should I choose? This is pretty new for me and I'm not sure what pros/cons of each are.

Easiest is probably just to host your own github runners

Thanks John! Will give it a shot. I'm realizing that due to the packaging system, the images may be much larger than required since we chose base images which included cuda + torch etc but pants now wants to manage those itself in the pex. I'm going to try to choose a slimmer base image and if that doesn't work, will probably try self-hosted runners

I've been pretty happy with Namespaces.so & their Bazel cache. We use both their GHA runners and their Bazel cache. https://namespace.so/docs/cli/pants-cache-setup (have not tried it with files as big as you have tho)

i am trying to setup namespace.so and am running into

14:58:29.70 [INFO] waiting for pantsd to start...
14:58:30.11 [INFO] pantsd started
14:58:30.83 [INFO] Initializing scheduler...
Error: 0.88 [ERROR] Could not initialize store for cache: "Error making env for store at \"/home/runner/.cache/pants/lmdb_store/cache/2\": Cannot allocate memory"

@some-insurance-58590 did you have any problems during the setup? the remote cache was easy to setup (seems to):

- name: Configure access to Namespace
        run: |
          nsc pants cache setup --pants-toml=/tmp/pants.toml

      - name: Install Pants
        uses: pantsbuild/actions/init-pants@v8
        with:
          gha-cache-key: v1
          named-caches-hash: ${{ hashFiles('python-default.lock') }}
          pants-ci-config: "['pants.toml', '/tmp/pants.toml']"

Can't remember why, but we switched off init-pants when switching to namespace.so This is basically how we're setting up pants:

- uses: namespacelabs/nscloud-setup@v0

    - name: Set up cache
      uses: namespacelabs/nscloud-cache-action@v1
      with:
        path: |
          ~/.cache/nce
          ~/.cache/pants/named_caches/

    - name: Install Pants
      run: |
          bin/get-pants.sh

    - name: Setup remote Pants (Bazel) cache
      run: |
          nsc pants cache setup --pants-toml=$GITHUB_WORKSPACE/pants.cache.toml

    - name: Configure pants to also use pants.ci.toml
      run: |
          echo "PANTS_CONFIG_FILES=['$GITHUB_WORKSPACE/pants.toml','$GITHUB_WORKSPACE/pants.ci.toml','$GITHUB_WORKSPACE/pants.cache.toml']" >> $GITHUB_ENV

thank you! will try it out here!

bin/get-pants.sh

comes from here - https://www.pantsbuild.org/stable/docs/getting-started/installing-pants (following the advice on that page for checking it in to the repo)

was going to ask this, thanks!

I guess so - hasn't been a problem for us

I updated our workflow to use the script and now the problem just moved to our next Pants invocation. I was able to print the exception.

Error: 1.58 [ERROR] Could not initialize store for cache: "Error making env for store at \"/home/runner/.cache/pants/lmdb_store/cache/2\": Cannot allocate memory"
Traceback (most recent call last):
  File "/home/runner/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/bin/daemon_pants_runner.py", line 132, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/pantsd/pants_daemon_core.py", line 174, in prepare
    self._initialize(
  File "/home/runner/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/pantsd/pants_daemon_core.py", line 117, in _initialize
    raise e
  File "/home/runner/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/pantsd/pants_daemon_core.py", line 107, in _initialize
    self._scheduler = EngineInitializer.setup_graph(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/init/engine_initializer.py", line 198, in setup_graph
    return EngineInitializer.setup_graph_extended(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/init/engine_initializer.py", line 341, in setup_graph_extended
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/home/runner/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/engine/internals/scheduler.py", line 236, in __init__
    self._py_scheduler = native_engine.scheduler_create(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Could not initialize store for cache: "Error making env for store at \"/home/runner/.cache/pants/lmdb_store/cache/2\": Cannot allocate memory"

Use -ldebug for more logs. 
See <https://www.pantsbuild.org/2.26/docs/using-pants/troubleshooting-common-issues> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/community/getting-help>

Thank you for your help, Dan

np! Yea, that error I have no idea about

i got in contact with support from namespace and it seems to be exclusive to arm runners. i don't know if that's your case, probably not.

ahh good to know - thanks for the update yea, we're using intel runners