We re having issues with our build stage in CI taking quite Pants #general

We're having issues with our build stage in CI tak...

ripe-architect-1001

04/09/2025, 1:49 PM

We're having issues with our build stage in CI taking quite a long time. Currently it is set up to run this:

Copy code

pants --filter-target-type='-docker_image' package ::

I'm looking into updating it to use

--changed-since

but it seems I cannot mix that with my filter. For more detail the build times really blew up after a pytorch was added to the project. We already have our pants setup set to use a cache and I've tried setting up a named cache for the other dependencies and it hasn't seemed to help anything. In the thread is the circleci build job that we have if anyone can point out improvements or what I might be doing wrong.

ripe-architect-1001

04/09/2025, 1:49 PM

Copy code

build_artifacts:
    parameters:
      base_cache_key:
        type: string
        default: v1
    resource_class: large
    docker:
      - image: cimg/python:3.12-node
    environment:
      DOCKER_BUILDKIT: 1
      PANTS_CONFIG_FILES: "pants.ci.toml"
    steps:
      - checkout
      - setup_credential_env_vars
      - run:
          name: Installing pants
          command: sudo ./get-pants.sh --bin-dir /usr/bin
      - run:
          name: Printing pants bootstrap cache key file
          command: PANTS_BOOTSTRAP_TOOLS=2 pants bootstrap-cache-key > pants-bootstrap-cache-key.txt
      - restore_cache:
          name: Restoring pants bootstrap cache
          key: pants-setup-<< parameters.base_cache_key >>-{{ checksum "pants-bootstrap-cache-key.txt" }}
      - restore_cache:
          name: Restoring pants named cache
          keys:
            - pants-named-cache-<< parameters.base_cache_key >>-{{ checksum "pants.toml" }}-{{ checksum "lockfiles/python.lock" }}-{{ .Branch }}
            - pants-named-cache-<< parameters.base_cache_key >>-{{ checksum "pants.toml" }}-{{ checksum "lockfiles/python.lock" }}
            - pants-named-cache-<< parameters.base_cache_key >>-{{ checksum "pants.toml" }}
            - pants-named-cache-<< parameters.base_cache_key >>
      - run:
          name: Setup incremental builds
          command: |
            git fetch origin prod
            MERGE_BASE=$(git merge-base HEAD origin/prod)
            echo "export PANTS_CHANGED_SINCE=$MERGE_BASE" >> $BASH_ENV
      - run:
          name: Build deployment artifacts
          command: |
            source $BASH_ENV
            pants --changed-since=$PANTS_CHANGED_SINCE --changed-dependents=transitive --filter-target-type='-docker_image' package ::
      - store_artifacts:
          path: dist/
          destination: deployment-packages
      - store_artifacts:
          path: .pants.d/pants.log
          destination: pants-log-build
      - save_cache:
          paths:
            - "~/.cache/nce"
          key: pants-setup-<< parameters.base_cache_key >>-{{ checksum "pants-bootstrap-cache-key.txt" }}
      - save_cache:
          paths:
            - ".pants.d/named_caches"
          key: pants-named-cache-<< parameters.base_cache_key >>-{{ checksum "pants.toml" }}-{{ checksum "lockfiles/python.lock" }}-{{ .Branch }}
      - persist_to_workspace:
          name: Saving built artifacts to workspace
          root: ./
          paths:
            - dist

some-insurance-58590

04/09/2025, 2:17 PM

Does something like this work for you? we use something similar in our CI pipeline (for a slightly different purpose, so using

list

instead of

package

)

pants --filter-target-type='-docker_image' --changed-since=origin/main --changed-dependents=transitive package

(note no

::

)

ripe-architect-1001

04/09/2025, 2:19 PM

I'll give it a shot

6 Views

Open in Slack

Previous Next