I've now had repeated instances of this happening in our CI system

23:06:17.95 [INFO] Long running tasks:
  3567.56s	Scheduling: Determine Python dependencies for astranis-python/astranis/gnc/odet/gmat_common.py:sources
  3567.68s	Scheduling: Determine Python dependencies for astranis-python/astranis/gnc/gmat/queried_data_files.py
  3568.93s	Scheduling: Parse Dockerfile.

Basically ran for an hour and timed out the build. I wasn't able to reproduce locally, but this did fail when we attempted a rebuild. We're just running a

./pants package ::

command with some tag filtering. Any ideas of where to go chasing for what could be holding up this scheduling? The logs show it makes it through building one of our artifacts and then hangs on this for the remaining of the time.

are you using a remote cache?

We do have one. There is a warning higher up that was issued prior to the first artifact being built successfully

19:34:41.41 [WARN] Failed to read from remote cache (1 occurrences so far): Unavailable: "Pants client timeout"

are you using the

cache_content_behavior

flag?

We have

cache_content_behavior = "validate"

in

pants.toml

i would remove that setting, and allow

cache_content_behavior

to default to `fetch`… it can experience this kind of issue.

Thanks for the advice! I'm actually going to disable the remote caching for this step entirely while also changing to fetch. The caching has been great for unit tests, but I'm not as concerned about it for these artifact generation steps. They generally happen fast enough that the caching isn't important.

hm. that shouldn’t be necessary, but up to you.

Right. Still going to leave it enabled for the testing. But we weren't using it for these steps anywhere.

i would view that as an opportunity to make things faster for free by fixing it then 😉

Hmm, yeah, you're right. I bet I can fix it all the way down properly.