I’m trying to add golang code to a polyglot repo, and I’m finding that pants downloads Go modules into it’s sandbox much more frequently than I would expect. Even when doing a

pants run some.py

which has no relationship with any go code results in a lengthy Go module download. Am I doing something obviously wrong, or is this the expected behaviour?

Which Pants version is this?

this is expected currently, because the go plugin generates targets for thirdparty, and so needs to have run

go list

for many operations

Is there a way to speed this up? For example it doesn't seem to be using the modules already cached from

$GOPATH

it should be cached in Pants’ cache: are you using a local or remote cache of builds?

I'm using a local cache. Now that I think about it the initial download of modules took 40mins, while most subsequent Go Module downloads take 5-10mins. If it's cached should I be getting the go module download messages, or should it jump straight to the go module metadata analysis?

you might see

Scheduling: $process

, during the cache lookup, but only for a few seconds at most. you shouldn’t see any processes actually running unless their inputs have changed.

Hmmm I might need to have a closer look then

I can imagine having to do some one-time Go work but then it should be completely quiescent if you only do Python stuff

Seeing this as well. “Download Go module” and “Analyze metadata for Go third-party module” takes a significant amount of time with an uncached state, which is expected, but even running something like

pants lint path/to/python/file.py

multiple times after another still adds about 10s. Running

pants -ldebug lint path/to/python/file.py

with some timestamps:

# first log output 
14:32:09.54 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x108769880>
# first reference to Go
14:32:15.39 [DEBUG] Running Searching for `go` on PATH=….
# last references to Go
14:32:25.90 [DEBUG] Completed: Download and analyze all third-party Go packages
14:32:26.01 [DEBUG] Completed: Generate `go_third_party_package` targets from `go_mod` target
# done
14:32:27.66 [DEBUG] Completed: `lint` goal

The same command without Go enabled takes 6s instead of 18s.

Yeah, I'm not totally sure how to deal with this. In theory, dep inference on Go code could generate Python-relevant dependencies. Even if there is no pathway for that today.

It logs

[DEBUG] Launching pantsd

each time, and

[DEBUG] pantsd is running at pid 49264, pailgun port is 64668

logs a different pid and port. So it seems so?

Damn it, the answer was so simple. We have a slightly bigger codebase, needed to bump it to 5GB, and now things are flying again. Before we having our python devs just have the ignore directive set for the golang directories in pants.toml:

pants_ignore.add = [“src/golang”,“tests/golang”,]

We really need to bump the default, 1GiB is way too low, people keep bumping in to this.

I had to increase the max number again today. We are now running with a max of 12GB. I know it’s probably a complicated question, but we have some kind of rough guideline for what is expected usage based on codebase size?

would you mind filing an issue containing the output of

--stats-memory-usage

for a relevant run? see https://github.com/pantsbuild/pants/pull/18389#issuecomment-1458610778

Whats interesting is that the runs on my machine don’t use more than 447MB. I’ve now read the Using Pants in CI article and will add stats and memory usage collection to our CI so it makes it easier to identify jobs with issues.