I think the mental model is part of what’s giving me trouble —

tailor

has tried to generate a build file for every directory, specifying individual `go_package`s and `go_binary`s. I don’t understand why it does this: from my perspective, building go binaries is just

go build

, so I assumed that the build step for a Go project would just be • specify input files • specify output file(s) • write a command that runs

go install

to install deps, then

go build

, and writes to the output location(s) And in theory, that should give you a hermetic build, as long as the Go toolchain is included as part of the sandbox. I don’t see why having a build-system-level abstraction for each package and for each third-party dependency is necessary — but when I tried Bazel I noticed that it had a very similar model, so I’m guessing there’s a reason for it I just haven’t grokked yet.

I also figure that I could do exactly what I described above with a shell rule, but again, there’s presumably some reason that the standard tooling for Go in these build systems doesn’t work that way, and I don’t want to blow past something important somebody else already figured out…

I don’t see why having a build-system-level abstraction for each package and for each third-party dependency is necessary

So with both Pants and Bazel, a "target" is how you give decentralized metadata for parts of your code. That metadata includes things like which packages/dirs depend on which others, and also optional metadata like the timeout for tests. A key point of build systems is to keep up as your codebase scales, and to do that, we generally believe metadata should live near the code it describes, rather than a centralized file. Hence, BUILD files. But Pants tries hard to make that boilerplate more minimal, such as inferring your dependencies for you. It also generates the

go_third_party_package

targets for you, rather than having to write them to a BUILD file or use a BUILD file generator We actually used to generate the

go_package

targets for you in Pants 2.8, rather than having one BUILD file per directory. But after a lot of discussion, we changed it in Pants 2.9 to be explicit. https://blog.pantsbuild.org/pants-2-9/ Does that make sense?

As for why Pants doesn’t just use

go build

, it is because

go build

is designed as a CLI tool for end users and makes many assumptions consistent with that design but that conflict with the needs of Pants. For example,

go build

does not provide any insight into the separate build cache that it maintains. Thus, Pants uses

go tool compile

and

go tool link

to invoke the Go compiler and linker directly. This allows Pants to specify where to output the “package archive” for a compiled package and then store that package archive in the Pants cache (which includes any configured remote cache which is something

go

does not support. Bazel rules_go had to make similar choices. Note that Bazel vendors or reimplements some of the

go

source. See https://github.com/bazelbuild/rules_go/tree/master/go/tools/builders.)

It would be great if the logic behind

go build

were fully available as a library, but that is not the case. For example, Pants originally used

go list

to analyze Go packages but that caused performance problems for Pants due to

go list

assuming that transitive dependencies would always be available. Pants ended up incorporating an adapted version the package analysis code from

go

to side step that issue. (Note that Bazel also made a similar choice, for what I will presume are similar reasons.)

I assume that’s substantially faster than building a whole binary with just

go build

even with a hot Go build cache, because it doesn’t have to do any checking that the cache is up to date?

If Pants had no insight into compiler output (which would be the case with using

go build

), Pants would have no way to provide the developer experience that users have to come to expect from other Pants language backends. And if the Pants Go backend cannot provide that experience, no particular reason to have a Go backend any more.

I’m imagining a hybrid system where a build run with

go build

is “sandboxed” by Pants, but access to the Go filesystem-based build cache is considered an allowed exception, like (I think?) the toolchain itself

which would allow a user to populate a less-granular build cache based on eg the hashes of the mod and sum files, and still get the DX benefits of pants

in that: • changes to the source would motivate a recompilation of the binary • changes to the dep files would download new deps, but the binaries of unchanged deps would still exist in the build cache • builds remain hermetic (as long as you don’t jam a bunch of unrelated stuff into the build cache to use it as an escape hatch)

And Pants's caching is much more fine-grained than caching the output of an entire linked binary

the latter would not be very useful in practice, as any change to any input file would invalidate the whole thing