nice-hydrogen-28689
02/20/2022, 5:43 AMtailor
generated a bunch of build files, but when I to run ./pants list ::
I get an error complaining that one of our dependencies does not exist — it’s something that we’ve vendored and linked through go.mod, and when I looked in the chroot for that step I found that only my mod and sum files were there. No sources or anything.tailor
has tried to generate a build file for every directory, specifying individual `go_package`s and `go_binary`s. I don’t understand why it does this: from my perspective, building go binaries is just go build
, so I assumed that the build step for a Go project would just be
• specify input files
• specify output file(s)
• write a command that runs go install
to install deps, then go build
, and writes to the output location(s)
And in theory, that should give you a hermetic build, as long as the Go toolchain is included as part of the sandbox.
I don’t see why having a build-system-level abstraction for each package and for each third-party dependency is necessary — but when I tried Bazel I noticed that it had a very similar model, so I’m guessing there’s a reason for it I just haven’t grokked yet.happy-kitchen-89482
02/20/2022, 6:50 AMnice-hydrogen-28689
02/20/2022, 9:11 AMhundreds-father-404
02/21/2022, 2:30 AMI don’t see why having a build-system-level abstraction for each package and for each third-party dependency is necessarySo with both Pants and Bazel, a "target" is how you give decentralized metadata for parts of your code. That metadata includes things like which packages/dirs depend on which others, and also optional metadata like the timeout for tests. A key point of build systems is to keep up as your codebase scales, and to do that, we generally believe metadata should live near the code it describes, rather than a centralized file. Hence, BUILD files. But Pants tries hard to make that boilerplate more minimal, such as inferring your dependencies for you. It also generates the
go_third_party_package
targets for you, rather than having to write them to a BUILD file or use a BUILD file generator
We actually used to generate the go_package
targets for you in Pants 2.8, rather than having one BUILD file per directory. But after a lot of discussion, we changed it in Pants 2.9 to be explicit. https://blog.pantsbuild.org/pants-2-9/
Does that make sense?fast-nail-55400
02/21/2022, 8:22 PMI get an error complaining that one of our dependencies does not exist — it’s something that we’ve vendored and linked through go.mod, and when I looked in the chroot for that step I found that only my mod and sum files were there.vendoring isn’t supported yet. https://github.com/pantsbuild/pants/issues/12768
go build
, it is because go build
is designed as a CLI tool for end users and makes many assumptions consistent with that design but that conflict with the needs of Pants. For example, go build
does not provide any insight into the separate build cache that it maintains. Thus, Pants uses go tool compile
and go tool link
to invoke the Go compiler and linker directly. This allows Pants to specify where to output the “package archive” for a compiled package and then store that package archive in the Pants cache (which includes any configured remote cache which is something go
does not support.
Bazel rules_go had to make similar choices. Note that Bazel vendors or reimplements some of the go
source. See https://github.com/bazelbuild/rules_go/tree/master/go/tools/builders.)go build
were fully available as a library, but that is not the case. For example, Pants originally used go list
to analyze Go packages but that caused performance problems for Pants due to go list
assuming that transitive dependencies would always be available. Pants ended up incorporating an adapted version the package analysis code from go
to side step that issue. (Note that Bazel also made a similar choice, for what I will presume are similar reasons.)nice-hydrogen-28689
02/22/2022, 12:59 AMgo build
itself is smart enough to recognize the presence of already-compiled copies of matching packages in that cache and not recompile them at all
I knew Go did build caching at the package level and that you can save and restore that cache from the filesystem (we do this in our current CI), but it had never occurred to me to populate the cache selectively. which it sounds like is what you’re saying Pants does.fast-nail-55400
02/22/2022, 1:00 AMgo tool link
with all of the package archives that are already stored in the Pants cache.nice-hydrogen-28689
02/22/2022, 1:01 AMgo tool link
was a thinggo build
even with a hot Go build cache, because it doesn’t have to do any checking that the cache is up to date?fast-nail-55400
02/22/2022, 1:22 AMgo build
since go build
is very optimized for its use case. But then again we haven’t micro-optimized the Go backend in Pants at all. The main reason for not using go build
is really around the fact that Pants would not be able to leverage its own cache at all, which is part of how Pants tracks dependencies and is able to invalidate build artifacts only when they change.go build
), Pants would have no way to provide the developer experience that users have to come to expect from other Pants language backends. And if the Pants Go backend cannot provide that experience, no particular reason to have a Go backend any more.nice-hydrogen-28689
02/22/2022, 1:35 AMgo build
is “sandboxed” by Pants, but access to the Go filesystem-based build cache is considered an allowed exception, like (I think?) the toolchain itselffast-nail-55400
02/22/2022, 2:26 AMOr is that what you mean by “no need for a backend” — at that point, the use case is well served by a simple genrule?I meant that no real need to implement the Go backend at all if Pants can’t provide the expected developer experience. The user is free to use
go build
without Pants.happy-kitchen-89482
02/22/2022, 5:31 AMgo
tooling can provide, but Pants can.nice-hydrogen-28689
02/22/2022, 6:01 AMhappy-kitchen-89482
02/22/2022, 5:04 PM