Interesting notes from experiments with pants and local buildfarms (why on earth would you do a local buildfarm? A) for testing, and B) so that pexes are built with a specified architecture without needing pre-built wheels when source dists are necessary). Having said that... I successfully got buildfarm, buildbarn, and buildgrid building pexes, generic docker containers, and dockerized pexes. Hooray for our side! Having said that, it's slower than "immediate" builds (although I should run proper benchmarks), and I did run across a curious error; not sure if it's a buildfarm error or a pants error so mentioning it here. An

ADD

and probably

COPY

instruction of the form

ADD ./my_file /tmp/install/my_file

where

my_file

is listed as part of a

files

dependency works fine on buildbarn and buildgrid but on buildfarm generates the error

#8 ERROR: "/my_file" not found: not found
------
 > [4/5] ADD ./my_file /tmp/install/my_file:
------
failed to compute cache key: "/my_file" not found: not found

...it could be I'm doing something strange, but given that it works on 2/3 build systems but fails on the third is interesting. Wanted to at least raise it for awareness but I'm not sure if I should file an issue given that I'm not sure which product the issue is with! I'm not sure given the size of generated artifacts if remote execution or caching even makes all that much sense for a lot of pants builds since that's an awful lot of I/O, but it's certainly been a worthwhile experiment. I still may try it for our CI solution. One of the buildfarm folks raised the idea that it may be worthwhile to add something to

pants

to disable remote execution (and maybe remote caching) for only certain steps, such as building pexes via remote execution but build the docker containers locally (also useful because as @fast-nail-55400 has mentioned in another thread, docker caches are machine-specific, so it may make sense to optionally turn off remote execution for docker builds; I was doing the buildfarms locally with a mounted

/var/run/docker.sock

so it was using my local machine anyway).

...it could be I’m doing something strange, but given that it works on 2/3 build systems but fails on the third is interesting. Wanted to at least raise it for awareness but I’m not sure if I should file an issue given that I’m not sure which product the issue is with!

It would be helpful if you could open an issue with a reproducible example triggering the error along with a Pants debug log (via

-ldebug

). (The debug log will have the command-line/environment for the “action” being executed remotely.) Sometimes these servers have subtle differences in the remote execution environment that contribute to the failure. It’s been known to happen in other cases that works on 2/3 and fails on the other. (See https://github.com/pantsbuild/pants/issues/13170 for an example of how thorny an issue can be sometimes.)

I'll try and do so; as with all development environments time is somewhat tight and I'm not really being encouraged to experiment with build farms 🙂 But that should be doable I think.

just even opening the issue with an example of the failing target and the Pants invocation would be very helpful. but yeah, time constraints abound everywhere!. even with a minimal reproduction, looking into it would be lower priority for me (unless someone were actively blocked by it), given how involved debugging these are.

Issue started as https://github.com/pantsbuild/pants/issues/14755; hope it's remotely useful!

“remotely” useful — nice 🙂