TL;DR: I am seeing errors related to

PATH

not being populated on remote exec requests as well

download_python_binary

as not working in different versions of Pants. Since

2.17.1

, I cannot get Remote Execution working. Hey Pants community 🙂 I am having a few issues with the Pants and Remote Execution feature (through BuildBarn Remote Execution) when upgrading from

2.14.0

to

2.17.1

(I actually tried a few intermediate versions in between). I am wondering if anyone had similar issues with Remote Execution. The issues that I faced: • In

2.15.x

global remote execution mode is broken with

PATH

not being set when a remote exec request is being sent, resulting in

Cannot find executable \"env\" in search paths \"\""

error (related to #17262 ) ◦ I believe this could be related to this call not setting environment ◦ I can enforce

PATH

on a worker but I believe this should be set from a client ◦ I also recall this issue #19375 which was adding

system-binaries

subsystem which might address that but I couldn't see it released • Since

2.16.0

, with a single

remote_environment

and setting

remote_execution = true

in

pants.toml

doesn't result in targets running remotely. This can be enabled by adding

__defaults__(all=dict(environment="remote"))

to the root

BUILD

file and having

fallback=local

environment ◦ If I understand correctly, this unnecessary as I should be able "globally" turn off and on remote exec through

remote_execution

, at least that was behavior in

2.15.2

• Since

2.17.0

the

download_python_binary

function fails with

cp: cannot create directory '.python-build-standalone/c12164f0e9228ec20704c1aba97eb31b8e2a482d41943d541cc8e3a9e84f7349': No such file or directory

◦ I had a look around the problematic code and it looks like the "installation_root" directory was never created but it could be that digest is incorrect I created this repository to replicate all the above issues. There is a chance that my config is missing a setting where I can set the remote environment variables. I will see if I can patch the mentioned code and run it again but I would love to hear your thoughts 🙂

cc @fast-nail-55400 who I think knows the most about this

◦ I can enforce

PATH

on a worker but I believe this should be set from a client

I disagree that Pants is the only place where

PATH

should be set. For example, if instead we were talking about a Docker image, then I would expect the Docker image to set the

PATH

in a way appropriate to itself, so that any containers using that image would not require users of the image to know what

PATH

was relevant. Similar reasoning applies to remote execution workers. If the worker is similar to an "image", then the worker should know where to find things.

Thanks @fast-nail-55400 for all the info. It makes sense right now. I will try

subprocess-environment-env-vars

(I thought I have tried that but it is not covered in my replication config so didn't happen 🙂) but probably I would rely on Buildbarn's config in that case. Interestingly, in the older version of Remote Execution protocol, it was discourage to rely on

PATH

(if I read that correctly).

// Changed in v2.3. v2.2 and older require that no PATH lookups are performed,

// and that relative paths are resolved relative to the input root. This

// behavior can, however, not be relied upon, as most implementations already

// followed the rules described above.

The earlier part is not very clear were the resolution should be happening.

// The first argument specifies the command to run, which may be either an

// absolute path, a path relative to the working directory, or an unqualified

// path (without path separators) which will be resolved using the operating

// system's equivalent of the PATH environment variable. Path separators

// native to the operating system running on the worker SHOULD be used. If the

//

environment_variables

list contains an entry for the PATH environment

// variable, it SHOULD be respected. If not, the resolution process is

// implementation-defined.

Especially this "If not, the resolution process is implementation-defined." BuildBarn recommends setting it on the client. Anyway, as I wrote, your explaination makes total sense. And any thougths on below?

• Since

2.17.0

the

download_python_binary

function fails with

cp: cannot create directory '.python-build-standalone/c12164f0e9228ec20704c1aba97eb31b8e2a482d41943d541cc8e3a9e84f7349': No such file or directory

// For better hermeticity, is preferable to have the environment

// controlled by the build client, such as Bazel's --action_env.

// --action_env, however, has limited scope that makes it not useful

// in some scenarios: https://github.com/bazelbuild/bazel/issues/3320

I don't think there is a right answer; seems like a decision with trade-offs. For example, trading off against "more hermeticity" by putting the

PATH

value into the cache key (i.e., the Action/Command protos) versus having to maintain that configuration across many repos instead of just in one place. Choose what works best for your organization. 🙂

So I tried

subprocess-environment-env-vars

(see all my config here) without luck (I am getting the same error about

env

not being found). I am fine with relying on BuildBarn to populate it. In regards

--remote-execution-append-only-caches-base-path

, I will try that tomorrow. Thanks for all the help! 🙏