https://pantsbuild.org/ logo
#development
Title
# development
h

hundreds-father-404

10/14/2022, 9:04 PM
@witty-crayon-22786 any suggestions for how to debug this remote execution failure?
Copy code
Engine traceback:
  in select
    ..
  in pants.core.goals.test.run_tests
    `test` goal
  in pants.backend.python.goals.pytest_runner.run_python_test
    Run Pytest - (environment:remote, src/python/pants/util/osutil_test.py:tests)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
    ..
  in pants.backend.python.util_rules.pex.create_venv_pex
    ..
  in pants.backend.python.util_rules.pex.build_pex
    ..
  in pants.engine.process.fallible_to_exec_result_or_raise
    ..

Traceback (most recent call last):
  File "/Users/ericarellano/code/pants/src/python/pants/engine/process.py", line 289, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
[Errno 2] No such file or directory: 'local_dists.pex/PEX-INFO'
w

witty-crayon-22786

10/14/2022, 9:21 PM
hm!
h

hundreds-father-404

10/14/2022, 9:21 PM
deterministic, and confirmed it's not coming from the cache by rebasing on top of https://github.com/pantsbuild/pants/pull/17198
w

witty-crayon-22786

10/14/2022, 9:21 PM
maybe see whether it fails in docker as well?
h

hundreds-father-404

10/14/2022, 9:21 PM
I'm using Docker from your PR, haven't tried Toolchain
w

witty-crayon-22786

10/14/2022, 9:21 PM
sorry, i meant: “in a docker environment”
👍 1
can also probably increase the pex verbosity a bit
h

hundreds-father-404

10/14/2022, 9:32 PM
docker was fine. Do you know if it's possible to materialize the input digest?
Copy code
action=Action { command_digest: Some(Digest { hash: "a4e682122b29d2202d4307d55c54c142df5fe1d96c6401f254d871c57076d068", size_bytes: 772 }), input_root_digest: Some(Digest { hash: "de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87", size_bytes: 508 }), timeout: None, do_not_cache: false, salt: b"", platform: None }; command=Command { arguments: ["/usr/local/bin/python3.9", "./pex", "--tmpdir", ".tmp", "--python-path", "/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "--output-file", "pytest_runner.pex", "--no-emit-warnings", "--venv", "--seed", "verbose", "--no-venv-site-packages-copies", "--python", "/usr/local/bin/python3.9", "--console-script", "pytest", "--pex-path", "pytest.pex:requirements.pex:local_dists.pex", "--sources-directory=source_files", "--no-pypi", "--index=<https://pypi.org/simple/>", "--manylinux", "manylinux2014", "--resolver-version", "pip-2020-resolver", "--layout", "packed"], environment_variables: [EnvironmentVariable { name: "CPPFLAGS", value: "" }, EnvironmentVariable { name: "LANG", value: "C.UTF-8" }, EnvironmentVariable { name: "LDFLAGS", value: "" }, EnvironmentVariable { name: "PANTS_CACHE_KEY_TARGET_PLATFORM", value: "linux_x86_64" }, EnvironmentVariable { name: "PATH", value: "/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin" }, EnvironmentVariable { name: "PEX_IGNORE_RCFILES", value: "true" }, EnvironmentVariable { name: "PEX_ROOT", value: ".cache/pex_root" }], output_files: [], output_directories: ["pytest_runner.pex"], output_paths: [], platform: Some(Platform { properties: [] }), working_directory: "", output_node_properties: [] }; execute_request=ExecuteRequest { instance_name: "", skip_cache_lookup: true, action_digest: Some(Digest { hash: "04952d45f525c3142fd86081a4a74f46adc65fb193d7f1af7777501525dfd8cc", size_bytes: 142 }), execution_policy: None, results_cache_policy: None }
w

witty-crayon-22786

10/14/2022, 9:33 PM
yea: can use `fs_util`: one sec.
👍 1
will be:
Copy code
./cargo run -p fs_util -- --help
❤️ 1
h

hundreds-father-404

10/14/2022, 9:33 PM
Copy code
Digest { hash: "de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87", size_bytes: 508 })
w

witty-crayon-22786

10/14/2022, 9:41 PM
so probably something like :
Copy code
./cargo run -p fs_util -- --remote-instance-name="" --server-address=127.0.0.1:50051 directory materialize de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87 508 dest
🙌 1
h

hundreds-father-404

10/14/2022, 9:45 PM
womp womp
Copy code
thread 'main' panicked at 'A WorkunitStore has not been set for this thread.', workunit_store/src/lib.rs:831:31
w

witty-crayon-22786

10/14/2022, 9:45 PM
set
RUST_BACKTRACE=1
and run again?
probably need
Copy code
diff --git a/src/rust/engine/fs/fs_util/src/main.rs b/src/rust/engine/fs/fs_util/src/main.rs
index 4f23a17ae5..03486ccefd 100644
--- a/src/rust/engine/fs/fs_util/src/main.rs
+++ b/src/rust/engine/fs/fs_util/src/main.rs
@@ -84,6 +84,8 @@ struct SummaryWithDigest {
 #[tokio::main]
 async fn main() {
   env_logger::init();
+  let workunit_store = WorkunitStore::new(false, log::Level::Debug);
+  workunit_store.init_thread_state(None);
 
   match execute(
     &Command::new("fs_util")
🙌 1
h

hundreds-father-404

10/14/2022, 9:48 PM
yep that fixed it! now "transport error", but progress
w

witty-crayon-22786

10/14/2022, 10:47 PM
is this against
build-support/reapi-sample-server/run.sh
or some other server?
i don’t repro
h

hundreds-father-404

10/14/2022, 10:48 PM
yeah, reapi-sample-server
w

witty-crayon-22786

10/14/2022, 10:48 PM
…hm! on what sha…?
h

hundreds-father-404

10/14/2022, 10:49 PM
Copy code
./pants --pants-config-files=pants.remote-execution.toml test src/python/pants/util/osutil_test.py --no-python-infer-conftests -ldebug
w

witty-crayon-22786

10/14/2022, 10:50 PM
yea, i don’t repro. hm.
you’re not proxying the relevant ports elsewhere, are you…?
h

hundreds-father-404

10/14/2022, 10:51 PM
I don't think so. I restarted my machine today and all kubectl port forwarding is off
w

witty-crayon-22786

10/14/2022, 10:53 PM
…oh, shoot. i probably had not set the environment on the target.
h

hundreds-father-404

10/14/2022, 10:53 PM
ah yeah,
main
doesn't have that set
w

witty-crayon-22786

10/14/2022, 10:54 PM
k. trying to repro. did you make it past the
fs_util
issue?
h

hundreds-father-404

10/14/2022, 10:55 PM
not the transient error, but trying to reproduce the error again on latest main. Great, still fails for me to run the Pants command
oh, but looks like even with the transport error it still wrote to
dest/
w

witty-crayon-22786

10/14/2022, 10:57 PM
ok, yea: i repro for
os_util.py
h

hundreds-father-404

10/14/2022, 10:58 PM
are you able to get fs util working? you'll need https://github.com/pantsbuild/pants/pull/17229. It looks like it is just saving the directories locally, no file content
w

witty-crayon-22786

10/14/2022, 10:59 PM
which digest were you using from the command?
i don’t see the “merged” input digest being logged
h

hundreds-father-404

10/14/2022, 11:02 PM
Copy code
command_digest: Some(Digest { hash: "e7f28fff7fd45be366ca220d6ef13a383851836a301803311a0921ccbc4dc5e8", size_bytes: 740 }), input_root_digest: Some(Digest { hash: "de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87", size_bytes: 508 })
I was using
input_root_digest
- is that not it?
w

witty-crayon-22786

10/14/2022, 11:03 PM
ah ... possibly? let me see. we do some amount of manipulation of the digest to create the request.
h

hundreds-father-404

10/14/2022, 11:04 PM
w

witty-crayon-22786

10/14/2022, 11:08 PM
hm: i’m pretty sure
:tcp
isn’t right in your
fs_util
command
trying to repro that bit now.
h

hundreds-father-404

10/14/2022, 11:09 PM
oh yeah bad copy and paste, it failed w/o it too. fixed the issue
w

witty-crayon-22786

10/14/2022, 11:10 PM
my guess is that it’s related to http(s)
1
yea… needs to be
<http://127.0.0.1:50051>
. sorry about that.
that allows me to dump to
dest
…or at least doesn’t error. hm.
oh, jeez. the dest needs to be absolute
yea, this works for me with:
Copy code
./cargo run -p fs_util -- --remote-instance-name="" --server-address=<http://127.0.0.1:50051> directory materialize $digest_fingerprint $digest_len $(pwd)/dest
…ahhh. can pass
--verbose
multiple times to make the server noisier. will get a change out for that.
doesn’t get you a whole lot more from the worker though.
hm… so. this
host-tools
runner that it is using is not the
buildbox-runner-hosttools
runner, which is an annoying bit of naming. the up shot is that it is easy to update to add additional debug output. the downside is that it may not be behaving exactly the same as the full
buildbox-runner-hosttools
runner. will investigate further on Monday.
👍 1