https://pantsbuild.org/ logo
Join Slack
Powered by
<@U06A03HV1> any suggestions for how to debug thi...
# development
h
@witty-crayon-22786 any suggestions for how to debug this remote execution failure?
Copy code
Engine traceback:
  in select
    ..
  in pants.core.goals.test.run_tests
    `test` goal
  in pants.backend.python.goals.pytest_runner.run_python_test
    Run Pytest - (environment:remote, src/python/pants/util/osutil_test.py:tests)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
    ..
  in pants.backend.python.util_rules.pex.create_venv_pex
    ..
  in pants.backend.python.util_rules.pex.build_pex
    ..
  in pants.engine.process.fallible_to_exec_result_or_raise
    ..

Traceback (most recent call last):
  File "/Users/ericarellano/code/pants/src/python/pants/engine/process.py", line 289, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
[Errno 2] No such file or directory: 'local_dists.pex/PEX-INFO'
w
hm!
h
deterministic, and confirmed it's not coming from the cache by rebasing on top of https://github.com/pantsbuild/pants/pull/17198
w
maybe see whether it fails in docker as well?
h
I'm using Docker from your PR, haven't tried Toolchain
w
sorry, i meant: “in a docker environment”
👍 1
can also probably increase the pex verbosity a bit
h
docker was fine. Do you know if it's possible to materialize the input digest?
Copy code
action=Action { command_digest: Some(Digest { hash: "a4e682122b29d2202d4307d55c54c142df5fe1d96c6401f254d871c57076d068", size_bytes: 772 }), input_root_digest: Some(Digest { hash: "de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87", size_bytes: 508 }), timeout: None, do_not_cache: false, salt: b"", platform: None }; command=Command { arguments: ["/usr/local/bin/python3.9", "./pex", "--tmpdir", ".tmp", "--python-path", "/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "--output-file", "pytest_runner.pex", "--no-emit-warnings", "--venv", "--seed", "verbose", "--no-venv-site-packages-copies", "--python", "/usr/local/bin/python3.9", "--console-script", "pytest", "--pex-path", "pytest.pex:requirements.pex:local_dists.pex", "--sources-directory=source_files", "--no-pypi", "--index=<https://pypi.org/simple/>", "--manylinux", "manylinux2014", "--resolver-version", "pip-2020-resolver", "--layout", "packed"], environment_variables: [EnvironmentVariable { name: "CPPFLAGS", value: "" }, EnvironmentVariable { name: "LANG", value: "C.UTF-8" }, EnvironmentVariable { name: "LDFLAGS", value: "" }, EnvironmentVariable { name: "PANTS_CACHE_KEY_TARGET_PLATFORM", value: "linux_x86_64" }, EnvironmentVariable { name: "PATH", value: "/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin" }, EnvironmentVariable { name: "PEX_IGNORE_RCFILES", value: "true" }, EnvironmentVariable { name: "PEX_ROOT", value: ".cache/pex_root" }], output_files: [], output_directories: ["pytest_runner.pex"], output_paths: [], platform: Some(Platform { properties: [] }), working_directory: "", output_node_properties: [] }; execute_request=ExecuteRequest { instance_name: "", skip_cache_lookup: true, action_digest: Some(Digest { hash: "04952d45f525c3142fd86081a4a74f46adc65fb193d7f1af7777501525dfd8cc", size_bytes: 142 }), execution_policy: None, results_cache_policy: None }
w
yea: can use `fs_util`: one sec.
👍 1
will be:
Copy code
./cargo run -p fs_util -- --help
❤️ 1
h
Copy code
Digest { hash: "de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87", size_bytes: 508 })
w
so probably something like :
Copy code
./cargo run -p fs_util -- --remote-instance-name="" --server-address=127.0.0.1:50051 directory materialize de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87 508 dest
🙌 1
h
womp womp
Copy code
thread 'main' panicked at 'A WorkunitStore has not been set for this thread.', workunit_store/src/lib.rs:831:31
w
set 
RUST_BACKTRACE=1
and run again?
probably need
Copy code
diff --git a/src/rust/engine/fs/fs_util/src/main.rs b/src/rust/engine/fs/fs_util/src/main.rs
index 4f23a17ae5..03486ccefd 100644
--- a/src/rust/engine/fs/fs_util/src/main.rs
+++ b/src/rust/engine/fs/fs_util/src/main.rs
@@ -84,6 +84,8 @@ struct SummaryWithDigest {
 #[tokio::main]
 async fn main() {
   env_logger::init();
+  let workunit_store = WorkunitStore::new(false, log::Level::Debug);
+  workunit_store.init_thread_state(None);
 
   match execute(
     &Command::new("fs_util")
🙌 1
h
yep that fixed it! now "transport error", but progress
w
is this against 
build-support/reapi-sample-server/run.sh
or some other server?
i don’t repro
h
yeah, reapi-sample-server
w
…hm! on what sha…?
h
Copy code
./pants --pants-config-files=pants.remote-execution.toml test src/python/pants/util/osutil_test.py --no-python-infer-conftests -ldebug
w
yea, i don’t repro. hm.
you’re not proxying the relevant ports elsewhere, are you…?
h
I don't think so. I restarted my machine today and all kubectl port forwarding is off
w
…oh, shoot. i probably had not set the environment on the target.
h
ah yeah, 
main
doesn't have that set
w
k. trying to repro. did you make it past the 
fs_util
issue?
h
not the transient error, but trying to reproduce the error again on latest main. Great, still fails for me to run the Pants command
oh, but looks like even with the transport error it still wrote to 
dest/
w
ok, yea: i repro for 
os_util.py
h
are you able to get fs util working? you'll need https://github.com/pantsbuild/pants/pull/17229. It looks like it is just saving the directories locally, no file content
w
which digest were you using from the command?
i don’t see the “merged” input digest being logged
h
Copy code
command_digest: Some(Digest { hash: "e7f28fff7fd45be366ca220d6ef13a383851836a301803311a0921ccbc4dc5e8", size_bytes: 740 }), input_root_digest: Some(Digest { hash: "de271807c1a24e8d41b0e843c0ece6626aee7f5dc3030c7405ff1b10ef723a87", size_bytes: 508 })
I was using 
input_root_digest
- is that not it?
w
ah ... possibly? let me see. we do some amount of manipulation of the digest to create the request.
h
w
hm: i’m pretty sure 
:tcp
isn’t right in your 
fs_util
command
trying to repro that bit now.
h
oh yeah bad copy and paste, it failed w/o it too. fixed the issue
w
my guess is that it’s related to http(s)
➕ 1
yea… needs to be 
<http://127.0.0.1:50051>
. sorry about that.
that allows me to dump to 
dest
…or at least doesn’t error. hm.
oh, jeez. the dest needs to be absolute
yea, this works for me with:
Copy code
./cargo run -p fs_util -- --remote-instance-name="" --server-address=<http://127.0.0.1:50051> directory materialize $digest_fingerprint $digest_len $(pwd)/dest
…ahhh. can pass 
--verbose
multiple times to make the server noisier. will get a change out for that.
doesn’t get you a whole lot more from the worker though.
hm… so. this 
host-tools
runner that it is using is not the 
buildbox-runner-hosttools
runner, which is an annoying bit of naming. the up shot is that it is easy to update to add additional debug output. the downside is that it may not be behaving exactly the same as the full 
buildbox-runner-hosttools
runner. will investigate further on Monday.
👍 1
Open in Slack
PreviousNext
Powered by