https://pantsbuild.org/ logo
Powered by
r

rapid-bird-79300

11/30/2022, 10:45 PM
Hey Pants Team, curious if anyone has seen this before in CI:
Copy code
2022-11-30T22:41:52Z] 22:41:52.07 [ERROR] Permission denied (publickey).
[2022-11-30T22:41:52Z] fatal: Could not read from remote repository.
[2022-11-30T22:41:52Z]
[2022-11-30T22:41:52Z] Please make sure you have the correct access rights
[2022-11-30T22:41:52Z] and the repository exists.
[2022-11-30T22:41:52Z] Permission denied (publickey).
[2022-11-30T22:41:52Z] fatal: Could not read from remote repository.
[2022-11-30T22:41:52Z]
[2022-11-30T22:41:52Z] Please make sure you have the correct access rights
[2022-11-30T22:41:52Z] and the repository exists.
[2022-11-30T22:41:52Z] fatal: unable to read 6a8276e5ec50a3dfc850f4c636e9bae80199aa73
[2022-11-30T22:41:52Z] Traceback (most recent call last):
[2022-11-30T22:41:52Z]   File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/daemon_pants_runner.py", line 131, in single_daemonized_run
[2022-11-30T22:41:52Z]     runner = LocalPantsRunner.create(
[2022-11-30T22:41:52Z]   File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 158, in create
[2022-11-30T22:41:52Z]     specs = calculate_specs(
[2022-11-30T22:41:52Z]   File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/init/specs_calculator.py", line 68, in calculate_specs
[2022-11-30T22:41:52Z]     changed_files = tuple(changed_options.changed_files(maybe_git_worktree.git_worktree))
[2022-11-30T22:41:52Z]   File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/vcs/changed.py", line 127, in changed_files
[2022-11-30T22:41:52Z]     git_worktree.changed_files(
[2022-11-30T22:41:52Z]   File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/vcs/git.py", line 91, in changed_files
[2022-11-30T22:41:52Z]     committed_changes = self._git_binary._invoke_unsandboxed(
[2022-11-30T22:41:52Z]   File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/core/util_rules/system_binaries.py", line 384, in _invoke_unsandboxed
[2022-11-30T22:41:52Z]     self._check_result(cmd, process.returncode, err.decode())
[2022-11-30T22:41:52Z]   File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/core/util_rules/system_binaries.py", line 393, in _check_result
[2022-11-30T22:41:52Z]     raise GitBinaryException(failure_msg or f"{cmd_str} failed with exit code {result}")
[2022-11-30T22:41:52Z] pants.core.util_rules.system_binaries.GitBinaryException: Permission denied (publickey).
[2022-11-30T22:41:52Z] fatal: Could not read from remote repository.
[2022-11-30T22:41:52Z]
[2022-11-30T22:41:52Z] Please make sure you have the correct access rights
[2022-11-30T22:41:52Z] and the repository exists.
[2022-11-30T22:41:52Z] Permission denied (publickey).
[2022-11-30T22:41:52Z] fatal: Could not read from remote repository.
[2022-11-30T22:41:52Z]
[2022-11-30T22:41:52Z] Please make sure you have the correct access rights
[2022-11-30T22:41:52Z] and the repository exists.
[2022-11-30T22:41:52Z] fatal: unable to read 6a8276e5ec50a3dfc850f4c636e9bae80199aa73
[2022-11-30T22:41:52Z]
[2022-11-30T22:41:52Z]
[2022-11-30T22:41:52Z] Use -ldebug for more logs.
[2022-11-30T22:41:52Z] See <https://www.pantsbuild.org/v2.14/docs/troubleshooting> for common issues.
[2022-11-30T22:41:52Z] Consider reaching out for help: <https://www.pantsbuild.org/v2.14/docs/getting-help>
the 
--change-since
option is failing randomly in builds.
version is 
2.14.0
h

hundreds-father-404

11/30/2022, 10:46 PM
this seems like an underlying git error and Pants is surfacing it. Can you try running some Git commands directly beforehand?
r

rapid-bird-79300

11/30/2022, 10:47 PM
We actually do, this is in buildkite so the repo is clone and we do a 
git fetch origin master
prior to this pants goal
all git commands show no issue
going to try to get some debug logs here
I have some debug logs. Okay to send in PM?
h

hundreds-father-404

11/30/2022, 11:03 PM
how hard would it be to purge the sensitive parts? so that others can weigh in too
r

rapid-bird-79300

11/30/2022, 11:04 PM
no problem at all one sec
Copy code
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] spawned local process as Some(7686) for Process { argv: ["./find_binary.sh", "git"], env: {"PATH": "/usr/bin:/bin:/usr/local/bin"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<74daa8164942a954dfa7b0e771e4a38e58b3dd98c3f2daffa9d54fcd98b6626a>, size_bytes: 91 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<74daa8164942a954dfa7b0e771e4a38e58b3dd98c3f2daffa9d54fcd98b6626a>, size_bytes: 91 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Searching for `git` on PATH=/usr/bin:/bin:/usr/local/bin", level: Debug, append_only_caches: {}, jdk_home: None, platform_constraint: None, cache_scope: PerRestartSuccessful, remote_cache_speculation_delay: 0ns }
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Completed: Searching for `git` on PATH=/usr/bin:/bin:/usr/local/bin
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Completed: Scheduling: Searching for `git` on PATH=/usr/bin:/bin:/usr/local/bin
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Completed: Finding the `git` binary
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] computed 1 nodes in 0.023991 seconds. there are 108 total nodes.
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Launching 1 roots (poll=false).
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Starting: Finding the `git` binary
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Completed: Finding the `git` binary
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Executing: /usr/bin/git rev-parse --show-toplevel
[2022-11-30T22:59:43Z] 22:59:43.69 [DEBUG] Executing: /usr/bin/git --git-dir=/var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests/.git --work-tree=/var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests rev-parse --abbrev-ref HEAD
[2022-11-30T22:59:43Z] 22:59:43.70 [DEBUG] Detected git repository at /var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests on branch None
[2022-11-30T22:59:43Z] 22:59:43.70 [DEBUG] computed 1 nodes in 0.011813 seconds. there are 111 total nodes.
[2022-11-30T22:59:43Z] 22:59:43.70 [DEBUG] Executing: /usr/bin/git --git-dir=/var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests/.git --work-tree=/var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests diff --name-only HEAD -- /var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests
[2022-11-30T22:59:44Z] 22:59:44.35 [DEBUG] Executing: /usr/bin/git --git-dir=/var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests/.git --work-tree=/var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests diff --name-only origin/master...HEAD -- /var/lib/buildkite-agent/builds/buildkite-agent-foo-bar-tests-batches-blue-stack-i-0cdebc318d73e685e-1/foo-bar/foo-bar-pr-tests
[2022-11-30T22:59:45Z] 22:59:45.27 [ERROR] Permission denied (publickey).
[2022-11-30T22:59:45Z] fatal: Could not read from remote repository.
[2022-11-30T22:59:45Z]
[2022-11-30T22:59:45Z] Please make sure you have the correct access rights
[2022-11-30T22:59:45Z] and the repository exists.
[2022-11-30T22:59:45Z] Permission denied (publickey).
[2022-11-30T22:59:45Z] fatal: Could not read from remote repository.
maybe it makes more sense to use the base commit sha rather than the 
origin/master
h

happy-kitchen-89482

11/30/2022, 11:12 PM
So, some git permissions error. But you’re saying it doesn’t happen every time, only intermittently?
How is the local git client authenticating against your git server?
r

rapid-bird-79300

11/30/2022, 11:13 PM
this is happening intermittently
one detail is that this is for a hotfix process, so branch are based off the hotfix branch. So that's why we're thinking to use base commit sha to fix this
but it passes most of the time
then sometimes fails with this
so we're really confused
what is the usage for the 
PANTS_CHANGED_SINCE
env var? Would we just set and run 
./pants tailor
?
only can get it working with 
./pants tailor --check lint --changed-since=${PANTS_CHANGED_SINCE}
h

hundreds-father-404

11/30/2022, 11:27 PM
PANTS_CHANGED_SINCE is a way of setting 
--changed-since
, because every option can be set by CLI, config file, and env var. 
--changed-since=${PANTS_CHANGED_SINCE}
has no purpose - the env var should already be setting the same option
r

rapid-bird-79300

11/30/2022, 11:27 PM
hmm just setting 
PANTS_CHANGED_SINCE
didn't seem to work tho
h

happy-kitchen-89482

11/30/2022, 11:28 PM
I don’t understand 
--changed-since=${PANTS_CHANGED_SINCE}
. What is the value of that env var?
r

rapid-bird-79300

11/30/2022, 11:28 PM
we see this error
Copy code
$ ./run_pants_lint.sh
origin/master
15:28:22.59 [WARN] No arguments specified with `./pants tailor`, so the goal will do nothing.

Instead, you should provide arguments like this:

  * `./pants tailor ::` to run on everything
  * `./pants tailor dir::` to run on `dir` and subdirs
  * `./pants tailor dir` to run on `dir`
  * `./pants --changed-since=HEAD tailor` to only run on changed and new files
✔ No lint issues found!
PANTS_CHANGED_SINCE
is 
origin/master
h

happy-kitchen-89482

11/30/2022, 11:29 PM
Ah OK
so maybe we’re on to something here
what is setting that?
Here’s what may be causing trouble
You can set any Pants option via flag, config or env var
So you could run 
./pants --changed-since=origin/master tailor
or
PANTS_CHANGED_SINCE=origin/master ./pants tailor
Same thing
r

rapid-bird-79300

11/30/2022, 11:30 PM
yup
h

happy-kitchen-89482

11/30/2022, 11:30 PM
But - it sounds like you’re hijacking that env var name
r

rapid-bird-79300

11/30/2022, 11:30 PM
we do the former 
./pants --changed-since=origin/master tailor
I'm just trying to update the script now to accept the env var
but it's not working
h

happy-kitchen-89482

11/30/2022, 11:31 PM
So what is 
./pants tailor --check lint --changed-since=${PANTS_CHANGED_SINCE}
?
r

rapid-bird-79300

11/30/2022, 11:31 PM
just me iterating in my local
so this is the script
Copy code
#!/bin/bash
set -e

PANTS_CHANGED_SINCE="${PANTS_CHANGED_SINCE:-origin/master}"
PANTS_CONFIG_FILES="${PANTS_CONFIG_FILES:-pants.toml}"

if [[ "$@" == "--auto-fix" || "$@" == "--fix" ]]; then
    ./pants tailor
    ./pants fmt lint
else
    ./pants tailor --check lint
fi
echo -e "\033[0;32m✔ No lint issues found!\033[0m"
this doesn't seem to work
should we export?
exporting fixed it
apologies first time using the env var here
h

happy-kitchen-89482

11/30/2022, 11:33 PM
ah ok
yeah, that is correct
--changed-since=${PANTS_CHANGED_SINCE}
is redundant and a bit confusing, but I guess would still work if the env var was set properly
r

rapid-bird-79300

11/30/2022, 11:34 PM
yeah it felt off for sure
and we had a shadowing issue like that in the past so not trying to go there again
this works tho
but still we don't know what the root cause is here
okay merge base did not help
we may just have to remove pants change subsystem and passed changed files directly
a bit of a pain
curious if this log line means anything
Copy code
[2022-11-30T22:41:52Z] fatal: unable to read 6a8276e5ec50a3dfc850f4c636e9bae80199aa73
I'm assuming that's an error from git or something
h

happy-kitchen-89482

12/01/2022, 12:13 AM
Hmmmmmm
We should try and get to the bottom of this
Can you open a bug for this at https://github.com/pantsbuild/pants ?
👍 1
It would be really important to nail down under what circumstances this happens, and get a consistent repro of it
r

rapid-bird-79300

12/01/2022, 1:00 AM
I will try to repo in an example repo if possible. Otherwise will file a bug report
🙏 1
Powered by