https://pantsbuild.org/ logo
Join Slack
Powered by
hi! got an error. when pants is running git comman...
# general
v
hi! got an error. when pants is running git commands internally, it seems to not be aware of my ~/.ssh/known_hosts file, and is getting permission denied errors. as part of a call like this:
Copy code
pants \
         --keep-sandboxes=always \
         -ldebug \
         --print-stacktrace \
         --changed-since=$BASE_REVISION \
         --changed-dependents=transitive \
         list
i'm getting this error
Copy code
19:18:43.90 [DEBUG] spawned local process as Some(4662) for Process { argv: ["./find_binary.sh", "git"], env: {"PATH": "/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<74daa8164942a954dfa7b0e771e4a38e58b3dd98c3f2daffa9d54fcd98b6626a>, size_bytes: 91 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<74daa8164942a954dfa7b0e771e4a38e58b3dd98c3f2daffa9d54fcd98b6626a>, size_bytes: 91 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Searching for `git` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin", level: Debug, append_only_caches: {}, jdk_home: None, cache_scope: PerRestartSuccessful, execution_environment: ProcessExecutionEnvironment { name: None, platform: Linux_x86_64, strategy: Local }, remote_cache_speculation_delay: 0ns }
19:18:43.91 [DEBUG] Completed: Searching for `git` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin
19:18:43.91 [DEBUG] Completed: Scheduling: Searching for `git` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin
19:18:43.91 [DEBUG] Completed: Finding the `git` binary
19:18:43.91 [DEBUG] computed 1 nodes in 0.056598 seconds. there are 55 total nodes.
19:18:43.91 [DEBUG] Launching 1 roots (poll=false).
19:18:43.91 [DEBUG] Starting: Finding the `git` binary
19:18:43.91 [DEBUG] Completed: Finding the `git` binary
19:18:43.91 [DEBUG] Executing: /usr/bin/git rev-parse --show-toplevel
19:18:43.92 [DEBUG] Executing: /usr/bin/git --git-dir=/home/circleci/affinity_blobless/.git --work-tree=/home/circleci/affinity_blobless rev-parse --abbrev-ref HEAD
19:18:43.93 [DEBUG] Detected git repository at /home/circleci/affinity_blobless on branch None
19:18:43.93 [DEBUG] computed 1 nodes in 0.022767 seconds. there are 58 total nodes.
19:18:43.93 [DEBUG] Executing: /usr/bin/git --git-dir=/home/circleci/affinity_blobless/.git --work-tree=/home/circleci/affinity_blobless diff --name-only HEAD -- /home/circleci/affinity_blobless/python
19:18:43.94 [DEBUG] Executing: /usr/bin/git --git-dir=/home/circleci/affinity_blobless/.git --work-tree=/home/circleci/affinity_blobless diff --name-only 8f761f8ab9994905881e87cd197042eb09ef2c47...HEAD -- /home/circleci/affinity_blobless/python
19:18:50.16 [ERROR] git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: could not fetch 0a5d25bf23e33c8c19b3eb6b5c49d68572cdb38e from promisor remote
Traceback (most recent call last):
  File "/home/circleci/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/bin/daemon_pants_runner.py", line 134, in single_daemonized_run
    runner = LocalPantsRunner.create(
  File "/home/circleci/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 153, in create
    specs = calculate_specs(
  File "/home/circleci/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/init/specs_calculator.py", line 71, in calculate_specs
    changed_files = tuple(changed_options.changed_files(maybe_git_worktree.git_worktree))
  File "/home/circleci/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/vcs/changed.py", line 136, in changed_files
    git_worktree.changed_files(
  File "/home/circleci/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/vcs/git.py", line 91, in changed_files
    committed_changes = self._git_binary._invoke_unsandboxed(
  File "/home/circleci/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/core/util_rules/system_binaries.py", line 400, in _invoke_unsandboxed
    self._check_result(cmd, process.returncode, err.decode())
  File "/home/circleci/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/core/util_rules/system_binaries.py", line 409, in _check_result
    raise GitBinaryException(failure_msg or f"{cmd_str} failed with exit code {result}")
pants.core.util_rules.system_binaries.GitBinaryException: git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: could not fetch 0a5d25bf23e33c8c19b3eb6b5c49d68572cdb38e from promisor remote
it should be noted that, because this is a blobless clone of the git repo, git diff i think will actually result in some fetching from origin. if i run the same final command outside of pants, it works fine. (and because git no longer needs to fetch, pant will run fine as well on a repeat call). is there a way to signal pants to the existence of my ssh keys? i've echoed them into known_hosts ahead of time
Copy code
echo 'github.com ssh-ed25519 ...' >> ~/.ssh/known_hosts
            echo 'github.com ecdsa-sha2-nistp256 ...' >> ~/.ssh/known_hosts
            echo 'github.com ssh-rsa ...' >> ~/.ssh/known_hosts
i could also be wrong and it is some other lack of SSH config
e
Yeah, try 
env | grep SSH
. Pants masks all env vars by default and you're missing a key one for ssh agent. That should be enough to continue your roll through docs about poking env var holes.
v
Thanks John
to the best of my ability, i think i have found the call site to git: https://github.com/pantsbuild/pants/blob/f8eda7668521ca7b55be2217bf124365105df457/src/python/pants/vcs/git.py#L90 it superficially appears that the git binary is invoked outside of a sandbox, and that there is no opportunity to poke holes through to any environment variables. OTOH, if it is not invoked in a sandbox, i have no idea what environment it is invoked with. whatever. pants does not appear to generate a sandbox in /tmp with a corresponding __run.sh and what not for these git commands
closest scope i could find is the Changed subsystem: https://github.com/pantsbuild/pants/blob/2476a4149e90d9a627ae289977f3c1c5ff9ef0fd/src/python/pants/vcs/changed.py#L142 but it does not have any env var related options i can find
--subprocess-environment-env-vars
also appears to have no effect
theoretically, i have my targets running in a docker_environment(), so just in case, tried 
--docker-env-vars
but that doesn't seem to work
aha! adding 
--no-pantsd
appears to solve the permission denied error. so, whatever environment the pantsd process is running is running in is what needs to be affected
interesting. so, this has been failing in CircleCI on a machine executor, but on my local mac laptop, i don't get this error afaict
pantsd might be run with this hermetic environment: https://github.com/pantsbuild/pants/blob/2476a4149e90d9a627ae289977f3c1c5ff9ef0fd/src/python/pants/pantsd/pants_daemon.py#L199 or it could be run here with this environment: https://github.com/pantsbuild/pants/blob/2476a4149e90d9a627ae289977f3c1c5ff9ef0fd/src/python/pants/pantsd/process_manager.py#L562 tbh, i don't really understand the code well enough to understand why --no-pantsd succeeds and --pantsd doesn't. and why this only happens on CircleCI
Perhaps this is a related discussion on what env variables are preserved for pantsd: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1684784924506069 @witty-crayon-22786 not sure if this is relevant to you, but when pantsd calls git, it does so in an unsandboxed manner. there are no env vars that can be set to affect git afaict. ssh-related environment variables can impact git’s ability to fetch from remote. The Pants Changed subsystem will run git diff for “—changed-since”, which can result in fetches of trees and blobs on demand from remote if one has a blobless or treeless git clone
e
@victorious-zebra-49010 it looks like you nailed it. Thanks for digging on the issue.
v
i'll file an issue shortly
2 Views
Open in Slack
PreviousNext
Powered by