https://pantsbuild.org/ logo
Join Slack
Powered by
Hey guys, trying to add a private git repo as a de...
# general
t
Hey guys, trying to add a private git repo as a dependency using poetry_requirements and seeing issues when I want to install add a package from a subdirectory. my poetry dependencies:
Copy code
[tool.poetry]
name = "hermes"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
pants-test-repo-2 = {git = "git@github.com:MyOrg/pants-test-repo.git", subdirectory = "my-package"}
pants-test-repo = {git = "git@github.com:MyOrg/pants-test-repo.git"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
I am running 
pants generate-lockfiles
to test it. And when I include pants-test-repo-2 I get errors:
Copy code
STDERR:
  WARNING: Generating metadata for package pants-test-repo-2 produced metadata for project name pants-test-repo. Fix your #egg=pants-test-repo-2 fragments.
WARNING: Discarding git+ssh://****@github.com/MyOrg/pants-test-repo.git. Requested pants-test-repo from git+ssh://****@github.com/MyOrg/pants-test-repo.git has different name in metadata: 'pants-test-repo'
ERROR: Could not find a version that satisfies the requirement pants-test-repo-2 (unavailable)
ERROR: No matching distribution found for pants-test-repo-2 (unavailable)
So looks like pants is only looking at the project root directory and ignoring the 'subdirectory' option, is there a way to override it in the BUILD file or specify it otherwise?
the pants-test-repo looks like:
Copy code
.
├── README.md
├── my-package
│   ├── README.md
│   ├── pants_test_repo_2
│   │   ├── __init__.py
│   │   └── main.py
│   ├── poetry.lock
│   └── pyproject.toml
├── pants_test_repo
│   ├── __init__.py
│   └── main.py
├── poetry.lock
└── pyproject.toml
I was able to generate-lockfiles by adding a python_requirements field to my BUILD file:
Copy code
poetry_requirements()

python_requirement(
    name="pants-test-repo-2",
    requirements=["pants-test-repo-2@ <git+ssh://git@github.com>:MyOrg/pants-test-repo.git@main#subdirectory=my-package"],
)
But it would be great if I could have just the poetry_requirements field and define the path there, not quite sure how to do it from the docs in https://www.pantsbuild.org/docs/python-third-party-dependencies
right after posting my comment, tried this and it worked:
Copy code
poetry_requirements(
    overrides={
        "pants-test-repo-2": {"requirements": ["pants-test-repo-2@ <git+ssh://git@github.com>:MyOrg/pants-test-repo.git@main#subdirectory=my-package"]}
    },
)
which is perfect for now, just looks like automatic inference does not work for: git repo subdirectory dependencies defined in poetry pyproject.toml file
Another question related to using private git dependency, I am able to execute 
pants run ...
goal with my local environment, however when I try to package it using a docker_environment I get ssh errors:
Copy code
pants package monitor_runner:: 
13:16:39.45 [INFO] Completed: Building 7 requirements for monitor_runner.monitor_runner/main@environment=linux_docker.pex from the python-default.lock resolve: aiokafka<0.9.0,>=0.8.1, aioredis<3.0.0,>=2.0.1, pants-test-repo-2@ git+ssh:... (179 characters truncated)
13:16:39.45 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building 7 requirements for monitor_runner.monitor_runner/main@environment=linux_docker.pex from the python-default.lock resolve: aiokafka<0.9.0,>=0.8.1, aioredis<3.0.0,>=2.0.1, pants-test-repo-2@ <git+ssh://git@github.com/MyOrg/pants-test-repo.git@main#subdirectory=my-package>, pandas<3.0.0,>=2.0.3, protobuf<5.0.0,>=4.23.4, pydantic-settings<3.0.0,>=2.0.2, pymongo<5.0.0,>=4.4.1' failed with exit code 1.
stdout:

stderr:
There was 1 error downloading required artifacts:
1. pants-test-repo-2 0.1 from <git+ssh://git@github.com/MyOrg/pants-test-repo.git@main#subdirectory=my-package>"
    pid 144 -> /pants-named-caches/pex_root/venvs/31fa86ae3546b96473ed1ff124014930432b64ee/cba9b10390762edd539c75c9f4b40c564335d72d/bin/python -sE /pants-named-caches/pex_root/venvs/31fa86ae3546b96473ed1ff124014930432b64ee/cba9b10390762edd539c75c9f4b40c564335d72d/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /pants-named-caches/pex_root/pip_cache download --dest /pants-named-caches/pex_root/downloads/resolver_download.td1wr01a/usr.local.bin.python3.10 --no-binary :all: --no-deps pants-test-repo-2 @ <git+ssh://git>+<ssh://git@github.com/MyOrg/pants-test-repo.git@main#subdirectory=my-package> --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  ERROR: Command errored out with exit status 128:
   command: git clone -q 'ssh://****@github.com/MyOrg/pants-test-repo.git' /pants-named-caches/pex_root/pip_cache/.tmp/pip-download-g8vrr632/pants-test-repo-2
       cwd: None
  Complete output (5 lines):
  Host key verification failed.
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.
  ----------------------------------------
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/MyOrg/pants-test-repo.git' /pants-named-caches/pex_root/pip_cache/.tmp/pip-download-g8vrr632/pants-test-repo-2 Check the logs for full command output.
I looked through related threads and GitHub issues but could not resolve, I added 
env_vars
to pants.toml and BUILD:
Copy code
# pants.toml
[subprocess-environment]
env_vars.add = [
  "SSH_AUTH_SOCK",
  "SSH_AGENT_PID",
]
Copy code
# BUILD
local_environment(name="local_osx", compatible_platforms=["macos_arm64"])

docker_environment(
    name="linux_docker",
    platform="linux_arm64",
    image="arm64v8/python:3.10",
    python_bootstrap_search_path=["<PATH>"],
    subprocess_environment_env_vars=["SSH_AUTH_SOCK", "SSH_AGENT_PID"],
    docker_env_vars=[
        "SSH_AUTH_SOCK",
        "SSH_AGENT_PID",
        ]
)
any help on this greatly appreciated, I'm sure it's resolved in one of the related issues, but jus can't piece it together
e
Close. Instead of 
[subprocess-environment]
try `[docker]`: https://www.pantsbuild.org/docs/reference-docker#env_vars The 
subprocess-environment
has a generic sounding name but is Python-subprocess-only IIRC. A legacy wart.
t
added that as well now:
Copy code
# pants.toml

[docker]
env_vars = ["SSH_AUTH_SOCK", "SSH_AGENT_PID"]
identical output was produced from 
pants package ...
also the main error is slightly different then in the docs:
Copy code
Host key verification failed.
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.
e
Ok, I'll leave further help to others. I'm heading AFK.
4 Views
Open in Slack
PreviousNext
Powered by