https://pantsbuild.org/ logo
#general
Title
# general
r

rapid-bird-79300

08/17/2021, 1:18 AM
Testing out
poetry_requirements
and seeing an issue when building with
git+ssh
on private repositories. Seeing errors like
ERROR: Could not find a version that satisfies the requirement privatelib (unavailable)
and
ERROR: No matching distribution found for privatelib (unavailable)
. Anyone see something similar or have suggestion on how to debug this?
h

hundreds-father-404

08/17/2021, 1:37 AM
All the macro does is parses pyproject.toml to create python_requirement_library targets. It won't handle thinks like setting up custom indexes First step is to check that the requirement string is what you expect. You can run
./pants dependencies --type=3rdparty ::
to see it
👀 1
r

rapid-bird-79300

08/17/2021, 9:25 PM
hmm I think it looks correct. For example:
privatelib@ <git+ssh://git@github.com/my-org/privatelib#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>
🤔
I have to remove the space between
@
and
git+ssh
but it works with a basic pip install command…not sure if that space can be the reason
here’s a full debug log from a local run: https://gist.github.com/njgrisafi/54f099ab1776c7e22210fe24b5a5af62 it really seems like it’s doing the right thing.
h

hundreds-father-404

08/17/2021, 9:31 PM
hm how come you have to remove the space? Pep 440 says it should be fine https://www.python.org/dev/peps/pep-0440/#direct-references
r

rapid-bird-79300

08/17/2021, 9:32 PM
I see this
Copy code
$ pip install django-mongoengine@ <git+ssh://git@github.com/Rippling/django-mongoengine#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>
ERROR: Invalid requirement: 'django-mongoengine@'
WARNING: You are using pip version 21.1.2; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/njgrisafi/.pyenv/versions/3.7.10/envs/aws-scripts-py3/bin/python3.7 -m pip install --upgrade pip' command.
👍 1
prob just cli parsing
h

hundreds-father-404

08/17/2021, 9:33 PM
this is what looks most relevant
Command errored out with exit status 128: git clone -q 'ssh://****@github.com/Rippling/django-mongoengine'
1
r

rapid-bird-79300

08/17/2021, 9:33 PM
adding quotes allows to install with space
h

hundreds-father-404

08/17/2021, 9:34 PM
r

rapid-bird-79300

08/17/2021, 9:34 PM
we don’t require proxy for our github tho, shouldn’t credentials be okay?
I think this issue is related to what I’m experiencing here: https://github.com/pantsbuild/pants/issues/11790
r

rapid-bird-79300

08/17/2021, 9:59 PM
It does not seem to be a permission denied exception like the docs states. It’s just saying it can’t find the version
h

hundreds-father-404

08/17/2021, 10:09 PM
Hm another step for debugging could be trying to install directly w/ Pex,
pex ' django-mongoengine@ <git+ssh://git@github.com/Rippling/django-mongoengine#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>'
. (I personally install
pex
with
pipx
). If that is working, then we only need to narrow down the difference from Pants invoking Pex
👀 1
r

rapid-bird-79300

08/17/2021, 10:12 PM
was able to reproduce this without using a private repository: https://github.com/njgrisafi/pants-example/tree/git-ssh-example
👀 1
running
./pants lint app/app.py
results in
Copy code
Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.lint.lint
  in pants.core.goals.lint.enrich_lint_results
  in pants.backend.python.lint.pylint.rules.pylint_lint
  in pants.backend.python.lint.pylint.rules.pylint_lint_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.1rc1_py37/lib/python3.7/site-packages/pants/engine/process.py", line 262, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 1 requirement: requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e>' failed with exit code 1.
stdout:

stderr:
WARNING: Discarding git+ssh://****@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e. Command errored out with exit status 128: git clone -q 'ssh://****@github.com/psf/requests' /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executionWLq8p2/.tmp/pip-download-ppq3qehq/requests_41bd534af3bf44ecacab730f90dd0f38 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement requests (unavailable)
ERROR: No matching distribution found for requests (unavailable)
pid 45572 -> /Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/77324398bdc90c65aabdf6af9d93e49cf4c6afeb/24c964c00add511df98351239d3d6f06cedf58dc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/njgrisafi/.cache/pants/named_caches/pex_root --log /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executionWLq8p2/.tmp/tmpi_lyho5q/pip.log download --dest /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executionWLq8p2/.tmp/tmpt95hzutj/usr.local.Cellar.python@3.9.3.9.6.Frameworks.Python.framework.Versions.3.9.bin.python3.9 requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e> --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
curious if you have the same issue on that branch (
git-ssh-example
)
I didn’t see any errors with the pex command:
Copy code
$ pex 'django-mongoengine@ <git+ssh://git@github.com/Rippling/django-mongoengine#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>'
/Users/njgrisafi/.pyenv/versions/3.7.10/envs/aws-scripts-py3/lib/python3.7/site-packages/pex/tools/commands/venv.py:141: PEXWarning: Encountered collision building venv at /Users/njgrisafi/.pex/venvs/short/231bdab5 from /Users/njgrisafi/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc:
1. /Users/njgrisafi/.pex/venvs/2a89f6e1881e5fafca055d973675d77a45986d75/cc48858524bf3820a737c19c7f14d57d4a5c4208.b5157ee01aa341749cb6ec83654ed0c4/lib/python3.7/site-packages/constraints.txt was provided by:
	/Users/njgrisafi/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/setuptools/constraints.txt
	/Users/njgrisafi/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/wheel/constraints.txt
  pex_warnings.warn(message)

Python 3.7.10 (default, Mar 17 2021, 11:08:29)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
I can import
django_mongoengine
in that shell just fine too
👍 1
h

hundreds-father-404

08/17/2021, 10:29 PM
Hm I do not reproduce using
requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e>
on `pantsbuild/pants`'s
main
branch. We're using default options for most of
[python-setup]
and
[python-repos]
have you maybe configured
[python-repos]
or anything that might be relevant? feel free to DM panst.toml if it's private
r

rapid-bird-79300

08/17/2021, 10:30 PM
yeah same we mostly use default option. This
pants.toml
is failing for me on the example branch: https://github.com/njgrisafi/pants-example/blob/git-ssh-example/pants.toml and that’s as simple as it gets
debug logs here for the example repo install requests: https://gist.github.com/njgrisafi/d0cebde9337266fc64168dd44ddfaf95
not sure if there’s any tells in there
okay just confirmed this only happening on my laptop. Not sure why this is happening tho
👀 1
h

hundreds-father-404

08/18/2021, 11:22 PM
using Requests as the example, right? I'm a little out of my depth on why it wouldn't be working @happy-kitchen-89482 any ideas based on your certs work why PEP 440 requirement would be failing to git clone w/ this?
Command errored out with exit status 128: git clone -q 'ssh://****@github.com/Rippling/django-mongoengine'
r

rapid-bird-79300

08/18/2021, 11:28 PM
yeah using the same requests example fails on my machine too.
Copy code
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 1 requirement: requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e>' failed with exit code 1.
stdout:

stderr:
WARNING: Discarding git+ssh://****@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e. Command errored out with exit status 128: git clone -q 'ssh://****@github.com/psf/requests' /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-execution4s0UPx/.tmp/pip-download-_85e7ts9/requests_4b9188b2af1d4823b52b0d8ecd412733 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement requests (unavailable)
ERROR: No matching distribution found for requests (unavailable)
pid 24640 -> /Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/77324398bdc90c65aabdf6af9d93e49cf4c6afeb/24c964c00add511df98351239d3d6f06cedf58dc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/njgrisafi/.cache/pants/named_caches/pex_root --log /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-execution4s0UPx/.tmp/tmpa88fq3g6/pip.log download --dest /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-execution4s0UPx/.tmp/tmpzmo6inf6/usr.local.Cellar.python@3.9.3.9.6.Frameworks.Python.framework.Versions.3.9.bin.python3.9 requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e> --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
h

happy-kitchen-89482

08/19/2021, 3:37 AM
Huh, this doesn't ring a bell...
Do you have some custom ssh setup in ~/.ssh ?
👍 1
r

rapid-bird-79300

08/19/2021, 3:46 PM
I did have a
~/.ssh/config
with the following contents:
Copy code
Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_ed25519
I tried removing and restarting the agent but still have the issue.
h

hundreds-father-404

08/19/2021, 3:52 PM
What about maybe your global git config?
👀 1
r

rapid-bird-79300

08/19/2021, 3:54 PM
🤔
Copy code
$ cat ~/.gitconfig
[user]
	signingkey = D162AA5FD24B8702
	name = Nick
	email = <mailto:ngrisafi@rippling.com|ngrisafi@rippling.com>
[submodule]
	recurse = true
okay I just ran into this again after switching repos and testing a plugin:
Copy code
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 1 requirement: pantsbuild.pants==2.6.1rc1' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.6.1rc1
ERROR: No matching distribution found for pantsbuild.pants==2.6.1rc1
pid 81228 -> /Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/77324398bdc90c65aabdf6af9d93e49cf4c6afeb/ccc61217db289adfd45201fae89a0868ad7824b7/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/njgrisafi/.cache/pants/named_caches/pex_root --log /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executiond0sAQD/.tmp/tmpa5sfmt62/pip.log download --dest /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executiond0sAQD/.tmp/tmp4bc3zx0s/Users.njgrisafi..pyenv.versions.3.6.10.bin.python3.6 pantsbuild.pants==2.6.1rc1 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
very strange it happened in the venv but not the other.
h

hundreds-father-404

08/24/2021, 12:27 AM
python3.6 is the culprit
1
this is a bit awkward: your plugin code has to within Py37-39, even if your own code isn't
👀 1
r

rapid-bird-79300

09/02/2021, 12:28 AM
picked this back up and unable to get this to work on my machine. Pretty sure it’s something related to how my laptop is configured but can use some help to debug. I tried setting interpreter compatibility but still seeing the error when install deps. Also confirmed this is happening for both Python 3.6 and 3.9.
h

hundreds-father-404

09/02/2021, 12:31 AM
It would help to get a run with
-ldebug --pex-verbosity=3
, then copy that output into a gist. warning it will be long. Also, running
./pants py-constraints ::
(https://www.pantsbuild.org/docs/python-interpreter-compatibility#tip-activate-pantsbackendpythonmixed_interpreter_constraints)
👀 1
r

rapid-bird-79300

09/02/2021, 1:08 AM
I wonder if this is an issue:
Copy code
error: could not expand include path '~/.gitcinclude'
  fatal: bad config line 44 in file /usr/local/git/etc/gitconfig
wow that was it actually! I changed any
~
->
/Users/njgrisafi
in my
/usr/local/git/etc/gitconfig
and now this works!
how did I end up in this state not sure but thank you for all the help @hundreds-father-404!
h

hundreds-father-404

09/02/2021, 1:36 AM
Uh that's weird. Cc @witty-crayon-22786. I can look next week if you don't beat me
👍 1
w

witty-crayon-22786

09/02/2021, 7:25 PM
hm… very funky.
is
gitcinclude
supposed to have that
c
in it? have never seen that before.
oooh. whoa. that’s include , not
ignore
. sorry about that, makes sense.
i can only guess that your git version was out of alignment with that config file, or that you made edits at some point without noticing their impacts
r

rapid-bird-79300

09/03/2021, 7:09 PM
setting up a new laptop today (Macos) and ran into this same issue. Updating the config fixed it
5 Views