https://pantsbuild.org/ logo
Join SlackCommunities
Powered by
Testing out `poetry_requirements` and seeing an is...
# general
r
Testing out 
poetry_requirements
and seeing an issue when building with 
git+ssh
on private repositories. Seeing errors like 
ERROR: Could not find a version that satisfies the requirement privatelib (unavailable)
and 
ERROR: No matching distribution found for privatelib (unavailable)
. Anyone see something similar or have suggestion on how to debug this?
h
All the macro does is parses pyproject.toml to create python_requirement_library targets. It won't handle thinks like setting up custom indexes First step is to check that the requirement string is what you expect. You can run 
./pants dependencies --type=3rdparty ::
to see it
👀 1
r
hmm I think it looks correct. For example: 
privatelib@ <git+ssh://git@github.com/my-org/privatelib#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>
🤔
I have to remove the space between 
@
and 
git+ssh
but it works with a basic pip install command…not sure if that space can be the reason
here’s a full debug log from a local run: https://gist.github.com/njgrisafi/54f099ab1776c7e22210fe24b5a5af62 it really seems like it’s doing the right thing.
h
hm how come you have to remove the space? Pep 440 says it should be fine https://www.python.org/dev/peps/pep-0440/#direct-references
r
I see this
Copy code
$ pip install django-mongoengine@ <git+ssh://git@github.com/Rippling/django-mongoengine#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>
ERROR: Invalid requirement: 'django-mongoengine@'
WARNING: You are using pip version 21.1.2; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/njgrisafi/.pyenv/versions/3.7.10/envs/aws-scripts-py3/bin/python3.7 -m pip install --upgrade pip' command.
👍 1
prob just cli parsing
h
this is what looks most relevant
Command errored out with exit status 128: git clone -q 'ssh://****@github.com/Rippling/django-mongoengine'
➕ 1
r
adding quotes allows to install with space
h
r
we don’t require proxy for our github tho, shouldn’t credentials be okay?
I think this issue is related to what I’m experiencing here: https://github.com/pantsbuild/pants/issues/11790
h
r
It does not seem to be a permission denied exception like the docs states. It’s just saying it can’t find the version
h
Hm another step for debugging could be trying to install directly w/ Pex, 
pex ' django-mongoengine@ <git+ssh://git@github.com/Rippling/django-mongoengine#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>'
. (I personally install 
pex
with 
pipx
). If that is working, then we only need to narrow down the difference from Pants invoking Pex
👀 1
r
was able to reproduce this without using a private repository: https://github.com/njgrisafi/pants-example/tree/git-ssh-example
👀 1
running 
./pants lint app/app.py
results in
Copy code
Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.lint.lint
  in pants.core.goals.lint.enrich_lint_results
  in pants.backend.python.lint.pylint.rules.pylint_lint
  in pants.backend.python.lint.pylint.rules.pylint_lint_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.1rc1_py37/lib/python3.7/site-packages/pants/engine/process.py", line 262, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 1 requirement: requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e>' failed with exit code 1.
stdout:

stderr:
WARNING: Discarding git+ssh://****@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e. Command errored out with exit status 128: git clone -q 'ssh://****@github.com/psf/requests' /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executionWLq8p2/.tmp/pip-download-ppq3qehq/requests_41bd534af3bf44ecacab730f90dd0f38 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement requests (unavailable)
ERROR: No matching distribution found for requests (unavailable)
pid 45572 -> /Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/77324398bdc90c65aabdf6af9d93e49cf4c6afeb/24c964c00add511df98351239d3d6f06cedf58dc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/njgrisafi/.cache/pants/named_caches/pex_root --log /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executionWLq8p2/.tmp/tmpi_lyho5q/pip.log download --dest /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executionWLq8p2/.tmp/tmpt95hzutj/usr.local.Cellar.python@3.9.3.9.6.Frameworks.Python.framework.Versions.3.9.bin.python3.9 requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e> --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
curious if you have the same issue on that branch (
git-ssh-example
)
I didn’t see any errors with the pex command:
Copy code
$ pex 'django-mongoengine@ <git+ssh://git@github.com/Rippling/django-mongoengine#6a4a3ecbcfe0f739c51794957877cfd3348cb34d>'
/Users/njgrisafi/.pyenv/versions/3.7.10/envs/aws-scripts-py3/lib/python3.7/site-packages/pex/tools/commands/venv.py:141: PEXWarning: Encountered collision building venv at /Users/njgrisafi/.pex/venvs/short/231bdab5 from /Users/njgrisafi/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc:
1. /Users/njgrisafi/.pex/venvs/2a89f6e1881e5fafca055d973675d77a45986d75/cc48858524bf3820a737c19c7f14d57d4a5c4208.b5157ee01aa341749cb6ec83654ed0c4/lib/python3.7/site-packages/constraints.txt was provided by:
	/Users/njgrisafi/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/setuptools/constraints.txt
	/Users/njgrisafi/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/wheel/constraints.txt
  pex_warnings.warn(message)

Python 3.7.10 (default, Mar 17 2021, 11:08:29)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
I can import 
django_mongoengine
in that shell just fine too
👍 1
h
Hm I do not reproduce using 
requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e>
on `pantsbuild/pants`'s 
main
branch. We're using default options for most of 
[python-setup]
and 
[python-repos]
have you maybe configured 
[python-repos]
or anything that might be relevant? feel free to DM panst.toml if it's private
r
yeah same we mostly use default option. This 
pants.toml
is failing for me on the example branch: https://github.com/njgrisafi/pants-example/blob/git-ssh-example/pants.toml and that’s as simple as it gets
debug logs here for the example repo install requests: https://gist.github.com/njgrisafi/d0cebde9337266fc64168dd44ddfaf95
not sure if there’s any tells in there
okay just confirmed this only happening on my laptop. Not sure why this is happening tho
👀 1
h
using Requests as the example, right? I'm a little out of my depth on why it wouldn't be working @happy-kitchen-89482 any ideas based on your certs work why PEP 440 requirement would be failing to git clone w/ this?
Command errored out with exit status 128: git clone -q 'ssh://****@github.com/Rippling/django-mongoengine'
r
yeah using the same requests example fails on my machine too.
Copy code
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 1 requirement: requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e>' failed with exit code 1.
stdout:

stderr:
WARNING: Discarding git+ssh://****@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e. Command errored out with exit status 128: git clone -q 'ssh://****@github.com/psf/requests' /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-execution4s0UPx/.tmp/pip-download-_85e7ts9/requests_4b9188b2af1d4823b52b0d8ecd412733 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement requests (unavailable)
ERROR: No matching distribution found for requests (unavailable)
pid 24640 -> /Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/77324398bdc90c65aabdf6af9d93e49cf4c6afeb/24c964c00add511df98351239d3d6f06cedf58dc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/njgrisafi/.cache/pants/named_caches/pex_root --log /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-execution4s0UPx/.tmp/tmpa88fq3g6/pip.log download --dest /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-execution4s0UPx/.tmp/tmpzmo6inf6/usr.local.Cellar.python@3.9.3.9.6.Frameworks.Python.framework.Versions.3.9.bin.python3.9 requests@ <git+ssh://git@github.com/psf/requests#b0e025ade7ed30ed53ab61f542779af7e024932e> --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
h
Huh, this doesn't ring a bell...
Do you have some custom ssh setup in ~/.ssh ?
👍 1
r
I did have a 
~/.ssh/config
with the following contents:
Copy code
Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_ed25519
I tried removing and restarting the agent but still have the issue.
h
What about maybe your global git config?
👀 1
r
🤔
Copy code
$ cat ~/.gitconfig
[user]
	signingkey = D162AA5FD24B8702
	name = Nick
	email = <mailto:ngrisafi@rippling.com|ngrisafi@rippling.com>
[submodule]
	recurse = true
okay I just ran into this again after switching repos and testing a plugin:
Copy code
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 1 requirement: pantsbuild.pants==2.6.1rc1' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.6.1rc1
ERROR: No matching distribution found for pantsbuild.pants==2.6.1rc1
pid 81228 -> /Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/77324398bdc90c65aabdf6af9d93e49cf4c6afeb/ccc61217db289adfd45201fae89a0868ad7824b7/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/njgrisafi/.cache/pants/named_caches/pex_root --log /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executiond0sAQD/.tmp/tmpa5sfmt62/pip.log download --dest /private/var/folders/bq/srpk91nj1b38053_9j7gqvpc0000gn/T/process-executiond0sAQD/.tmp/tmp4bc3zx0s/Users.njgrisafi..pyenv.versions.3.6.10.bin.python3.6 pantsbuild.pants==2.6.1rc1 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
very strange it happened in the venv but not the other.
h
python3.6 is the culprit
➕ 1
this is a bit awkward: your plugin code has to within Py37-39, even if your own code isn't
👀 1
r
picked this back up and unable to get this to work on my machine. Pretty sure it’s something related to how my laptop is configured but can use some help to debug. I tried setting interpreter compatibility but still seeing the error when install deps. Also confirmed this is happening for both Python 3.6 and 3.9.
h
It would help to get a run with 
-ldebug --pex-verbosity=3
, then copy that output into a gist. warning it will be long. Also, running 
./pants py-constraints ::
(https://www.pantsbuild.org/docs/python-interpreter-compatibility#tip-activate-pantsbackendpythonmixed_interpreter_constraints)
👀 1
r
I wonder if this is an issue:
Copy code
error: could not expand include path '~/.gitcinclude'
  fatal: bad config line 44 in file /usr/local/git/etc/gitconfig
wow that was it actually! I changed any 
~
-> 
/Users/njgrisafi
in my 
/usr/local/git/etc/gitconfig
and now this works!
how did I end up in this state not sure but thank you for all the help @hundreds-father-404!
h
Uh that's weird. Cc @witty-crayon-22786. I can look next week if you don't beat me
👍 1
w
hm… very funky.
is 
gitcinclude
supposed to have that 
c
in it? have never seen that before.
oooh. whoa. that’s include , not 
ignore
. sorry about that, makes sense.
i can only guess that your git version was out of alignment with that config file, or that you made edits at some point without noticing their impacts
r
setting up a new laptop today (Macos) and ran into this same issue. Updating the config fixed it
5 Views
Open in Slack
PreviousNext
Powered by