https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all, could anyone give me any hint on what coul...
# general
l
Hi all, could anyone give me any hint on what could I do to debug this problem? At work we're using a private Python index (devpi) that fails when resolving ruff because it seems it tries to compile it from source instead of downloading a wheel, but it works fine if: • I use official pypi • I force the wheel in 
pants.toml
it works fine:
Copy code
[python.resolves_to_only_binary]
python-default = ["ruff"]
The original error I'm getting is this:
Copy code
pants generate-lockfiles --resolve=python-default                                                   [18:26:03]
18:27:06.62 [INFO] Completed: Generate lockfile for python-default
18:27:06.62 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 3665 -> /Users/anler/.cache/pants/named_caches/pex_root/venvs/839dd85d778df5927a872ace6e92e15b4f301d3d/287fbd6663e1bc9ef762f521460cbf66ca0b5bb1/bin/python -sE /Users/anler/.cache/pants/named_caches/pex_root/venvs/839dd85d778df5927a872ace6e92e15b4f301d3d/287fbd6663e1bc9ef762f521460cbf66ca0b5bb1/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /Users/anler/.cache/pants/named_caches/pex_root/pip_cache --log /private/var/folders/h5/cd67_q5j2zs7s7lc9dr_72fc0000gn/T/pants-sandbox-WtglAE/.tmp/pex-pip-log.sm_lr2j4/pip.log download --dest /Users/anler/.cache/pants/named_caches/pex_root/downloads/fingerprint_artifact.3ze1waeg --no-deps https://<company devpi>/root/pypi/%2Bf/1b0/9f29b16c6ead5/ruff-0.1.6.tar.gz --index-url https://<company devpi>/+simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
I've been comparing the responses of both mirrors looking for differences but I cannot find any 😞
b
You might need to dig into the commands pants is running to work out what is going on, e.g. using 
--keep-sandboxes=on_failure
and then playing around with the 
__run.sh
script in it.
l
That has been very useful. It turns out that turning the -vv option for pex I'm seeing traces of this error:
Copy code
super(HTTPSConnection, self).__init__(host, port, timeout,
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 857, in __init__
    (self.host, self.port) = self._get_hostport(host, port)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 900, in _get_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 900, in _get_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
http.client.InvalidURL: nonnumeric port: '****@pypi.mycompany.com'
So it seems that my private devpi URL 
"<https://user:password@devpi.mycompany.com/simple/>"
cannot be parsed by Python's http client, has anyone had issues with this?
b
I've run into this issue. There are two ways we've dealt with it. The first is to use an API token which doesn't contain special characters. This works for us as our repo is artifactory based, so there is an easy way to generate those. the other way of doing this is via .netrc. Once again, you can't have some characters in it (like spaces), but that works too.
l
Hi @blue-city-97042 I've been trying to make it work via .netrc but it seems to not being picked up by pex/pip (but if I use standalone pip it does work)
Copy code
pex: Fetching PEP-691 index metadata from <https://pypi.company.com/+simple/pytest/> for application/vnd.pypi.simple.v1+json :: Fetching PEP-691 index metadata from <https://pypi.company.com/+simple/coverage/> for application/vnd.pypi.simple.v1+json :: Fetching PEP-691 index metadata from <https://pypi.rollbar.tools/rollbar/pex>: Fetching PEP-691 index metadata from <https://pypi.company.com/+simple/prompt-toolkit/> for application/vnd.pypi.simple.v1+json :: Fetching PEP-691 index metadata from <https://pypi.company.com/+simple/mypy-extensions/> for application/vnd.pypi.simple.v1+json :: Fetching PEP-691 index metadata from <https://pypi.rollbarTraceback> (most recent call last):ggy/ for application/vnd.pypi.simple.v1+json
Have you done something else than creating the ~/.netrc file?
b
generate-lockfiles worked once we had the .netrc created. Did you set the host to the proper machine
l
I think I did because standalone pip is working fine:
Copy code
machine <http://pypi.mycompany.com|pypi.mycompany.com>
login read
password 1234
but running 
pants generate-lockfiles
fails 😞
b
and you removed the entry for the urls in the pants.toml, correct?
what do you have in your pip.conf?
l
hmm what do you mean? I have this set in pants.tom
Copy code
[python-repos]
indexes = ["<https://pypi.mycompany.com/subpath/+simple/>"]
no pip.conf nope
b
is the '+simple' what you expect in the url?
l
yes, it has the + for some reason 😅
b
try encoding the '+' instead of the bare '+'
l
No luck, it looks like whatever pex is doing is not using the credentials in .netrc file, is this possible?
Copy code
./__run.sh                                                                                                [18:28:42]
pex: Resolving interpreters :: Resolving for:
pex: Calculating compatible python versions for ('>=3.11',): 9.7ms
pex: Hashing pex: 312.9ms
pex: Isolating pex: 0.1ms
pex: Resolving interpreters :: Resolving for:
pex: Found 0 fingerprints cached in database.
pex: Resolving interpreters :: Resolving for:
  /Users/anler/.pyenv/versions/3.11.4/bin/python3.11 :: Making 40 PEP-691 JSON API requests across 10 threads to fingerpex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/execnet/> for applicationpex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/prompt-toolkit/> for applpex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/parso/> for application/vpex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/matplotlib-inline/> for apex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/pure-eval/> for applicatipex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/pytest-xdist/> for applicpex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/pytest/> for application/pex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/typing-extensions/> for apex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/click/> for application/vpex: Fetching PEP-691 index metadata from <https://pypi.company.com/subpath/%2Bsimple/black/> for application/vTraceback (most recent call last):
  File "/Users/anler/.cache/pants/named_caches/pex_root/installed_wheels/a060ee4f080f4e681abec7f26b26f05a9b58b0ad572c3cf94d8e394d2c99ba95/pex-2.1.137-py2.py3-none-any.whl/pex/resolve/pep_691/api.py", line 72, in request
    with self._url_fetcher.get_body_stream(
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/Users/anler/.cache/pants/named_caches/pex_root/installed_wheels/a060ee4f080f4e681abec7f26b26f05a9b58b0ad572c3cf94d8e394d2c99ba95/pex-2.1.137-py2.py3-none-any.whl/pex/fetcher.py", line 109, in get_body_stream
    fp = cast(BinaryIO, opener.open(request, timeout=self._timeout))
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 555, in error
    result = self._call_chain(*args)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden060ee4f080f4e681abec7f26b26f05a9b58b0ad572c3cf94d8e394d2c99ba95
b
are you using a docker image to do the build?
or straight from your workspace?
l
nope, my intel mac
b
the pip command uses the exact same stuff pex is using as under the hood pex is just calling pip.
what version of pants are you using?
l
2.18
b
lemme double check my configs really quick
yeah, just using a .netrc in my home directory. indexes are defined in the indexes. The only other thing is that I have a pip.conf for some other stuff too.
but from what I've seen of the pex, it just sets the indexes via the --extra-index-url command line flag
l
yes this is what it is sending in my case: 
--no-pypi $'--index=<https://pypi.company.com/subpath/%2Bsimple/>'
and running pip in my shell works fine:
Copy code
$ pip install --no-cache --dry-run --index-url=$PYPI_INDEX company-protos                           [18:43:32]
Looking in indexes: <https://read>:****@pypi.company.com/subpath/+simple/
Collecting rollbar-protos
  Downloading <https://pypi.company.com/subpath/%2Bf/de6/afa594fed1fbf/company_protos-0.23.8-py2.py3-none-any.whl> (112 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.5/112.5 kB 414.6 kB/s eta 0:00:00
Collecting grpcio (from company-protos)
  Downloading <https://pypi.company.com/root/pypi/%2Bf/cb4/e9cbd9b7388fc/grpcio-1.59.3-cp311-cp311-macosx_10_10_universal2.whl> (9.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.6/9.6 MB 17.6 MB/s eta 0:00:00
Collecting protobuf (from company-protos)
  Downloading <https://pypi.company.com/root/pypi/%2Bf/0bf/384e75b92c428/protobuf-4.25.1-cp37-abi3-macosx_10_9_universal2.whl> (394 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 394.2/394.2 kB 96.8 MB/s eta 0:00:00
Would install grpcio-1.59.3 protobuf-4.25.1 company-protos-0.23.8
so it must be something with my pants configuration or something
I think I got it. Playing with invoking the pex directly by modifying the command in __run.sh I've noticed that 
--pip-version vendored
or 
--pip-version 20.3.4-patched
works. It seems that I cannot use 
pip_version = "vendored"
in pants.tom, but using 
pip_version = "20.3.4-patched"
works like a charm! @blue-city-97042 sorry to bother you again, could you tell me which pip version are you using?
b
is the latest specified by the pip versions for pex.
I can't remember which one. Also, I am using the latest possible version of the pex-cli
l
I'll try upgrading to the latest pex to see if it works with the latest pip also, thanks!
h
Pants has its own pip version default, independent of pex's (pex is conservative about such changes, Pants is not). Pants is currently at pip==23.1.2.
b
👍
l
@happy-kitchen-89482 is that version the one that is changed whenever I use
Copy code
[python]
...
pip_version = "20.3.4-patched"
in pants.toml?
for me anything above the 20.3.4-patched caused 403 errors I guess because it wasn't source the ~/.netrc file
Just tried again upgrading pex-cli and pip to the latest versions and still having issues:
Copy code
[python]
pip_version = "latest"

[pex-cli]
version = "v2.1.143"
known_versions = [
  "v2.1.143|macos_arm64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291",
  "v2.1.143|macos_x86_64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291",
  "v2.1.143|linux_x86_64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291",
  "v2.1.143|linux_arm64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291"
]
This is the pex invocation inside `__run.sh`:
Copy code
PEX_SCRIPT=pex3 /.../venvs/2.18.0/bin/python3.9 ./pex lock create --tmpdir .tmp --python-path $'...' $'--output=lock.json' --no-emit-warnings $'--style=universal' --pip-version latest --resolver-version pip-2020-resolver --target-system linux --target-system mac $'--indent=2' --no-pypi $'--index=<https://pypi.mycompany.com/subpath/+simple/>' $'--find-links=/Users/anler/src/monorepo/python_wheels' --manylinux manylinux2014 $'--path-mapping=WHEELS_DIR|/Users/anler/src/monorepo/python_wheels' --interpreter-constraint $'CPython>=3.11' $'pytest==7.4.3' ...
and if I run it again with verbose options 
-vv
I can see 403 error traces:
b
Hey @lively-gpu-26436 I haven't seen your pants.toml but the way I solved this on JFrog is to add this to the 
[python-repos]
section, along with the environment variables. I also used the API key as suggested above to avoid special characters:
Copy code
[python-repos]
indexes = [
    "https://%(env.JFROG_USER)s:%(env.JFROG_API_KEY)s@my.jfrog.io/artifactory/api/pypi/libs-local/simple",
    "<https://pypi.org/simple>",
]
Here are the docs that I followed https://www.pantsbuild.org/docs/reference-python-repos#indexes
❤️ 1
h
yes, that setting is how you control the underlying pip version from within Pants
👌 1
l
Hi @breezy-mouse-20493 that's what I had initially:
Copy code
[python-repos]
indexes = ["<https://user:pass@pypi.company.com/path/+simple/>"]
find_links = ["%(buildroot)s/python_wheels"]
path_mappings = ["WHEELS_DIR|%(buildroot)s/python_wheels"]
but with that our resolving went from <2min with pypi, up to 12min, and trying to debug it is how I found that issue is multiple requests sent to our devpi are failing and retried multiple times until after 12min errors completely. Then, when going into the sandbox and adding 
-vv
to the pex command inside __run.sh, I get this error logged multiple times:
Copy code
File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 900, in _get_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
Traceback (most recent call last):
and my suspicion is that somehow that repo url gets passed to python's http.client.HTTPConnection which doesn't support the user:password@host notation, since it considers the stuff after the colon a port value
this is the full beginning of the log:
Copy code
anler@dynames /private/var/folders/h5/cd67_q5j2zs7s7lc9dr_72fc0000gn/T/pants-sandbox-aKqt3u
$ ./__run.sh                                                          [0:54:18]
pex: Resolving interpreters :: Resolving for:
pex: Calculating compatible python versions for ('>=3.11',): 14.7ms
pex: Hashing pex: 417.7ms
pex: Isolating pex: 0.1ms
pex: Resolving interpreters :: Resolving for:
pex: Found 0 fingerprints cached in database.
pex: Resolving interpreters :: Resolving for:
  /Users/anler/.pyenv/versions/3.11.3/bin/python3.11 :: Making 37 PEP-691 JSON API requests across 10 threads to fingerpex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/parso/ fpex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/traitletpex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/pygmentspex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/asttokenpex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/future/ pex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/pathspecpex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/iniconfipex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/ptyprocepex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/setuptoopex: Fetching PEP-691 index metadata from <https://username>****@pypi.company.com/path/+simple/decoratoTraceback (most recent call last):.v1+json
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 895, in _get_hostport
    port = int(host[i+1:])
Traceback (most recent call last):
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 895, in _get_hostport
    port = int(host[i+1:])
ValueError: invalid literal for int() with base 10: '****@pypi.company.com'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/anler/.cache/pants/named_caches/pex_root/installed_wheels/a060ee4f080f4e681abec7f26b26f05a9b58b0ad572c3cf94d8e394d2c99ba95/pex-2.1.137-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/Users/anler/.cache/pants/named_caches/pex_root/installed_wheels/a060ee4f080f4e681abec7f26b26f05a9b58b0ad572c3cf94d8e394d2c99ba95/pex-2.1.137-py2.py3-none-any.whl/pex/resolve/pep_691/api.py", line 72, in request
    with self._url_fetcher.get_body_stream(
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/Users/anler/.cache/pants/named_caches/pex_root/installed_wheels/a060ee4f080f4e681abec7f26b26f05a9b58b0ad572c3cf94d8e394d2c99ba95/pex-2.1.137-py2.py3-none-any.whl/pex/fetcher.py", line 109, in get_body_stream
    fp = cast(BinaryIO, opener.open(request, timeout=self._timeout))
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 895, in _get_hostport
    port = int(host[i+1:])
ValueError: invalid literal for int() with base 10: '****@pypi.company.com'
Traceback (most recent call last):
Traceback (most recent call last):
ValueError: invalid literal for int() with base 10: '****@pypi.company.com'
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/urllib/request.py", line 1315, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
Traceback (most recent call last):
  File "/Users/anler/Library/Caches/nce/82231cb77d4a5c8081a1a1d5b8ae440abe6993514eb77a926c826e9a69a94fb1/cpython-3.9.18+20231002-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/http/client.py", line 895, in _get_hostport
    port = int(host[i+1:])
ValueError: invalid literal for int() with base 10: '****@pypi.company.com'
Traceback (most recent call last):

During handling of the above exception, another exception occurred:
@happy-kitchen-89482 ☝️
h
Which log is this? what is in your netrc and what did you set the pip version to?
l
After pants generate-locks --resolve=my-resolve fails, it points me to the sandbox:
Copy code
00:53:25.56 [INFO] Preserving local process execution dir /private/var/folders/h5/cd67_q5j2zs7s7lc9dr_72fc0000gn/T/pants-sandbox-aKqt3u for Generate lockfile for python-dev
Inside of it are these files
Copy code
anler@dynames /private/var/folders/h5/cd67_q5j2zs7s7lc9dr_72fc0000gn/T/pants-sandbox-aKqt3u
$ ls -a                                                                                                      [1:09:49]
.                        .cache                   .tmp                     pex
..                       .python-build-standalone __run.sh
__run.sh contains:
Copy code
anler@dynames /private/var/folders/h5/cd67_q5j2zs7s7lc9dr_72fc0000gn/T/pants-sandbox-aKqt3u
$ cat __run.sh

#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /private/var/folders/h5/cd67_q5j2zs7s7lc9dr_72fc0000gn/T/pants-sandbox-aKqt3u
env -i CPPFLAGS= ... PEX_SCRIPT=pex3 /Users/anler/Library/Caches/nce/.../bindings/venvs/2.18.0/bin/python3.9 ./pex lock create --tmpdir .tmp --python-path $'/Users/anler/.pyenv/versions/2.7.18/bin:/Users/anler/.pyenv/versions/3.11.3/bin:/Users/anler/.pyenv/versions/3.11.4/bin:/usr/local/opt/mysql-client/bin:/Users/anler/.pyenv/shims:...' $'--output=lock.json' --no-emit-warnings $'--style=universal' --pip-version 23.1.2 --resolver-version pip-2020-resolver --target-system linux --target-system mac $'--indent=2' --no-pypi $'--index=<https://user:password@pypi.company.com/path/+simple/>' $'--find-links=/Users/anler/src/monorepo/python_wheels' --manylinux manylinux2014 $'--path-mapping=WHEELS_DIR|/Users/anler/src/monorepo/python_wheels' -vv --interpreter-constraint $'CPython>=3.11' black docformatter $'future==0.18.3' ipdb $'pytest-cov==4.1.0' $'pytest-xdist==3.5.0' $'pytest==7.4.3' ruff setuptools wheel
and I just change that file to add the 
-vv
flag to pex invocation. The logs then come from running that file again: 
./__run.sh
and my .netrc looks like this:
Copy code
$ cat ~/.netrc                                                                                               [1:10:26]
machine <http://pypi.company.com|pypi.company.com>
login user
password passwd
But note that I'm talking about the case where I configure the indexes to contain the url that contains the user and password embedded like 
indexes = ["https://%(env.DEVPI_USER)s:%(env.DEVPI_API_KEY)<mailto:s@pypi.company.com|s@pypi.company.com>/path/+simple/"]
. Because of the error I pointed above about the 
Invalid literal for int()
I started thinking on a way of authenticating without embedding the credentials in the url, and this when I ended up with 
indexes = ["<https://pypi.company.com/path/+simple/>"]
and the credentials in the ~/.netrc file, which again failed but this time the logs I saw showed 403 Forbidden errors, and the only way I got passed that was by setting 
pip_version = "20.3.4-patched"
don't know, maybe I could try to set a test devpi mirror with credentials I can share, and try to come up with an example project that reproduces it and try to explain the whole thing better
h
Yeah, I think the issue is less the explanation, you've been pretty clear there, and provided good detail, and more that I can't debug without a reproduction...
l
I'm trying to create a reproduction environment but I'm failing hard at it, I just ran a local devpi and using the 
indexes = ["<http://users:jPZfUKHZ0N6aWqpWjdhwjUsBm2K80kis@localhost:3141/root/pypi/>"]
along with 
pip_version = "latest"
works without any issues, but as soon as I change that back to 
indexes = ["<https://user:passwd@pypi.company.com/path/+simple/>"]
it fails 😵‍💫
but there's a twist, if I write the port explicitly like: 
indexes = ["<https://user:passwd@pypi.company.com:443/path/+simple/>"]
and downgrade pip with 
pip_version = "2.3.4-patched"
it all works 🤣
anyways, thanks a lot for the help. I'll keep trying to find the root cause but at least I'm not blocked by this anymore 🙂
w
Has it not struck anyone that the index URL is not a URL? It's a weird half-breed html tag thing.
b
We haven't missed an obvious problem like that. The weirdness looks to be an artifact of the linen mirror (first image), it's not visible on Slack (second).
32 Views
Open in Slack
PreviousNext
Powered by