https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all, I've been using Pants off and on for a fe...
# general
p
Hey all, I've been using Pants off and on for a few weeks now at work. I was previously using version 2.6.1rc3 without issue. I just tried updating to pants version 2.7.0 so I could try out some of the new features, however I got the following error when trying to use ./pants --version to bootstrap my pants 2.7.0 install:
Copy code
./pants --version
15:48:17.17 [ERROR] 1 Exception encountered:

  Exception: Error downloading file: error sending request for url (<https://github-releases.githubusercontent.com/22072746/f2e54b89-00b6-4e73-bfdd-1edea9f28a45?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210930T204704Z&X-Amz-Expires=300&X-Amz-Signature=43ad57850d558c35c2b87b4a1299c8ab5627359de7ad7f59b9cbf5f2e473755d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=22072746&response-content-disposition=attachment%3B%20filename%3Dpex&response-content-type=application%2Foctet-stream>): error trying to connect: invalid certificate: UnknownIssuer


(Use --print-stacktrace for more error details and/or --no-process-execution-local-cleanup to inspect chroots and/or -ldebug for more logs. See <https://www.pantsbuild.org/v2.7/docs/troubleshooting> for common issues. Consider reaching out for help: <https://www.pantsbuild.org/v2.7/docs/getting-help.>)
I tried switching my pants.toml to point back to 2.6.1rc3 and that is now throwing the same error. I have tried deleting my ~/.cache/pants directory without any success. Can someone help point me in the right direction?
f
I had encountered a similar issue in another context in trying to 
curl
a GitHub URL for the release of a project. It was a bug in GitHub’s generation of AWS signed URLs. I reported it to their support system and they fixed.
Given the presence of AWS S3 signature fields, I am making the guess it could be a similar issue on Github’s systems.
although 
error trying to connect: invalid certificate: UnknownIssuer
is a difference from my own issue
can you run with 
-ldebug
? I’d like to see what it is trying to download since that URL does not give a clue.
p
I am behind a corporate proxy with a custom certificate, however to my knowledge none of that setup has changed
f
actually I’m wrong, the URL has “pex” in there so probably a pex download
Copy code
filename%3Dpex&response-content-type=application%2Foctet-stream):
w
--print-stacktrace
will give you a bit more
p
standby, slack seems to be having issues pasting the entire output
This seems to be the last thing it tried to do before throwing the error:
Copy code
16:03:50.42 [DEBUG] Starting: pants.backend.python.util_rules.pex.build_pex
16:03:50.42 [DEBUG] Starting: pants.backend.python.util_rules.pex.build_pex_component
16:03:50.42 [DEBUG] Starting: pants.core.util_rules.external_tool.download_external_tool
16:03:50.42 [DEBUG] Starting: Downloading: DownloadFile(url='<https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>', expected_digest=FileDigest(fingerprint='41f15d08ae5f5e9e8fe25ea4a1f9ee4fc59e342accd5a9d94bdb376d2cc102e... (37 characters truncated)
16:03:50.83 [DEBUG] Completed: Downloading: DownloadFile(url='<https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>', expected_digest=FileDigest(fingerprint='41f15d08ae5f5e9e8fe25ea4a1f9ee4fc59e342accd5a9d94bdb376d2cc102e... (37 characters truncated)
16:03:50.83 [DEBUG] Completed: pants.core.util_rules.external_tool.download_external_tool
16:03:50.83 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex_component
16:03:50.83 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
16:03:50.83 [DEBUG] computed 1 nodes in 0.482784 seconds. there are 70 total nodes.
16:03:50.84 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex_component (pants_plugins.pex)
  in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
  in pants.backend.python.util_rules.pex_cli.download_pex_pex
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Error downloading file: error sending request for url (<https://github-releases.githubusercontent.com/22072746/f2e54b89-00b6-4e73-bfdd-1edea9f28a45?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210930T210213Z&X-Amz-Expires=300&X-Amz-Signature=900e1e31fb8a2bf8520f50524ab3ce4014fbf3e47561d536e39c7095477ecd2b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=22072746&response-content-disposition=attachment%3B%20filename%3Dpex&response-content-type=application%2Foctet-stream>): error trying to connect: invalid certificate: UnknownIssuer
Traceback (most recent call last):
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/bin/daemon_pants_runner.py", line 124, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/pantsd/pants_daemon_core.py", line 129, in prepare
    build_config, options = self._options_initializer.build_config_and_options(
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/init/options_initializer.py", line 107, in build_config_and_options
    build_config = _initialize_build_configuration(
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/init/options_initializer.py", line 47, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper, env)
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/init/plugin_resolver.py", line 123, in resolve
    for resolved_plugin_location in self._resolve_plugins(options_bootstrapper, env):
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/init/plugin_resolver.py", line 146, in _resolve_plugins
    session.product_request(ResolvedPluginDistributions, [self._request])[0],
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 562, in product_request
    self._raise_on_error([t for _, t in throws])
  File "/home/brian/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 497, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex_component (pants_plugins.pex)
  in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
  in pants.backend.python.util_rules.pex_cli.download_pex_pex
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Error downloading file: error sending request for url (<https://github-releases.githubusercontent.com/22072746/f2e54b89-00b6-4e73-bfdd-1edea9f28a45?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210930T210213Z&X-Amz-Expires=300&X-Amz-Signature=900e1e31fb8a2bf8520f50524ab3ce4014fbf3e47561d536e39c7095477ecd2b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=22072746&response-content-disposition=attachment%3B%20filename%3Dpex&response-content-type=application%2Foctet-stream>): error trying to connect: invalid certificate: UnknownIssuer

(Use --no-process-execution-local-cleanup to inspect chroots. See <https://www.pantsbuild.org/v2.7/docs/troubleshooting> for common issues. Consider reaching out for help: <https://www.pantsbuild.org/v2.7/docs/getting-help.>)
I was able to successfully download that file using my browser as well
f
does your corporate proxy do anything to modify TLS? like insert its own certificate chain? https://github.com/pantsbuild/pants/issues/12000 might be relevant then.
p
I was able to previously use pants 2.6.1.rc3 behind this proxy
h
Since it sounds like reverting back to 2.6.1.rc3 showed the same error, I'm wondering if this is caused by some issue with the interaction with that proxy that may have occurred in the past, but you're only hitting it now because of the upgrade, and previously you happened to have an already-bootstrapped cached copy of pex
👍 1
The issue @fast-nail-55400 pointed to may be relevant - your browser, and cUrl, probably use openssl, which is more tolerant than rustls of technically invalid certs (and that issue points out that at least one well-known SSL proxy rewrites certs invalidly)
OTOH this did work for you at one point, and it seems unlikely that the proxy was made less standards-conformant over time...
I suppose it's possible that it happens to sometimes generate invalid certs and other times not (by "invalid" I mean "technically nonconformant, but accepted by openssl)
So one way to diagnose this would be to run 
curl -L -O <https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>
twice (from behind the proxy), once with openssl as the TLS backend and once with rustls. To do so, you'll probably have to build curl with both those backends: • Install [rust](https://rustup.rs/) if necessary • In a tmpdir: ◦ 
cargo install cbindgen
◦ 
git clone <https://github.com/rustls/rustls-ffi> -b v0.7.0
◦ 
cd rustls-ffi
◦ 
make
◦ 
make DESTDIR=${HOME}/rustls-ffi-built/ install
• In another tmpdir: ◦ 
git clone <https://github.com/curl/curl> -b curl-7_79_0
◦ 
cd curl
◦ 
./buildconf
◦ 
LIBS=-lm ./configure --with-rustls=${HOME}/rustls-ffi-built --with-openssl --prefix ${HOME}/curlbin
◦ 
make
◦ 
make install
Now you can run
Copy code
CURL_SSL_BACKEND=openssl ~/curlbin/bin/curl -L -O <https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>
vs
Copy code
CURL_SSL_BACKEND=rustls ~/curlbin/bin/curl -L -O <https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex|https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>
This all worked for me in a docker container (created from the 
gcc
image)
So
Copy code
$ docker pull gcc
$ docker run -ti gcc bash
But it should work on raw MacOS too
So when you get a chance if you could run all that and see if 
rustls
is indeed the issue, that would be super helpful
p
Thanks for the input @happy-kitchen-89482 I'll try that today
h
PS the instructions are based on https://github.com/curl/curl/blob/master/docs/RUSTLS.md but with some modifications I had to add because those instructions are broken as-is
Such as needing to link the math library with -lm
And it's even more broken at HEAD, so working off the release tag is crucial
p
I've been messing around with this for the last few hours, I followed your above instructions inside of a gcc:latest docker container. When I tried downloading the specified file with curl, both backend successfully downloaded the pex file. So I then decided to try installing pants inside of the docker container, to see if that was now working as well. I made a dummy pants.toml file as follows:
Copy code
[GLOBAL]
pants_version = "2.7.0"


[subprocess-environment]
env_vars.add = ["http_proxy", "https_proxy"]
Pants was able to install successfully with both version 2.7.0 and 2.6.1. I then decided to try copying my actual project into this docker container as well. With my actual project, I got the same error UnknownIssuer error that I originally reported.
So next I tried turning off my corporate vpn/proxy configs (I'm working from home today) on my machine. I was then able to install pants within my project with version 2.7.0 and 2.6.1
h
Oh boy, OK
p
However I seem to having another unrelated issue with 2.7.0 when I try to create a repl in one of my python library targets:
Copy code
./pants repl libs/redfish_models/::
11:57:01.26 [INFO] Completed: Resolving constraints.txt
11:57:03.08 [INFO] Completed: Extracting pydantic<2.0.0,>=1.8.2 from repository.pex
11:57:03.50 [INFO] Completed: Composing 1 requirement to build requirements.pex from repository.pex
Traceback (most recent call last):
  File "/home/brian/git/compute-platform-api/.pants.d/tmpnmkq2ovz/requirements.pex/.bootstrap/pex/pex.py", line 482, in execute
  File "/home/brian/git/compute-platform-api/.pants.d/tmpnmkq2ovz/requirements.pex/.bootstrap/pex/pex.py", line 139, in activate
  File "/home/brian/git/compute-platform-api/.pants.d/tmpnmkq2ovz/requirements.pex/.bootstrap/pex/pex.py", line 125, in _activate
  File "/home/brian/git/compute-platform-api/.pants.d/tmpnmkq2ovz/requirements.pex/.bootstrap/pex/pex.py", line 109, in _loaded_envs
  File "/home/brian/git/compute-platform-api/.pants.d/tmpnmkq2ovz/requirements.pex/.bootstrap/pex/pex_info.py", line 103, in from_pex
FileNotFoundError: [Errno 2] No such file or directory: '__reqs/pydantic<2.0.0,>=1.8.2.pex/PEX-INFO'
2.6.1 seems to be fully working after installing it with my proxy off
so I'm able to continue developing
however, not an ideal solution for the rest of my team to always have to install pants from off the corporate network
h
Oh yes, clearly we must fix this
But at least you're not immediately blocked
That unrelated issue, as a guess, do you have 
pydantic<2.0.0,>=1.8.2
in your 
constraints.txt
?
constraints.txt
files must contain only exact requirements (
pydantic==1.8.2
for example)
p
pydantic==1.8.2; python_full_version >= "3.6.1"
that's whats in my constraints.txt
h
Hmmmmmmmmmm
@enough-analyst-54434 thoughts on this one? The 
pydantic
issue that is?
p
I'm using the poetry_requirements in my top level BUILD
did something change in poetry_requirements in 2.7.0?
e
Well, this is the one and only release series that used the composition of pexs via PEX_PATH trick and that's implicated in the backtrace.
Just a sec for a pants_version to try to help narrow if that's it...
@powerful-addition-15861 do you mind trying 
2.8.0.dev1
? If that works, that leaves the PEX_PATH thing as a likely suspect, if not, it might help narrow in a bit.
p
Trying now
I'm now getting the following:
Copy code
./pants repl libs/redfish_models/::
12:06:22.24 [INFO] Completed: Resolving constraints.txt
12:06:22.68 [INFO] Completed: Extracting 1 requirement to build requirements.pex from repository.pex: pydantic<2.0.0,>=1.8.2
Traceback (most recent call last):
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 288, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 632, in _activate
    resolved = self.resolve()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 468, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 474, in resolve_dists
    self._update_candidate_distributions(self._load_internal_cache())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 245, in _update_candidate_distributions
    ranked_dist = self._can_add(dist)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 255, in _can_add
    filename, ext = os.path.splitext(os.path.basename(dist.location))
AttributeError: 'NoneType' object has no attribute 'location'
e
@powerful-addition-15861 does constraints.txt have an entry for pydantic?
Ah sorry - answered above.
Aha, right. So pydantic is platform-specific. This must mean requirements were resolved for one platform and repl is using another. Say resolve for 3.8, repl on 3.7. Trying to sort the easiest way to prove this / debug...
@powerful-addition-15861 can you re-run the 
repl
command with 
--no-process-execution-local-cleanup
? That will give us a chroot to poke around in. I'll then want you to run a few more commands.
p
sorry was away from my computer
back now
I ran it with that flag
h
It should have logged a path to the sandbox directory in which in ran the resolving process?
e
The last sandbox path logged should be good enough.
p
Here is the output:
Copy code
./pants --no-process-execution-local-cleanup repl libs/redfish_models/::
12:49:42.98 [INFO] preserving local process execution dir `"/tmp/process-executiony7toEg"` for "Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin"
12:49:42.99 [INFO] preserving local process execution dir `"/tmp/process-executionnLKVRN"` for "Test binary /bin/bash."
12:49:42.99 [INFO] preserving local process execution dir `"/tmp/process-executionttZp5k"` for "Test binary /usr/bin/bash."
12:49:43.01 [INFO] preserving local process execution dir `"/tmp/process-executionTHfgSx"` for "Searching for `python2` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
12:49:43.01 [INFO] preserving local process execution dir `"/tmp/process-executionz3RktZ"` for "Searching for `unzip` on PATH=/usr/bin:/bin:/usr/local/bin"
12:49:43.01 [INFO] preserving local process execution dir `"/tmp/process-executioncLibf3"` for "Searching for `python` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
12:49:43.01 [INFO] preserving local process execution dir `"/tmp/process-executionKKkmkC"` for "Searching for `python3` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
12:49:43.01 [INFO] preserving local process execution dir `"/tmp/process-executionftk8fg"` for "Searching for `tar` on PATH=/usr/bin:/bin:/usr/local/bin"
12:49:43.01 [INFO] preserving local process execution dir `"/tmp/process-execution3W7jfp"` for "Test binary /bin/unzip."
12:49:43.01 [INFO] preserving local process execution dir `"/tmp/process-executionfClXxj"` for "Test binary /usr/bin/unzip."
12:49:43.02 [INFO] preserving local process execution dir `"/tmp/process-executionJIf3ij"` for "Test binary /usr/bin/tar."
12:49:43.02 [INFO] preserving local process execution dir `"/tmp/process-executionVbVghD"` for "Test binary /bin/tar."
12:49:43.02 [INFO] preserving local process execution dir `"/tmp/process-execution7i0m0h"` for "Test binary /bin/python3."
12:49:43.02 [INFO] preserving local process execution dir `"/tmp/process-executionDvGqJh"` for "Test binary /usr/bin/python3."
12:49:43.04 [INFO] preserving local process execution dir `"/tmp/process-executionJGRsqd"` for "Find interpreter for constraints: CPython>=3.9"
Traceback (most recent call last):
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 288, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 632, in _activate
    resolved = self.resolve()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 468, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 474, in resolve_dists
    self._update_candidate_distributions(self._load_internal_cache())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 245, in _update_candidate_distributions
    ranked_dist = self._can_add(dist)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 255, in _can_add
    filename, ext = os.path.splitext(os.path.basename(dist.location))
AttributeError: 'NoneType' object has no attribute 'location'
e
You'll need to throw in a 
--no-process-execution-local-cache
too it looks like. This will be slower and redo lots of work but you should get a more useful last sandbox.
p
heading into a meeting quick, can run that in a bit
e
Ok, thanks.
p
Copy code
./pants --no-process-execution-local-cleanup --no-process-execution-local-cache repl libs/redfish_models/::
Traceback (most recent call last):
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 288, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 632, in _activate
    resolved = self.resolve()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 468, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 474, in resolve_dists
    self._update_candidate_distributions(self._load_internal_cache())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 245, in _update_candidate_distributions
    ranked_dist = self._can_add(dist)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 255, in _can_add
    filename, ext = os.path.splitext(os.path.basename(dist.location))
AttributeError: 'NoneType' object has no attribute 'location'
e
Hrm. Also add in 
--no-pantsd
.
We try really hard to do no work!
p
Copy code
./pants --no-process-execution-local-cleanup --no-process-execution-local-cache --no-pantsd repl libs/redfish_models/::
13:28:24.54 [WARN] File handle limit is capped to: 4096. To avoid 'too many open file handle' errors, we recommend a limit of at least 10000: please see <https://www.pantsbuild.org/docs/troubleshooting#too-many-open-files-error> for more information.
13:28:24.66 [INFO] preserving local process execution dir `"/tmp/process-execution9Fc0pw"` for "Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin"
13:28:24.67 [INFO] preserving local process execution dir `"/tmp/process-executionOO7cU0"` for "Test binary /usr/bin/bash."
13:28:24.67 [INFO] preserving local process execution dir `"/tmp/process-executioniusFQ4"` for "Test binary /bin/bash."
13:28:24.67 [INFO] preserving local process execution dir `"/tmp/process-executionOYcnEN"` for "Searching for `python` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
13:28:24.67 [INFO] preserving local process execution dir `"/tmp/process-executionhyNqHx"` for "Searching for `python2` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
13:28:24.67 [INFO] preserving local process execution dir `"/tmp/process-executionh0QyAh"` for "Searching for `python3` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
13:28:24.68 [INFO] preserving local process execution dir `"/tmp/process-executionydgBpi"` for "Test binary /bin/python3."
13:28:24.68 [INFO] preserving local process execution dir `"/tmp/process-executionfSiB26"` for "Test binary /usr/bin/python3."
13:28:24.70 [INFO] preserving local process execution dir `"/tmp/process-executionzTyLuJ"` for "Searching for `tar` on PATH=/usr/bin:/bin:/usr/local/bin"
13:28:24.71 [INFO] preserving local process execution dir `"/tmp/process-execution0jS6Fq"` for "Searching for `unzip` on PATH=/usr/bin:/bin:/usr/local/bin"
13:28:24.71 [INFO] preserving local process execution dir `"/tmp/process-executionlFEXvE"` for "Test binary /usr/bin/unzip."
13:28:24.71 [INFO] preserving local process execution dir `"/tmp/process-executionCyMcO2"` for "Test binary /usr/bin/tar."
13:28:24.71 [INFO] preserving local process execution dir `"/tmp/process-executionka5QnN"` for "Test binary /bin/tar."
13:28:24.71 [INFO] preserving local process execution dir `"/tmp/process-executionEYQVkb"` for "Test binary /bin/unzip."
13:28:24.71 [INFO] Starting: Resolving plugins: GitPython
13:28:24.71 [INFO] preserving local process execution dir `"/tmp/process-executionIc4GHo"` for "Resolving plugins: GitPython"
13:28:25.84 [INFO] Completed: Resolving plugins: GitPython
13:28:25.84 [INFO] preserving local process execution dir `"/tmp/process-executioneoibf0"` for "Extracting plugin locations"
13:28:26.01 [WARN] File handle limit is capped to: 4096. To avoid 'too many open file handle' errors, we recommend a limit of at least 10000: please see <https://www.pantsbuild.org/docs/troubleshooting#too-many-open-files-error> for more information.
13:28:26.26 [INFO] preserving local process execution dir `"/tmp/process-execution3HngNh"` for "Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin"
13:28:26.27 [INFO] preserving local process execution dir `"/tmp/process-executionrxO446"` for "Test binary /usr/bin/bash."
13:28:26.27 [INFO] preserving local process execution dir `"/tmp/process-executionFlqhcq"` for "Test binary /bin/bash."
13:28:26.29 [INFO] preserving local process execution dir `"/tmp/process-executionf0fidg"` for "Searching for `python3` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
13:28:26.29 [INFO] preserving local process execution dir `"/tmp/process-execution6Y9Cg2"` for "Searching for `tar` on PATH=/usr/bin:/bin:/usr/local/bin"
13:28:26.29 [INFO] preserving local process execution dir `"/tmp/process-executionhl0b06"` for "Searching for `unzip` on PATH=/usr/bin:/bin:/usr/local/bin"
13:28:26.29 [INFO] preserving local process execution dir `"/tmp/process-execution8UESgo"` for "Searching for `python` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
13:28:26.29 [INFO] preserving local process execution dir `"/tmp/process-executionDu5j3G"` for "Searching for `python2` on PATH=/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin"
13:28:26.30 [INFO] preserving local process execution dir `"/tmp/process-execution1DVRrc"` for "Test binary /bin/tar."
13:28:26.30 [INFO] preserving local process execution dir `"/tmp/process-executionskB8Fs"` for "Test binary /bin/unzip."
13:28:26.30 [INFO] preserving local process execution dir `"/tmp/process-executionI8QsJr"` for "Test binary /usr/bin/tar."
13:28:26.30 [INFO] preserving local process execution dir `"/tmp/process-executionmx6gVu"` for "Test binary /usr/bin/unzip."
13:28:26.30 [INFO] preserving local process execution dir `"/tmp/process-executionOC63Ds"` for "Test binary /bin/python3."
13:28:26.30 [INFO] preserving local process execution dir `"/tmp/process-executionsWJrMP"` for "Test binary /usr/bin/python3."
13:28:26.33 [INFO] preserving local process execution dir `"/tmp/process-executionqtY9AF"` for "Find interpreter for constraints: CPython>=3.9"
13:28:27.12 [INFO] preserving local process execution dir `"/tmp/process-executionjPGiW5"` for "Determine Python imports for libs/redfish_models/redfish_models/root.py"
13:28:27.13 [INFO] preserving local process execution dir `"/tmp/process-executionlSF5tY"` for "Determine Python imports for libs/redfish_models/redfish_models/status.py"
13:28:27.13 [INFO] preserving local process execution dir `"/tmp/process-execution64ONL1"` for "Determine Python imports for libs/redfish_models/redfish_models/base.py"
13:28:27.13 [INFO] preserving local process execution dir `"/tmp/process-executionBkDHWH"` for "Determine Python imports for libs/redfish_models/redfish_models/session.py"
13:28:27.13 [INFO] preserving local process execution dir `"/tmp/process-executionnRc8w4"` for "Determine Python imports for libs/redfish_models/redfish_models/event.py"
13:28:27.13 [INFO] preserving local process execution dir `"/tmp/process-executionpSMRxF"` for "Determine Python imports for libs/redfish_models/redfish_models/__init__.py"
13:28:27.27 [INFO] preserving local process execution dir `"/tmp/process-executionb7PC4T"` for "Resolving constraints.txt"
13:28:30.71 [INFO] Completed: Resolving constraints.txt
13:28:30.71 [INFO] preserving local process execution dir `"/tmp/process-execution9aPuOv"` for "Extracting 1 requirement to build requirements.pex from repository.pex: pydantic<2.0.0,>=1.8.2"
13:28:31.13 [INFO] Completed: Extracting 1 requirement to build requirements.pex from repository.pex: pydantic<2.0.0,>=1.8.2
Traceback (most recent call last):
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 288, in activate
    self._activated_dists = self._activate()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 632, in _activate
    resolved = self.resolve()
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 468, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 474, in resolve_dists
    self._update_candidate_distributions(self._load_internal_cache())
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 245, in _update_candidate_distributions
    ranked_dist = self._can_add(dist)
  File "/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py", line 255, in _can_add
    filename, ext = os.path.splitext(os.path.basename(dist.location))
AttributeError: 'NoneType' object has no attribute 'location'
e
Ok, excellent. Please 
cd  /tmp/process-execution9aPuOv
and if you have 
tree
dump that output here.
p
Copy code
tree
.
├── __run.sh
├── constraints.txt
├── pex
├── repository.pex
│   ├── PEX-INFO
│   └── __main__.py
├── requirements.pex
│   ├── PEX-INFO
│   └── __main__.py
└── source_files

3 directories, 7 files
e
Ok, great. Can you provide the output of 
jq . repository.pex/PEX-INFO
? Or else just upload the file?
p
Copy code
jq . repository.pex/PEX-INFO
{
  "bootstrap_hash": "bfe889820782562c90cf1fccb3029d75e2b1a458",
  "build_properties": {
    "pex_version": "2.1.48"
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "GitPython-3.1.24-py3-none-any.whl": "77355767101276ed6ceaf9ebf9606c995ce1672b",
    "PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64.whl": "bee146b7b338f215cf12e6c28c8ece8c798ec0e5",
    "aiohttp-3.7.4.post0-cp39-cp39-manylinux2014_x86_64.whl": "8001cea875648172cf6fb9b68cd2c6cb738dcd63",
    "asgiref-3.4.1-py3-none-any.whl": "7759c06f05ee0f7bed239ee1675f65eb62f10630",
    "async_timeout-3.0.1-py3-none-any.whl": "c2eee6978705f4068a2daef43918f29c0659204f",
    "attrs-21.2.0-py2.py3-none-any.whl": "2d30ac163afe61a501f56152d991607463db80f9",
    "chardet-4.0.0-py2.py3-none-any.whl": "89dd44593cce7d569db3cc546cb47e293a8340b2",
    "click-8.0.1-py3-none-any.whl": "60cbe72bab38df1f22792bc2d8b6b3a072d02dbf",
    "fastapi-0.68.1-py3-none-any.whl": "62d9f7dfa60f60cea4cae76040fbf3c58fa4b60f",
    "gitdb-4.0.7-py3-none-any.whl": "cbb0a1a2c7f68bf77771d0c21546565b34ad655a",
    "h11-0.12.0-py3-none-any.whl": "be8485222ee3db4b7e44be71f6bef67554953af6",
    "httptools-0.2.0-cp39-cp39-manylinux1_x86_64.whl": "fa13a188daa7996bd0cb3495a41a7a2752368b84",
    "idna-3.2-py3-none-any.whl": "f6f18646cd1fcc3b8d439bd113b7cc9ad4ebc3c0",
    "multidict-5.1.0-cp39-cp39-manylinux2014_x86_64.whl": "45f765df549c49a1e6d2cd4f2cb52145df463f65",
    "pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl": "b485ce89fe715a85c562f7bfc7aa950610725456",
    "python_dotenv-0.19.0-py2.py3-none-any.whl": "5f7d2e549aa14a4a2af957e7f7492e56588e0fd8",
    "smmap-4.0.0-py2.py3-none-any.whl": "3bc43d31c55385dd9f9f8ac4df4dc9575abb93f1",
    "starlette-0.14.2-py3-none-any.whl": "edcf11d948014beb6f59506f8e6b9f905174001c",
    "typing_extensions-3.10.0.2-py3-none-any.whl": "e1248851ba7610196a1039644897ce611bd8536d",
    "uvicorn-0.15.0-py3-none-any.whl": "d7a1e9d1a1b090347efafc14ff141d69b03e283f",
    "uvloop-0.16.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl": "1089c0d91d5b8bdc850950baf5dc3981e53ac1f6",
    "watchgod-0.7-py3-none-any.whl": "7b399f7e11c74d75315b55cdc91e80291d3a17aa",
    "websockets-10.0-cp39-cp39-manylinux2010_x86_64.whl": "f314e32f46f7234023327add4b589fea483bda99",
    "yarl-1.6.3-cp39-cp39-manylinux2014_x86_64.whl": "f02d992cafe98ff312955315c7f6ee01a9b16c2d"
  },
  "emit_warnings": false,
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "51ce09df72ac75e9b8204c6581477745f9db111b",
  "pex_path": null,
  "requirements": [
    "aiohttp==3.7.4.post0; python_version >= \"3.6\"",
    "asgiref==3.4.1; python_version >= \"3.6\"",
    "async-timeout==3.0.1; python_full_version >= \"3.5.3\" and python_version >= \"3.6\"",
    "attrs==21.2.0; python_version >= \"3.6\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.5.0\" and python_version >= \"3.6\"",
    "chardet==4.0.0; python_version >= \"3.6\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.5.0\" and python_version >= \"3.6\"",
    "click==8.0.1; python_version >= \"3.6\"",
    "fastapi==0.68.1; python_version >= \"3.6\"",
    "gitdb==4.0.7; python_version >= \"3.7\"",
    "gitpython==3.1.24; python_version >= \"3.7\"",
    "h11==0.12.0; python_version >= \"3.6\"",
    "httptools==0.2.0",
    "idna==3.2; python_version >= \"3.6\"",
    "multidict==5.1.0; python_version >= \"3.6\"",
    "pydantic==1.8.2; python_full_version >= \"3.6.1\"",
    "python-dotenv==0.19.0; python_version >= \"3.5\"",
    "pyyaml==5.4.1; python_version >= \"2.7\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.6.0\"",
    "smmap==4.0.0; python_version >= \"3.7\"",
    "starlette==0.14.2; python_version >= \"3.6\"",
    "typing-extensions==3.10.0.2; python_version < \"3.10\" and python_version >= \"3.7\" and python_full_version >= \"3.6.1\"",
    "uvicorn==0.15.0",
    "uvloop==0.16.0; sys_platform != \"win32\" and sys_platform != \"cygwin\" and platform_python_implementation != \"PyPy\" and python_version >= \"3.7\"",
    "watchgod==0.7; python_version >= \"3.5\"",
    "websockets==10.0; python_version >= \"3.7\"",
    "yarl==1.6.3; python_version >= \"3.6\""
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false
}
e
Ok - so only cp39 dists.
So what does __run.sh say?
p
do you mean what is the contents of __run.sh? or do you want me to execute it
e
You can execute it but should get same failure. I'd like to see the contents.
p
Copy code
cat __run.sh
#!/bin/bash
# This command line should execute the same process as pants did internally.
export CPPFLAGS= LANG=C.UTF-8 LDFLAGS= PATH=$'/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin' PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root
cd /tmp/process-execution9aPuOv
/usr/bin/python3.9 ./pex --python-path $'/home/brian/.local/bin:/home/brian/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/223034492/AppData/Local/Microsoft/WindowsApps:/snap/bin' --tmpdir .tmp --output-file requirements.pex --pex-repository repository.pex --python /usr/bin/python3.9 --no-emit-warnings --jobs 4 --manylinux manylinux2014 $'--sources-directory=source_files' --constraints constraints.txt $'pydantic<2.0.0,>=1.8.2' --layout packed
e
Ok. Can you ``jq . requirements.pex/PEX-INFO` ?
p
Copy code
jq . requirements.pex/PEX-INFO
{
  "bootstrap_hash": "bfe889820782562c90cf1fccb3029d75e2b1a458",
  "build_properties": {
    "pex_version": "2.1.48"
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl": "b485ce89fe715a85c562f7bfc7aa950610725456",
    "typing_extensions-3.10.0.2-py3-none-any.whl": "e1248851ba7610196a1039644897ce611bd8536d"
  },
  "emit_warnings": false,
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8",
  "pex_path": null,
  "requirements": [
    "pydantic<2.0.0,>=1.8.2"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false
}
e
Ok, all looks good 😕 The interesting bit remaining is what /usr/bin/python3.9 reports for its tags. This does appear to be a wsl setup, so maybe that's interseting. It certainly affects the bit of code where the error is coming from (the interpreter tags). Please try:
Copy code
/usr/bin/python3.9 -mvenv test.venv
test.venv/bin/pip debug -v
The output of the second command will be what I'm after. It will have ~600 lines or so.
p
yes this is Ubuntu in WSL
Copy code
/usr/bin/python3.9 -mvenv test.venv
Error: Command '['/tmp/process-execution9aPuOv/test.venv/bin/python3.9', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
e
Hrm.
Ok, saving that attack for later, lets try looking at the requirement.pex layout. Can you 
tree -a requirements.pex
?
p
Copy code
tree -a requirements.pex
requirements.pex
├── .bootstrap
├── .deps
│   ├── pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl
│   └── typing_extensions-3.10.0.2-py3-none-any.whl
├── PEX-INFO
└── __main__.py

1 directory, 5 files
e
Ok, and ~same for the unzipped version of that PEX: 
ls -l /home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.deps
There should be 2 symlinks and you'll want to check those are both good symlinks.
p
Copy code
ls -l /home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.deps
total 8
lrwxrwxrwx 1 brian brian 116 Oct  1 12:06 pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl -> ../../../installed_wheels/b485ce89fe715a85c562f7bfc7aa950610725456/pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl
lrwxrwxrwx 1 brian brian 110 Oct  1 12:06 typing_extensions-3.10.0.2-py3-none-any.whl -> ../../../installed_wheels/e1248851ba7610196a1039644897ce611bd8536d/typing_extensions-3.10.0.2-py3-none-any.whl
e
Are the symlinks good? They should both point to non-empty directories with the corresponding wheel installation there.
p
Copy code
ls ../../../installed_wheels/b485ce89fe715a85c562f7bfc7aa950610725456/pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl
pydantic  pydantic-1.8.2.dist-info
Copy code
ls ../../../installed_wheels/e1248851ba7610196a1039644897ce611bd8536d/typing_extensions-3.10.0.2-py3-none-any.whl
typing_extensions-3.10.0.2.dist-info  typing_extensions.py
e
Hrm - ok, all looks in order. Back to the interpreter tags...
Can you apply this patch to 
/home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.bootstrap/pex/environment.py
?:
Copy code
$ git diff
diff --git a/pex/environment.py b/pex/environment.py
index 03b7d09..128a2bb 100644
--- a/pex/environment.py
+++ b/pex/environment.py
@@ -237,6 +237,7 @@ class PEXEnvironment(object):
         with TRACER.timed("Searching dependency cache: %s" % internal_cache, V=2):
             for distribution_name in self._pex_info.distributions:
                 dist_path = os.path.join(internal_cache, distribution_name)
+                print(f"Yielding dep {dist_path} which exists?: {os.path.exists(dist_path)}", file=sys.stderr)
                 yield DistributionHelper.distribution_from_path(dist_path)
 
     def _update_candidate_distributions(self, distribution_iter):
Then 
./__run.sh
p
Hmm, I edited that file and ran __run.sh...but got no output
e
Ok, so 
.__run.sh
completes with 0 exit code then?
p
seems that way
e
Ok, a few seconds here while I re-familiarize myself with repl chroots - they are not in the same place....
Ok, can you mix in 
-ldebug
to the 
./pants repl
command? I'd like to see what interpreter constraints Pants thinks it should be using when launching your repl.
p
still want the other args we've been doing for testing?
e
Yes - lets leave constant what we've built up to.
p
Untitled.txt
Untitled.txt
It made me make a snippet
e
Copy code
Yielding dep /home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/.deps/pex-2.1.48-py2.py3-none-any.whl which exists?: False
So the edit you added did print and it printed something unexpected. Your requirements.pex had no pex dependency last we checked looking at PEX-INFO.
So I'd like to review your two PEX-INFO jq dumps and poosibly I'll be asking for re-dos.
p
np
e
OK, let's see what this has: 
jq . /home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/PEX-INFO
p
Copy code
jq . /home/brian/.cache/pants/named_caches/pex_root/unzipped_pexes/4c72174837f01d2d26e05fcdb6fd8bc07a0c21f8/PEX-INFO
{
  "bootstrap_hash": "05dc9963564f8e8b6c67803a2bcd88a73244617a",
  "build_properties": {
    "pex_version": "2.1.48"
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "pex-2.1.48-py2.py3-none-any.whl": "332e69dd24a824a9104b3b7ec78e7e5445e54e2e"
  },
  "emit_warnings": true,
  "entry_point": "pex.bin.pex:main",
  "ignore_errors": false,
  "includes_tools": true,
  "inherit_path": "false",
  "interpreter_constraints": [
    ">=2.7,<3.10,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*"
  ],
  "pex_hash": "d49e0d878ea870dc5d0aa3593e33aeea08b9067c",
  "pex_path": null,
  "requirements": [
    "pex==2.1.48"
  ],
  "strip_pex_env": false,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false
}
e
Ok - that's wrong.
You'll note the pex_hash != to the parent dir name for one.
So, can you try moving aside the pex_root? 
mv /home/brian/.cache/pants/named_caches/pex_root /home/brian/.cache/pants/named_caches/pex_root.sav
Then re-run the 
./pants repl
.
p
I'm now getting the repl
so seems to be working
e
Ok. Thanks for sticking with me on that.
p
I have been moving between 2.6.1 and 2.7.0 quite a bit today while trying to diagnose the original issue in this thread
e
So clearly there was some bad cache entry making done by Pants / Pex at some point. I'd like to harden against that / root cause. But I think I can do that without delaying you further.
p
although I've been periodically deleting this cache
let me know if you need anymore input
e
I think this sort of thing is hardened against in ~latest Pants / Pex but Pex should log more useful information when it can't find an expected distribution like this. That I can take action on in the short term.
I should be good to go. Hopefully you are too?
p
just tried rolling back to 2.7.0 version
h
Thanks for sticking with us as we work through this @powerful-addition-15861! The debugging info is really useful
❤️ 1
p
getting that same original error FileNotFoundError: [Errno 2] No such file or directory: '__reqs/pydantic<2.0.0,>=1.8.2.pex/PEX-INFO'
should I delete the cache and try again?
e
Yeah - you can try that.
If it re-curs still, we probably have a bug in the slice of time we did the PEX_PATH thing 2.7 does.
p
yea looks like still having that issue in 2.7 after removing my ~/.cache/pants directory
e
Hrm, ok. So, I'll try to get a repro of that then. We need to find a fix we can apply to 2.7 for this.
p
and yea going back to 2.8.0.dev1 does work
(after removing cache again)
e
Do you have any viable workaround? Is 2.8.0.devX Ok or dropping back?
p
I'll go back to 2.6.1 for now
I initially moved to 2.7 with the intent of trying out the experimental docker build target
but I don't need that yet
e
Gotcha. Ok. Thanks again for your time on this. I'll link issues here once I have them filed.
p
@happy-kitchen-89482 going back to my original issue, I'll have another person on my team try installing pants behind our corporate proxy early next week to confirm that this issue is not specific to my machine
h
Great, definitely still need to diagnose and fix that one
Sounds like it's not rustls after all
p
So far I've been enjoying pants a lot, so happy to help diagnose issues for you guys
🎉 1
e
The good thing is this is easy to repro: https://github.com/pantsbuild/pants/issues/13075
👀 1
😬 1
h
@powerful-addition-15861 Do you mind opening a ticket (https://github.com/pantsbuild/pants/issues) with all the info we have so far about the SSL issue? That one might take some fixing, and it's better to track less ephemerally than on Slack.
p
I should be able to do that early next week
e
Ok, to update the 
./pants repl
issue. The fixes have been cherry-picked to 2.7.x (https://github.com/pantsbuild/pants/commit/305ab9602838c3e1092ffcb85dbd6f46af548885). I need to cherry-pick one further Pex upgrade there to avoid potential issues with unicode characters in distribution scripts, but after that I'll wrangle up a 2.7.1rc0 after consulting the other maintainers.
h
Thanks for following up!
e
h
@powerful-addition-15861 Did you have a chance to file that issue? Thanks.
5 Views
Open in Slack
PreviousNext
Powered by