https://pantsbuild.org/ logo
#general
Title
# general
p

plain-carpet-73994

09/07/2021, 4:43 PM
I'm having trouble getting my own pypi repo to work. I've set up a GCP PyPi-like simple repo and I can install packages from it fine one the command line via
pip install
. Specifically
pip install pyrealsense2==2.49.0
works but if I put that in my
constraints.txt
Pants can't install it. I re-ran the command pants was running and removed CLI options to get it down to the minimum:
Copy code
/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/oliverdain/.cache/pants/named_caches/pex_root --log /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmpfchex0j2/pip.log download --dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmp9bliehja/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7   pyrealsense2==2.49.0+companion   --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
Note that in the above I'm not using my
constraints.txt
but instead directly passing the package version I need. I get this error:
Copy code
2021-09-07T09:38:09,804 <https://us-central1-python.pkg.dev:443> "GET /infrastructure-314723/python-deps/simple/pyrealsense2/ HTTP/1.1" 200 1028
2021-09-07T09:38:09,806 Updating cache with response from "<https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>"
2021-09-07T09:38:09,834   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0
2021-09-07T09:38:09,835   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0.3313-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0.3313
2021-09-07T09:38:09,835   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:38:09,836   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:38:09,836 Given no hashes to check 0 links for project 'pyrealsense2': discarding no candidates
2021-09-07T09:38:09,836 ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0+companion
It sure looks like that package has the right tags so I don't understand the
none of the wheel's tags match
errors. Any thoughts?
This repo will eventually be private but for testing purposes I've made it public. The .whl here is a version of pyrealsense2 compiled for OSX (as Intel only provides binaries for Windows and Linux).
e

enough-analyst-54434

09/07/2021, 4:47 PM
So, https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/ has only exactly cp37m wheels - how sure are you the Pants resolve is running for CPython 3.7?
p

plain-carpet-73994

09/07/2021, 4:49 PM
There's also a Linux wheel up there but there is a cp37m wheel. Pretty sure it's running the resolve for CPython 3.7. See the following in the logs:
Copy code
2021-09-07T09:38:09,835   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
It lists the tags it's looking for and it lists what it found and they match to my eyes.
Also: I've tried it without the
+companion
suffix and get the same result.
e

enough-analyst-54434

09/07/2021, 4:52 PM
It lists the tags it's looking for and it lists what it found and they match to my eyes.
I think your interpretation is wrong. It lists the summary of the tags it found on the left of the line for the link on the right of the line.
Can you provide full log output?
There should be a command line in the output that invloves pip download - that's the key one that will reveal what Python Pants is resolving for.
p

plain-carpet-73994

09/07/2021, 4:53 PM
Full logs:
Copy code
2021-09-07T09:50:54,961 Created temporary directory: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Initialized build tracking at /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Created build tracker: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Entered build tracker: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Created temporary directory: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-download-gpowzxs5
2021-09-07T09:50:54,970 Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
2021-09-07T09:50:54,970 1 location(s) to search for versions of pyrealsense2:
2021-09-07T09:50:54,970 * <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>
2021-09-07T09:50:54,971 Fetching project page and analyzing links: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>
2021-09-07T09:50:54,971 Getting page <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>
2021-09-07T09:50:54,972 Found index url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
2021-09-07T09:50:55,233 Looking up "<https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>" in the cache
2021-09-07T09:50:55,234 Request header has "max_age" as 0, cache bypassed
2021-09-07T09:50:55,235 Starting new HTTPS connection (1): us-central1-python.pkg.dev:443
2021-09-07T09:50:55,690 <https://us-central1-python.pkg.dev:443> "GET /infrastructure-314723/python-deps/simple/pyrealsense2/ HTTP/1.1" 200 1028
2021-09-07T09:50:55,692 Updating cache with response from "<https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>"
2021-09-07T09:50:55,721   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0
2021-09-07T09:50:55,721   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0.3313-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0.3313
2021-09-07T09:50:55,722   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:50:55,722   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:50:55,722 Given no hashes to check 0 links for project 'pyrealsense2': discarding no candidates
2021-09-07T09:50:55,722 ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0
2021-09-07T09:50:55,723 ERROR: No matching distribution found for pyrealsense2==2.49.0
2021-09-07T09:50:55,723 Exception information:
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion
2021-09-07T09:50:55,723     crit = self.state.criteria[name]
2021-09-07T09:50:55,723 KeyError: 'pyrealsense2'
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 During handling of the above exception, another exception occurred:
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in resolve
2021-09-07T09:50:55,723     name, crit = self._merge_into_criterion(r, parent=None)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _merge_into_criterion
2021-09-07T09:50:55,723     crit = Criterion.from_requirement(self._p, requirement, parent)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 83, in from_requirement
2021-09-07T09:50:55,723     raise RequirementsConflicted(criterion)
2021-09-07T09:50:55,723 pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('pyrealsense2==2.49.0')
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 During handling of the above exception, another exception occurred:
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 122, in resolve
2021-09-07T09:50:55,723     requirements, max_rounds=try_to_avoid_resolution_too_deep,
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
2021-09-07T09:50:55,723     state = resolution.resolve(requirements, max_rounds=max_rounds)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 320, in resolve
2021-09-07T09:50:55,723     raise ResolutionImpossible(e.criterion.information)
2021-09-07T09:50:55,723 pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('pyrealsense2==2.49.0'), parent=None)]
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 The above exception was the direct cause of the following exception:
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
2021-09-07T09:50:55,723     status = self.run(options, args)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
2021-09-07T09:50:55,723     return func(self, options, args)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/commands/download.py", line 131, in run
2021-09-07T09:50:55,723     reqs, check_supported_wheels=True
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 127, in resolve
2021-09-07T09:50:55,723     six.raise_from(error, e)
2021-09-07T09:50:55,723   File "<string>", line 3, in raise_from
2021-09-07T09:50:55,723 pip._internal.exceptions.DistributionNotFound: No matching distribution found for pyrealsense2==2.49.0
2021-09-07T09:50:55,724 Removed build tracker: '/private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
e

enough-analyst-54434

09/07/2021, 4:54 PM
Yeah - that does not look full, just looks like pip logs.
What pants command line are you using to get this log output?
p

plain-carpet-73994

09/07/2021, 4:54 PM
Copy code
/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/oliverdain/.cache/pants/named_caches/pex_root --log /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmpfchex0j2/pip.log download --dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmp9bliehja/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7   pyrealsense2==2.49.0+companion   --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
Note that includes
--dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmp9bliehja/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7
And my
pants.toml
includes:
Copy code
[python-setup]
interpreter_constraints = ['CPython==3.7.*']
requirement_constraints = 'constraints.txt'
e

enough-analyst-54434

09/07/2021, 4:56 PM
Yup - so a macOS provided python. ... OK
p

plain-carpet-73994

09/07/2021, 4:56 PM
It's a
pyenv
built python.
e

enough-analyst-54434

09/07/2021, 4:56 PM
Nope
It is definitely not
p

plain-carpet-73994

09/07/2021, 4:57 PM
That's super strange. (1) This is a new laptop that I just set up and (2) it's big sur which only provides 3.9. I had to install pyenv and build 3.7.
e

enough-analyst-54434

09/07/2021, 4:57 PM
It's the one here: /Library/Frameworks/Python/framework/Versions/3.7/bin/python3.7
I assume pyenv does not install there.
Do you have XCode?
That installs its own above and beyond the OS
p

plain-carpet-73994

09/07/2021, 4:57 PM
Copy code
$ ls /Library/Frameworks/Python/framework/Versions/3.7/bin/python3.7
ls: /Library/Frameworks/Python/framework/Versions/3.7/bin/python3.7: No such file or directory
e

enough-analyst-54434

09/07/2021, 4:58 PM
Hrm
p

plain-carpet-73994

09/07/2021, 4:58 PM
There's a bunch of directories before
/Library/Frameworks
in that command line that I'm pretty sure correspond to the pants sandbox.
Copy code
$ type python3.7
python3.7 is /Users/oliverdain/.pyenv/shims/python3.7
I did an
xcode-select --install
which I think gives me some, but not all, of xcode stuff.
e

enough-analyst-54434

09/07/2021, 5:00 PM
Can you humor me and inspect the /Library/ ... path bit by bit? I have case insensitive fs ptsd with that lowercase library in the path I gave you.
p

plain-carpet-73994

09/07/2021, 5:01 PM
ha! Sure.
Copy code
$ ls /Library/Frameworks/Python
ls: /Library/Frameworks/Python: No such file or directory
Copy code
$ ls /Library/Frameworks/Python.framework/
Headers		Python		Resources	Versions
Copy code
$ ls /Library/Frameworks/Python.framework/Versions/
3.7	Current
e

enough-analyst-54434

09/07/2021, 5:02 PM
Aha!
p

plain-carpet-73994

09/07/2021, 5:02 PM
So it does look like there's a 3.7 there.
e

enough-analyst-54434

09/07/2021, 5:02 PM
Ok I just translated all dots to /
Ok - that is what it's using
So, switching back to pyenv
Do you have Pants configured to look for pyenv?
p

plain-carpet-73994

09/07/2021, 5:04 PM
Dunno. How do I tell?
p

plain-carpet-73994

09/07/2021, 5:04 PM
But also, why would that matter. I want it using Python3.7, the whl is built for python3.7. Do I care which 3.7 it's using?
e

enough-analyst-54434

09/07/2021, 5:04 PM
Well, you do because macOS interpreters are notorious
p

plain-carpet-73994

09/07/2021, 5:05 PM
I see. I do not have the
<PYENV>
bit in my
interpreter_search_paths
. Will add.
e

enough-analyst-54434

09/07/2021, 5:05 PM
We could spin a while and figure out what's up with that interpreter though if you want to drill in. Otherwise, pyenv setup may work better for you and others.
p

plain-carpet-73994

09/07/2021, 5:07 PM
Updates search paths and trying to re-build to see if that'll make a difference. Just curious, where'd that other 3.7 interpreter come from?
Do I need to clear the cache after adding the search paths bit?
e

enough-analyst-54434

09/07/2021, 5:09 PM
No cache clearing needed.
p

plain-carpet-73994

09/07/2021, 5:09 PM
No success either I'm afraid.
Copy code
$ ./pants run realsense/cli/ -- -h
10:06:46.68 [INFO] Initializing scheduler...
10:06:47.28 [INFO] Scheduler initialized.
10:07:45.58 [INFO] Completed: Resolving constraints.txt
10:07:45.58 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 190, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 533, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 501, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Resolving constraints.txt' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0+companion
ERROR: No matching distribution found for pyrealsense2==2.49.0+companion
pid 4666 -> /Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/oliverdain/.cache/pants/named_caches/pex_root --log /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionSppZaq/.tmp/tmptlzd7yp3/pip.log download --dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionSppZaq/.tmp/tmpwuyigplf/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7 --constraint constraints.txt absl-py==0.13.0 aiodns==3.0.0 aiofile==3.7.1 aiofiles==0.5.0 aiohttp==3.7.4.post0 aniso8601==7.0.0 astor==0.8.1 async-exit-stack==1.0.1 async-generator==1.10 async-timeout==3.0.1 attrs==21.2.0 backoff==1.10.0 bleach==4.1.0 boto3==1.18.32 botocore==1.21.32 cachecontrol==0.12.6 cachetools==4.2.2 caio==0.9.3 certifi==2021.5.30 cffi==1.14.6 chardet==3.0.4 charset-normalizer==2.0.4 chevron==0.14.0 click==7.1.2 colorama==0.4.4 cryptography==3.4.8 cycler==0.10.0 dnspython==2.1.0 docutils==0.17.1 email-validator==1.1.3 fastapi==0.68.1 firebase-admin==5.0.2 gast==0.2.2 google-api-core==1.31.2 google-api-python-client==2.19.0 google-auth-httplib2==0.1.0 google-auth==1.35.0 google-cloud-bigquery==2.25.1 google-cloud-core==2.0.0 google-cloud-firestore==2.3.1 google-cloud-storage==1.42.0 google-crc32c==1.1.3 google-pasta==0.2.0 google-resumable-media==2.0.0 googleapis-common-protos==1.53.0 graphene==2.1.9 graphql-core==2.3.2 graphql-relay==2.0.1 grpcio==1.39.0 h11==0.12.0 h2==4.0.0 h5py==2.10.0 hpack==4.0.0 httplib2==0.19.1 httptools==0.1.2 hypercorn==0.11.2 hyperframe==6.0.1 idna==3.2 importlib-metadata==4.8.1 iniconfig==1.1.1 itsdangerous==1.1.0 jeepney==0.7.1 jinja2==2.11.3 jmespath==0.10.0 keras-applications==1.0.8 keras-preprocessing==1.1.2 keyring==23.1.0 keyrings.google-artifactregistry-auth==0.0.3 kiwisolver==1.3.2 labelbox==3.2.0 markdown==3.3.4 markupsafe==2.0.1 matplotlib==3.4.3 msgpack==1.0.2 multidict==5.1.0 ndjson==0.3.1 numpy==1.18.5 opencv-python-headless==4.5.3.56 opt-einsum==3.3.0 orjson==3.6.3 packaging==21.0 pillow==8.3.1 pkginfo==1.7.1 pluggy==1.0.0 priority==2.0.0 promise==2.3 proto-plus==1.19.0 protobuf==3.17.3 py==1.10.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pycares==4.0.0 pycparser==2.20 pycryptodomex==3.10.1 pydantic==1.8.2 pygments==2.10.0 pyjwt==2.1.0 pyparsing==2.4.7 pyrealsense2==2.49.0+companion pytest-asyncio==0.15.1 pytest==6.2.5 python-dateutil==2.8.2 python-dotenv==0.19.0 python-multipart==0.0.5 pytz==2021.1 pyyaml==5.4.1 readme-renderer==29.0 requests-toolbelt==0.9.1 requests==2.26.0 rfc3986==1.5.0 rsa==4.7.2 rx==1.6.1 s3transfer==0.5.0 secretstorage==3.3.1 semantic-version==2.8.5 six==1.16.0 starlette==0.14.2 stream-chat==3.12.1 tensorboard==1.15.0 tensorflow-estimator==1.15.1 tensorflow==1.15.5 termcolor==1.1.0 toml==0.10.2 tqdm==4.62.2 twine==3.4.2 types-futures==3.3.0 types-protobuf==3.17.4 types-pyyaml==5.4.8 types-requests==2.25.6 typing-extensions==3.10.0.2 ujson==4.1.0 uritemplate==3.0.1 urllib3==1.26.6 uvicorn==0.13.4 uvloop==0.16.0 watchgod==0.7 webencodings==0.5.1 websockets==8.1 werkzeug==2.0.1 wheel==0.37.0 wrapt==1.12.1 wsproto==1.0.0 yarl==1.6.3 zipp==3.5.0 --index-url <https://pypi.org/simple/> --extra-index-url <https://us-central1-python.pkg.dev:443/infrastructure-314723/python-deps/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
None
That's the full command line that failed. It's trying to resolve lots of other dependencies but starting from there I was able to get the errors you saw. I can whittle it down again if you like.
e

enough-analyst-54434

09/07/2021, 5:10 PM
It's still using the same Python.
So, lets switch tactics temporarily.
Can you try this as a sanity check?:
Copy code
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -mvenv test.venv
test.venv/bin/pip install -U pip==20.3.4
test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
p

plain-carpet-73994

09/07/2021, 5:14 PM
interesting:
Copy code
$ test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0
ERROR: No matching distribution found for pyrealsense2==2.49.0
WARNING: You are using pip version 20.3.4; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/oliverdain/Documents/code/central/python/test.venv/bin/python3.7 -m pip install --upgrade pip' command.
e

enough-analyst-54434

09/07/2021, 5:14 PM
Ok - awesome - this just confirms borked macOS Python - as I said notorious.
p

plain-carpet-73994

09/07/2021, 5:14 PM
But if I do the equivalent with pyenv Python I'm pretty sure it works. Testing. Note: I added
<PYENV>
but it still picked up the system python.
e

enough-analyst-54434

09/07/2021, 5:14 PM
Can you also provide your relevant pants.toml snip. Most crucially, I'd guess "<PYENV>" is not 1st in the list, and that's why.
p

plain-carpet-73994

09/07/2021, 5:15 PM
Copy code
[python-setup]
  interpreter_constraints = ['CPython==3.7.*']
  requirement_constraints = 'constraints.txt'
  interpreter_search_paths = ["<PYENV>", "<PATH>"]
Looks first to me...
e

enough-analyst-54434

09/07/2021, 5:15 PM
Hrm
Msay require
pkill pantsd
- binary search paths are generally calculated just once per pants session - and with pantsd (the default) the session is long-lived.
p

plain-carpet-73994

09/07/2021, 5:17 PM
same test fails with pyenv python.
Copy code
Olivers-MBP:python oliverdain$ type python3.7
python3.7 is hashed (/Users/oliverdain/.pyenv/shims/python3.7)
Olivers-MBP:python oliverdain$ /Users/oliverdain/.pyenv/shims/python3.7 -mvenv test.venv
Olivers-MBP:python oliverdain$ test.venv/bin/pip install -U pip==20.3.4
Collecting pip==20.3.4
  Using cached pip-20.3.4-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed pip-20.3.4
Olivers-MBP:python oliverdain$ test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0
ERROR: No matching distribution found for pyrealsense2==2.49.0
Yup, I'll kill the pants daemon but per above it does not appear to be a system python issue.
e

enough-analyst-54434

09/07/2021, 5:20 PM
Putting on the breaks more - instead of using a shim in your latest experiment, please use
/Users/oliverdain/.pyenv/versions/3.7.x/bin/python
This will avoid shim unexpected redirection shenanigans and prove the pyenv interpreter works or does not.
p

plain-carpet-73994

09/07/2021, 5:21 PM
will do
restarted
pantsd
did not solve the problem.
e

enough-analyst-54434

09/07/2021, 5:22 PM
And ... do you have the same problem with pyrealsense2 2.47.0?
p

plain-carpet-73994

09/07/2021, 5:22 PM
yes
e

enough-analyst-54434

09/07/2021, 5:23 PM
OK, so gathering outstanding requests: 1. Please try .penv/versions/.../bin/python exact path with pop experiment and report 2. Please provide full output of pants run post killing of pantsd to confirm the --dest in the pip command looks like its using pyenv
p

plain-carpet-73994

09/07/2021, 5:24 PM
Copy code
$ /Users/oliverdain/.pyenv/versions/3.7.10/bin/python -mvenv test.venv
$ test.venv/bin/pip install -U pip==20.3.4
$ test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
Collecting pyrealsense2==2.49.0
  Downloading <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl>
     | 16.6 MB 891 kB/s
Installing collected packages: pyrealsense2
Successfully installed pyrealsense2-2.49.0+companion
e

enough-analyst-54434

09/07/2021, 5:24 PM
Ok - so pyenv works
p

plain-carpet-73994

09/07/2021, 5:24 PM
So it does work with pyenv 3.7!
e

enough-analyst-54434

09/07/2021, 5:24 PM
Now we just need to get Pants seeing pyenv 1st on the search path.
p

plain-carpet-73994

09/07/2021, 5:24 PM
yeah. So why don't my shims work and where did the other 3.7 come from?
e

enough-analyst-54434

09/07/2021, 5:24 PM
The other 3.7 is presumably on your PATH
p

plain-carpet-73994

09/07/2021, 5:25 PM
Yeah, very strange. I can re-run
pyenv global
to set up interpreters...
e

enough-analyst-54434

09/07/2021, 5:25 PM
which -a python3.7
p

plain-carpet-73994

09/07/2021, 5:25 PM
Copy code
$ which -a python3.7
/Users/oliverdain/.pyenv/shims/python3.7
/usr/local/bin/python3.7
Yup.
e

enough-analyst-54434

09/07/2021, 5:25 PM
which -a Python
?
p

plain-carpet-73994

09/07/2021, 5:26 PM
So per above, if I use
/Users/oliverdain/.pyenv/shims/python3.7
it fails but if I just type
python3.7
I do get the pyenv version and it works. which is super odd, right?
e

enough-analyst-54434

09/07/2021, 5:26 PM
or
which -a python
one of those should list the
/Libarary/...
Python.
p

plain-carpet-73994

09/07/2021, 5:26 PM
Copy code
Olivers-MBP:python oliverdain$ which -a python
/Users/oliverdain/.pyenv/shims/python
/usr/bin/python
Olivers-MBP:python oliverdain$ which -a Python
/Users/oliverdain/.pyenv/shims/Python
/usr/bin/Python
tried caps and lower-case.
Copy code
$ ls -l /usr/bin/python
lrwxr-xr-x  1 root  wheel  75 Jan  1  2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
So that's py2.7 which I'm almost positive isn't what's getting used.
e

enough-analyst-54434

09/07/2021, 5:27 PM
which -a python3
?
p

plain-carpet-73994

09/07/2021, 5:27 PM
Copy code
$ which -a python3
/Users/oliverdain/.pyenv/shims/python3
/usr/local/bin/python3
/usr/bin/python3
so pyenv -> brew -> system
Copy code
$ /usr/bin/python3 --version
Python 3.8.2
e

enough-analyst-54434

09/07/2021, 5:28 PM
Ok, but - I'm slow - I'm still wanting to know where
/Library/...
comes from.
p

plain-carpet-73994

09/07/2021, 5:28 PM
Copy code
$ /Users/oliverdain/.pyenv/shims/python3 --version
Python 3.7.9
But 3.7.9 isn't what I installed with pyenv:
Copy code
$ pyenv versions
* system (set by /Users/oliverdain/.pyenv/version)
* 3.7.10 (set by /Users/oliverdain/.pyenv/version)
* 3.8.9 (set by /Users/oliverdain/.pyenv/version)
yeah. It looks to me like the shim'd python isn't working correctly.
FWIW, this should be a very clean setup. Brand new laptop and I have an Ansible script that provisions it for development so I didn't run a bunch of random commands to set things up. The Ansible takes care of the pyenv stuff for me.
I'm kind of inclined to re-run
pyenv global
to see if I can get the shim corrected but happy to do something else if you prefer.
BTW: really appreciate your help on this. Been banging may head on this for days now!
e

enough-analyst-54434

09/07/2021, 5:31 PM
Gotcha. The unclean bit is the macOS Python 3.7 and that's leaking in.
p

plain-carpet-73994

09/07/2021, 5:32 PM
looks like. I'm not familiar enough with pyenv magic but I think somehow its shims got borked.
e

enough-analyst-54434

09/07/2021, 5:32 PM
Sure - try re-running
pyenv global
, but IIRC Pants doesn't use shims at all, it runs
pyenv root
and then adds versions/*/bin to the search path.
p

plain-carpet-73994

09/07/2021, 5:33 PM
Copy code
Olivers-MBP:python oliverdain$ pyenv global system 3.8.9 3.7.10
Olivers-MBP:python oliverdain$ /Users/oliverdain/.pyenv/shims/python3 --version
Python 3.7.9
e

enough-analyst-54434

09/07/2021, 5:33 PM
Yeah - it's all versions dir based. Pants just ignores shims altogether.
p

plain-carpet-73994

09/07/2021, 5:34 PM
Did not fix. Maybe it thinks
system
is 3.7 even though it's supposed to be 3.9 on Big Sur??
e

enough-analyst-54434

09/07/2021, 5:34 PM
So that should not have helped anything.
p

plain-carpet-73994

09/07/2021, 5:34 PM
yeah, penv thinks 3.7 is the
system
.
e

enough-analyst-54434

09/07/2021, 5:35 PM
Any pants doesn't look at global local or shell - so it should not be relevant.
p

plain-carpet-73994

09/07/2021, 5:35 PM
yeah. I get that. Seems like maybe a clue though that 3.9 is supposed to be big sur's system but we somehow have 3.7 installed and pyenv thinks that's the system. So wondering if whatever has confused pyenv is also confusing pants.
e

enough-analyst-54434

09/07/2021, 5:36 PM
So, can you try this - the output will be hugeish - and post a gist or something like that?:
Copy code
./pants -ldebug --no-pantsd --no-process-execution-local-cache ...your command...
What Pants version is this by the way?
p

plain-carpet-73994

09/07/2021, 5:39 PM
2.6.0
It's still running but this looks like it's using the right version of python:
Copy code
10:38:18.29 [DEBUG] spawned local process as Some(7477) for Process { argv: ["/Users/oliverdain/.pyenv/versions/3.7.10/bin/python3.7", "./__parse_python_imports.py", "utils/pydantic_utils.py"], env: {"STRING_IMPORTS": "n"}, working_directory: None, input_files: Digest { hash: Fingerprint<979a1f6fadfd74b55e364d7ceaef29f3e552d6087b9b8f653a9721c2759465cc>, size_bytes: 179 }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, description: "Determine Python imports for utils/pydantic_utils.py", level: Debug, append_only_caches: {}, jdk_home: None, platform_constraint: None, is_nailgunnable: false, cache_scope: Successful }
same error though.
Here's the full logs.
Appears to be failing with the same command that started this thread but now it's pretty clearly using the correct python3.7. This is confusing!
e

enough-analyst-54434

09/07/2021, 5:46 PM
Its actually not clearly using it. The last ERROR line shows the --dest /Library thing.
Ok, one more idea - try
rm -rf ~/.cache/pants/named_caches/pex_root/{interpreters,pip.pex}
Then re-run the Pants command you just ran.
p

plain-carpet-73994

09/07/2021, 5:48 PM
How 'bout I just nuke
~/.cache/pants
?
e

enough-analyst-54434

09/07/2021, 5:48 PM
Um, edit to above:
Copy code
rm -rf ~/.cache/pants/named_caches/pex_root/{interpreters,pip.pex,venvs}
Because that's not needed / bad habit looking to debug. I can understand your impatience though.
p

plain-carpet-73994

09/07/2021, 5:49 PM
👍
running...
Same result.
run.log
e

enough-analyst-54434

09/07/2021, 5:54 PM
And, not that this is feasible, but what happend if you just use ["<PYENV>"] for the interpreter search path - no "<PATH>" entry at all?
p

plain-carpet-73994

09/07/2021, 5:55 PM
Do the
rm -rf
bit again after making that change?
e

enough-analyst-54434

09/07/2021, 5:55 PM
Sure.
p

plain-carpet-73994

09/07/2021, 5:56 PM
running...
SUCCESS!
e

enough-analyst-54434

09/07/2021, 5:58 PM
Ok. 1st off for your moving on - is that solution acceptable or to onerous for other devs setups?
p

plain-carpet-73994

09/07/2021, 6:00 PM
hmmm.... I'm not sure. I know that some of our devs are not using pyenv. We could try to migrate them but it'd be best if we didn't have to.
But I'm guessing this is a pants bug (it's not honoring the order of the search paths) and can't be fixed immediately??
e

enough-analyst-54434

09/07/2021, 6:00 PM
Right - that's expected.
The other's not being able to use pyenv bit is the expected thing.
p

plain-carpet-73994

09/07/2021, 6:02 PM
Is there a way to have per-dev preferences? I did see somewhere that there's an env variable you can set with something like
PANTS_CONFIG_FILES
and the fact that it was plural (FILES vs FILE) made me think we could maybe have the main config in the repo but the search paths could be per dev??
Right - that's expected.
Do you mean that the list of search paths is not supposed to be ordered?
@enough-analyst-54434 is this the correct summary: • The search paths are not meant to be ordered so if we have
<PATH>
in the list it'll find an use the incorrect version. Currently that means we need to get everybody using pyenv. • However, there's maybe a way to have per-dev dependencies (see 2 messages up) which might let different devs have different setups? • It's not clear if the pants team thinks this is a bug or not. Personally it seems like
interpreter_search_paths
should be an ordered list. Thanks again for all your help. I don't think I would have ever figured that one out on my own...
e

enough-analyst-54434

09/07/2021, 6:23 PM
So - the search path is ordered. There's something else subtle going on here.
There is no sane way to have per-dev setups. You can use ~/.pants.rc - which is the same format as pants.toml - to override things, but if I were in your shoes I'd not be happy with that solution.
I definitely think this is a bug. That's slightly besides the point though. Worse - there is not even an acceptable workaround yet.
I'm staring at your latest logs and code with the aim at filing a Pants or Pex bug once I can narrow where things must be going wrong. I'll update this thread when I get that far.
p

plain-carpet-73994

09/07/2021, 6:28 PM
Thanks @enough-analyst-54434. REALLY appreciate it.
You can use ~/.pants.rc - which is the same format as pants.toml - to override things, but if I were in your shoes I'd not be happy with that solution.
Why not? It's not ideal but it sounds like we can remove the
interpreter_search_paths
and then have devs who use pyenv just add 2 lines to their
~/.pants.rc
. Agreed it's not ideal but it doesn't seem terrible. Are there other ramifications I'm missing?
e

enough-analyst-54434

09/07/2021, 6:28 PM
Just that you will forget about that file and then later you get action at a distance behvior that's hard to debug.
p

plain-carpet-73994

09/07/2021, 6:31 PM
yeah, that makes sense. Thanks.
e

enough-analyst-54434

09/07/2021, 6:38 PM
Ok, If you can try one more thing and gather all logs that would be great. Get pants.toml back in the
["<PYENV>", "<PATH>"]
state,
pkill pantsd
and make sure its dead,
rm -rf ~/.cache/pants/named_caches/pex_root/{interpreters,pip.pex,venvs}
and then run your Pants command with
./pants -ldebug --pex-verbosity=9 --no-pantsd --no-process-execution-local-cache ...
- There's an extra
--pex-verbosity=9
bit in there that should provide insight into why Pex is apparently picking
/Library/...
when doing that resolve despite being run with a pyenv python and having that pyenv python being 1st on the `PEX_PYTHON_PATH`/`--python-path` it's given.
And then before running any other commands after capturing log output, copy / save aside
~/.cache/pants/named_caches/pex_root
in case I come back with questions about files in there.
p

plain-carpet-73994

09/07/2021, 6:52 PM
running now...
@enough-analyst-54434 it actually worked with the
<PATH>
in there!
I'm very confused.
e

enough-analyst-54434

09/07/2021, 6:57 PM
Ok. Me too. Hopefully this does not come up again, but if it does for some other dev, then this sort of log output will be useful to debug.
I'ts pretty hard to remote-control debug these things without giving instructions that stomp on evidence inadvertantly and I did that here.
p

plain-carpet-73994

09/07/2021, 6:59 PM
Thanks again for the help.
It just came back and I had to remove the
<PATH>
again and that fixed it.
I'm actually supposed to be on vacation as I type and I really need to get going but I can help you with this some more next week if you've got more things I should try.
e

enough-analyst-54434

09/07/2021, 7:28 PM
Ok. I'd like to do that. Thanks Oliver.
9 Views