https://pantsbuild.org/ logo
Join Slack
Powered by
I'm having trouble getting my own pypi repo to wor...
# general
p
I'm having trouble getting my own pypi repo to work. I've set up a GCP PyPi-like simple repo and I can install packages from it fine one the command line via 
pip install
. Specifically 
pip install pyrealsense2==2.49.0
works but if I put that in my 
constraints.txt
Pants can't install it. I re-ran the command pants was running and removed CLI options to get it down to the minimum:
Copy code
/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/oliverdain/.cache/pants/named_caches/pex_root --log /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmpfchex0j2/pip.log download --dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmp9bliehja/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7   pyrealsense2==2.49.0+companion   --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
Note that in the above I'm not using my 
constraints.txt
but instead directly passing the package version I need. I get this error:
Copy code
2021-09-07T09:38:09,804 <https://us-central1-python.pkg.dev:443> "GET /infrastructure-314723/python-deps/simple/pyrealsense2/ HTTP/1.1" 200 1028
2021-09-07T09:38:09,806 Updating cache with response from "<https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>"
2021-09-07T09:38:09,834   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0
2021-09-07T09:38:09,835   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0.3313-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0.3313
2021-09-07T09:38:09,835   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:38:09,836   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:38:09,836 Given no hashes to check 0 links for project 'pyrealsense2': discarding no candidates
2021-09-07T09:38:09,836 ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0+companion
It sure looks like that package has the right tags so I don't understand the 
none of the wheel's tags match
errors. Any thoughts?
This repo will eventually be private but for testing purposes I've made it public. The .whl here is a version of pyrealsense2 compiled for OSX (as Intel only provides binaries for Windows and Linux).
e
So, https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/ has only exactly cp37m wheels - how sure are you the Pants resolve is running for CPython 3.7?
p
There's also a Linux wheel up there but there is a cp37m wheel. Pretty sure it's running the resolve for CPython 3.7. See the following in the logs:
Copy code
2021-09-07T09:38:09,835   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
It lists the tags it's looking for and it lists what it found and they match to my eyes.
Also: I've tried it without the 
+companion
suffix and get the same result.
e
It lists the tags it's looking for and it lists what it found and they match to my eyes.
I think your interpretation is wrong. It lists the summary of the tags it found on the left of the line for the link on the right of the line.
Can you provide full log output?
There should be a command line in the output that invloves pip download - that's the key one that will reveal what Python Pants is resolving for.
p
Full logs:
Copy code
2021-09-07T09:50:54,961 Created temporary directory: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Initialized build tracking at /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Created build tracker: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Entered build tracker: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
2021-09-07T09:50:54,962 Created temporary directory: /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-download-gpowzxs5
2021-09-07T09:50:54,970 Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
2021-09-07T09:50:54,970 1 location(s) to search for versions of pyrealsense2:
2021-09-07T09:50:54,970 * <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>
2021-09-07T09:50:54,971 Fetching project page and analyzing links: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>
2021-09-07T09:50:54,971 Getting page <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>
2021-09-07T09:50:54,972 Found index url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
2021-09-07T09:50:55,233 Looking up "<https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>" in the cache
2021-09-07T09:50:55,234 Request header has "max_age" as 0, cache bypassed
2021-09-07T09:50:55,235 Starting new HTTPS connection (1): us-central1-python.pkg.dev:443
2021-09-07T09:50:55,690 <https://us-central1-python.pkg.dev:443> "GET /infrastructure-314723/python-deps/simple/pyrealsense2/ HTTP/1.1" 200 1028
2021-09-07T09:50:55,692 Updating cache with response from "<https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>"
2021-09-07T09:50:55,721   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0
2021-09-07T09:50:55,721   Found link <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.47.0.3313-cp37-cp37m-macosx_10_15_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>), version: 2.47.0.3313
2021-09-07T09:50:55,722   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:50:55,722   Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_11_0_x86_64: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0-cp37-cp37m-macosx_11_0_x86_64.whl> (from <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple/pyrealsense2/>)
2021-09-07T09:50:55,722 Given no hashes to check 0 links for project 'pyrealsense2': discarding no candidates
2021-09-07T09:50:55,722 ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0
2021-09-07T09:50:55,723 ERROR: No matching distribution found for pyrealsense2==2.49.0
2021-09-07T09:50:55,723 Exception information:
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion
2021-09-07T09:50:55,723     crit = self.state.criteria[name]
2021-09-07T09:50:55,723 KeyError: 'pyrealsense2'
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 During handling of the above exception, another exception occurred:
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in resolve
2021-09-07T09:50:55,723     name, crit = self._merge_into_criterion(r, parent=None)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _merge_into_criterion
2021-09-07T09:50:55,723     crit = Criterion.from_requirement(self._p, requirement, parent)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 83, in from_requirement
2021-09-07T09:50:55,723     raise RequirementsConflicted(criterion)
2021-09-07T09:50:55,723 pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('pyrealsense2==2.49.0')
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 During handling of the above exception, another exception occurred:
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 122, in resolve
2021-09-07T09:50:55,723     requirements, max_rounds=try_to_avoid_resolution_too_deep,
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
2021-09-07T09:50:55,723     state = resolution.resolve(requirements, max_rounds=max_rounds)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 320, in resolve
2021-09-07T09:50:55,723     raise ResolutionImpossible(e.criterion.information)
2021-09-07T09:50:55,723 pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('pyrealsense2==2.49.0'), parent=None)]
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 The above exception was the direct cause of the following exception:
2021-09-07T09:50:55,723
2021-09-07T09:50:55,723 Traceback (most recent call last):
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
2021-09-07T09:50:55,723     status = self.run(options, args)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
2021-09-07T09:50:55,723     return func(self, options, args)
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/commands/download.py", line 131, in run
2021-09-07T09:50:55,723     reqs, check_supported_wheels=True
2021-09-07T09:50:55,723   File "/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 127, in resolve
2021-09-07T09:50:55,723     six.raise_from(error, e)
2021-09-07T09:50:55,723   File "<string>", line 3, in raise_from
2021-09-07T09:50:55,723 pip._internal.exceptions.DistributionNotFound: No matching distribution found for pyrealsense2==2.49.0
2021-09-07T09:50:55,724 Removed build tracker: '/private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/pip-req-tracker-c1gjf69t
e
Yeah - that does not look full, just looks like pip logs.
What pants command line are you using to get this log output?
p
Copy code
/Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/oliverdain/.cache/pants/named_caches/pex_root --log /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmpfchex0j2/pip.log download --dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmp9bliehja/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7   pyrealsense2==2.49.0+companion   --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
Note that includes 
--dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionZcFGYp/.tmp/tmp9bliehja/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7
And my 
pants.toml
includes:
Copy code
[python-setup]
interpreter_constraints = ['CPython==3.7.*']
requirement_constraints = 'constraints.txt'
e
Yup - so a macOS provided python. ... OK
p
It's a 
pyenv
built python.
e
Nope
It is definitely not
p
That's super strange. (1) This is a new laptop that I just set up and (2) it's big sur which only provides 3.9. I had to install pyenv and build 3.7.
e
It's the one here: /Library/Frameworks/Python/framework/Versions/3.7/bin/python3.7
I assume pyenv does not install there.
Do you have XCode?
That installs its own above and beyond the OS
p
Copy code
$ ls /Library/Frameworks/Python/framework/Versions/3.7/bin/python3.7
ls: /Library/Frameworks/Python/framework/Versions/3.7/bin/python3.7: No such file or directory
e
Hrm
p
There's a bunch of directories before 
/Library/Frameworks
in that command line that I'm pretty sure correspond to the pants sandbox.
Copy code
$ type python3.7
python3.7 is /Users/oliverdain/.pyenv/shims/python3.7
I did an 
xcode-select --install
which I think gives me some, but not all, of xcode stuff.
e
Can you humor me and inspect the /Library/ ... path bit by bit? I have case insensitive fs ptsd with that lowercase library in the path I gave you.
p
ha! Sure.
Copy code
$ ls /Library/Frameworks/Python
ls: /Library/Frameworks/Python: No such file or directory
Copy code
$ ls /Library/Frameworks/Python.framework/
Headers		Python		Resources	Versions
Copy code
$ ls /Library/Frameworks/Python.framework/Versions/
3.7	Current
e
Aha!
p
So it does look like there's a 3.7 there.
e
Ok I just translated all dots to /
Ok - that is what it's using
So, switching back to pyenv
Do you have Pants configured to look for pyenv?
p
Dunno. How do I tell?
e
p
But also, why would that matter. I want it using Python3.7, the whl is built for python3.7. Do I care which 3.7 it's using?
e
Well, you do because macOS interpreters are notorious
p
I see. I do not have the 
<PYENV>
bit in my 
interpreter_search_paths
. Will add.
e
We could spin a while and figure out what's up with that interpreter though if you want to drill in. Otherwise, pyenv setup may work better for you and others.
p
Updates search paths and trying to re-build to see if that'll make a difference. Just curious, where'd that other 3.7 interpreter come from?
Do I need to clear the cache after adding the search paths bit?
e
No cache clearing needed.
p
No success either I'm afraid.
Copy code
$ ./pants run realsense/cli/ -- -h
10:06:46.68 [INFO] Initializing scheduler...
10:06:47.28 [INFO] Scheduler initialized.
10:07:45.58 [INFO] Completed: Resolving constraints.txt
10:07:45.58 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 190, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 533, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/oliverdain/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 501, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Resolving constraints.txt' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0+companion
ERROR: No matching distribution found for pyrealsense2==2.49.0+companion
pid 4666 -> /Users/oliverdain/.cache/pants/named_caches/pex_root/venvs/ed22d73f2adcc11d2242086e3c714a5181886e41/ea52facb9baa3bab8b4e9c79c1aa16d07c3e12e5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/oliverdain/.cache/pants/named_caches/pex_root --log /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionSppZaq/.tmp/tmptlzd7yp3/pip.log download --dest /private/var/folders/tb/5hcfs_nx5g72yt8wz8c42_4w0000gn/T/process-executionSppZaq/.tmp/tmpwuyigplf/Library.Frameworks.Python.framework.Versions.3.7.bin.python3.7 --constraint constraints.txt absl-py==0.13.0 aiodns==3.0.0 aiofile==3.7.1 aiofiles==0.5.0 aiohttp==3.7.4.post0 aniso8601==7.0.0 astor==0.8.1 async-exit-stack==1.0.1 async-generator==1.10 async-timeout==3.0.1 attrs==21.2.0 backoff==1.10.0 bleach==4.1.0 boto3==1.18.32 botocore==1.21.32 cachecontrol==0.12.6 cachetools==4.2.2 caio==0.9.3 certifi==2021.5.30 cffi==1.14.6 chardet==3.0.4 charset-normalizer==2.0.4 chevron==0.14.0 click==7.1.2 colorama==0.4.4 cryptography==3.4.8 cycler==0.10.0 dnspython==2.1.0 docutils==0.17.1 email-validator==1.1.3 fastapi==0.68.1 firebase-admin==5.0.2 gast==0.2.2 google-api-core==1.31.2 google-api-python-client==2.19.0 google-auth-httplib2==0.1.0 google-auth==1.35.0 google-cloud-bigquery==2.25.1 google-cloud-core==2.0.0 google-cloud-firestore==2.3.1 google-cloud-storage==1.42.0 google-crc32c==1.1.3 google-pasta==0.2.0 google-resumable-media==2.0.0 googleapis-common-protos==1.53.0 graphene==2.1.9 graphql-core==2.3.2 graphql-relay==2.0.1 grpcio==1.39.0 h11==0.12.0 h2==4.0.0 h5py==2.10.0 hpack==4.0.0 httplib2==0.19.1 httptools==0.1.2 hypercorn==0.11.2 hyperframe==6.0.1 idna==3.2 importlib-metadata==4.8.1 iniconfig==1.1.1 itsdangerous==1.1.0 jeepney==0.7.1 jinja2==2.11.3 jmespath==0.10.0 keras-applications==1.0.8 keras-preprocessing==1.1.2 keyring==23.1.0 keyrings.google-artifactregistry-auth==0.0.3 kiwisolver==1.3.2 labelbox==3.2.0 markdown==3.3.4 markupsafe==2.0.1 matplotlib==3.4.3 msgpack==1.0.2 multidict==5.1.0 ndjson==0.3.1 numpy==1.18.5 opencv-python-headless==4.5.3.56 opt-einsum==3.3.0 orjson==3.6.3 packaging==21.0 pillow==8.3.1 pkginfo==1.7.1 pluggy==1.0.0 priority==2.0.0 promise==2.3 proto-plus==1.19.0 protobuf==3.17.3 py==1.10.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pycares==4.0.0 pycparser==2.20 pycryptodomex==3.10.1 pydantic==1.8.2 pygments==2.10.0 pyjwt==2.1.0 pyparsing==2.4.7 pyrealsense2==2.49.0+companion pytest-asyncio==0.15.1 pytest==6.2.5 python-dateutil==2.8.2 python-dotenv==0.19.0 python-multipart==0.0.5 pytz==2021.1 pyyaml==5.4.1 readme-renderer==29.0 requests-toolbelt==0.9.1 requests==2.26.0 rfc3986==1.5.0 rsa==4.7.2 rx==1.6.1 s3transfer==0.5.0 secretstorage==3.3.1 semantic-version==2.8.5 six==1.16.0 starlette==0.14.2 stream-chat==3.12.1 tensorboard==1.15.0 tensorflow-estimator==1.15.1 tensorflow==1.15.5 termcolor==1.1.0 toml==0.10.2 tqdm==4.62.2 twine==3.4.2 types-futures==3.3.0 types-protobuf==3.17.4 types-pyyaml==5.4.8 types-requests==2.25.6 typing-extensions==3.10.0.2 ujson==4.1.0 uritemplate==3.0.1 urllib3==1.26.6 uvicorn==0.13.4 uvloop==0.16.0 watchgod==0.7 webencodings==0.5.1 websockets==8.1 werkzeug==2.0.1 wheel==0.37.0 wrapt==1.12.1 wsproto==1.0.0 yarl==1.6.3 zipp==3.5.0 --index-url <https://pypi.org/simple/> --extra-index-url <https://us-central1-python.pkg.dev:443/infrastructure-314723/python-deps/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
None
That's the full command line that failed. It's trying to resolve lots of other dependencies but starting from there I was able to get the errors you saw. I can whittle it down again if you like.
e
It's still using the same Python.
So, lets switch tactics temporarily.
Can you try this as a sanity check?:
Copy code
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -mvenv test.venv
test.venv/bin/pip install -U pip==20.3.4
test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
p
interesting:
Copy code
$ test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0
ERROR: No matching distribution found for pyrealsense2==2.49.0
WARNING: You are using pip version 20.3.4; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/oliverdain/Documents/code/central/python/test.venv/bin/python3.7 -m pip install --upgrade pip' command.
e
Ok - awesome - this just confirms borked macOS Python - as I said notorious.
p
But if I do the equivalent with pyenv Python I'm pretty sure it works. Testing. Note: I added 
<PYENV>
but it still picked up the system python.
e
Can you also provide your relevant pants.toml snip. Most crucially, I'd guess "<PYENV>" is not 1st in the list, and that's why.
p
Copy code
[python-setup]
  interpreter_constraints = ['CPython==3.7.*']
  requirement_constraints = 'constraints.txt'
  interpreter_search_paths = ["<PYENV>", "<PATH>"]
Looks first to me...
e
Hrm
Msay require 
pkill pantsd
- binary search paths are generally calculated just once per pants session - and with pantsd (the default) the session is long-lived.
p
same test fails with pyenv python.
Copy code
Olivers-MBP:python oliverdain$ type python3.7
python3.7 is hashed (/Users/oliverdain/.pyenv/shims/python3.7)
Olivers-MBP:python oliverdain$ /Users/oliverdain/.pyenv/shims/python3.7 -mvenv test.venv
Olivers-MBP:python oliverdain$ test.venv/bin/pip install -U pip==20.3.4
Collecting pip==20.3.4
  Using cached pip-20.3.4-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed pip-20.3.4
Olivers-MBP:python oliverdain$ test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0
ERROR: No matching distribution found for pyrealsense2==2.49.0
Yup, I'll kill the pants daemon but per above it does not appear to be a system python issue.
e
Putting on the breaks more - instead of using a shim in your latest experiment, please use 
/Users/oliverdain/.pyenv/versions/3.7.x/bin/python
This will avoid shim unexpected redirection shenanigans and prove the pyenv interpreter works or does not.
p
will do
restarted 
pantsd
did not solve the problem.
e
And ... do you have the same problem with pyrealsense2 2.47.0?
p
yes
e
OK, so gathering outstanding requests: 1. Please try .penv/versions/.../bin/python exact path with pop experiment and report 2. Please provide full output of pants run post killing of pantsd to confirm the --dest in the pip command looks like its using pyenv
p
Copy code
$ /Users/oliverdain/.pyenv/versions/3.7.10/bin/python -mvenv test.venv
$ test.venv/bin/pip install -U pip==20.3.4
$ test.venv/bin/pip install --index-url <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple> pyrealsense2==2.49.0
Looking in indexes: <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/simple>
Collecting pyrealsense2==2.49.0
  Downloading <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/pyrealsense2/pyrealsense2-2.49.0%2Bcompanion-cp37-cp37m-macosx_11_0_x86_64.whl>
     | 16.6 MB 891 kB/s
Installing collected packages: pyrealsense2
Successfully installed pyrealsense2-2.49.0+companion
e
Ok - so pyenv works
p
So it does work with pyenv 3.7!
e
Now we just need to get Pants seeing pyenv 1st on the search path.
p
yeah. So why don't my shims work and where did the other 3.7 come from?
e
The other 3.7 is presumably on your PATH
p
Yeah, very strange. I can re-run 
pyenv global
to set up interpreters...
e
which -a python3.7
p
Copy code
$ which -a python3.7
/Users/oliverdain/.pyenv/shims/python3.7
/usr/local/bin/python3.7
Yup.
e
which -a Python
?
p
So per above, if I use 
/Users/oliverdain/.pyenv/shims/python3.7
it fails but if I just type 
python3.7
I do get the pyenv version and it works. which is super odd, right?
e
or 
which -a python
one of those should list the 
/Libarary/...
Python.
p
Copy code
Olivers-MBP:python oliverdain$ which -a python
/Users/oliverdain/.pyenv/shims/python
/usr/bin/python
Olivers-MBP:python oliverdain$ which -a Python
/Users/oliverdain/.pyenv/shims/Python
/usr/bin/Python
tried caps and lower-case.
Copy code
$ ls -l /usr/bin/python
lrwxr-xr-x  1 root  wheel  75 Jan  1  2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
So that's py2.7 which I'm almost positive isn't what's getting used.
e
which -a python3
?
p
Copy code
$ which -a python3
/Users/oliverdain/.pyenv/shims/python3
/usr/local/bin/python3
/usr/bin/python3
so pyenv -> brew -> system
Copy code
$ /usr/bin/python3 --version
Python 3.8.2
e
Ok, but - I'm slow - I'm still wanting to know where 
/Library/...
comes from.
p
Copy code
$ /Users/oliverdain/.pyenv/shims/python3 --version
Python 3.7.9
But 3.7.9 isn't what I installed with pyenv:
Copy code
$ pyenv versions
* system (set by /Users/oliverdain/.pyenv/version)
* 3.7.10 (set by /Users/oliverdain/.pyenv/version)
* 3.8.9 (set by /Users/oliverdain/.pyenv/version)
yeah. It looks to me like the shim'd python isn't working correctly.
FWIW, this should be a very clean setup. Brand new laptop and I have an Ansible script that provisions it for development so I didn't run a bunch of random commands to set things up. The Ansible takes care of the pyenv stuff for me.
I'm kind of inclined to re-run 
pyenv global
to see if I can get the shim corrected but happy to do something else if you prefer.
BTW: really appreciate your help on this. Been banging may head on this for days now!
e
Gotcha. The unclean bit is the macOS Python 3.7 and that's leaking in.
p
looks like. I'm not familiar enough with pyenv magic but I think somehow its shims got borked.
e
Sure - try re-running 
pyenv global
, but IIRC Pants doesn't use shims at all, it runs 
pyenv root
and then adds versions/*/bin to the search path.
p
Copy code
Olivers-MBP:python oliverdain$ pyenv global system 3.8.9 3.7.10
Olivers-MBP:python oliverdain$ /Users/oliverdain/.pyenv/shims/python3 --version
Python 3.7.9
e
Yeah - it's all versions dir based. Pants just ignores shims altogether.
p
Did not fix. Maybe it thinks 
system
is 3.7 even though it's supposed to be 3.9 on Big Sur??
e
So that should not have helped anything.
p
yeah, penv thinks 3.7 is the 
system
.
e
Any pants doesn't look at global local or shell - so it should not be relevant.
p
yeah. I get that. Seems like maybe a clue though that 3.9 is supposed to be big sur's system but we somehow have 3.7 installed and pyenv thinks that's the system. So wondering if whatever has confused pyenv is also confusing pants.
e
So, can you try this - the output will be hugeish - and post a gist or something like that?:
Copy code
./pants -ldebug --no-pantsd --no-process-execution-local-cache ...your command...
What Pants version is this by the way?
p
2.6.0
It's still running but this looks like it's using the right version of python:
Copy code
10:38:18.29 [DEBUG] spawned local process as Some(7477) for Process { argv: ["/Users/oliverdain/.pyenv/versions/3.7.10/bin/python3.7", "./__parse_python_imports.py", "utils/pydantic_utils.py"], env: {"STRING_IMPORTS": "n"}, working_directory: None, input_files: Digest { hash: Fingerprint<979a1f6fadfd74b55e364d7ceaef29f3e552d6087b9b8f653a9721c2759465cc>, size_bytes: 179 }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, description: "Determine Python imports for utils/pydantic_utils.py", level: Debug, append_only_caches: {}, jdk_home: None, platform_constraint: None, is_nailgunnable: false, cache_scope: Successful }
same error though.
Here's the full logs.
run.log
Appears to be failing with the same command that started this thread but now it's pretty clearly using the correct python3.7. This is confusing!
e
Its actually not clearly using it. The last ERROR line shows the --dest /Library thing.
Ok, one more idea - try 
rm -rf ~/.cache/pants/named_caches/pex_root/{interpreters,pip.pex}
Then re-run the Pants command you just ran.
p
How 'bout I just nuke 
~/.cache/pants
?
e
Um, edit to above:
Copy code
rm -rf ~/.cache/pants/named_caches/pex_root/{interpreters,pip.pex,venvs}
Because that's not needed / bad habit looking to debug. I can understand your impatience though.
p
👍
running...
Same result.
run.log
run.log
e
And, not that this is feasible, but what happend if you just use ["<PYENV>"] for the interpreter search path - no "<PATH>" entry at all?
p
Do the 
rm -rf
bit again after making that change?
e
Sure.
p
running...
SUCCESS!
e
Ok. 1st off for your moving on - is that solution acceptable or to onerous for other devs setups?
p
hmmm.... I'm not sure. I know that some of our devs are not using pyenv. We could try to migrate them but it'd be best if we didn't have to.
But I'm guessing this is a pants bug (it's not honoring the order of the search paths) and can't be fixed immediately??
e
Right - that's expected.
The other's not being able to use pyenv bit is the expected thing.
p
Is there a way to have per-dev preferences? I did see somewhere that there's an env variable you can set with something like 
PANTS_CONFIG_FILES
and the fact that it was plural (FILES vs FILE) made me think we could maybe have the main config in the repo but the search paths could be per dev??
Right - that's expected.
Do you mean that the list of search paths is not supposed to be ordered?
@enough-analyst-54434 is this the correct summary: • The search paths are not meant to be ordered so if we have 
<PATH>
in the list it'll find an use the incorrect version. Currently that means we need to get everybody using pyenv. • However, there's maybe a way to have per-dev dependencies (see 2 messages up) which might let different devs have different setups? • It's not clear if the pants team thinks this is a bug or not. Personally it seems like 
interpreter_search_paths
should be an ordered list. Thanks again for all your help. I don't think I would have ever figured that one out on my own...
e
So - the search path is ordered. There's something else subtle going on here.
There is no sane way to have per-dev setups. You can use ~/.pants.rc - which is the same format as pants.toml - to override things, but if I were in your shoes I'd not be happy with that solution.
I definitely think this is a bug. That's slightly besides the point though. Worse - there is not even an acceptable workaround yet.
I'm staring at your latest logs and code with the aim at filing a Pants or Pex bug once I can narrow where things must be going wrong. I'll update this thread when I get that far.
p
Thanks @enough-analyst-54434. REALLY appreciate it.
You can use ~/.pants.rc - which is the same format as pants.toml - to override things, but if I were in your shoes I'd not be happy with that solution.
Why not? It's not ideal but it sounds like we can remove the 
interpreter_search_paths
and then have devs who use pyenv just add 2 lines to their 
~/.pants.rc
. Agreed it's not ideal but it doesn't seem terrible. Are there other ramifications I'm missing?
e
Just that you will forget about that file and then later you get action at a distance behvior that's hard to debug.
p
yeah, that makes sense. Thanks.
e
Ok, If you can try one more thing and gather all logs that would be great. Get pants.toml back in the 
["<PYENV>", "<PATH>"]
state, 
pkill pantsd
and make sure its dead, 
rm -rf ~/.cache/pants/named_caches/pex_root/{interpreters,pip.pex,venvs}
and then run your Pants command with 
./pants -ldebug --pex-verbosity=9 --no-pantsd --no-process-execution-local-cache ...
- There's an extra 
--pex-verbosity=9
bit in there that should provide insight into why Pex is apparently picking 
/Library/...
when doing that resolve despite being run with a pyenv python and having that pyenv python being 1st on the `PEX_PYTHON_PATH`/`--python-path` it's given.
And then before running any other commands after capturing log output, copy / save aside 
~/.cache/pants/named_caches/pex_root
in case I come back with questions about files in there.
p
running now...
@enough-analyst-54434 it actually worked with the 
<PATH>
in there!
I'm very confused.
e
Ok. Me too. Hopefully this does not come up again, but if it does for some other dev, then this sort of log output will be useful to debug.
I'ts pretty hard to remote-control debug these things without giving instructions that stomp on evidence inadvertantly and I did that here.
p
Thanks again for the help.
It just came back and I had to remove the 
<PATH>
again and that fixed it.
I'm actually supposed to be on vacation as I type and I really need to get going but I can help you with this some more next week if you've got more things I should try.
e
Ok. I'd like to do that. Thanks Oliver.
9 Views
Open in Slack
PreviousNext
Powered by