https://pantsbuild.org/ logo
Powered by
b

broad-dentist-80514

03/27/2023, 2:08 PM
Hello friends, glad to have found this community! 🙂 I currently want to add a wheel file to a python project that I'm building with pants. Unfortunately, whenever I want to 
pip install
the .whl into a given env, the process gets stuck resolving deps. It installs fine when I pass pip the 
--use-deprecated=legacy-resolver
flag. My question: Is there a way to pass 
--use-deprecated=legacy-resolver
into 
pants
? Because currently, pants (like pip) seems to just time out at 
Building requirements.pex with 1 requirement: ...@ file://
e

enough-analyst-54434

03/27/2023, 2:14 PM
Unfortunately Pants does not allow you to pass this. It delegates to Pex (which delegates to Pip). Pex in fact defaults to the legacy resolver and Pants purposely requests the 
pip-2020-resolver
.
👀 1
Can you share the wheel METADATA file?
👍 1
🙏 1
b

broad-dentist-80514

03/27/2023, 2:16 PM
Copy code
Metadata-Version: 2.1
Name: ...
Version: 0.1.2
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Requires-Dist: black (~=22.0)
Requires-Dist: flake8 (<4.0,>=3.7.9)
Requires-Dist: ipython (<8.0,>=7.31.1)
Requires-Dist: isort (~=5.0)
Requires-Dist: jupyter (~=1.0)
Requires-Dist: jupyterlab (~=3.0)
Requires-Dist: kedro[pandas.csvdataset,pandas.exceldataset,pandas.parquetdataset] (~=0.18.3)
Requires-Dist: kedro-telemetry (~=0.2.0)
Requires-Dist: kedro-viz (~=4.0)
Requires-Dist: nbstripout (~=0.4)
Requires-Dist: pandera (~=0.13.3)
Requires-Dist: pytest-cov (~=3.0)
Requires-Dist: pytest-mock (<2.0,>=1.7.1)
Requires-Dist: pytest (~=6.2)
Requires-Dist: scikit-learn (~=1.0)
Provides-Extra: docs
Requires-Dist: docutils (<0.18.0) ; extra == 'docs'
Requires-Dist: sphinx (~=3.4.3) ; extra == 'docs'
Requires-Dist: sphinx-rtd-theme (==0.5.1) ; extra == 'docs'
Requires-Dist: nbsphinx (==0.8.1) ; extra == 'docs'
Requires-Dist: nbstripout (~=0.4) ; extra == 'docs'
Requires-Dist: myst-parser (~=0.17.2) ; extra == 'docs'
Requires-Dist: sphinx-autodoc-typehints (==1.11.1) ; extra == 'docs'
Requires-Dist: sphinx-copybutton (==0.3.1) ; extra == 'docs'
Requires-Dist: ipykernel (<7.0,>=5.3) ; extra == 'docs'

UNKNOWN
e

enough-analyst-54434

03/27/2023, 2:18 PM
It's extremely odd that a wheel depends on pytest related things as non-extras. Surely the production code does not use pytest?! What is the provenance of this wheel?
b

broad-dentist-80514

03/27/2023, 2:19 PM
It's essentially just an examples project called 
spaceflights
https://docs.kedro.org/en/stable/tutorial/spaceflights_tutorial.html
I'm trying to find if this whl is public or work internal ...
The specific whl is work internal
Difficult to judge atm for me why pytest is in there. Do you reckon this could be the reason for pip timing out?
e

enough-analyst-54434

03/27/2023, 2:21 PM
Ok. So, can you flesh out the scenario more? Do you actually care about this wheel or are you learning Pants with it?
b

broad-dentist-80514

03/27/2023, 2:23 PM
I'm currently setting up a monorepo for a new project. The projects aim is to load any given whl that contains a https://github.com/kedro-org/kedro data science project and generate a supporting codebase/interface.
So this specific whl is not a make or break, but it serves as a proxy of more of this type to come along our way.
e

enough-analyst-54434

03/27/2023, 2:24 PM
In short, it's extremely frustrating when Pants does things like this, takes away a knob it doesn't really care about. It could allow using the legacy resolver just fine. That said, if a project needs the Pip legacy resolver to resolve, that's a very bad sign because the Pip legacy resolver is so broken.
💡 1
b

broad-dentist-80514

03/27/2023, 2:25 PM
Sounds like I better go and fix my whl. 🙂
e

enough-analyst-54434

03/27/2023, 2:25 PM
I'm AFK but can dig more on the wheel itself in 30 minutes or so.
b

broad-dentist-80514

03/27/2023, 2:26 PM
Thank you so much for your timely (instant) help. 🙏 🙏 🙏
Quick update: I modified the wheels source, 
==
pinned its dependencies and now it seems to work. 🙂
e

enough-analyst-54434

03/27/2023, 2:57 PM
Ok. For my investigation - what are your interpreter constraints for the resolve?
@broad-dentist-80514 I get a pretty fast resolve failure using CPython 3.9 at ~30 seconds. I'm using this requirements.txt to simulate the wheel:
Copy code
black~=22.0
flake8<4.0,>=3.7.9
ipython<8.0,>=7.31.1
isort~=5.0
jupyter~=1.0
jupyterlab~=3.0
kedro[pandas.csvdataset,pandas.exceldataset,pandas.parquetdataset]~=0.18.3
kedro-telemetry~=0.2.0
kedro-viz~=4.0
nbstripout~=0.4
pandera~=0.13.3
pytest-cov~=3.0
pytest-mock<2.0,>=1.7.1
pytest~=6.2
scikit-learn~=1.0
🙏 1
b

broad-dentist-80514

03/27/2023, 3:17 PM
Copy code
[python]
interpreter_constraints = ["CPython>=3.8"]
I'm also trying 
==3.9
now
e

enough-analyst-54434

03/27/2023, 3:23 PM
So, do you really mean >=3.8? If you have 3.12 alpha on your machine, you want it to use that (and 3.10 and 3.11, ...)?
Yeah, I get ~30s resolves for each of 
--interpreter-constraint
"==3.{8,9,10}.*"
. Trying the full open ended range...
Ok, the open-ended ">=3.8" does hang for me in 100% cpu somewhere. I have 7 interpreters locally that fit the ">=3.8" bill; so its trying to build a PEX that includes dependencies for all 7 of those.
I failed after 7 minutes with:
Copy code
...
    CPU dispatch  :
      Requested   : 'max -xop -fma4'
      Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
      Generated   : none
    INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /home/jsirois/.pex/pip_cache/.tmp/pip-install-t2_1tr0u/scipy_fc65dcc08b87435d8889b2b9724aa304/build/temp.linux-x86_64-3.9/ccompiler_opt_cache_clib.py
    Building wheel for scipy (PEP 517): finished with status 'error'
    ERROR: Failed building wheel for scipy
  Failed to build scipy
  ERROR: Could not build wheels for scipy which use PEP 517 and cannot be installed directly
  Exception information:
  Traceback (most recent call last):
    File "/home/jsirois/.pex/venvs/c9ddfa173396da5869fe34ecc0594b87dd4565db/d43b4081fdded907dd84e965934e4d9159642320/lib/pypy3.9/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
      status = self.run(options, args)
    File "/home/jsirois/.pex/venvs/c9ddfa173396da5869fe34ecc0594b87dd4565db/d43b4081fdded907dd84e965934e4d9159642320/lib/pypy3.9/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
      return func(self, options, args)
    File "/home/jsirois/.pex/venvs/c9ddfa173396da5869fe34ecc0594b87dd4565db/d43b4081fdded907dd84e965934e4d9159642320/lib/pypy3.9/site-packages/pip/_internal/commands/install.py", line 362, in run
      raise InstallationError(
  pip._internal.exceptions.InstallationError: Could not build wheels for scipy which use PEP 517 and cannot be installed directly
...
So all the time was in building wheels from sdists for off-brand interpreters - PyPy 3.9 in my case. Perhaps you use a Mac M1 or something similar (Linux ARM) which tends not to have pre-built wheels published for it?
b

broad-dentist-80514

03/27/2023, 4:38 PM
Yes, M1. Thanks for your investigations! 🙂
Powered by