Hello friends glad to have found this community slightly smi Pants #general

Hello friends, glad to have found this community! ...

broad-dentist-80514

03/27/2023, 2:08 PM

Hello friends, glad to have found this community! 🙂 I currently want to add a wheel file to a python project that I'm building with pants. Unfortunately, whenever I want to

pip install

the .whl into a given env, the process gets stuck resolving deps. It installs fine when I pass pip the

--use-deprecated=legacy-resolver

flag. My question: Is there a way to pass
--use-deprecated=legacy-resolver
into
pants
? Because currently, pants (like pip) seems to just time out at

Building requirements.pex with 1 requirement: ...@ file://

enough-analyst-54434

03/27/2023, 2:14 PM

Unfortunately Pants does not allow you to pass this. It delegates to Pex (which delegates to Pip). Pex in fact defaults to the legacy resolver and Pants purposely requests the

pip-2020-resolver

👀 1

enough-analyst-54434

03/27/2023, 2:14 PM

Can you share the wheel METADATA file?

👍 1

🙏 1

broad-dentist-80514

03/27/2023, 2:16 PM

Copy code

Metadata-Version: 2.1
Name: ...
Version: 0.1.2
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Requires-Dist: black (~=22.0)
Requires-Dist: flake8 (<4.0,>=3.7.9)
Requires-Dist: ipython (<8.0,>=7.31.1)
Requires-Dist: isort (~=5.0)
Requires-Dist: jupyter (~=1.0)
Requires-Dist: jupyterlab (~=3.0)
Requires-Dist: kedro[pandas.csvdataset,pandas.exceldataset,pandas.parquetdataset] (~=0.18.3)
Requires-Dist: kedro-telemetry (~=0.2.0)
Requires-Dist: kedro-viz (~=4.0)
Requires-Dist: nbstripout (~=0.4)
Requires-Dist: pandera (~=0.13.3)
Requires-Dist: pytest-cov (~=3.0)
Requires-Dist: pytest-mock (<2.0,>=1.7.1)
Requires-Dist: pytest (~=6.2)
Requires-Dist: scikit-learn (~=1.0)
Provides-Extra: docs
Requires-Dist: docutils (<0.18.0) ; extra == 'docs'
Requires-Dist: sphinx (~=3.4.3) ; extra == 'docs'
Requires-Dist: sphinx-rtd-theme (==0.5.1) ; extra == 'docs'
Requires-Dist: nbsphinx (==0.8.1) ; extra == 'docs'
Requires-Dist: nbstripout (~=0.4) ; extra == 'docs'
Requires-Dist: myst-parser (~=0.17.2) ; extra == 'docs'
Requires-Dist: sphinx-autodoc-typehints (==1.11.1) ; extra == 'docs'
Requires-Dist: sphinx-copybutton (==0.3.1) ; extra == 'docs'
Requires-Dist: ipykernel (<7.0,>=5.3) ; extra == 'docs'

UNKNOWN

enough-analyst-54434

03/27/2023, 2:18 PM

It's extremely odd that a wheel depends on pytest related things as non-extras. Surely the production code does not use pytest?! What is the provenance of this wheel?

broad-dentist-80514

03/27/2023, 2:19 PM

It's essentially just an examples project called

spaceflights

https://docs.kedro.org/en/stable/tutorial/spaceflights_tutorial.html

broad-dentist-80514

03/27/2023, 2:19 PM

I'm trying to find if this whl is public or work internal ...

broad-dentist-80514

03/27/2023, 2:20 PM

https://github.com/quantumblacklabs/kedro-starter-spaceflights

broad-dentist-80514

03/27/2023, 2:20 PM

The specific whl is work internal

broad-dentist-80514

03/27/2023, 2:21 PM

Difficult to judge atm for me why pytest is in there. Do you reckon this could be the reason for pip timing out?

enough-analyst-54434

03/27/2023, 2:21 PM

Ok. So, can you flesh out the scenario more? Do you actually care about this wheel or are you learning Pants with it?

broad-dentist-80514

03/27/2023, 2:23 PM

I'm currently setting up a monorepo for a new project. The projects aim is to load any given whl that contains a https://github.com/kedro-org/kedro data science project and generate a supporting codebase/interface.

broad-dentist-80514

03/27/2023, 2:24 PM

So this specific whl is not a make or break, but it serves as a proxy of more of this type to come along our way.

enough-analyst-54434

03/27/2023, 2:24 PM

In short, it's extremely frustrating when Pants does things like this, takes away a knob it doesn't really care about. It could allow using the legacy resolver just fine. That said, if a project needs the Pip legacy resolver to resolve, that's a very bad sign because the Pip legacy resolver is so broken.

💡 1

broad-dentist-80514

03/27/2023, 2:25 PM

Sounds like I better go and fix my whl. 🙂

enough-analyst-54434

03/27/2023, 2:25 PM

I'm AFK but can dig more on the wheel itself in 30 minutes or so.

broad-dentist-80514

03/27/2023, 2:26 PM

Thank you so much for your timely (instant) help. 🙏 🙏 🙏

broad-dentist-80514

03/27/2023, 2:56 PM

Quick update: I modified the wheels source,

==

pinned its dependencies and now it seems to work. 🙂

enough-analyst-54434

03/27/2023, 2:57 PM

Ok. For my investigation - what are your interpreter constraints for the resolve?

enough-analyst-54434

03/27/2023, 3:03 PM

@broad-dentist-80514 I get a pretty fast resolve failure using CPython 3.9 at ~30 seconds. I'm using this requirements.txt to simulate the wheel:

Copy code

black~=22.0
flake8<4.0,>=3.7.9
ipython<8.0,>=7.31.1
isort~=5.0
jupyter~=1.0
jupyterlab~=3.0
kedro[pandas.csvdataset,pandas.exceldataset,pandas.parquetdataset]~=0.18.3
kedro-telemetry~=0.2.0
kedro-viz~=4.0
nbstripout~=0.4
pandera~=0.13.3
pytest-cov~=3.0
pytest-mock<2.0,>=1.7.1
pytest~=6.2
scikit-learn~=1.0

🙏 1

broad-dentist-80514

03/27/2023, 3:17 PM

Copy code

[python]
interpreter_constraints = ["CPython>=3.8"]

broad-dentist-80514

03/27/2023, 3:18 PM

I'm also trying

==3.9

now

enough-analyst-54434

03/27/2023, 3:23 PM

So, do you really mean >=3.8? If you have 3.12 alpha on your machine, you want it to use that (and 3.10 and 3.11, ...)?

enough-analyst-54434

03/27/2023, 3:26 PM

Yeah, I get ~30s resolves for each of

--interpreter-constraint

"==3.{8,9,10}.*"

. Trying the full open ended range...

enough-analyst-54434

03/27/2023, 3:28 PM

@broad-dentist-80514 are you using lock files? https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles

enough-analyst-54434

03/27/2023, 3:30 PM

Ok, the open-ended ">=3.8" does hang for me in 100% cpu somewhere. I have 7 interpreters locally that fit the ">=3.8" bill; so its trying to build a PEX that includes dependencies for all 7 of those.

enough-analyst-54434

03/27/2023, 4:07 PM

I failed after 7 minutes with:

Copy code

...
    CPU dispatch  :
      Requested   : 'max -xop -fma4'
      Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
      Generated   : none
    INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /home/jsirois/.pex/pip_cache/.tmp/pip-install-t2_1tr0u/scipy_fc65dcc08b87435d8889b2b9724aa304/build/temp.linux-x86_64-3.9/ccompiler_opt_cache_clib.py
    Building wheel for scipy (PEP 517): finished with status 'error'
    ERROR: Failed building wheel for scipy
  Failed to build scipy
  ERROR: Could not build wheels for scipy which use PEP 517 and cannot be installed directly
  Exception information:
  Traceback (most recent call last):
    File "/home/jsirois/.pex/venvs/c9ddfa173396da5869fe34ecc0594b87dd4565db/d43b4081fdded907dd84e965934e4d9159642320/lib/pypy3.9/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
      status = self.run(options, args)
    File "/home/jsirois/.pex/venvs/c9ddfa173396da5869fe34ecc0594b87dd4565db/d43b4081fdded907dd84e965934e4d9159642320/lib/pypy3.9/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
      return func(self, options, args)
    File "/home/jsirois/.pex/venvs/c9ddfa173396da5869fe34ecc0594b87dd4565db/d43b4081fdded907dd84e965934e4d9159642320/lib/pypy3.9/site-packages/pip/_internal/commands/install.py", line 362, in run
      raise InstallationError(
  pip._internal.exceptions.InstallationError: Could not build wheels for scipy which use PEP 517 and cannot be installed directly
...

So all the time was in building wheels from sdists for off-brand interpreters - PyPy 3.9 in my case. Perhaps you use a Mac M1 or something similar (Linux ARM) which tends not to have pre-built wheels published for it?

broad-dentist-80514

03/27/2023, 4:38 PM

Yes, M1. Thanks for your investigations! 🙂

Open in Slack

Previous Next