I m struggling to generate lockfile with pants when I have

I'm struggling to generate lockfile with pants whe...

gentle-flower-25372

03/12/2024, 2:41 PM

I'm struggling to generate lockfile with pants when I have fasttext is in

pyproject.toml

(

poetry_requirements

). The error is complaining that pybind11 is not available. I'm guessing I need to make it available to the lockfile process somehow. I tried installing via pipx and making available on path, but it's still not working. repro repository: https://github.com/jasonwbarnett/repro-pants-lockfile-fasttext

Copy code

08:46:13.33 [INFO] Completed: Generate lockfile for python-default
08:46:13.34 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
/Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/bin/python: No module named pip
Traceback (most recent call last):
  File "<string>", line 38, in __init__
ModuleNotFoundError: No module named 'pybind11'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/pex", line 268, in <module>
    exec(ast, globals_map, locals_map)
  File "-c <cmd>", line 10, in <module>
  File "/Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
  File "/Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/lib/python3.9/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
    self.run_setup()
  File "/Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
    exec(code, locals())
  File "<string>", line 72, in <module>
  File "<string>", line 41, in __init__
RuntimeError: pybind11 install failed.
Could not gather lock metadata for 1 project with source artifacts:
1. /private/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/pants-sandbox-XkdlhZ/.tmp/tmpct0cd_ap/opt.homebrew.Cellar.python@3.9.3.9.18_1.Frameworks.Python.framework.Versions.3.9.bin.python3.9/fasttext-0.9.2.tar.gz: Executing /Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/bin/python -sE /Users/jasonbarnett/.cache/pants/named_caches/pex_root/venvs/ad73efd44905eafe035f20860cf75c0f2c4b2e3f/bf23592afc6bbf9e59da204a4c864270f9819638/pex -c import json
import sys

import setuptools.build_meta


if not hasattr(setuptools.build_meta.__legacy__, 'get_requires_for_build_wheel'):
    sys.exit(75)

result = setuptools.build_meta.__legacy__.get_requires_for_build_wheel(*(), **{})
with open('/private/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/pants-sandbox-XkdlhZ/.tmp/tmprins5mp5', "w") as fp:
    json.dump(result, fp)
 failed with 1



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

gentle-flower-25372

03/12/2024, 2:51 PM

It also doesn't work to install via poetry, but it does work when I install via pip.

Copy code

$ poetry install
Using python3 (3.9.18)
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing fasttext (0.9.2): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel

  /private/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/tmpy3v47qm6/.venv/bin/python: No module named pip
  Traceback (most recent call last):
    File "<string>", line 38, in __init__
  ModuleNotFoundError: No module named 'pybind11'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/jasonbarnett/Library/Application Support/pipx/venvs/poetry/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/jasonbarnett/Library/Application Support/pipx/venvs/poetry/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/jasonbarnett/Library/Application Support/pipx/venvs/poetry/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/tmpy3v47qm6/.venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/tmpy3v47qm6/.venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/tmpy3v47qm6/.venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
      super().run_setup(setup_script=setup_script)
    File "/private/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/tmpy3v47qm6/.venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 72, in <module>
    File "<string>", line 41, in __init__
  RuntimeError: pybind11 install failed.


  at ~/Library/Application Support/pipx/venvs/poetry/lib/python3.12/site-packages/poetry/installation/chef.py:152 in _prepare
      148│
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│
      151│             if error is not None:
    → 152│                 raise error from None
      153│
      154│             return path
      155│
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with fasttext (0.9.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "fasttext (==0.9.2)"'.

gentle-flower-25372

03/12/2024, 2:53 PM

Copy code

$ pip wheel --use-pep517 "fasttext (==0.9.2)"
Collecting fasttext==0.9.2
  Downloading fasttext-0.9.2.tar.gz (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.8/68.8 kB 685.5 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pybind11>=2.2 (from fasttext==0.9.2)
  File was already downloaded /Users/jasonbarnett/git/repro-pants-lockfile-fasttext/pybind11-2.11.1-py3-none-any.whl
Collecting setuptools>=0.7.0 (from fasttext==0.9.2)
  File was already downloaded /Users/jasonbarnett/git/repro-pants-lockfile-fasttext/setuptools-69.1.1-py3-none-any.whl
Collecting numpy (from fasttext==0.9.2)
  File was already downloaded /Users/jasonbarnett/git/repro-pants-lockfile-fasttext/numpy-1.26.4-cp39-cp39-macosx_11_0_arm64.whl
Building wheels for collected packages: fasttext
  Building wheel for fasttext (pyproject.toml) ... done
  Created wheel for fasttext: filename=fasttext-0.9.2-cp39-cp39-macosx_14_0_arm64.whl size=302956 sha256=edaf82496b9a76ef878e800dadc707b081cbfcf6bde7596b9a8df3781b4d07be
  Stored in directory: /Users/jasonbarnett/Library/Caches/pip/wheels/64/57/bc/1741406019061d5664914b070bd3e71f6244648732bc96109e
Successfully built fasttext

gentle-flower-25372

03/12/2024, 3:21 PM

@bitter-ability-32190 From looking through the history it seems like you have experience getting this working? I read you pulled in pybind11 via http, but I'm not sure if your use case is different or not. I'm not compiling first party source, just depending on 3rd party python libraries that depend on pybind11.

bitter-ability-32190

03/12/2024, 11:21 PM

Hmm nothing sticks out at me. Either time has weathered my memory or what you're doing is different

bitter-ability-32190

03/12/2024, 11:22 PM

I'm also not using pants at $dayjob anymore, so that doesn't help my memory either

gentle-flower-25372

03/12/2024, 11:23 PM

ahh, makes sense. Thanks for engaging and at least briefly reviewing. I've "solved" my problem by swapping fasttext with fasttext-wheel. Not ideal for production, but it will unblock me for now.

6 Views

Open in Slack

Previous Next