https://pantsbuild.org/ logo
#general
Title
# general
s

sparse-pharmacist-76603

05/27/2022, 7:05 AM
Can someone help with this error:
Copy code
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/dpetrov/.cache/pants/na
med_caches/pex_root/unzipped_pexes/aa08b4e865b8aa322f2e33f409ce7348b0188c91.
Needed cp38-cp38-manylinux_2_35_x86_64 compatible dependencies for:
 1: numpy
    Required by:
      nufft 0.0.3
    But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
I have pin ponided the issue to install_requires in setup.py
Copy code
install_requires=[
        'numpy',
    ],
If I remove that requirement I am able to use that dist as dependency in other parts of the monorepo
./pants package - works correctly but using it as dependency causes that error
r

refined-addition-53644

05/27/2022, 11:34 AM
Have you added this as requirement to
python_requirements/poetry_requirements
? Which translates to adding it to requirements.txt/pyproject.toml
s

sparse-pharmacist-76603

05/27/2022, 11:42 AM
Yes
h

happy-kitchen-89482

05/27/2022, 5:46 PM
Which version of Pants are you on, and are you using lockfiles?
Also, can you post your BUILD file and setup.py (eliding anything you need to)?
s

sparse-pharmacist-76603

05/28/2022, 4:44 AM
Sure
Copy code
#!/usr/bin/env python

import os
import sys
from numpy.distutils.core import setup, Extension

# Hackishly inject a constant into builtins to enable importing of the
# package even if numpy isn't installed. Only do this if we're not
# running the tests!
if sys.version_info[0] < 3:
    import __builtin__ as builtins
else:
    import builtins
builtins.__NUFFT_SETUP__ = True
import nufft

# Publish the library to PyPI.
if "publish" in sys.argv[-1]:
    os.system("python setup.py sdist upload")
    sys.exit()

# Push a new tag to GitHub.
if "tag" in sys.argv:
    os.system("git tag -a {0} -m 'version {0}'".format(version))
    os.system("git push --tags")
    sys.exit()

# Set up the compiled extension.
extensions = []
if not os.environ.get('READTHEDOCS', None) == 'True':
    sources = list(map(os.path.join("src", "nufft", "{0}").format,
                       ["dfftpack.f",
                        "dirft1d.f",
                        "dirft2d.f",
                        "dirft3d.f",
                        "next235.f",
                        "nufft1df90.f",
                        "nufft2df90.f",
                        "nufft3df90.f"]))
    sources += [os.path.join("nufft", "nufft.pyf")]
    extensions = [Extension("nufft._nufft", sources=sources)]

setup(
    name="nufft",
    version="0.0.3",
    author="Daniel Foreman-Mackey",
    author_email="<mailto:danfm@nyu.edu|danfm@nyu.edu>",
    url="<https://github.com/dfm/python-nufft>",
    license="MIT",
    packages=["nufft"],
    install_requires=[
        "numpy"
    ],
    setup_requires=[
        'numpy'
    ],
    ext_modules=extensions,
    description="non-uniform FFTs",
    long_description=open("README.rst").read(),
    package_data={"": ["README.rst", "LICENSE"]},
    test_suite='tests',
    tests_require=[
        'nose',
        'unittest2'
    ],
    include_package_data=True,
    classifiers=[
        # "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "Intended Audience :: Science/Research",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
        "Programming Language :: Python",
    ],
)
Copy code
poetry_requirements(name="nufft")

python_sources(name="nufft0")

resource(name="pyproject", source="pyproject.toml")
resources(
    name="nufft_files",
    sources=[
        "README.rst",
	"LICENSE",
	"<http://MANIFEST.in|MANIFEST.in>",
	"src/nufft/*",
	"nufft/nufft.pyf",
])

python_distribution(
    name="nufft_dist",
    dependencies=[
        ":pyproject",
	":nufft_files",
	"py/libs/nufft/setup.py:nufft0",
    ],
    provides=python_artifact(
        name="nufft",
        version="0.0.3",
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
    generate_setup = False,
)
Yes I am using lock files
./pants package works and correctly builds the package
But if I use it as dependency in othe BUILD I see the issue, for example in tests
Copy code
python_tests(
    dependencies=[
        "py/libs/nufft:nufft_dist",
    ]
)
h

happy-kitchen-89482

05/28/2022, 6:31 AM
Out of curiosity, why depend on the dist and not directly on the in-repo code?
s

sparse-pharmacist-76603

05/28/2022, 6:35 AM
it has extension that has to be built
I do not want to comit binaeies
here is also pyproject.yaml if that helps
Copy code
# Make lockfile
# $ dephell deps convert --env=lock
[tool.dephell.lock]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "poetrylock", path = "poetry.lock"}

[tool.poetry]
name = "nufft"
version = "0.0.3"
description = "FFT Fortran Module"
authors = ["None <none@none.com>"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Environment :: Console",
    "Framework :: Setuptools Plugin",
    "Intended Audience :: Developers",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.6",
    "Programming Language :: Python :: 3.7",
]

[tool.poetry.dependencies]
python = ">=3.8,<3.11"

numpy = { version = "1.21.6" }
pytest = "^7.1.2"

[build-system]
requires = ["setuptools", "numpy<1.22", "poetry==1.1.13", "wheel"]
build-backend = "setuptools.build_meta"
h

happy-kitchen-89482

05/28/2022, 6:38 AM
Ah right, that is exactly why we implemented consumption of "local dists"
And if you unzip the METADATA out of the wheel you build, do you see a requirement on numpy?
s

sparse-pharmacist-76603

05/28/2022, 6:45 AM
Let me check that
Copy code
❯ unzip -p dist/nufft-0.0.3-cp38-cp38-linux_x86_64.whl nufft-0.0.3.dist-info/METADATA -- | head -15
caution: filename not matched:  --
Metadata-Version: 2.1
Name: nufft
Version: 0.0.3
Summary: non-uniform FFTs
Home-page: <https://github.com/dfm/python-nufft>
Author: Daniel Foreman-Mackey
Author-email: <mailto:danfm@nyu.edu|danfm@nyu.edu>
License: MIT
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
License-File: LICENSE
Requires-Dist: numpy
I guess yes?
h

happy-kitchen-89482

05/28/2022, 7:34 AM
Yep, looks like it
Oh, I think I see why
Hmmm
Will need to dig into why we set that there, because that would prevent things from working naturally
But I'm sure there was a reason
Meanwhile as a workaround, what happens if you add numpy as an explicit dep of that
python_tests
target? Not suggesting this as a long-term solution, but let's see if that works
s

sparse-pharmacist-76603

05/28/2022, 7:43 AM
Copy code
python_tests(
    dependencies=[
        "py/libs/nufft:nufft_dist",
        "py/libs/nufft#numpy",
    ]
)
This gives the same error
Copy code
❯ ./pants list py/libs/nufft:: | grep -i numpy
py/libs/nufft#numpy
numpy is detected as dep
Is that what you meant or I misunderstood?
btw, I am on this version: 2.11.1rc1
2.11.0 - same error
I also tried to comment the line you mentioned
Copy code
dists_pex = await Get(
        Pex,
        PexRequest(
            output_filename="local_dists.pex",
            requirements=PexRequirements(wheels),
            interpreter_constraints=request.interpreter_constraints,
            additional_inputs=wheels_digest,
            internal_only=request.internal_only,
            # additional_args=["--intransitive"],
        ),
    )
in ~/.cache/pants/setup/...../local_dists.py
same error
h

happy-kitchen-89482

05/28/2022, 3:27 PM
🤔
After commenting out that line did you run with
--no-local-cache
? Otherwise changing the code like that won't cause the pex to be rebuilt
s

sparse-pharmacist-76603

05/28/2022, 11:00 PM
Tested with that option
Same result
h

happy-kitchen-89482

05/28/2022, 11:27 PM
Hmm
And did you kill
pantsd
?
We have to aggressively force Pants to run cached stuff again
But anyway, the manually added dep should have also fixed this and didn't, so something else is going on
Are you able to post a toy example repo that reproduces the problem but doesn't expose any proprietary code?
Then we can debug properly
s

sparse-pharmacist-76603

05/29/2022, 6:19 AM
Hi Benjy, Killing pantsd after removing
--intransitive
, fixed the issue
h

happy-kitchen-89482

05/30/2022, 2:25 PM
aha
that's good information
Can you file this as a bug at https://github.com/pantsbuild/pants and post a link here? Thanks!
s

sparse-pharmacist-76603

06/10/2022, 11:36 AM
h

happy-kitchen-89482

06/10/2022, 3:54 PM
Thanks!
5 Views