https://pantsbuild.org/ logo
Join Slack
Powered by
Can someone help with this error: ```pex.environme...
# general
s
Can someone help with this error:
Copy code
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/dpetrov/.cache/pants/na
med_caches/pex_root/unzipped_pexes/aa08b4e865b8aa322f2e33f409ce7348b0188c91.
Needed cp38-cp38-manylinux_2_35_x86_64 compatible dependencies for:
 1: numpy
    Required by:
      nufft 0.0.3
    But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
I have pin ponided the issue to install_requires in setup.py
Copy code
install_requires=[
        'numpy',
    ],
If I remove that requirement I am able to use that dist as dependency in other parts of the monorepo
./pants package - works correctly but using it as dependency causes that error
r
Have you added this as requirement to 
python_requirements/poetry_requirements
? Which translates to adding it to requirements.txt/pyproject.toml
s
Yes
h
Which version of Pants are you on, and are you using lockfiles?
Also, can you post your BUILD file and setup.py (eliding anything you need to)?
s
Sure
Copy code
#!/usr/bin/env python

import os
import sys
from numpy.distutils.core import setup, Extension

# Hackishly inject a constant into builtins to enable importing of the
# package even if numpy isn't installed. Only do this if we're not
# running the tests!
if sys.version_info[0] < 3:
    import __builtin__ as builtins
else:
    import builtins
builtins.__NUFFT_SETUP__ = True
import nufft

# Publish the library to PyPI.
if "publish" in sys.argv[-1]:
    os.system("python setup.py sdist upload")
    sys.exit()

# Push a new tag to GitHub.
if "tag" in sys.argv:
    os.system("git tag -a {0} -m 'version {0}'".format(version))
    os.system("git push --tags")
    sys.exit()

# Set up the compiled extension.
extensions = []
if not os.environ.get('READTHEDOCS', None) == 'True':
    sources = list(map(os.path.join("src", "nufft", "{0}").format,
                       ["dfftpack.f",
                        "dirft1d.f",
                        "dirft2d.f",
                        "dirft3d.f",
                        "next235.f",
                        "nufft1df90.f",
                        "nufft2df90.f",
                        "nufft3df90.f"]))
    sources += [os.path.join("nufft", "nufft.pyf")]
    extensions = [Extension("nufft._nufft", sources=sources)]

setup(
    name="nufft",
    version="0.0.3",
    author="Daniel Foreman-Mackey",
    author_email="<mailto:danfm@nyu.edu|danfm@nyu.edu>",
    url="<https://github.com/dfm/python-nufft>",
    license="MIT",
    packages=["nufft"],
    install_requires=[
        "numpy"
    ],
    setup_requires=[
        'numpy'
    ],
    ext_modules=extensions,
    description="non-uniform FFTs",
    long_description=open("README.rst").read(),
    package_data={"": ["README.rst", "LICENSE"]},
    test_suite='tests',
    tests_require=[
        'nose',
        'unittest2'
    ],
    include_package_data=True,
    classifiers=[
        # "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "Intended Audience :: Science/Research",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
        "Programming Language :: Python",
    ],
)
Copy code
poetry_requirements(name="nufft")

python_sources(name="nufft0")

resource(name="pyproject", source="pyproject.toml")
resources(
    name="nufft_files",
    sources=[
        "README.rst",
	"LICENSE",
	"<http://MANIFEST.in|MANIFEST.in>",
	"src/nufft/*",
	"nufft/nufft.pyf",
])

python_distribution(
    name="nufft_dist",
    dependencies=[
        ":pyproject",
	":nufft_files",
	"py/libs/nufft/setup.py:nufft0",
    ],
    provides=python_artifact(
        name="nufft",
        version="0.0.3",
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
    generate_setup = False,
)
Yes I am using lock files
./pants package works and correctly builds the package
But if I use it as dependency in othe BUILD I see the issue, for example in tests
Copy code
python_tests(
    dependencies=[
        "py/libs/nufft:nufft_dist",
    ]
)
h
Out of curiosity, why depend on the dist and not directly on the in-repo code?
s
it has extension that has to be built
I do not want to comit binaeies
here is also pyproject.yaml if that helps
Copy code
# Make lockfile
# $ dephell deps convert --env=lock
[tool.dephell.lock]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "poetrylock", path = "poetry.lock"}

[tool.poetry]
name = "nufft"
version = "0.0.3"
description = "FFT Fortran Module"
authors = ["None <none@none.com>"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Environment :: Console",
    "Framework :: Setuptools Plugin",
    "Intended Audience :: Developers",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.6",
    "Programming Language :: Python :: 3.7",
]

[tool.poetry.dependencies]
python = ">=3.8,<3.11"

numpy = { version = "1.21.6" }
pytest = "^7.1.2"

[build-system]
requires = ["setuptools", "numpy<1.22", "poetry==1.1.13", "wheel"]
build-backend = "setuptools.build_meta"
h
Ah right, that is exactly why we implemented consumption of "local dists"
And if you unzip the METADATA out of the wheel you build, do you see a requirement on numpy?
s
Let me check that
Copy code
❯ unzip -p dist/nufft-0.0.3-cp38-cp38-linux_x86_64.whl nufft-0.0.3.dist-info/METADATA -- | head -15
caution: filename not matched:  --
Metadata-Version: 2.1
Name: nufft
Version: 0.0.3
Summary: non-uniform FFTs
Home-page: <https://github.com/dfm/python-nufft>
Author: Daniel Foreman-Mackey
Author-email: <mailto:danfm@nyu.edu|danfm@nyu.edu>
License: MIT
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
License-File: LICENSE
Requires-Dist: numpy
I guess yes?
h
Yep, looks like it
Oh, I think I see why
Hmmm
Will need to dig into why we set that there, because that would prevent things from working naturally
But I'm sure there was a reason
Meanwhile as a workaround, what happens if you add numpy as an explicit dep of that 
python_tests
target? Not suggesting this as a long-term solution, but let's see if that works
s
Copy code
python_tests(
    dependencies=[
        "py/libs/nufft:nufft_dist",
        "py/libs/nufft#numpy",
    ]
)
This gives the same error
Copy code
❯ ./pants list py/libs/nufft:: | grep -i numpy
py/libs/nufft#numpy
numpy is detected as dep
Is that what you meant or I misunderstood?
btw, I am on this version: 2.11.1rc1
2.11.0 - same error
I also tried to comment the line you mentioned
Copy code
dists_pex = await Get(
        Pex,
        PexRequest(
            output_filename="local_dists.pex",
            requirements=PexRequirements(wheels),
            interpreter_constraints=request.interpreter_constraints,
            additional_inputs=wheels_digest,
            internal_only=request.internal_only,
            # additional_args=["--intransitive"],
        ),
    )
in ~/.cache/pants/setup/...../local_dists.py
same error
h
🤔
After commenting out that line did you run with 
--no-local-cache
? Otherwise changing the code like that won't cause the pex to be rebuilt
s
Tested with that option
Same result
h
Hmm
And did you kill 
pantsd
?
We have to aggressively force Pants to run cached stuff again
But anyway, the manually added dep should have also fixed this and didn't, so something else is going on
Are you able to post a toy example repo that reproduces the problem but doesn't expose any proprietary code?
Then we can debug properly
s
Hi Benjy, Killing pantsd after removing 
--intransitive
, fixed the issue
h
aha
that's good information
Can you file this as a bug at https://github.com/pantsbuild/pants and post a link here? Thanks!
s
h
Thanks!
5 Views
Open in Slack
PreviousNext
Powered by