https://pantsbuild.org/ logo
Powered by
j

jolly-oyster-17014

10/18/2022, 6:57 PM
I've been trying to introduce 
torch-geometric
into my monorepo, and I've come across an interesting problem. Installing this depends on 
torch-scatter
and 
torch-sparse
(docs here), but the issues arises here with M1 macbooks. There are no official arm64 wheels available on the index. This is usually fine - i can install from source, but 
setup.py
on these packages have a hard dependency on 
torch
(link), so trying to do this would lead to 
module not found: torch
. I followed the docs for undeclared dependencies, https://www.pantsbuild.org/docs/python-third-party-dependencies#requirements-with-undeclared-dependencies, but this still generates an issue with 
python setup.py egg_info
, stack trace:
Copy code
WARNING: Discarding git+<https://github.com/rusty1s/pytorch_scatter@62a958f467bf80c3f566083b4949ec21e2be57d0>. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding git+<https://github.com/rusty1s/pytorch_sparse@6143af2112cd404adca42111daf14397a45f2d7e>. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement torch-scatter (unavailable)
ERROR: No matching distribution found for torch-scatter (unavailable)
Pinning a single page link source is also problematic since it should build for setups with cuda & without cuda on cpu - there are single page source for each of these cases for the wheels https://data.pyg.org/whl/torch-1.11.0+cpu.html and https://data.pyg.org/whl/torch-1.11.0+cu113.html Any tips/ideas on how to get around this issue? (This usually is no problem in conda because of the package metadata, but there's no way to incorporate conda into pants right?)
Hmm I'm wondering if there is something wrong I'm doing with declaring dependencies. I currently have a requirements.txt in 
3rdparty/python
and the buildfile contains:
Copy code
python_requirements(
    name="reqs",
    overrides={
        "torch-geometric": {"dependencies": [":reqs#torch-scatter", ":reqs#torch-sparse"]},
        "torch-scatter": {"dependencies": [":reqs#torch"]},
        "torch-sparse": {"dependencies": [":reqs#torch"]},
    }
)
and the requirements.txt have:
Copy code
torch==1.11.0
torch-geometric==2.0.4

torch-scatter==2.0.9
torch-sparse==0.6.15
# also tried installing from source w pep404 syntax
#torch-scatter @ git+<https://github.com/rusty1s/pytorch_scatter@62a958f467bf80c3f566083b4949ec21e2be57d0>
#torch-sparse @ git+<https://github.com/rusty1s/pytorch_sparse@6143af2112cd404adca42111daf14397a45f2d7e>
e

enough-analyst-54434

10/18/2022, 7:24 PM
The Pants documentation you found only deals with handling undeclared runtime dependencies, the issue here is an undeclared buildtime dependency. The fundamental issue here is Pants (Pex) builds distributions in isolation. It does not, mutate a venv like Pip does. In other words, if the very building of a package requires some other package (other than setuptools or wheel), that build must say so. If a 
setup.py
build, this means using `setup_requires`(which has known issues - see the note here) or else using PEP-518 builds where build requirements are handled 1st class. Without this, the build will be happening from a clean slate and it will find no torch present in the build environment.
The only way to work around all this is to pre-build the wheel and host it in your own index or find-links repo and then configure Pants to look at that additional index or find-links repo using https://www.pantsbuild.org/docs/reference-python-repos
j

jolly-oyster-17014

10/18/2022, 7:58 PM
I had a hunch that it would come to 1. forking the repo or 2. porting over + adding the arm wheel and hosting them myself.. thanks for the response.
5 Views
Powered by