https://pantsbuild.org/ logo
#general
Title
# general
c

clean-manchester-83682

02/10/2023, 11:40 PM
Hi everyone! I'm trying to figure out how to exclude a transitive dependency. I'm depending on a python library called talon and it then depends on a library called scipy. First thing is when i call dependencies with --dependencies-transitive flag, it only shows one layer of transitive dependencies, which is talon. It doesn't the entire dependency tree.
./pants dependencies sailebot/referral:referral --dependencies-transitive
So i'm able to exclude talon fine, but i want to exclude a dependency that talon depends on. This works, to exclude talon
Copy code
python_awslambda(
    name="referral",
    runtime="python3.7",
    handler="referral_lambda.py:lambda_handler",
    dependencies=["//:reqs#psycopg2-binary","!!//:reqs#talon"],
)
But this doesn't work
Copy code
python_awslambda(
    name="referral",
    runtime="python3.7",
    handler="referral_lambda.py:lambda_handler",
    dependencies=["//:reqs#psycopg2-binary","!!//:reqs#scipy"],
)
e

enough-analyst-54434

02/11/2023, 1:11 AM
You cannot exclude transitive third party Python dependencies with Pants, Pex or Pip (or in the Python world generally that I'm aware of).
Why do you want to do this?
c

clean-manchester-83682

02/11/2023, 1:54 PM
Thanks for the response @enough-analyst-54434! It is a lambda function, and pulling in scipy doubles the size and puts it over the size limit. The function we are using within talon, doesn't use scipy. So we want to pull in talon, but not scipy. We haven't been successful at finding a replacement for talon I'm able to exclude talon, so you can exclude 3rd party that you directly depend on within your monorepo, just not a 3rd party that a 3rd party depends on? project-a (lambda) --project-b (library) ----talon (3rd party referenced in requirements.txt) --------scipy (3rd party not referenced in requirements.txt)
e

enough-analyst-54434

02/11/2023, 1:56 PM
Correct, you cannot exclude a transitive 3rdparty dependency.
Well, you can always get creative. Just switch your talon dep to a VCS dep on a fork of it's source: https://github.com/mailgun/talon/tree/master - in your fork, edit setup.py to not include scipy.
The Rust world uses forks for patching a good deal, the Python world doesn't seem to even though the ecosystem supports it. Not sure why.
c

clean-manchester-83682

02/11/2023, 2:03 PM
I appreciate your time in responding. I'll give it a try. Thanks!
c

clean-manchester-83682

02/11/2023, 3:45 PM
That worked! As I was wrapping up, i ran across this in the setup.py within talon. https://github.com/mailgun/talon/blob/master/setup.py
Copy code
class InstallCommand(install):
    user_options = install.user_options + [
        ('no-ml', None, "Don't install without Machine Learning modules."),
    ]
    ...
    for not_required in ["numpy", "scipy", "scikit-learn==0.24.1"]:
                dist.install_requires.remove(not_required)
This is what i needed. Is there a way to specify no-ml within pants?
e

enough-analyst-54434

02/11/2023, 3:52 PM
Hrm , they are doing that in an entirely non-standard way 😕 The standard way is to provide extras, and then you'd say
talon[ml]
as a requirement to get ML deps but they would be left off otherwise. All Python tools speak extras. As it is .. I don't think so. You could with Pex, but I believe Pants gets in the way here. With Pex / Pip you use a requirements file that passes the options as documented here: https://pip.pypa.io/en/stable/reference/requirements-file-format/#per-requirement-options
Something like this line in a requirements.txt file for the talon dependency:
talon --install-option=no-ml
Maybe you could pitch in an issue / PR to help talon do this the right way. Perhaps they know the right way but want the ml extras to be default though.
👍 1
5 Views