https://pantsbuild.org/ logo
#general
Title
# general
l

lemon-oxygen-72498

11/08/2022, 9:30 AM
Hello
pants
developers πŸ‘‹, I would like to get rid of the following WARN messsages when I run tests:
Copy code
The target libs/image_processing/kaiko/image_processing/mask/statistics.py:../../../kaiko-image-processing imports `numpy`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['libs/geometry:kaiko-geometry-poetry-reqs#numpy', 'libs/geometry:kaiko-geometry-reqs#numpy', 'libs/image_processing:kaiko-image-processing-reqs#numpy'].
My project structure is as follows:
Copy code
β”œβ”€β”€ BUILD
β”œβ”€β”€ libs
β”‚   β”œβ”€β”€ geometry
β”‚   β”‚   β”œβ”€β”€ BUILD
β”‚   β”‚   β”œβ”€β”€ pyproject.toml
β”‚   β”‚   └── requirements.txt
β”‚   └── image_processing
β”‚       β”œβ”€β”€ BUILD
β”‚       β”œβ”€β”€ pyproject.toml
β”‚       └── requirements.txt
└── pants.toml
And my
root
pattern is:
Copy code
[source] # <https://www.pantsbuild.org/docs/source-roots>
marker_filenames = ["pyproject.toml"]
My
libs/image_processing/BUILD
file is as follows:
Copy code
python_requirements(name="kaiko-image-processing-reqs")

python_sources(
    name="kaiko-image-processing",
    sources=["kaiko/image_processing/**/*.py"],
    dependencies=[
        ":kaiko-image-processing-reqs#opencv-python",
        "libs/geometry:kaiko-geometry",
    ],
)
I could list in
dependencies
all external dependencies, like I did for
opencv-python
but my developers push back quite hard on boilerplate, and this will make my life to integrate
pants
harder. Is there a workaround so that
pants
doesn't go looking for dependencies higher up than the enclosing root? I would like it picks up
'libs/image_processing:kaiko-image-processing-reqs#numpy'
without further ado here. (edited)
g

gifted-agency-25998

11/08/2022, 9:33 AM
I ran into this problem yesterday actually. It's because you have a
pyproject.toml
AND a
requirements.txt
. I removed the
requirements.txt
file to fix it. Would that work for you?
l

lemon-oxygen-72498

11/08/2022, 9:34 AM
Damn, no that wouldn't because we use
requirements.txt
for provisioning local development sandboxes, but
pyproject.toml
for deployment πŸ˜•
But it's good to know the reason nevertheless, thanks @gifted-agency-25998 πŸ™
g

gifted-agency-25998

11/08/2022, 9:36 AM
No worries - You could remove the
reqs
lines in the BUILD file, which would fix it at least until you
tailor
again. There is probably a better solution - as my experience with pants (the build system) is limited to the last few days
c

curved-television-6568

11/08/2022, 1:36 PM
When you have multiple requirements like this, you need to separate them into their own dedicated resolves. Then for your source targets, you specify which resolve they should use (may be multiple using
parametrize
). See: https://www.pantsbuild.org/docs/python-third-party-dependencies#warning-multiple-versions-of-the-same-dependency https://www.pantsbuild.org/docs/targets#parametrizing-targets https://www.pantsbuild.org/docs/targets#field-default-values <- is useful if you need to set the resolve for a whole subtree in your project. https://www.pantsbuild.org/docs/reference-python#enable_resolves
Ps. I’d recommend to not add your dependencies explicitly in your BUILD files, they should Pants be able to infer for you, the exception being for
files
or
resources
targets in most cases. If a third party is not inferred properly, it may need a
module_mapping
configuration on the
python_requirements
target to help guide Pants from the import name to the distribution name when it is not 1 to 1.
l

lemon-oxygen-72498

11/08/2022, 2:10 PM
Thanks @curved-television-6568, I'll try your suggestions πŸ‘ Regarding not declaring the dependencies, I don't see how
pants
could decide which dependencies to pick up (between
requirements.txt
or
pyproject.toml
. We want to use the
requirements.txt
when running tests for example, but not when packaging (
requirements.txt
is for sandboxes, but not for distribution).
c

curved-television-6568

11/08/2022, 2:14 PM
Cool. So to answer the last question there, you’d provide one resolve to your
python_tests
targets, matching the resolve for your
python_requirements
target you want to use, then for your
python_distribution
you include your
python_sources
with the parametrized version that uses the resolve you want to use for packaging, which will need to be parameterized on resolve like:
python_sources(…, resolve=parametrize("a", "b"), …)
where β€œa” and β€œb” are your resolve names. https://www.pantsbuild.org/docs/reference-python_tests#coderesolvecode
l

lemon-oxygen-72498

11/08/2022, 2:16 PM
I see, awesome, will try it out πŸ‘
πŸ‘ 1
h

happy-kitchen-89482

11/08/2022, 3:37 PM
You definitely can use multiple resolves, as Andreas mentioned. But one thing Pants enables, and is often simpler, is to have a single universe of requirements for your entire repo (often expressed in a single global requirements.txt or similar). Pants then automatically uses the relevant subset of those requirements in each situation (e.g., when running a test or packaging a pex). The advantage of this is that you get compatibility across the codebase, and there is a lot less manual management of requirements.
(The multiple resolves feature is still needed for when you can’t share requirements across the entire repo, because different parts of it genuinely need incompatible versions of some requirements)
You absolutely can use multiple resolves to continue to model β€œone set of requirements per project”
l

lemon-oxygen-72498

11/08/2022, 3:38 PM
Hello Benjy, funnilly this "global requirements.txt" was the initial state of our monorepo, but all developers except myself expressed that they'd rather have smaller sandboxes and hence multiple requirements.txt files.
h

happy-kitchen-89482

11/08/2022, 3:39 PM
Ah, Pants creates those smaller sandboxes for you!
Automatically
based on the exact
imports
and other deps in the code
l

lemon-oxygen-72498

11/08/2022, 3:39 PM
But how would different projects have different (sometimes conflicting) requirements then?
h

happy-kitchen-89482

11/08/2022, 3:39 PM
It’s definitely a different way of thinking about Python requirements than the standard way
If the requirements need to conflict, then you do need multiple of these β€œuniverses”, one per β€œset of conflicts”
But either way - Pants will select for each project the subset of requirements it needs from the β€œuniverse” with which it is compatible.
It’s a bit of a paradigm shift, so it takes some getting used to, but it has a lot of advantages
4 Views