I'm sorry if my question seems foolish. I have been using namespace packages (https://packaging.python.org/en/latest/guides/packaging-namespace-packages/) in my Python repository to ensure a consistent namespace while allowing me to choose from several smaller packages when building Docker images, thus avoiding the inclusion of a large and unnecessary set of libraries. I use Poetry to manage the dependencies among my libraries and third-party packages. The folder structure is organized as follows:

.
├── libs
│   ├── lib1
│   │   ├── company_name
│   │   │   └── lib1
│   │   │       ├── __init__.py
│   │   │       └── something.py
│   │   │   
│   │   └── tests
│   │       └── test1.py
│   └── lib2
│       ├── company_name
│       │   └── lib2
│       │       ├── __init__.py
│       │       └── something.py
│       └── tests
│           └── test2.py
│           
└── executable_apps
    ├── app1
    │   └── app1.py
    └── app2
        └── app2.py

If I was to adopt pants, would there be any need for using namespace packages? Would Pants be able to handle only pulling in the necessary libraries or files?

Pants can select just the files needed regardless of namespace packages or repo layout. You could have a 10000 file repo with 100 "binaries" and for each individual binary Pants will only package the exact 1st party files that comprise that binary and the exact (transitive) 3rd party dependencies needed by those files.

That is what I thought. I was just not sure if it would pick just the right files, or just the right libraries (which is basically what I am doing with poetry).

Yeah, your binary is a single entry point and import statements are parsed and recurse.

Thanks for the help; it is invaluable to get over the relatively steep learning curve. I will probably then remove the namespace package structure (just use company_name/lib1, company_name/lib2...) and let pants figure out the files needed for my binaries. But in that case, are there any best practices when it comes to test locations (specifically for pants)?

Not really. Whatever you like that works well with pytest, Pants aside. Pants is pretty un-opinionated about repo layout.

Yes, in the binaries we are using company_name.lib1 and company_name.lib2; but I might have to look more carefully at the use of

__init__.py

files.

Yeah, Pants doesn't care - it just makes guessing repo layout doing remote support hard. There is no way to look at a tree and guess where its sys.path roots should be in a PEP-420 world. An empty directory could be just a container for a project or it could be a Python package.

I see. This might be the reason I am having issues running my tests through pants 😕

You need to tell it using

[source]

root patterns. It can't guess either!

What would be a good structure in this case, for making it easy for pants to guess the roots? Also, in the /libs/lib1 folder I have a pyproject.toml file which defines (with poetry) dev dependencies needed for the tests (hypothesis for example). If the source roots were correct, would pants load them up?

As to the latter - no, you need a BUILD target to tell Pants to load up dependencies. In this case

poetry_requirements()

in a BUILD sibling of the

pyproject.toml

would do it: https://www.pantsbuild.org/docs/reference-poetry_requirements But one thing you need to shake is the impression that placement implies anything. It does not. If you have 13 pyproject.toml scattered about, each with a BUILD target, then all the deps from all 13 pyproject.toml will be visible to all source files anywhere in the repo. Since Pants parses imports to determine dependencies, this means everything gets exactly what it needs and no more.

Thank you so much. I think this might be coming together. I am trying to get some of the tests running (I already get most of the linters and formatters working just fine). It constantly complaints about missing source roots, so I am just adding those for now; and hopefully get them working. Once I have resolved the missing source roots, I plan to reorganize my repository. Specifically, I plan to remove the unnecessary nesting of my libraries. Instead of having the structure

/libs/lib1/company_name/lib1

, I will use the structure

/libs/company_name/lib1

. The current structure was created to make my packages smaller for building Docker images, but it seems like Pants handles that issue for me.

Sounds about right to me!

Well. Pants is no longer complaining about missing source roots when I run

> pants test libs/lib1::

but complains about missing dependencies, numpy used in lib1/something.py and hypothesis used in lib1/tests/test1. Both libraries are specified in /libs/lib1/pyproject.toml (numpy under dependencies and hypothesis under dev-dependencies). The pyproject.toml has a sibling BUILD file that looks like this

poetry_requirements(
    name="poetry",
)

Any ideas?

Is numpy used via import statement or via something more dynamic like

__import("...")__

?

Yes, I used ./pants tailor for this. Unfortunately, this is all private. I shared above the content of the /libs/lib1/BUILD file. Other relevant files might be /libs/lib1/company_name/lib1/BUILD

python_sources()

and /libs/lib1/tests/BUILD/

python_tests()

What does

./pants dependencies --transitive libs/libs1/tests/test1.py

show?

It shows the list of files it depends on under /libs/lib1/company_name/lib1 but one thing looks weird.

libs/lib2/pyproject.toml:poetry

Ok, I'm going to go slow here since I'm blind; so forgive me for that. Can you confirm at least 1 of the files in the transitive dependency list has an

import numpy

or

from numpy import ...

in it?

Nothing to forgive, and again, thank you for your help. I am using pants version 2.15.0.rc5. No, I am not using any lockfiles or resolves.

Ok, and does libs/lib1/pyproject.toml declare a numpy dependency?

Yes, it does. But the wrong libs//lib2/pyproject.toml file does not.

Ok, well there is no right or wrong - Pants should be looking at all of them if they have an associated BUILD target.

This is normal dependency, but I specify a particular version of numpy (^1.22); but there might be conflicting numpy versions used in different pyproject.toml files.

Ok. And what does this say?:

./pants list libs/lib1:

It includes both dependencies:

libs/lib1:poetry#hypothesis
libs/lib1:poetry#numpy

Ok. All looks good.

Yes. I can remove the BUILD target, but what should that do?

If there is an issue due to numpy ambiguity it removes it temporarily.

It did not change anything. I was just trying to understand how you are approaching this 🙂

Well, you've removed the lib2 BUILD hopefully so that we can ignore that for now?

I misunderstood you a bit when you told me to remove the BUILD file. I did it again, and now I get a log warning message from pants concerning numpy. Something along the lines of "The target libs/lib1/company_name/lib1/somefile.py imports

numpy

, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use ..." And then a list of other libraries which use numpy. This might be useful; but I will have to take a better look at this tomorrow. It is getting late her.

Aha!

Close to 20 🤯

And are they real, or project devs being capricious? COuld you force any of these 20 conflicts to merge?

I am using numpy all over the place; and I probably can merge most of them. Most of them are probably not real.

Ok, great.

I will have to take a look at this again tomorrow. Thank you so much for your help. I now have something to actually look at 🙂 I will try to check-in again tomorrow and let you know how it goes.

Eventually you'll be left with - say 3 conflicts and will need 3 resolves, read here: https://www.pantsbuild.org/docs/python-third-party-dependencies#multiple-lockfiles And definitely start using lockfiles!

not sure if this is helpful - but responding to original question - im using a bunch of published (pypi) namespaced packages with pants to set it up i had to figure out a few things in terms of my pants understanding, but also in terms of how i wanted the packages to interact with each other - ie using repo versions <> published versions in the end i added a couple of pants plugins to simplify the setup, and as is often the case, there are trade offs with the decisions i made, but it works pretty well for our needs if sometimes imperfectly/quirkily the repo is https://github.com/envoyproxy/pytooling