Hey all. I'm testing out pants atm. Does pants support namespace packaging (Lets say I want every package prefixed with "coolcompany", like

coolcompany.lib1

) ? Is there a demo project anywhere that shows how to do this? I currently have a project that was inspired by this: https://medium.com/@jherreras/python-microlibs-5be9461ad979 which I am attempting to migrate to pants. Goal is to have all company packages in same namespace, with option to install a subset of them (and by extension their respective subdependencies).

Hello! Welcome. Yes name space packages should work, and in fact pants itself uses them. The main thing to set up is your source routes. Check out these docs for more information including some example layouts. Please don't hesitate to ask for any help, we are very happy to help you figure this out 🙂 https://www.pantsbuild.org/docs/source-roots

We have a substantial project which uses pants and namespace packages; as Eric says, the source roots are the key to getting that to work, but we haven't had any trouble at all (and pants does a fine job of merging things at a fine grain of dependency, if you will, so that namespaces actually work very well indeed)

Awesome. Okay, I threw together a repo to show what I'm trying to do. If anyone gets a chance to peek at it and let me know if I'm on the right track or if I'm doing something very un-pants, I would greatly appreciate it. https://github.com/AGiantSquid/pants_test

What you want is based on whatever your imports are. I see that you have

from acme_corp.lib.lib1_module

. What is the full file path that that should be coming from?

the full path of

lib1_module

would be

python_projects/acme_corp/lib1/src/acme_corp/lib1/lib1_module.py

Then yes, I do think that your source routes are set up the right way. Note however that in a mono repo, code can import from anywhere. That's part of the purpose, to facilitate code sharing. There may be a less strong separation between your projects than expected Very related is how third-party python requirements are handled with pants. If you have not yet read these sections I recommend checking out https://www.pantsbuild.org/docs/python-third-party-dependencies#warning-multiple-versions-of-the-same-dependency. We are still playing with better ways to explain this and to give tips on how mono repos work, so no worries if this doesn't make total sense and please I will try to explain it in a different way

when I attempt to run the pex, I get an import error:

$ ./pants run python_projects/acme_corp/lib1:lib1_pex
Traceback (most recent call last):
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 503, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 408, in _wrap_coverage
    return runner(*args)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 439, in _wrap_profiling
    return runner(*args)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 559, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 695, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 726, in execute_pkg_resources
    runner = entry.resolve()
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'acme_corp'

See this for some tips on how to debug https://www.pantsbuild.org/docs/troubleshooting#import-errors-and-missing-dependencies

Okay, I ran

./pants tailor

which added BUILD files which I was evidently missing, and that fixed it. 😎

Sweet! Once you get to this stage, we recommend adding those commands to your CI https://www.pantsbuild.org/docs/using-pants-in-ci#recommended-commands

okay, testing out a distribution. It looks like the

python_sources

will automatically "name" itself after its parent directory. Is it okay to override this to make it use the namespace prefix? I think I gain something by labeling this package by the way it will be imported, but not sure if I'm breaking the rules:

Yes it is OK to set the name field. Although it does not impact your application at all. It only impacts how you refer to your code when using pants https://www.pantsbuild.org/docs/targets#target-addresses

I'm also a little confused why the pex_binary entry_point uses dot notation, but the python_distribution dependencies needs a file path. Am I doing that correctly?

for pex binary, you can either use the dot notation or use the file path. We actually recommend the file path because it's simpler and allows you to use file arguments where you say pants package and then the file path, rather than the target address https://www.pantsbuild.org/docs/python-package-goal#entry_point-with-a-file-name

I'm seeing it work when the entry_point is relative to where pants detects a root, but dependencies is a path relative to the monorepo root. does that sound correct?

Yes so the entry point field is a relative file path, relative to the BUILD file directory. Meanwhile, the dependencies field uses the entire path. (If the dependency is defined the same BUILD file, you can also use a shortcut to only say

:target_name

)

oh, I see. For the

entry_point

I didn't have

src

on there when I was using the dot notation, but when I use a path to the file I need to add it.

This is not required, but it is often idiomatic to define the pex binary target in the same directory as the script that it is for. then entry point would look like main.py for example

ah, interesting. I think my brain is used to having a setup py do most of the project definition stuff at the "project" root.

That makes a lot of sense! I alluded to this in the earlier message, that there's a big mental model change with using pants that what matters more are individual files rather than big projects. Individual files are the atomic unit, unlike normal workflows where your setup.py project is the atomic unit

sorry, still trying to wrap my head around this. Do I actually need the folder structure to use my namespace, or is the namespace just coming from the

python_sources

name? If I changed

python_sources(name='acme_corp.lib1')

to

python_sources(name='some.random.prefixes.lib1')

, would that affect how I import the library from

from acme_corp.lib1.lib1_module import lib1_func

to

from some.random.prefixes.lib1.lib1_module import lib1_func

?

Exactly. The pants target name does not impact your own code at all

okay, I think I'm getting it now. It helps a lot playing around with it with a toy example like this.

Sorry this is confusing, but this is helpful to us for improving the docs! As Eric mentioned, names of things in BUILD files are just for referencing them (e.g., on the Pants cmdline) or sometimes for naming outputs (e.g., the name of the .pex file is derived from the name of its

pex_binary

target) but it has no effect on naming inside your code, e.g., imports.