https://pantsbuild.org/ logo
#general
Title
# general
b

bland-yak-74764

04/05/2022, 7:50 PM
Hey all. I'm testing out pants atm. Does pants support namespace packaging (Lets say I want every package prefixed with "coolcompany", like
coolcompany.lib1
) ? Is there a demo project anywhere that shows how to do this? I currently have a project that was inspired by this: https://medium.com/@jherreras/python-microlibs-5be9461ad979 which I am attempting to migrate to pants. Goal is to have all company packages in same namespace, with option to install a subset of them (and by extension their respective subdependencies).
h

hundreds-father-404

04/05/2022, 8:08 PM
Hello! Welcome. Yes name space packages should work, and in fact pants itself uses them. The main thing to set up is your source routes. Check out these docs for more information including some example layouts. Please don't hesitate to ask for any help, we are very happy to help you figure this out 🙂 https://www.pantsbuild.org/docs/source-roots
e

echoing-farmer-15630

04/05/2022, 8:36 PM
We have a substantial project which uses pants and namespace packages; as Eric says, the source roots are the key to getting that to work, but we haven't had any trouble at all (and pants does a fine job of merging things at a fine grain of dependency, if you will, so that namespaces actually work very well indeed)
💯 1
b

bland-yak-74764

04/05/2022, 9:14 PM
Awesome. Okay, I threw together a repo to show what I'm trying to do. If anyone gets a chance to peek at it and let me know if I'm on the right track or if I'm doing something very un-pants, I would greatly appreciate it. https://github.com/AGiantSquid/pants_test
when I do
$ ./pants roots
I get
Copy code
.
python_projects/acme_corp/app1/src
python_projects/acme_corp/lib1/src
which I think is what I want
h

hundreds-father-404

04/05/2022, 9:16 PM
What you want is based on whatever your imports are. I see that you have
from acme_corp.lib.lib1_module
. What is the full file path that that should be coming from?
Also will each of those pyproject files correspond to a poetry project?
b

bland-yak-74764

04/05/2022, 9:19 PM
the full path of
lib1_module
would be
python_projects/acme_corp/lib1/src/acme_corp/lib1/lib1_module.py
I was thinking I would make
lib1
and
app1
each a separate "project", which could use poetry or pip.
so either a setup.py or a pyproject toml could go there, but that location is where I am trying to indicate is the "repo root" for a traditional python package if that makes sense.
h

hundreds-father-404

04/05/2022, 9:25 PM
Then yes, I do think that your source routes are set up the right way. Note however that in a mono repo, code can import from anywhere. That's part of the purpose, to facilitate code sharing. There may be a less strong separation between your projects than expected Very related is how third-party python requirements are handled with pants. If you have not yet read these sections I recommend checking out https://www.pantsbuild.org/docs/python-third-party-dependencies#warning-multiple-versions-of-the-same-dependency. We are still playing with better ways to explain this and to give tips on how mono repos work, so no worries if this doesn't make total sense and please I will try to explain it in a different way
b

bland-yak-74764

04/05/2022, 9:28 PM
when I attempt to run the pex, I get an import error:
Copy code
$ ./pants run python_projects/acme_corp/lib1:lib1_pex
Traceback (most recent call last):
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 503, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 408, in _wrap_coverage
    return runner(*args)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 439, in _wrap_profiling
    return runner(*args)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 559, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 695, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/pex.py", line 726, in execute_pkg_resources
    runner = entry.resolve()
  File "/c/Users/AshleyShultz/.cache/pants/named_caches/pex_root/unzipped_pexes/6151344b859fa5ca6377ed7f6029524c286be3a6/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'acme_corp'
h

hundreds-father-404

04/05/2022, 9:29 PM
b

bland-yak-74764

04/05/2022, 9:36 PM
Okay, I ran
./pants tailor
which added BUILD files which I was evidently missing, and that fixed it. 😎
🙌 1
❤️ 1
h

hundreds-father-404

04/05/2022, 9:38 PM
Sweet! Once you get to this stage, we recommend adding those commands to your CI https://www.pantsbuild.org/docs/using-pants-in-ci#recommended-commands
b

bland-yak-74764

04/05/2022, 10:16 PM
okay, testing out a distribution. It looks like the
python_sources
will automatically "name" itself after its parent directory. Is it okay to override this to make it use the namespace prefix? I think I gain something by labeling this package by the way it will be imported, but not sure if I'm breaking the rules:
(It looked like the BUILD file on the right was defaulting to name "lib1" rather than "acme_corp.lib1")
h

hundreds-father-404

04/05/2022, 10:20 PM
Yes it is OK to set the name field. Although it does not impact your application at all. It only impacts how you refer to your code when using pants https://www.pantsbuild.org/docs/targets#target-addresses
❤️ 1
b

bland-yak-74764

04/05/2022, 10:21 PM
I'm also a little confused why the pex_binary entry_point uses dot notation, but the python_distribution dependencies needs a file path. Am I doing that correctly?
h

hundreds-father-404

04/05/2022, 10:23 PM
for pex binary, you can either use the dot notation or use the file path. We actually recommend the file path because it's simpler and allows you to use file arguments where you say pants package and then the file path, rather than the target address https://www.pantsbuild.org/docs/python-package-goal#entry_point-with-a-file-name
b

bland-yak-74764

04/05/2022, 10:27 PM
I'm seeing it work when the entry_point is relative to where pants detects a root, but dependencies is a path relative to the monorepo root. does that sound correct?
h

hundreds-father-404

04/05/2022, 10:33 PM
Yes so the entry point field is a relative file path, relative to the BUILD file directory. Meanwhile, the dependencies field uses the entire path. (If the dependency is defined the same BUILD file, you can also use a shortcut to only say
:target_name
)
b

bland-yak-74764

04/05/2022, 10:37 PM
oh, I see. For the
entry_point
I didn't have
src
on there when I was using the dot notation, but when I use a path to the file I need to add it.
👍 1
h

hundreds-father-404

04/05/2022, 10:38 PM
This is not required, but it is often idiomatic to define the pex binary target in the same directory as the script that it is for. then entry point would look like main.py for example
Generally, pants encourages having metadata live as close to the code that it defines as possible. We have found over the years that this often scales the best. It's not required but a generally good practice. The tailor goal generally handles this for you
b

bland-yak-74764

04/05/2022, 10:40 PM
ah, interesting. I think my brain is used to having a setup py do most of the project definition stuff at the "project" root.
2
so, should I put the
pex_binary
and the
python_distribution
in the BUILD that is a sibling of the actual code?
h

hundreds-father-404

04/05/2022, 10:41 PM
That makes a lot of sense! I alluded to this in the earlier message, that there's a big mental model change with using pants that what matters more are individual files rather than big projects. Individual files are the atomic unit, unlike normal workflows where your setup.py project is the atomic unit
so, should I put the pex_binary and the python_distribution in the BUILD that is a sibling of the actual code?
it's not required to do that, but it's generally how folks would do this. (Pants will work just fine either way)
b

bland-yak-74764

04/05/2022, 10:54 PM
sorry, still trying to wrap my head around this. Do I actually need the folder structure to use my namespace, or is the namespace just coming from the
python_sources
name? If I changed
python_sources(name='acme_corp.lib1')
to
python_sources(name='some.random.prefixes.lib1')
, would that affect how I import the library from
from acme_corp.lib1.lib1_module import lib1_func
to
from some.random.prefixes.lib1.lib1_module import lib1_func
?
okay, I tested and it looks like that doesn't work
h

hundreds-father-404

04/05/2022, 11:02 PM
Exactly. The pants target name does not impact your own code at all
The only thing that should impact how your imports work is your source roots, the full paths of your files, and making sure that you have build files for everything (tailor)
b

bland-yak-74764

04/05/2022, 11:12 PM
okay, I think I'm getting it now. It helps a lot playing around with it with a toy example like this.
❤️ 1
💯 1
h

happy-kitchen-89482

04/06/2022, 12:18 AM
Sorry this is confusing, but this is helpful to us for improving the docs! As Eric mentioned, names of things in BUILD files are just for referencing them (e.g., on the Pants cmdline) or sometimes for naming outputs (e.g., the name of the .pex file is derived from the name of its
pex_binary
target) but it has no effect on naming inside your code, e.g., imports.
The BUILD files are the eggshells, your code is the cute little snakes inside the eggs...
🥚 2
🐍 2
😂 1
4 Views