Apologies if this question is overly broad, but le...
# general
n
Apologies if this question is overly broad, but let's say I have a source root of ["/"] with directories py/shared and py/src underneath it. I have a file /py/shared/logging.py and a subdirectory of an app directory at /py/src/apps/app01 where I want to import a function from logging.py (basically some utility function I want to easily share anywhere in my python apps). What's the leanest way for me to make /py/shared/logging.py available to be declared as a dependency and the leanest way for me to declare it as a dependency in app01's BUILD file? Bonus points if the answer allows me to just do something like "import configure from logging" inside files in that app directory instead of "import configure from py.src.logging". If this is more of a Python question than a Pants question, please feel free to show me the door. I just feel like I'm missing some clean, convenience manner of declaring non-inferrable dependencies in BUILD files and so I always end up just putting the full path from the source root to the dependency in the python_sources dependency array. It's not like that hasn't been working, but I can't shake this nagging feeling that I'm missing some special sauce or didn't parse the documentation properly.
b
That source root configuration is telling Pants that
py
(and any other immediate children of
/
) is the "entry point" of that library, so Pants will understand an import like
from py.shared.logging import configure
as referring to the
logging.py
file... which sounds like it is wrong for you! Potentially you want a source root configuration
roots = ["/py/shared", "py/src/apps"]
. This means that Pants will understand that the immediate children of those directories are the top-level libraries for import statements. (For the specifics of this case, calling a file
logging.py
to import like
import logging ...
is likely to be very confusing, as it conflicts with the standard library
logging
module)
n
Yes, I understand. I picked the name somewhat randomly. Could think of it as monkeys_bananas.py instead if it makes the question clearer.
👍 1
And thank you for the other part of your answer.
I will most definitely experiment with some source root setups other than just ["/"]. One curiosity I have is how you put "/py/shared" for the first of your example roots but "py/src/apps" without the leading forward slash for the other. I've noticed a difference in practice without being able to understand it so exactly. What makes it apply to my example. I believe you when you say to try it that way, I just don't have a fine-grained understanding there yet.
And while I'm just shooting questions from the hip, I really like being able to use the ":target" syntax when I can. Is there a way I can organize my code to use it more often, or is it more purpose-built and organizing around it would be kind of silly?
b
ah, that was just a typo; no reason for them to be different. I believe leading
/
means it only applies to that exact path starting at the
pants.toml
file ("build root") which might be fine for you; while no leading
/
means it finds any directory path that matches. e.g. a common source root config is
roots = ["src"]
to find any
src/
directory anywhere in the repo.
n
Ah, so if I'm below any "src" directory, I can start my module references from paths directly below it. That makes sense and I think would help me out a bunch.
e
Are you making up a new repo or is the pre-existing?
n
New repo. New pants.
e
Ok. If pre-existing the whole point of source roots is you change nothing of the existing layout; you just teach Pants about that layout.
The source roots should not drive pre-existing imports to be different - its exactly the opposite.
n
Right. No pre-existing imports.
Part of this might belong in #C01SPQQ2WK1, but I liked pants after experimenting, reading the docs, experimenting some more and watching the video referenced in Slack. I'm hoping to build my repo in such a way that, among other things, I don't have to define super long paths for non-inferable dependencies in my BUILD files. I'm doing greenfield development and am generally fishing for ways that I could keep pants + python in mind while structuring my repo so that my BUILD files and target references are as lean as possible.
e
There should be very few non-inferrable dependencies. It seems an odd thing to optimize around. Do you have tons of data files and / or resource files as dependencies or else do you have a custom Python importer that does magic?
If your source roots are wrong - don't match your imports - then you have to declare ~everything of course. Perhaps that's relevant to your source roots question? You absolutely must get your source roots right to start or all else falls apart.
n
It's that second message there I think I'm most interested in. Many of the examples I've seen have just gone with ["/"] and moved on with their life. So my question becomes: Are there other combinations of source roots that might be better than ["/"] if the repo will primarily be used for writing a big pile of apps and a big pile of shared libs in 1-2 languages? If it just boils down to taste and experimentation, I'm OK with that. Just curious if there are some well-worn patterns I've missed in my research that might be worth trying since I don't have to adapt my source roots to an existing code base.
Thanks for taking the time to reply, btw. I have some work I need to focus on for the next hour or so, but I'll be sure to loop back.
e
This is all totally arbitrary - that's the 1st thing to fully understand. It absolutely positively does not matter - except to people who have tastes. Now, almost always just
/
only makes sense for a single language + single project - exactly what the example repos all are. So those repos may be horrible guidance for you when confused about source roots and imports. As far as providing guidance to the arbitrary decision for a repo with many languages and/or projects - I've personally lived in 2 approaches. Google did
src/java
,
src/python
,
src/cpp
- 1 root per language, all 1000s of projects / teams / apps living under there together. Twitter had 1 monorepo that did that too, and another monorepo (insert joke here) that did 1 source root per project - little fiefdoms:
finch/src/scala
,
finch/src/python
,
crow/src/scala
, etc... (where projects tended to have damn bird names.
🤔 1
I personally vastly prefer the 1st approach. It sociologically promotes fixing anything you come across / use. Its not "your teams" code - its more everyones. The latter approach can be used that way, but tends to re-inforce fiefdoms from what I've seen. Depends on the folks using the layout really and their attitudes.
🙏 1
n
OK so in Google's case (hush hush such bigtime IP wow), did they have something like "/", "src", "/src" as their source root? Did they go with something like ["java", "python", "cpp"]? On one hand I feel like that one specifying the languages could work but also kind of requires discipline to not put those terms elsewhere in a path. Am I even thinking about this correctly? (Also that's hilarious that Twitter had multiple monorepos) Please disregard. I am too dumb to realize you said what their source roots were. Thank you. I need to go eat some paste now.