https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to generate `BUILD` files only in s...
# general
c
Is there a way to generate 
BUILD
files only in semi-top level folders, rather than in every directory? Something like the following?
Copy code
my_project/
├── awesome_service/
│   ├── internals/
│   |   ├── utils.py
│   |   ├── constants.py
│   ├── client/
│   |   ├── client.py
│   └── BUILD
└── another_awesome_service/
    ├── internals/
    |   ├── utils.py
    |   ├── constants.py
    ├── client/
    |   ├── client.py
    └── BUILD
h
not currently, but it's been discussed a bit. See the tooltip "One BUILD file per directory" for our rationale of why we do it this way: https://www.pantsbuild.org/docs/targets#target-generation But not every maintainer agrees as strongly with that recommendation, and a common frustration w/ Pants is the # of BUILD files
👀 1
w
This is actually the reason I don't use 
tailor
I use 1 BUILD per service/project/larger unit of code - requires globbing though, so 🤷 https://github.com/sureshjoshi/pants-plugins/blob/main/hellocpp/BUILD.pants
h
@wide-midnight-78598 are you using 
overrides
extensively? If so, how is that going?
c
Is there any way to provide arguments to 
tailor
such that we specify path rules on where to generate 
BUILD
directories?
h
(My main priority is encouraging folks to use precise metadata. It's important for cache hits, and I think important for maintainability. Otherwise, I don't care much about 111)
Is there any way to provide arguments to tailor such that we specify path rules on where to generate BUILD directories?
You can say 
./pants dir
and it will only generate for 
dir
and everything below it. (I've proposed changing that to only generate for 
dir
, and then you use 
dir::
for everything)
w
@hundreds-father-404 Nope - don't think I've ever used an override except in maybe the PyOxidizer build file? I basically have 1 BUILD file representing an artifact (executable, lib, whatever). Having too many BUILD files is like having all those 
init.py
files everywhere. Makes searching for the specific one a pain, and just contributes to visual glut on the filesystem.
c
In my example, can I tell it to do 
my_project/awesome_service
and 
my_project/another_awesome_service
, but not the subdirectories 
my_project/awesome_service/internals
, etc.?
And basically, the subdirectories are included as sources in the parent directory?
h
Not currently. Feel free to open a feature request for 
tailor
to support non-1:1-1 mode: https://github.com/pantsbuild/pants/issues/new/choose Also FYI I recently added options to disable every built-in 
tailor
implementation, which means that you can turn those off and replace them w/ your own plugin
c
What does "1:1-1 mode" mean?
h
1 BUILD file per directory
c
I see. Why is it named "1:1-1"?
h
1 BUILD file, 1 directory, 1 target. (Which isn't totally accurate, often it's 
python_sources
+ 
python_tests
)
👍 1
The gist of the motivation is: we want to encourage people to set as precise of metadata as possible. 111 encourages that And pt 2: even if you start w/ all defaults, we've found that over time you often add more, such as test timeouts
but the motivation is MUCH less compelling than in Pants v1, before we had dependency inference
w
I should note, my dislike of 1 build file per directory comes from CMake - where they also recommended/insisted on a 
CMakeFiles.txt
per directory, which caused massive ballooning and making it impossible to find the correct one in VSCode or other IDE search. I'm willing to take compile-time/test-time cache misses vs keyboard-time usability any day of the week
c
I added a feature request here: https://github.com/pantsbuild/pants/issues/15429
Do you know what the timeline of being able to support something like this would be, i.e.: • Is it difficult to do? • Is it perhaps controversial?
w
which caused massive ballooning and making it impossible to find the correct one in VSCode or other IDE search.
i’d argue that it’s harder to find the right BUILD file to edit if they aren’t consistently next to the code that they apply to. you need to add metadata for a file, so which of the parent directories do you go looking in? having said that: where to put `python_distribution`s is still unanswered with 111.
👍 1
Do you know what the timeline of being able to support something like this would be, i.e.:
• Is it difficult to do?
• Is it perhaps controversial?
probably both… the big challenge is just: how do you describe where the metadata should live via config, if not “next to the relevant file”? i suppose that if there were a clear answer to that question, it wouldn’t be controversial. but there are a lot of assumptions baked in in various places around 111 being the happy path: you’ll have less boilerplate per-file of default globs, shorter addresses, etc
w
In my case, as I mentioned, I use the "artifact". So, for micro-services, each service has 1 BUILD file. From a folder naming perspective, it's pretty intuitive, because the other way I look at it... If I wasn't using a monorepo, then each of the separate git repos would each have 1 BUILD file. No advocating it for anyone else, as there are pros/cons - but for me, this is how I treat most BUILD files in all my projects/languages. The exception being stupid 
KConfig
files for embedded linux/u-boot - and those infuriate me to no end, because they're everywhere.
👍 1
f
The big win of 111 is granular change detection IMO. If you use a single BUILD file to define a ton of targets, every time you touch that build file, if you're using 
--changed-dependees=transitive
to select tests, you're going to add every target defined by that build file to the dependent closure.
👍 1
w
So, for micro-services, each service has 1 BUILD file. From a folder naming perspective, it’s pretty intuitive, because the other way I look at it...
@wide-midnight-78598: where would a shared library live? and how would its metadata be nested?
w
@witty-crayon-22786 Just a top-level lib that gets pulled in as a 1st party dep
Copy code
python_library(
    name="libtransferrer",
    sources=["**/*.py", "!*_test.py"],
    dependencies=[
        "//:protobuf",
        "//:python-dotenv",
        "//libs/core",
    ],
)
c
My use case is actually exactly the same as @wide-midnight-78598’s. We are looking to configure a single BUILD file for each service.
w
@cold-soccer-63228 I don't use any tools to do it, I just write them by hand. Since there are like... 10 total in my project, it takes about 5 minutes
And they're only a dozen lines or so.
w
@witty-crayon-22786 Just a top-level lib that gets pulled in as a 1st party dep
if the library is monolithic, and the entire thing depends on those dependencies, then that can work. but to actually have fine-grained metadata (i.e., another library which doesn’t need 
protobuf
), you need new top-level directories, and can’t really use namespacing
f
In my case, having a top-level BUILD file works fine for things that don't need overrides and don't change much
For things that require a lot of overrides, it's a nightmare
But I too have seen quite a bit of knee-jerk reaction to small BUILD files that contain only 
python_sources()
or what have you
👍 1
w
@witty-crayon-22786 In my case, I have multiple internal libraries at varying levels of granularity. I treat them like I would any external library. So my "libs" have several shared libraries, services and top-level applications pick them up. I don't do much overriding - so haven't run into any problems. I think with very large services or apps, maybe it's more of an issue - but I keep a lot of this pretty tight
f
I think a lot of this knee-jerk reaction is misplaced tbh, but it's common enough that I'm not totally dismissive of it. I think it comes across as a code smell to some people
👍 1
c
Is it possible to support the option for both 1:1-1 mode and non-1:1-1 mode to exist? It feels that there are pros and cons to both, and it would be a nice feature to leave it up to the developer to decide which route to go with
w
anyway: as mentioned above: i don’t think that anyone would block a patch if someone came up with a sufficiently good API/config-layout to instruct 
tailor
of what to do. i expect that that is challenging because the boundaries of projects are challenging to define in a monorepo… but if someone wants to take a swing at it, then by all means. just note that other UX biases toward 111, so creating the BUILD files is probably ~75% of the story
w
@bitter-ability-32190 As mentioned above, a lot of my hesitation comes from CMake, where each of my files actually had a decent amount of boilerplate - and narrowing them down to fewer files basically allowed my to DRY up my code. It's mildly annoying when I'm putting anything in the Pants repo to have a couple of files that NEED to be in each directory, but I don't care too much - since I don't spend a lot of time there. With client repos, the added code, maintainability, visual clutter - and then onboarding teams. All that builds up. I guess the simplest way to look at it, is that I've never had someone complain that there weren't enough BUILD files 🙂
f
Yeah I get what you're saying. I like it though because it lets me go on a fun rant about why DRY is a bad principle lmao
😆 1
b
ohhhh jey @wide-midnight-78598 wrong Josh 😂 I was like 'WTF?"
w
dangit Josh!
🙈 1
😉 1
w
Ahhhhh cannon!
🚀 1
b
FWIW DRY is grossly misinterpreted IMO. (Now you can @-me)
(Also enjoy the acronym salad)
f
Idk, the same devs I have who complain about a bunch of build files will also happily copy paste the same stuff into a ton of Makefiles
➕ 2
Having 111 as default I think is fine, and it's actually fairly easy to step out of that paradigm if you manually create recursive globs for some well-defined and relatively stable lib
➕ 2
w
Dont even get me started about Makefiles
f
./pants tailor
won't create BUILD files for files that are already encompassed in some generator definition
👍 1
w
Really? I need to test that again - I tried running a tailor over one of my stable projects, and it added like 8 redundant BUILD files (couple months ago)
f
I had the same difficulty initially but I think I had some things configured wrong
for reference, I have basically the bulk of our production code in one generator
Copy code
python_sources(
    name="aiven",
    sources=["aiven/**/*.py"],
    overrides={
        ...
    },
)
where
Copy code
❯ ./pants count-loc //:aiven
──────────────────────────────────────────
Language                 Files     Lines  
──────────────────────────────────────────
Python                    1693    409366
and 
./pants tailor
doesn't add a single BUILD file
The overrides (which I've omitted) make this a nightmare though, this is just a temp step for us
h
./pants tailor won't create BUILD files for files that are already encompassed in some generator definition
Yep, that is pretty crucial to 
./pants tailor
working
w
Okay, so I guess what I'm seeing might make sense, since sources aren't created, but rather it was a pex_binary: I have this: https://github.com/sureshjoshi/pants-plugins/blob/main/hellofib/BUILD.pants From my repo root, I run 
./pants tailor
and it just created:
Copy code
Created hellofib/hellofib/BUILD:
  - Add pex_binary target main
Copy code
pex_binary(
    name="main",
    entry_point="main.py",
)
f
ah I have
Copy code
[python]
tailor_pex_binary_targets = false
in my 
pants.toml
w
Ahhhh
Okay, that makes sense then
f
I think that was the setting that made it work okay for me
w
Yeah, there we go - that makes more sense. It also just created a python_requirements that I already have, but this now makes more sense
h
Also [tailor].ignore_paths and [tailor].ignore_adding_targets were added in the past 2-3 months And I added this week options to turn off every individual tailor implementation, like 
[python].tailor_python_sources
👍 1
It also just created a python_requirements that I already have, but this now makes more sense
Just now? That's a bug
w
I'm using a stale rc Pants - needed to test this on the latest, and see what the nuances are I'm missing. I've spent about 0 time looking at most of the tailor commands, so need to make sure I'm not missing anything obvious before I file a bug
👍 1
4 Views
Open in Slack
PreviousNext
Powered by