https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to get Pants to work properly, even...
# general
c
Is there a way to get Pants to work properly, even if we don't check in 
BUILD
files via Git? I noticed that if I have 
*/**/BUILD
in the 
.gitignore
, then Pants commands will complain that a 
BUILD
file doesn't exist, even though it does exist in the local filesystem.
h
h
ooc, what's the rationale for not putting build files in git and using pants?
➕ 1
h
+1, they are meant to be shared by everyone on the team so that everyone has the same setup
c
We want to generate custom logic in our 
BUILD
files using 
_package_config.py
files, where engineers can define other things as well. We want those to be the source of truth. 
BUILD
files are to be generated locally (since it doesn't take long), and we want one single source of truth
It's not a hard requirement—we can just generate the 
BUILD
files, and add headers telling people to not touch them or edit them manually
👍 1
h
I would encourage that approach instead. I think it will be helpful to have in Git diffs how BUILD files have changed, as that is Pants's source of truth
c
Mhm, makes sense. Thanks!
w
We want to generate custom logic in our 
BUILD
files using 
_package_config.py
files, where engineers can define other things as well. We want those to be the source of truth.
if this is meant to be a long term strategy, you might also be able to define a target generator to do this automatically.
➕ 1
h
fyi there is an option 
[tailor].build_file_header
👍 1
you might also be able to define a target generator to do this automatically
tailor
is also extensible
c
if this is meant to be a long term strategy, you might also be able to define a target generator to do this automatically.
tailor
is also extensible
There are currently no examples of anyone extending 
tailor
at the moment, right? We would ideally have a custom generator that does all of the basic stuff and some additional things we want to do
w
so, Pants itself uses its own plugin API to implement all language support: that means that any set of `@rule`s for 
tailor
in https://github.com/pantsbuild/pants/ can be used as examples for extension in your own repo
👍 1
but plugins to 
tailor
can only create a target type which already exists: if you want to create a new target type, and then declare those with 
tailor
, you’d do that using https://www.pantsbuild.org/docs/target-api
c
https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/tailor.py is being invoked when running 
./pants tailor
, for all Python files, correct? What would make the most sense if I wanted to extend this? For each 
BUILD
file generated, I would like to also look at its sibling 
_package_config.py
file in the same directory, and generate dependencies conditional on that 
_package_config.py
file.
h
do you want to be editing existing entries in the BUILD file?
c
Ideally, engineers should never have to manually edit the 
BUILD
files that are generated
h
but when you change 
_package_config.py
, then rerun your <insert mechanism to regenerate the BUILD file>, it might change existing content, right? If so, 
tailor
won't be the right fit
A macro or target generator might be. You can create whatever DSL you want, like:
Copy code
my_org_package(
   some_data_that_will_be_transformed=["a", "b"]
)
w
generate dependencies conditional on that 
_package_config.py
file.
this sounds like a dependency inference plugin then maybe. that would allow you to add dependencies automatically to a 
python_sources
target in the same directory
➕ 1
h
Taking a step back, what is your motivation here with generating the BUILD flies?
c
this sounds like a dependency inference plugin then maybe. that would allow you to add dependencies automatically to a 
python_sources
target in the same directory
Yes, that's exactly what we want to do!
Taking a step back, what is your motivation here with generating the BUILD flies?
We want to generate 
BUILD
files so that we can construct a dependency tree and run tests only for code that has changed in the current branch or most recent commit, to reduce the number of tests that need to run on CI/CD
h
so that we can construct a dependency tree and run tests only for code that has changed in the current branch or most recent commit, to reduce the number of tests that need to run on CI/CD
That is the crux of what Pants is designed for. For example: https://www.pantsbuild.org/docs/using-pants-in-ci#approach-1-only-run-over-changed-files Are you envisioning using Pants for that project introspection, or you are going to create the dependency graph yourself?
c
We're using Pants for that itself! But ideally we want engineers to never have to edit the generated BUILD files, but rather the files used to generate the BUILD files. Does that make sense?
h
How come? BUILD files are meant to be extremely minimal, usually only 1-4 lines, thanks to a) dependency inference b) sensible defaults c) "target generation" Pants already does for you I fear that adding 
_package_config.py
would be adding a new layer of indirection. For example, your teammates will have a harder time following along with the Pants docs when they have questions like "How do I change the timeout of a test?"
c
Right now, we use 
_package_config.py
everywhere in our code already to specify things like owners, code reviews, etc. We specify some other things in here like dependencies for asynchronous code. We define these in Python because we import them in our codebase in middleware implementation. Ideally, we would like to add some additional logic, perhaps in a dependency inference plugin that would allow us to add dependencies automatically to a 
python_sources
target in the same directory. Is something like this possible, and are there examples?
h
c
Great! Thanks for sending these over. I'll take a read—I think this may accomplish exactly what I'm looking for
🙌 1
h
Could you please share an example 
_package_config.py
? I'm curious how you're definining the dependencies now: might help us to give you more specific guidance
c
Yeah, of course. Give me one second
h
I'll take a read—I think this may accomplish exactly what I'm looking for
Cool! It does sound like this is the idiomatic way to achieve this We are definitely happy to help if you have questions along the way. Plugins can have a bit of a learning curve - we're wanting to improve our examples and docs
c
Here's an example of a 
_pacakge_config.py
.
Copy code
from front_porch.modules.common.modules.types import (
    MessagingConfig,
    OwnerValues,
    PackageConfig,
    PubSubConfig,
    SubscriptionConfig,
)

package_config = PackageConfig(
    owner=OwnerValues.PLATFORM,
    package_name="some_package_name",
    messaging=MessagingConfig(
        pubsub=PubSubConfig(
            subscriptions=[
                SubscriptionConfig(
                    subscription_id="SOME_PUBSUB_SUBSCRIPTION_ID",
                    filepath="path/to/some_handler.py",
                )
            ]
        )
    ),
)
In this configuration file, we define the owner, package name, and messaging dependencies. When generating the 
BUILD
file, we would like to add the 
filepath
specified in the messaging dependencies as dependencies in the 
BUILD
file
h
Okay cool. So the simplest—but fragile—way to do this will be to parse the file using 
tokenize
or 
ast
to extract the 
filepath="path/to/some_handler.py",
line Then add a dependency on 
Address(os.path.dir(path_to_file), relative_file_path=os.path.basename(path_to_file))
, which translates to the address 
path/to/some_handler.py
. This is a little fragile and wonky that it assumes your 
python_sources
is leaving off the 
name=
field, which is the default behavior with `tailor`; otherwise the address would be 
path/to/some_handler.py:tgt_name
. That seems fine to me to assume for now, and make more robust later
What files depend on the 
filepath
? For example, every single thing underneath the directory?
c
What files depend on the 
filepath
? For example, every single thing underneath the directory?
For now, yes. In the future, we would like it to be a bit more granular, but for now we could have everything underneath the directory have dependencies on it
Okay cool. So the simplest—but fragile—way to do this will be to parse the file using 
tokenize
or 
ast
to extract the 
filepath="path/to/some_handler.py",
line
Would I need to add a new target as well, after creating a new rule? Or could I tell 
tailor
to use this rule afterwards?
h
As long as you use 
tailor
to create all your 
python_sources
and 
python_tests
etc, no need for new targets. Your dep inference rule will enhance those targets
c
h
Question: how many of these 
_package_config.py
files do you have? I wonder if it would be feasible to start w/ simply hardcoding the 
filepath
in the 
dependencies
file, at least to start while you're iterating I was expecting there to be a lot more entries in 
_package_config.py
How do I reference my rule?
You are going to be adding a new entry into the dep inference plugin hook. Which gets registered with something like 
UnionRule(InferDependenciesRequest, InferInitDependencies),
at the bottom of 
python/dependency_inference/rules.py
Sorry we don't have a dedicated guide for this - a few people have been asking about this plugin hook this month and I'm itching to write a guide
👍 1
c
Question: how many of these 
_package_config.py
files do you have?
We have one for every root-level directory.
I was expecting there to be a lot more entries in 
_package_config.py
Some of these files have a many entries. I wanted to send a simple one so it'd be easy to understand what it looks like
You are going to be adding a new entry into the dep inference plugin hook. Which gets registered with something like 
UnionRule(InferDependenciesRequest, InferInitDependencies),
at the bottom of 
python/dependency_inference/rules.py
Does this mean I'll be contributing to the source code itself, as opposed to adding the rule in my own repository?
h
No, you will develop it internally. See https://www.pantsbuild.org/docs/plugins-overview if you have not yet, along with The Rules API section
c
Got it, thanks!
❤️ 1
5 Views
Open in Slack
PreviousNext
Powered by