How does

./pants tailor

know when to generate

python_sources

as opposed to

python_test_utils

, i.e. what heuristics of the directory tree, filenames, etc. is Pants looking for when deciding whether to generate

python_sources

for a directory as opposed to

python_test_utils

?

Test utils is mainly

conftest.py

, so python sources are everything but tests and test utils

I see. Does Pants support writing "hooks" for commands like

tailor

? For example, I'm interested in: • generating

python_test_utils

instead of

python_sources

if the directory name matches

test_utils

, • adding an additional auto-generated

python_source

to each

BUILD

file if the directory path matches a specific pattern. I would like the above two bullet points to happen every time I run

tailor

. Currently, I'm doing it outside of the Pants ecosystem by running a script that invokes

./pants tailor

, and then afterwards going in and editing the

BUILD

files, which feels more brittle than doing it within the Pants ecosystem.

Pants is pluggable, but I've never tried plugging tailor. I'll let someone else who is more knowledgeable answer this

tailor should be matching things that roughly correspond to the default

sources

field values of each target type: so python_sources: https://www.pantsbuild.org/docs/reference-python_tests#codesourcescode … vs python_test_utils: https://www.pantsbuild.org/docs/reference-python_test_utils#codesourcescode

Thanks @witty-crayon-22786. For my first bullet point, I precisely want to distinguish test utility files.

This target generator is intended for test utility files like

conftest.py

or

my_test_utils.py

. [...] this target can be helpful to better model your code by keeping separate test support files vs. production files.

Would renaming the files that I want to be test-only to end with

*_test_utils.py

be enough to trigger it so that

tailor

generates the

python_test_utils

instead of

python_sources

? Regarding your second comment: to be clear, "adding an additional auto-generated

python_source

to each

BUILD

file if the directory path matches a specific pattern" is not supported today, correct? Is it on the roadmap and expected to be supported at some point in the future?

in the link to

python_test_utils

above, the default value of the sources field is:

default: ('conftest.py', 'test_*.pyi', '*_test.pyi', 'tests.pyi')

Got it. Thank you!

it’s just intended to handle the common case of a

conftest.py

file.

Separately, I noticed that files I define with a leading underscore, e.g.

_foo.py

are ignored in

python_sources()

. This is intentional, correct?

Regarding your second comment: to be clear, “adding an additional auto-generated

python_source

to each

BUILD

file if the directory path matches a specific pattern” is not supported today, correct? Is it on the roadmap and expected to be supported at some point in the future?

a

tailor

@rule

could do this today: the code that classifies things is here, and more rules could be installed to do other things. the issue is more that we already have rules installed that will do something else. and that would lead to a conflict (tailor would likely error with “two different rules wanted to add targets for this file: uh oh”)

hm… no, not intentional. i think that this is the first time i’ve seen that. can you file a bug?

Ahh, never mind—there's no issue here. I was incorrectly retrieving the dependency, which was just

path/to/_foo.py

, instead of

path/to:_foo.py

. We're all good here. Thanks for all of your help Stu! 🙂

sure thing. when you say “retrieving the dependency”, do you mean that you’re explicitly specifying

dependencies=

to targets? generally if you have to do that for firstparty python sources, it’s a bit of a red flag… inference should be picking up 98% of deps.

Yes, but only for a very specific case. The reason I need to explicitly specify dependencies is because we have some hairy code in our codebase where we use

importlib

to import some modules from a given path, instead of importing them normally. So I wrote a quick script to run for one of the generated

BUILD

files, to iterate through all top-level directories in our codebase that have a

_package_config.py

file defined (which is the module being imported), and then inject dependencies that point to those, in the

BUILD

file.

Interesting. FWIW you could also write a plugin to add custom dep inference logic.

@happy-kitchen-89482, what do you mean?

Instead of a script that writes those dependencies into the BUILD files, you could write a Pants plugin to extend the automatic dependency inference to infer those dependencies.

I see. Do you know of any examples that are available that extend the automatic dependency inference? (Sorry, I'm very new to Pants and haven't quite figured out where to look for what just yet.)

@hundreds-father-404 Do we have an example of custom dep inference?

No dedicated examples. It's one of the first things I want to write for our revamp of the example plug-in repository. Would you be willing to share your script? I'm curious how you're doing things now. That might help to find a more useful example from the pants repository

Sure. Should I share it here, or perhaps somewhere else?

This thread works! Feel free to post directly or use something like a github gist etc.

It's small enough that I can paste it here. In the below code, the fictitious

SubprocessLibrary

is just a library used to run shell commands.

path/to/special/case/BUILD

is the special-case

BUILD

file that has dependencies on a bunch of other

path/to/_package_config.py

files that are defined by convention in other top-level directories. One of the modules within the

path/to/special/case

directory uses

importlib

to import the

_package_config.py

files. So what we're doing here is after

./pants tailor

finishes, we go in and manually add the dependencies.

from typing import List
import os

def _get_paths() -> List[str]:
    # Here, we remove the "./" prefix because using it in BUILD files denotes a relative path,
    # whereas we really want the absolute path starting from the root of the front-porch repo.
    return [
        os.path.join(root, dir).removeprefix("./") 
        for root, dirs, _ in os.walk(".") for dir in dirs
    ]

# Execute the "./pants tailor" command.
SubprocessLibrary.exec("./pants tailor")

# Get a list of all directories containing a BUILD file.
build_dirs: List[str] = []
for path in _get_paths():
    build_path = os.path.join(path, "BUILD")
    if os.path.exists(build_path):
        build_dirs.append(path)

# HACK: overwrite the tailor generated file
with open("path/to/special/case/BUILD", "w") as f:
    f.write("python_sources(\n")
    f.write("\tdependencies=[\n")

    for path in build_dirs:
        if os.path.exists(os.path.join(path, "_package_config.py")):
            f.write(f'\t\t"{path}/_package_config.py",\n')

    f.write("\t]\n")
    f.write(")\n")

BTW I have also just noticed that we do have an issue with source files with leading underscores. Am investigating.