cold-soccer-63228
04/29/2022, 7:32 PM./pants tailor
know when to generate python_sources
as opposed to python_test_utils
, i.e. what heuristics of the directory tree, filenames, etc. is Pants looking for when deciding whether to generate python_sources
for a directory as opposed to python_test_utils
?bitter-ability-32190
04/29/2022, 7:35 PMconftest.py
, so python sources are everything but tests and test utilscold-soccer-63228
04/29/2022, 7:37 PMtailor
?
For example, I'm interested in:
• generating python_test_utils
instead of python_sources
if the directory name matches test_utils
,
• adding an additional auto-generated python_source
to each BUILD
file if the directory path matches a specific pattern.
I would like the above two bullet points to happen every time I run tailor
. Currently, I'm doing it outside of the Pants ecosystem by running a script that invokes ./pants tailor
, and then afterwards going in and editing the BUILD
files, which feels more brittle than doing it within the Pants ecosystem.bitter-ability-32190
04/29/2022, 7:38 PMwitty-crayon-22786
04/29/2022, 7:43 PMsources
field values of each target type: so python_sources: https://www.pantsbuild.org/docs/reference-python_tests#codesourcescode … vs python_test_utils: https://www.pantsbuild.org/docs/reference-python_test_utils#codesourcescode@rules
for tailor… but you might also have to uninstall the default @rules
, which isn’t supported right nowcold-soccer-63228
04/29/2022, 7:47 PMThis target generator is intended for test utility files likeWould renaming the files that I want to be test-only to end withorconftest.py
. [...] this target can be helpful to better model your code by keeping separate test support files vs. production files.my_test_utils.py
*_test_utils.py
be enough to trigger it so that tailor
generates the python_test_utils
instead of python_sources
?
Regarding your second comment: to be clear, "adding an additional auto-generated python_source
to each BUILD
file if the directory path matches a specific pattern" is not supported today, correct? Is it on the roadmap and expected to be supported at some point in the future?witty-crayon-22786
04/29/2022, 7:48 PMpython_test_utils
above, the default value of the sources field is:
default: ('conftest.py', 'test_*.pyi', '*_test.pyi', 'tests.pyi')
This target generator is intended for test utility files like…so… it doesn’t actually matter whether your utilities areorconftest.py
. Technically, it generatesmy_test_utils.py
targets in the exact same way as thepython_source
target generator does, only that thepython_sources
field has a different default. So it is valid to usesources
instead. However, this target can be helpful to better model your code by keeping separate test support files vs. production files.python_sources
python_test_utils
or not. it is equivalent to python_sources
.cold-soccer-63228
04/29/2022, 7:50 PMwitty-crayon-22786
04/29/2022, 7:50 PMconftest.py
file.cold-soccer-63228
04/29/2022, 7:51 PM_foo.py
are ignored in python_sources()
. This is intentional, correct?witty-crayon-22786
04/29/2022, 7:52 PMRegarding your second comment: to be clear, “adding an additional auto-generatedato eachpython_source
file if the directory path matches a specific pattern” is not supported today, correct? Is it on the roadmap and expected to be supported at some point in the future?BUILD
tailor
@rule
could do this today: the code that classifies things is here, and more rules could be installed to do other things.
the issue is more that we already have rules installed that will do something else. and that would lead to a conflict (tailor would likely error with “two different rules wanted to add targets for this file: uh oh”)Separately, I noticed that files I define with a leading underscore, e.g.hm… no, not intentional. i think that this is the first time i’ve seen that. can you file a bug?are ignored in_foo.py
. This is intentional, correct?python_sources()
python_sources
are:
default: ('*.py', '*.pyi', '!test_*.py', '!*_test.py', '!tests.py', '!conftest.py', '!test_*.pyi', '!*_test.pyi', '!tests.pyi')
cold-soccer-63228
04/29/2022, 7:59 PMhm… no, not intentional. i think that this is the first time i’ve seen that. can you file a bug?Ahh, never mind—there's no issue here. I was incorrectly retrieving the dependency, which was just
path/to/_foo.py
, instead of path/to:_foo.py
. We're all good here.
Thanks for all of your help Stu! 🙂witty-crayon-22786
04/29/2022, 8:00 PMdependencies=
to targets? generally if you have to do that for firstparty python sources, it’s a bit of a red flag… inference should be picking up 98% of deps.cold-soccer-63228
04/29/2022, 8:02 PMimportlib
to import some modules from a given path, instead of importing them normally.
So I wrote a quick script to run for one of the generated BUILD
files, to iterate through all top-level directories in our codebase that have a _package_config.py
file defined (which is the module being imported), and then inject dependencies that point to those, in the BUILD
file.happy-kitchen-89482
04/29/2022, 8:32 PMcold-soccer-63228
04/29/2022, 8:34 PMhappy-kitchen-89482
04/29/2022, 8:42 PMcold-soccer-63228
04/29/2022, 8:44 PMhappy-kitchen-89482
04/29/2022, 9:45 PMhundreds-father-404
04/29/2022, 10:57 PMcold-soccer-63228
04/29/2022, 10:59 PMhundreds-father-404
04/29/2022, 11:00 PMcold-soccer-63228
04/29/2022, 11:06 PMSubprocessLibrary
is just a library used to run shell commands.
path/to/special/case/BUILD
is the special-case BUILD
file that has dependencies on a bunch of other path/to/_package_config.py
files that are defined by convention in other top-level directories. One of the modules within the path/to/special/case
directory uses importlib
to import the _package_config.py
files.
So what we're doing here is after ./pants tailor
finishes, we go in and manually add the dependencies.
from typing import List
import os
def _get_paths() -> List[str]:
# Here, we remove the "./" prefix because using it in BUILD files denotes a relative path,
# whereas we really want the absolute path starting from the root of the front-porch repo.
return [
os.path.join(root, dir).removeprefix("./")
for root, dirs, _ in os.walk(".") for dir in dirs
]
# Execute the "./pants tailor" command.
SubprocessLibrary.exec("./pants tailor")
# Get a list of all directories containing a BUILD file.
build_dirs: List[str] = []
for path in _get_paths():
build_path = os.path.join(path, "BUILD")
if os.path.exists(build_path):
build_dirs.append(path)
# HACK: overwrite the tailor generated file
with open("path/to/special/case/BUILD", "w") as f:
f.write("python_sources(\n")
f.write("\tdependencies=[\n")
for path in build_dirs:
if os.path.exists(os.path.join(path, "_package_config.py")):
f.write(f'\t\t"{path}/_package_config.py",\n')
f.write("\t]\n")
f.write(")\n")
_package_config.py
files. But this doesn't work for if one were to not know what is being imported. If that's the case, then it seems like there is no choice but to add everything as a dependency, which is suboptimal, but I don't think there's any way around that (besides perhaps rewriting the code altogether).happy-kitchen-89482
05/01/2022, 1:32 PM