Is there a way to make Pants include all Python files (or files in general) in a directory, without an explicit reference to them being used? I want to do some dynamic imports, but can't get my files to be included in the resulting pex.

use

python_sources(sources=["./**/*.py"])

and it should find all the python files you need. It will also recognize them as python files so that they can be linted, typechecked, etc.

Alternatively, if you use a BUILD file in that directory, just using

python_sources()

will result in the same, non-recursive, result

Running

pants run run.py

with

os.listdir

with

os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__))))

does not print any of the files that aren't explicitly imported or referenced, using either of the syntaxes, hmm. The result is:

['__pycache__', 'run.py', '__init__.py']

I see the problem. First, following our advice above, you make sure that each file is referenced by the target generator and "exists" as a

python_source

target that is known by pants. This is done. Next, since

run.py

is your entrypoint, pants needs to know what files are required by that entrypoint. Typically, dependency inference (ie. reading the import statements) can do this for you. But, as you've identified, with dynamic imports going on, pants can't tell. So you'll need to add the dependencies yourself. I'll walk through a step by step of my thought process here: • Following @wide-midnight-78598’s suggestion. Put

python_sources()

right in the same directory, and change it to

python_sources(name="source_files", dependencies=[":source_files"])

• It does seem a little strange for a target to reference itself, and indeed this may cause circular reference issues, so it might make more sense to do: ◦ `python_sources(name="deps", sources=["./*.py", "!./run.py"])`This is sources for all the files except

run.py

◦

python_sources(name="run", sources="[./run.py"], dependencies=[":deps"])

◦ Now, there is no overlap, but maintaining this will be not a lot of fun in the future. • IMO, this is the best way to do it: Though it will require changing your source a little to account. ◦ put the dynamically imported files into a subdirectory and have subdir/BUILD:

python_sources()

◦ BUILD file in directory with

run.py

,

decorators.py

, and

providers.py

(presumably these are the statically imported files):

python_sources(dependencies=["./subdir:subdir"])

◦ Now your dynamic stuff is nicely contained and can easily be covered by a single target generator, and its straightforward to make any other [group of] targets depend upon it. ◦ This would imply needing to change the search path used in your code to discover the files for import

It seems like your comment was cut short @elegant-florist-94385

Can you show your

pex_binary

part of the build file as well?

@acoustic-library-86413 Yeah, I always forget to Shift+enter when writing bullet points 🤦

Surprisingly

python_sources(name="source_files", dependencies=[":source_files"])

works excellently. Thank you so much for the insight! I don't think I would have thought of that myself.

Great!