Hi everyone I m trying to understand how to add all files to Pants #general

Hi everyone! I'm trying to understand how to add a...

square-psychiatrist-19087

07/09/2023, 2:35 PM

Hi everyone! I'm trying to understand how to add all files to python_distribution recursively. Here is my layout

Copy code

BUILD
src/
  BUILD
  mylib/
    BUILD
    *.py
    one/
      BUILD
      *.py
    two/
      BUILD
      *.py

each BUILD file has just

python_sources(name='src')

in it, except for the top level BUILD which has:

Copy code

python_distribution(
    name="dist",
    dependencies=[
        "./src/mylib:src",
        "./src/mylib/one:src",
        "./src/mylib/two:src",
    ],
    ...
)

As you can see, I have to specify each submodule explicitly. Is there a way to add all source targets to dependencies in python_distribution recursively?

fresh-cat-90827

07/16/2023, 9:11 PM

hey there, unless I misunderstand what you are trying to achieve, you could use the double colon notation

::

to address all targets in a directory, recursively. So this would be

src/mylib::

fresh-cat-90827

07/16/2023, 11:04 PM

also worth reviewing https://www.pantsbuild.org/docs/python-distributions#mapping-source-files-to-distributions as there are some gotchas if you are going to include everything (if there are multiple distributions)

square-psychiatrist-19087

07/20/2023, 11:15 AM

Thanks!

fresh-cat-90827

07/20/2023, 11:16 AM

hm let me test using

src/mylib::

, I vaguely remember this wasn't allowed

fresh-cat-90827

07/20/2023, 11:17 AM

right.

Copy code

UnsupportedWildcardError: The address `cheeseshop::` from the `dependencies` field from the target //:cheeseshop-query-wheel ended in a wildcard (`::`), which is not supported.

fresh-cat-90827

07/20/2023, 11:18 AM

does

src/mylib/two:src

depend on

src/mylib/one:src

and up all the way to the root?

fresh-cat-90827

07/20/2023, 11:19 AM

or are they independent components you simply want to put into a Python distribution?

square-psychiatrist-19087

07/20/2023, 11:20 AM

nope, they are independent

square-psychiatrist-19087

07/20/2023, 11:20 AM

and I want to package all mylib/**

square-psychiatrist-19087

07/20/2023, 11:24 AM

basically I'm trying to reproduce the behavior of a basic

python3 -m build

with pyproject.toml https://packaging.python.org/en/latest/tutorials/packaging-projects/#generating-distribution-archives

👀 1

fresh-cat-90827

07/20/2023, 11:38 AM

I see what you mean now, thanks for explaining. I've discussed this in https://github.com/pantsbuild/pants/issues/18217 earlier. If your list of targets doesn't update often, you may want to do:

Copy code

$ pants list --filter-target-type=python_sources cheeseshop::
cheeseshop:cheeseshop
cheeseshop/cli:cli
cheeseshop/cli/utils:utils
cheeseshop/repository:repository
cheeseshop/repository/parsing:parsing

and then copy those (perhaps with a bit of Bash/Python to produce a list of strings. Another workaround is to create a target that will own files:

Copy code

python_sources(
    name="all",
    sources=["**/*"]
)

which you could refer to as in

Copy code

python_distribution(
    name="simple-wheel",
    dependencies=[
        "cheeseshop:all",
    ],
    provides=python_artifact(
        name="cheeseshop-query",
    ),
    sdist=False,
    generate_setup=False,
)

fresh-cat-90827

07/20/2023, 11:40 AM

you can add more target types in the querying command e.g.

pants list --filter-target-type='python_sources,resource' cheeseshop::

fresh-cat-90827

07/20/2023, 11:41 AM

create an "all-owning" target isn't great because it creates ambiguity - your targets such as Python sources will be owned by multiple targets which may cause you some trouble in the dependency inference:

Copy code

❯ pants list cheeseshop/version.py                                      
cheeseshop/version.py
cheeseshop/version.py:all

square-psychiatrist-19087

07/20/2023, 11:42 AM

got you, thank you for help!

square-psychiatrist-19087

07/20/2023, 11:43 AM

I think I will go with the explicit list

fresh-cat-90827

07/20/2023, 11:50 AM

sure, but you can also use a handwritten setup.py https://www.pantsbuild.org/docs/python-distributions#setuppy

fresh-cat-90827

07/20/2023, 11:51 AM

the

packages=find_packages(exclude=["tests", "tests.*"]),

in the setup.py would package everything, recursively

square-psychiatrist-19087

07/20/2023, 11:52 AM

yeah, but then I could just use a regular pyproject.toml for packaging without pants

fresh-cat-90827

07/20/2023, 11:58 AM

sure. FWIW you may also find the

target

target helpful so that you could group them in a single place and refer to from elsewhere:

Copy code

target(
    name="all",
    dependencies=[
        "cheeseshop:cheeseshop",
        "cheeseshop:project-version",
        "cheeseshop/cli:cli",
        "cheeseshop/cli/utils:utils",
        "cheeseshop/repository:repository",
        "cheeseshop/repository/parsing:parsing",
    ],
)

https://www.pantsbuild.org/docs/reference-target

fresh-cat-90827

07/20/2023, 11:58 AM

and then

Copy code

python_distribution(
    name="simple-wheel",
    dependencies=[
        "//:all",
    ],
    provides=python_artifact(
        name="cheeseshop-query",
    ),
    sdist=False,
    generate_setup=False,
)

square-psychiatrist-19087

07/20/2023, 11:59 AM

cool, thanks 👍

fresh-cat-90827

07/20/2023, 12:02 PM

I have filed https://github.com/pantsbuild/pants/issues/18351 , but I am not sure when/if this will be implemented, so you may rely on the workarounds we sketched together above

5 Views

Open in Slack

Previous Next