Q When I run ` pants package` with a `python distribution di Pants #general

Q. When I run `./pants package` with a `python_dis...

rhythmic-morning-87313

05/19/2022, 5:03 AM

Q. When I run

./pants package

with a

python_distribution("dist")

target in

src/ai/backend/common/BUILD

which depends on a

python_sources("lib")

target in the same file which depends on a

python_sources("stubs")

target in

stubs/trafaret/BUILD

, it says "No python_distribution target found to own stubs/trafaret/__init__.pyi:stubs". Shouldn't it be included in the original

src/ai/backend/common:dist

target transitively? I don't want to make a separate package for the type stubs but just embed it inside the

src/ai/backend/common:dist

package. How do I avoid the error without changing the directory structure?

rhythmic-morning-87313

05/19/2022, 5:04 AM

Note that it just works fine with

./pants check

curved-television-6568

05/19/2022, 5:40 AM

Package will not include sources that is not in the subtree of the distribution I believe.

rhythmic-morning-87313

05/19/2022, 5:41 AM

Then.. can I just omit the stubs?

curved-television-6568

05/19/2022, 5:42 AM

Yes, you can exclude dependencies with

!tgt

curved-television-6568

05/19/2022, 5:42 AM

And transistive ones with

!!tgt

curved-television-6568

05/19/2022, 5:47 AM

See “Mapping source files to distributions” in Python/Building distributions docs (can't get proper link on mobile :/ ) And the notes under “dependencies field” from Key concepts/targets and BUILD files.

curved-television-6568

05/19/2022, 5:48 AM

See related issue (but for pex binary targets) https://github.com/pantsbuild/pants/issues/15454

rhythmic-morning-87313

05/19/2022, 5:54 AM

hmm

rhythmic-morning-87313

05/19/2022, 5:55 AM

Excluding the dependency using "!!" seems to make it working, but another error arises:

Copy code

NoSourceRootError: No source root found for `.`

rhythmic-morning-87313

05/19/2022, 5:55 AM

my source roots are

src

tests

stubs

tools/pants-plugins

happy-kitchen-89482

05/19/2022, 6:08 AM

For background: When you rely on Pants to generate a setup.py for you, it has to decide which of your sources to put in the distribution, vs which ones to depend on as an external requirement. This is important because we don't want to put the same source file in multiple distributions, as that can lead to runtime loading problems. The algorithm Pants uses to decide which sources go into a distribution is described here: https://www.pantsbuild.org/docs/python-distributions#mapping-source-files-to-distributions

👀 1

happy-kitchen-89482

05/19/2022, 6:09 AM

And a consequence of that algorithm is that a python_distribution can only contain sources that are next to or below that BUILD file in the filesystem.

rhythmic-morning-87313

05/19/2022, 6:09 AM

I'll take a look at (now I'm on a business meeting..)

happy-kitchen-89482

05/19/2022, 6:10 AM

stubs/trafaret/BUILD

is not under

src/ai/backend/common

, so it cannot be in that distribution

happy-kitchen-89482

05/19/2022, 6:10 AM

Your options are:

happy-kitchen-89482

05/19/2022, 6:11 AM

1) Move code around (I understand that you don't want to) 2) Publish stubs/trafaret in a separate python_distribution 3) Exclude the stubs dependency, which you shouldn't need at runtime anyway 4) Don't rely on an autogenerated setup.py, and instead write one manually. Then that algorithm doesn't come into play.

happy-kitchen-89482

05/19/2022, 6:11 AM

It sounds like 3) is the right answer for your case

👍 1

happy-kitchen-89482

05/19/2022, 6:12 AM

Using the

!!

to exclude sounds like the way to go

rhythmic-morning-87313

05/19/2022, 6:12 AM

image.png

rhythmic-morning-87313

05/19/2022, 6:12 AM

yes, i did that

happy-kitchen-89482

05/19/2022, 6:12 AM

Will need to figure out why you're getting this new error, and if it's related

rhythmic-morning-87313

05/19/2022, 6:12 AM

but it says the no source root found error

happy-kitchen-89482

05/19/2022, 6:12 AM

Can you post the full stack trace (run with --print-stacktrace)

rhythmic-morning-87313

05/19/2022, 6:13 AM

Copy code

Engine traceback:
  in select
  in pants.core.goals.package.package_asset
  in pants.backend.python.goals.setup_py.package_python_dist (src/ai/backend/common:dist)
  in pants.source.source_root.get_source_roots
Traceback (most recent call last):
  File "/home/joongi/workspace/pants/src/python/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
    res = func.send(arg)
  File "/home/joongi/workspace/pants/src/python/pants/source/source_root.py", line 249, in get_source_roots
    raise NoSourceRootError(path)
pants.source.source_root.NoSourceRootError: No source root found for `.`. See <https://www.pantsbuild.org/v2.13/docs/source-roots> for how to define source roots.

rhythmic-morning-87313

05/19/2022, 6:13 AM

Copy code

[source]
root_patterns = ["src", "stubs", "tests", "tools/pants-plugins"]

rhythmic-morning-87313

05/19/2022, 3:49 PM

Adding

to the

root_patterns

seems to resolve the issue, but i get the missing

python_distribution

error when executing

./pants package ::

because

stubs

directory has

python_sources()

without

python_distribution()

. How can I let pants "ignore" it?

rhythmic-morning-87313

05/19/2022, 3:54 PM

hmm

rhythmic-morning-87313

05/19/2022, 3:54 PM

it turns out that i have to add

!!stubs/trafaret:stubs

to all existing

python_distribution()

declarations

rhythmic-morning-87313

05/19/2022, 3:54 PM

how can i reduce this duplication?

5 Views

Open in Slack

Previous Next