When I build a distribution how do I include data files Pants #general

Join Slack

When I build a distribution, how do I include data...

# general

high-yak-85899

10/14/2022, 1:27 AM

When I build a distribution, how do I include data files?

✅ 1

high-yak-85899

10/14/2022, 1:30 AM

Normally this is controlled by

<http://MANIFEST.in|MANIFEST.in>

, but I'm not seeing mentions of that in pants docs.

fast-nail-55400

10/14/2022, 1:51 AM

https://www.pantsbuild.org/docs/assets

high-yak-85899

10/14/2022, 2:00 AM

I don't think that's the same thing

high-yak-85899

10/14/2022, 2:01 AM

I'm trying to build a

python_distribution

wheel that includes non-python files. When I build a pex of my system, it's exactly what I expect and includes my data files.

fast-nail-55400

10/14/2022, 2:05 AM

The code in

src/python/pants/backend/python/goals/setup_py.py

makes uses of

ResourceSourceField

, so my inclination that it could be relevant.

fast-nail-55400

10/14/2022, 2:06 AM

What happens if you add the

resources

target as a direct dependency of the

python_distribution

target?

high-yak-85899

10/14/2022, 2:22 AM

It's already a transitive dependency, but I can try it.

happy-kitchen-89482

10/14/2022, 4:34 AM

Are you using Pants-generated setup.py, or writing your own?

happy-kitchen-89482

10/14/2022, 4:35 AM

If the former, then it should include resources

high-yak-85899

10/14/2022, 5:00 AM

A pants-generated one

high-yak-85899

10/14/2022, 5:03 AM

I can see that my distribution does depend on these files

Copy code

./pants paths --from=//:ops_distribution --to=astranis-python/astranis/gnc/external/iers/iau2000A_finals_ab.txt 
[
  [
    "//:ops_distribution",
    "ops/scripts/geosat_1/odet/spacex_opm_to_odet_prior.py:../../all_scripts",
    "astranis-python/astranis/gnc/common/coordinate_transforms.py",
    "astranis-python/astranis/gnc/refsys/refsys_earth_orientation.py",
    "astranis-python/astranis/gnc/external/iers/iau2000A_finals_ab.txt:iau_finals"
  ],
  [
    "//:ops_distribution",
    "ops/scripts/geosat_1/payload_iot/antenna_characterization/antenna_characterization_slew.py:../../../all_scripts",
    "astranis-python/astranis/gnc/payload/payload_coordinate_transforms.py",
    "astranis-python/astranis/gnc/refsys/refsys_earth_orientation.py",
    "astranis-python/astranis/gnc/external/iers/iau2000A_finals_ab.txt:iau_finals"
  ],
  [
    "//:ops_distribution",
    "ops/scripts/geosat_1/payload_iot/antenna_characterization/antenna_characterization_slew.py:../../../all_scripts",
    "astranis-python/astranis/gnc/payload/payload_coordinate_transforms.py",
    "astranis-python/astranis/gnc/ephem/ephem.py",
    "astranis-python/astranis/gnc/refsys/refsys_earth_orientation.py",
    "astranis-python/astranis/gnc/external/iers/iau2000A_finals_ab.txt:iau_finals"
  ]
]

high-yak-85899

10/14/2022, 5:03 AM

But if I

./pants package

my distribution, they aren't in the wheel

high-yak-85899

10/14/2022, 5:04 AM

Here's what my distribution looks like

Copy code

python_distribution(
    name="ops_distribution",
    sdist=False,
    provides=python_artifact(name="ops_dist", version="0"),
    dependencies=[
        "//ops/scripts:all_scripts",
        "//astranis-python/astranis/ground/target_interface/script_runner.py",
        "//astranis-python/astranis/shell/backdoor.py",
    ],
    entry_points={
        "console_scripts": {
            "run_ops_script":
            "astranis.ground.target_interface.script_runner:main",
            "backdoor":
            "astranis.shell.backdoor:main"
        }
    })

high-yak-85899

10/14/2022, 5:05 AM

all_scripts

is a

python_sources

target with a glob pattern like

**/*.py

in case that's helpful.

fast-nail-55400

10/14/2022, 5:10 AM

what is the target type for

astranis-python/astranis/gnc/external/iers/iau2000A_finals_ab.txt:iau_finals

fast-nail-55400

10/14/2022, 5:11 AM

(just to confirm for clarity)

high-yak-85899

10/14/2022, 5:11 AM

resources

high-yak-85899

10/14/2022, 5:12 AM

specifically

Copy code

resources(
    name="iau_finals",
    sources=["*iau2000A_finals_ab.txt"],
)

high-yak-85899

10/14/2022, 5:14 AM

I use this in my system by making a separate pex that I can copy into my container that looks like this

Copy code

pex_binary(
    name="executable_scripts",
    script="conscript",
    dependencies=[":ops_distribution", ":reqs#conscript"])

high-yak-85899

10/14/2022, 5:14 AM

If I package that and unzip it, I see the

iau

files I want in that unzipped directory

high-yak-85899

10/14/2022, 5:21 AM

Explicitly adding

generate_setup=True

did not change the behavior

high-yak-85899

10/14/2022, 5:32 AM

Seems like this is the same as https://github.com/pantsbuild/pants/issues/13300

high-yak-85899

10/14/2022, 5:37 AM

Browsing the file contents of the wheel, I do see non-python files in other places. Various prototxt files and such. I'm not sure why this is different.

fast-nail-55400

10/14/2022, 5:48 AM

I wonder if this comment at https://github.com/pantsbuild/pants/blob/f6b2832539e26ddffb19a7d665b9cef697abd0da/src/python/pants/backend/python/goals/setup_py.py#L1047 is relevant:

Copy code

# If resource is not in a package, ignore it. There's no principled way to load it anyway.

fast-nail-55400

10/14/2022, 5:50 AM

is the resources target you had mentioned under a Python package?

high-yak-85899

10/14/2022, 4:29 PM

I don't think so. It's just a transitive dependency of one of our modules, but it lives in a different folder.

high-yak-85899

10/14/2022, 4:29 PM

I do not understand that comment at all, though. We have plenty of non-py files that lives under directories that only contain non-py files and they can get pulled into other repos just fine because of our MANIFEST.in

high-yak-85899

10/14/2022, 4:31 PM

As a silly example, here's kind of what this looks like

Copy code

foo/
  - bar.py
tree/
  - file.txt

bar.py

depends on

file.txt

in my scenario.

happy-kitchen-89482

10/14/2022, 5:32 PM

Hmmm, this is vexing. Would you be able to create a small dummy github repo that reproduces this, with anything proprietary redacted? That’s usually the easiest way to debug stuff like this

happy-kitchen-89482

10/14/2022, 5:33 PM

Probably something nuanced is going on

high-yak-85899

10/14/2022, 9:49 PM

A little crunched on time to fix this so odds are low

happy-kitchen-89482

10/14/2022, 10:27 PM

astranis-python/astranis/gnc/external/iers/iau2000A_finals_ab.txt

under a source root? (as seen by running

./pants roots

)

happy-kitchen-89482

10/14/2022, 10:28 PM

And, how is that file being used in your code?

happy-kitchen-89482

10/14/2022, 10:28 PM

i.e., by what relative path is it being accessed?

high-yak-85899

10/14/2022, 10:31 PM

Yes, it is. We're able to use it happily in unit tests and such and that's why it shows up in

./pants paths

. We use it by doing things like

pathlib.Path(__file__)

in one place and then navigating relative to that module to grab it.

high-yak-85899

10/14/2022, 10:31 PM

I've confirmed that it's not just a path mixup error as the

astranis/gnc/external

folder never shows up in my wheel when I unzip and inspect it.

high-yak-85899

10/14/2022, 10:32 PM

I also did a

find

with the unzipped wheel to make sure it wasn't anywhere in the wheel contents.

happy-kitchen-89482

10/14/2022, 11:42 PM

Very odd, if it’s under a source root, and wrapped in a

resources()

target then it should be there, as those other files are. Sounds like a subtle bug.

happy-kitchen-89482

10/15/2022, 12:26 AM

If you’re able to provide a repro, I’m happy to dive in and debug

high-yak-85899

10/15/2022, 12:31 AM

I appreciate it. I'll be sure to tag you in if I'm able to pull something together.

fast-nail-55400

10/15/2022, 12:31 AM

do you have the ability to run Pants from a sibling directory to your repository via a

pants_from_sources

script?

fast-nail-55400

10/15/2022, 12:32 AM

if so, this diff may provide some additional logging that could be of use

Untitled.diff

fast-nail-55400

10/15/2022, 12:32 AM

pants_from_sources

script can be found here: https://github.com/pantsbuild/example-python/blob/main/pants_from_sources

high-yak-85899

10/26/2022, 5:21 AM

As an update, I'm not able to reproduce with a simpler repository. Still poking...

high-yak-85899

10/26/2022, 5:42 AM

I have learned more though. We currently have something like

Copy code

lib/
  external/
    data/
       -> BUILD.pants
       -> my_data.txt
  data_user/
    -> BUILD.pants
    -> my_module.py

In this scenario, despite using

resources

lib/external/data/BUILD.pants

and properly linking that to

lib/data_user/my_module.py

in its BUILD file, we do not see the data file in the built distribution.

high-yak-85899

10/26/2022, 5:43 AM

However, if I change this to

Copy code

lib/
  data_user/
    -> BUILD.pants
    -> my_module.py
    -> my_data.txt

and declare the resource in

BUILD.pants

and link it to

my_module.py

, I do see it in my output distribution.

high-yak-85899

10/26/2022, 5:59 AM

Is it possible to inspect the generated

setup.py

and

<http://MANIFEST.in|MANIFEST.in>

? I think that would go a long way in debugging this.

high-yak-85899

10/26/2022, 6:02 AM

nvm, figured that out

high-yak-85899

10/26/2022, 6:11 AM

Okay, more info

high-yak-85899

10/26/2022, 6:13 AM

If I change to this

Copy code

lib/
  external/
    data/
       -> BUILD.pants
       -> my_data.txt
       -> dummy.py
  data_user/
    -> BUILD.pants
    -> my_module.py

and then say that

my_module.py

depends on

dummy.py

lib/data_user/BUILD.pants

, then that gets included in my built distribution.

high-yak-85899

10/26/2022, 6:15 AM

So there's something going on here related to what pants is thinking counts as a package or not.

high-yak-85899

10/26/2022, 6:16 AM

Worth noting: we don't use

__init__.py

in our repo. We rely on namespace packaging.

high-yak-85899

10/26/2022, 6:17 AM

So I think we're hitting https://github.com/pantsbuild/pants/blob/f6b2832539e26ddffb19a7d665b9cef697abd0da/src/python/pants/backend/python/goals/setup_py.py#L1047

high-yak-85899

10/26/2022, 6:24 AM

Confirmed that if I add logging at that section, all my data files that are missing are being ignored. So I think one outcome (and maybe can be my first pull request if folks agree) is to be a bit louder about resources that couldn't be included in the distribution rather than silently returning.

high-yak-85899

10/26/2022, 6:29 AM

Is this an indication that a pants distribution would not always intuitively work with a pep 420 compliant repo?

happy-kitchen-89482

10/26/2022, 7:27 AM

Thanks for the debugging effort. Yeah, that sounds like a footgun… A PR to add a warning there makes sense.

happy-kitchen-89482

10/26/2022, 7:27 AM

But even more sense is fixing the underlying issue, I guess

happy-kitchen-89482

10/26/2022, 7:28 AM

In your scenario, how are you loading my_data.txt? I.e., by which path, relative to which package?

happy-kitchen-89482

10/26/2022, 7:28 AM

In other words, what would you expect to see in the generated setup.py’s package_data?

high-yak-85899

10/26/2022, 4:35 PM

We add the

resources

target associated with the file we want to the dependencies of a

python_sources

target in another directory. We expect the built package to place things in a similar relative path so we do something like

pathlib.Path(__file__).parent / '<path to thing>'

to grab.

high-yak-85899

10/26/2022, 4:36 PM

Personally, I'm more familiar with using

<http://MANIFEST.in|MANIFEST.in>

to control data files when building distributions. I'm not sure how I would expect

package_data

to look in this scenario.

happy-kitchen-89482

10/26/2022, 4:38 PM

I see.

<http://MANIFEST.in|MANIFEST.in>

doesn’t distinguish between source and data files, IIUC

high-yak-85899

10/26/2022, 4:41 PM

Correct. Also, I think I know how to reproduce this. Need to tweak the example repo some more.

high-yak-85899

10/26/2022, 4:49 PM

Copy code

~/devel/pants-distribution-bug (distribution-bug-repro)$ ./pants dependencies --transitive //:test_distribution
//:reqs#ansicolors
//:reqs#setuptools
//:reqs#types-setuptools
//requirements.txt:reqs
helloworld/external/data/top_secret.txt:secret_data
helloworld/greet/__init__.py:lib
helloworld/greet/greeting.py:lib
helloworld/greet:translations
helloworld/translator/translator.py:lib
~/devel/pants-distribution-bug (distribution-bug-repro)$ ./pants package //:test_distribution
10:48:48.50 [INFO] Wrote dist/test_dist-0-py3-none-any.whl
10:48:48.50 [INFO] Wrote dist/test_dist-0.tar.gz
~/devel/pants-distribution-bug (distribution-bug-repro)$ unzip -l dist/test_dist-0-py3-none-any.whl 
Archive:  dist/test_dist-0-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2022-10-26 16:47   helloworld/greet/__init__.py
     1317  2022-10-26 16:47   helloworld/greet/greeting.py
      323  2022-10-26 16:47   helloworld/greet/translations.json
      249  2022-10-26 16:47   test_dist-0.dist-info/METADATA
       92  2022-10-26 16:47   test_dist-0.dist-info/WHEEL
       62  2022-10-26 16:47   test_dist-0.dist-info/entry_points.txt
        1  2022-10-26 16:47   test_dist-0.dist-info/namespace_packages.txt
       11  2022-10-26 16:47   test_dist-0.dist-info/top_level.txt
      737  2022-10-26 16:47   test_dist-0.dist-info/RECORD
---------                     -------
     2792                     9 files

high-yak-85899

10/26/2022, 4:49 PM

Okay, reproduced. I'll push to my repo and open a bug ticket so we can discuss further there.

high-yak-85899

10/26/2022, 5:13 PM

Dirty details in https://github.com/pantsbuild/pants/issues/17353

🙏 1

5 Views

Open in Slack

Previous Next