Hi I have a question about how to package the config static Pants #general

Hi, I have a question about how to package the con...

refined-addition-53644

12/13/2021, 6:01 PM

Hi, I have a question about how to package the config/static files with a python distribution. Right now I add it as a

resources

dependency explicitly in

python_distribution

target, but it still doesn't get shipped with the source code. What is the way to achieve this?

enough-analyst-54434

12/13/2021, 6:10 PM

One way to think of things to make more general sense of it all: Python libraries should be self contained in the root

python_sources

(used to be call python_library) target that defines the root of the library. The

python_distribution

target then just points to that

python_sources

target. The reason this makes sense is - presumably - the library should be useable without ever publishing it (you should be able to use the library locally in a repl session and have the library successfully load its resources). So a

python_distribution

target should just contain the extra metadata needed to publish a library and nothing more.

enough-analyst-54434

12/13/2021, 6:12 PM

So, concretely, move your resources dependency to the

python_sources

target owning the files that actually load or use the resources.

happy-kitchen-89482

12/13/2021, 6:31 PM

Good reminder that we really should log a warning about this! @refined-addition-53644 would you mind filing an issue at https://www.github.com/pantsbuild/pants/issues ?

👍 1

refined-addition-53644

12/14/2021, 4:03 PM

@enough-analyst-54434 I still am not able to package the static resources, even after providing all the static resources as explicit dependency when building

python_sources

This is how the python package looks like.

Copy code

pyfleet-vehicle-spec
├── BUILD
├── config/
├── database/
├── pyfleet_vehicle_spec/
├── pyproject.toml
├── README.md
├── resources/
├── tests/

The static resources are inside • config/ • database/ • resources/ and the python source code is inside

pyfleet_vehicle_spec

. I have checked the generated tar by unzipping it, and it includes everything which is inside

pyfleet_vehicle_spec

but nothing from any other resource. This is how the BUILD file looks like

Copy code

resources(
    name="req_config",
    sources=["config/**/*.py", "config/**/*.yaml"]
)

resources(
    name="req_resources",
    sources=["resources/**/*.py", "resources/**/*.csv"]
)

resources(
    name="req_db",
    sources=["database/**/*.py", "database/**/*.db"]
)

resources(
    name="req_version",
    sources=["pyfleet_vehicle_spec/VERSION"]
)

python_sources(
    name="lib",
    sources=["pyfleet_vehicle_spec/**/*.py"],
    dependencies=[":req_config", ":req_db", ":req_resources", ":req_version"],
)

resources(
    name="test_resources",
    sources=["tests/resources/**/*.py", "tests/resources/**/*.csv"]
)

python_tests(
    name="tests",
    sources=["tests/**/test_*.py"],
    dependencies=[":test_resources"],
    runtime_package_dependencies=[":pyfleet-vehicle-spec-dist"])

# Needed together for building the distribution
python_distribution(
    name="pyfleet-vehicle-spec-dist",
    dependencies=[":lib"],
    wheel=True,
    sdist=True,
    provides=setup_py(
        name="pyfleet-vehicle-spec",
        description="Vehicle spec",
        include_package_data=True,
    ),
    generate_setup=True,
)

enough-analyst-54434

12/14/2021, 6:08 PM

Thanks for the detailed data @refined-addition-53644. I repro and this exposes a bug in our calculation of package_data for the generated

setup.py

. We ensure the resource files live in a package. This idea is correct, but the implementation is bad since we only consider packages as those directories containing .py files owned by

python_sources

targets. This need not be the case as this example demonstrates:

Copy code

$ tree
.
├── config
│   ├── config.yaml
│   └── __init__.py
└── pyfleet_vehicle_spec
    └── source.py

2 directories, 3 files
$ cat config/config.yaml 
hello: world
$ cat pyfleet_vehicle_spec/source.py 
import pkgutil

print(pkgutil.get_data("config", "config.yaml"))
$ python -c 'import pyfleet_vehicle_spec.source'
b'hello: world\n'

Here the complete contents of the config/ directory could be owned by a

resources

target as they likely are in your case.

enough-analyst-54434

12/14/2021, 6:08 PM

I'll file here shortly.

enough-analyst-54434

12/14/2021, 6:12 PM

@happy-kitchen-89482 & @hundreds-father-404 do you agree its perfectly fine to have a

resources

target own any file type even if we think of certain file types as being generally owned by other targets. In this case a

resources

target owning a resource file + an

__init__.py

file? We can certainly force a user to type out two targets here and then add a manual dependency on both instead of just the resources target, but that seems a bit much to me. And really, to definitely harp on a concept, do we agree this is perfectly fine in the context of the package goal. Target types almost never make sense. The use of metadata by a specific verb is what makes sense of that metadata. In the case of the package goal, the python impl should only care that packages comes from any directory with at least 1 py file, not what target owns that .py file.

enough-analyst-54434

12/14/2021, 6:17 PM

So, @refined-addition-53644 pending bug status here, you can work around by splitting all your resources targets in two. One

resources

target to own non-python resource files and one

python_sources

target to own the python files in the resource directory. Then just add both to your explicit dependency list for

lib

. Does that make sense?

enough-analyst-54434

12/14/2021, 6:20 PM

FWIW, the bug - if we agree this is a bug - is here: https://github.com/pantsbuild/pants/blob/0f6b523eebda15315c0f92bf4e5e1e1421c084ac/src/python/pants/backend/python/goals/setup_py.py#L848-L861 That packages set is only formed from

python_source(s)

targets.

refined-addition-53644

12/14/2021, 6:53 PM

Do you mean something like this @enough-analyst-54434

Copy code

python_sources(
    name="lib_config",
    sources=["config/**/*.py"]
)
resources(
    name="req_config",
    sources=["config/**/*.yaml"]
)

python_sources(
    name="lib_resources",
    sources=["resources/**/*.py"]
)
resources(
    name="req_resources",
    sources=["resources/**/*.csv"]
)

python_sources(
    name="lib_db",
    sources=["database/**/*.py"]
)
resources(
    name="req_db",
    sources=["database/**/*.db"]
)

resources(
    name="req_version",
    sources=["pyfleet_vehicle_spec/VERSION"]
)

python_sources(
    name="lib",
    sources=["pyfleet_vehicle_spec/**/*.py"],
    dependencies=[
        ":lib_config",
        ":req_config",
        ":lib_db",
        ":req_db",
        ":lib_resources",
        ":req_resources",
        ":req_version"],
)

resources(
    name="test_resources",
    sources=["tests/resources/**/*.py", "tests/resources/**/*.csv"]
)

python_tests(
    name="tests",
    sources=["tests/**/test_*.py"],
    dependencies=[":test_resources"],
    runtime_package_dependencies=[":pyfleet-vehicle-spec-dist"])

# Needed together for building the distribtion
python_distribution(
    name="pyfleet-vehicle-spec-dist",
    dependencies=[":lib"],
    wheel=True,
    sdist=True,
    provides=setup_py(
        name="pyfleet_vehicle_spec",
        description="Provides vehicle specification and CO2 emissions.",
        include_package_data=True,
    ),
)

enough-analyst-54434

12/14/2021, 6:56 PM

Yes. Try that out. It could be simplified, but that's a personal choice and we can talk about that after you confirm the workaround works.

enough-analyst-54434

12/14/2021, 6:57 PM

Oh, two things: 1.

setup_py

needs to have

version

- you should be getting an error without that. 2. Remove

include_package_data

refined-addition-53644

12/14/2021, 7:13 PM

1. It still doesn't work with the way you suggested - separating python files and resources. 2. I had modified the

setup_py

to read version from local VERSION file using an internal plugin. I just disabled the plugin and ran with the default

setup_py

and still doesn't work. Also I had removed the

include_package_data

refined-addition-53644

12/14/2021, 7:14 PM

I see no non python file being included. I tried to add README.md as a resource which doesn't work either.

enough-analyst-54434

12/14/2021, 7:28 PM

I do not repro. Here is my setup - what is substantially different from yours?:

Copy code

$ tree .
.
├── pants
├── pants_from_sources
├── pants.toml
└── pyfleet-vehicle-spec
    ├── BUILD
    ├── config
    │   ├── config.yaml
    │   └── random.py
    ├── pyfleet_vehicle_spec
    │   └── source.py
    └── resources
        ├── __init__.py
        └── resource.csv

4 directories, 9 files
$ cat pants.toml 
[GLOBAL]
pants_version = "2.8.0"

backend_packages = [
  "pants.backend.python",
]

[anonymous-telemetry]
enabled = false

[source]
root_patterns = ["/pyfleet-vehicle-spec"]
$ cat pyfleet-vehicle-spec/BUILD 
resources(
    name="req_config",
    sources=["config/**/*.yaml"]
)

python_sources(
    name="req_config_srcs",
    sources=["config/**/*.py"]
)

resources(
    name="req_resources",
    sources=["resources/**/*.csv"]
)

python_sources(
    name="req_resources_srcs",
    sources=["resources/**/*.py"]
)

python_sources(
    name="lib",
    sources=["pyfleet_vehicle_spec/**/*.py"],
    dependencies=[
        ":req_config",
        ":req_config_srcs",
        ":req_resources",
        ":req_resources_srcs",
    ],
)

python_distribution(
    name="pyfleet-vehicle-spec-dist",
    dependencies=[":lib"],
    wheel=True,
    sdist=True,
    provides=setup_py(
        name="pyfleet-vehicle-spec",
        version="0.0.1",
    ),
    generate_setup=True,
)
$ ./pants package ::
11:27:45.17 [INFO] Wrote dist/pyfleet-vehicle-spec-0.0.1.tar.gz
11:27:45.17 [INFO] Wrote dist/pyfleet_vehicle_spec-0.0.1-py3-none-any.whl
$ tar -tzf dist/pyfleet-vehicle-spec-0.0.1.tar.gz
pyfleet-vehicle-spec-0.0.1/
pyfleet-vehicle-spec-0.0.1/setup.cfg
pyfleet-vehicle-spec-0.0.1/PKG-INFO
pyfleet-vehicle-spec-0.0.1/setup.py
pyfleet-vehicle-spec-0.0.1/backend_shim.py
pyfleet-vehicle-spec-0.0.1/MANIFEST.in
pyfleet-vehicle-spec-0.0.1/resources/
pyfleet-vehicle-spec-0.0.1/resources/resource.csv
pyfleet-vehicle-spec-0.0.1/resources/__init__.py
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec.egg-info/
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec.egg-info/top_level.txt
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec.egg-info/namespace_packages.txt
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec.egg-info/dependency_links.txt
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec.egg-info/SOURCES.txt
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec.egg-info/PKG-INFO
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec/
pyfleet-vehicle-spec-0.0.1/pyfleet_vehicle_spec/source.py
pyfleet-vehicle-spec-0.0.1/config/
pyfleet-vehicle-spec-0.0.1/config/random.py
pyfleet-vehicle-spec-0.0.1/config/config.yaml

enough-analyst-54434

12/14/2021, 7:29 PM

N.B @refined-addition-53644 I definitely do repro before splitting my two resources targets into resources / python sources pairs.

refined-addition-53644

12/14/2021, 7:50 PM

That was very stupid. I was unzipping some old dist 🤦 With the separation of python_sources and resources, it works. Thank you for your time and patience! 🙏

enough-analyst-54434

12/14/2021, 7:51 PM

You're welcome. I did not take the time to file an issue, because I'm not sure if my bug assessment will be agreed upon, but this is the fix to be debated: https://github.com/pantsbuild/pants/pull/13878

🙌 1

11 Views

Open in Slack

Previous Next