https://pantsbuild.org/ logo
Join Slack
Powered by
Hi, I have a general question about python packagi...
# general
l
Hi, I have a general question about python packaging with pants. We want to include non-source files (defined as 
resouces
added to the dependencies of the 
python_sources
) into the wheel. When I run the code as a 
.pex
file everything works nicely and we can get the files from 
importlib
. However, if we generate a wheel, the files don't get packaged and the 
python_resources
keyword in setup.py is an empty dict. Is including resources in wheel in general not supported?
h
Disclaimer: I'm not a maintainer. Hi, Yes, it is definitely supported. Do you let Pants use the existing 
setup.py
or its own generated one? In my case, I have the following definition template:
Copy code
python_distribution(
    name="PROJECT_NAME-dist-pure-wheel",
    dependencies=[
        ":pyproject",
        ":PROJECT_NAME-sources",
    ],
    provides=python_artifact(
        name="PROJECT_NAME",
    ),
    wheel_config_settings={
        "--build-option": ["--python-tag", "py3"]
    },
    generate_setup=True,
    sdist=False,
    wheel=True,
)
Where 
PROJECT_NAME
is replaced with the projects' name. And you need the dependency on your resource to be somehow linked back to the 
python_distribution
directive. For instance, it could be listed under 
dependencies
, or be a dependency of something that is under 
dependencies
.
l
I use 
generate_setup=True,
and have it listed in the dependencies (both of the distribution and the sources targets). It seems to be understood by pants (otherwise it would not be in the pex venv), so I wonder what else could be the culprit
The resource files are also listed by the 
pants dependencies
command for that target
h
When you 
pants package TARGET
and unzip the result, you don't see the file?
l
no, and the kwargs to the setup function is empty too
h
Sorry, I'm out of ideas... 🤷
l
Thanks anyways!!
c
if you run your package command with 
--keep-sandboxes=always
and see if you have the resources in there.. can help follow the thread to the root of the issue.
h
resources
should be packed into the wheel
If they're not either something is misconfigured or there is a bug
A small repro would be golden
l
Thanks. I will continue trying; I'll report here or come up with a minimal example
Hi @curved-television-6568, if I add the parameter, where would I find the sandboxes? A few directories in 
/private
(macos) are printed but they only contain a single file (such as 
__run.sh
)
Minimal example
Copy code
# projects/minimal/BUILD
resources(name="minimal_templates", sources=["templates/*/*.*"])

python_distribution(
    name="pkg",
    dependencies=[
        ":minimal_templates",  # templates
        "./src/minimal:minimal",  # sources
    ],
    provides=python_artifact(name="minimal", version="0.0.1"),
    generate_setup=True,
)
and there is only an init in the soures:
Copy code
python_sources(
    dependencies=["//projects/minimal:minimal_templates"]
)
Commenting out the dependencies has no effect. In fact, having it here does not list the template in 
pants dependencies project/minimal:pkg
The template(s) are in a folder 
/project/minimal/templates/template.yaml
If I uncomment the template in the distribution, it will vanish from 
pants dependencies project/minimal.pkg
otherwise the result is:
Copy code
projects/minimal/src/minimal/__init__.py
projects/minimal:minimal_templates
The template is not in the package
Copy code
pants package projects/minimal:pkg 
tar xvzf dist/minimal-0.0.1.tar.gz 
x minimal-0.0.1/
x minimal-0.0.1/MANIFEST.in
x minimal-0.0.1/PKG-INFO
x minimal-0.0.1/backend_shim.py
x minimal-0.0.1/minimal/
x minimal-0.0.1/minimal/__init__.py
x minimal-0.0.1/minimal.egg-info/
x minimal-0.0.1/minimal.egg-info/PKG-INFO
x minimal-0.0.1/minimal.egg-info/SOURCES.txt
x minimal-0.0.1/minimal.egg-info/dependency_links.txt
x minimal-0.0.1/minimal.egg-info/namespace_packages.txt
x minimal-0.0.1/minimal.egg-info/top_level.txt
x minimal-0.0.1/setup.cfg
x minimal-0.0.1/setup.py
c
pants will print the paths for all the saved sandboxes to the console as you run. in case it’s cached it may not show, however, in which case you can try with 
--no-local-cache
(or 
--no-cache
)
l
ok, now I found it. There is no trace of the template in the 
chroot
subfolder
c
so, double check all assumptions, and that there’s a dependency to your file from the target you are packaging. Could be a good idea to try 
pants paths --from=a --to=b
to ensure it is what you think it is.
l
Copy code
pants paths --from=projects/minimal:minimal_templates --to=projects/minimal:pkg
Like that (given the example above)? This gives me an empty result 
[]
c
yes, so now we’ve narrowed it down to be an issue with the dependency chain
do you get a hit with that if you check from your python source to the pkg?
l
Copy code
pants paths --from=projects/minimal/src/minimal:minimal --to=projects/minimal:pkg
[]
This I did not expect. The sources end up in the package
btw. I created a minimal pants repo with the files above and get the same results.. it should all be very standard by now
Copy code
pants peek projects/minimal:pkg
[
  {
    "address": "projects/minimal:pkg",
    "target_type": "python_distribution",
    "dependencies": [
      "projects/minimal/templates/tmpl.yaml:../minimal_templates",
      "projects/minimal/src/minimal/__init__.py:lib"
    ],
    "dependencies_raw": [
      ":minimal_templates",
      "./src/minimal:lib"
    ],
    "description": null,
    "entry_points": null,
    "env_vars": null,
    "generate_setup": true,
    "interpreter_constraints": null,
    "long_description_path": null,
    "output_path": "",
    "provides": {
      "name": "minimal",
      "version": "0.0.1"
    },
    "sdist": true,
    "sdist_config_settings": null,
    "tags": null,
    "wheel": true,
    "wheel_config_settings": null
  }
]
(I added 
name='lib'
to the sources) .. this all looks fine, doesn't it? Where would I expect the yaml file to appear?
I created a minimal repo. Can anyone try to reproduce please? https://github.com/StefanUlbrich/pants_minimal
đź‘€ 1
c
not sure if this is a conclusive general rule, but I found it didn’t include the resources until I moved them into the source tree of the code being packaged.
Copy code
❯ tar -tzf dist/minimal-0.0.1.tar.gz
minimal-0.0.1/
minimal-0.0.1/MANIFEST.in
minimal-0.0.1/PKG-INFO
minimal-0.0.1/backend_shim.py
minimal-0.0.1/minimal/
minimal-0.0.1/minimal/__init__.py
minimal-0.0.1/minimal/templates/
minimal-0.0.1/minimal/templates/tmpl.yaml
minimal-0.0.1/minimal.egg-info/
minimal-0.0.1/minimal.egg-info/PKG-INFO
minimal-0.0.1/minimal.egg-info/SOURCES.txt
minimal-0.0.1/minimal.egg-info/dependency_links.txt
minimal-0.0.1/minimal.egg-info/namespace_packages.txt
minimal-0.0.1/minimal.egg-info/top_level.txt
minimal-0.0.1/setup.cfg
minimal-0.0.1/setup.py
l
cool! It does work without the source tree if you create a pex file / application. Can we conclude that this is a bug?
c
I’m a packaging noob, so can’t say if it’s a bug, but filing a ticket for this is absolutely legit to do so we can track this (silently ignoring it is at least wrong, it should either work or warn, I think)
l
fair enough. I will open one tomorrow morning. Thanks a ton for your support
đź‘Ť 2
h
Thanks for the repro! A ticket would be great so we can document all this. I will take a look
l
Open in Slack
PreviousNext
Powered by