https://pantsbuild.org/ logo
Join Slack
Powered by
I am seeing the following error when trying to add...
# general
s
I am seeing the following error when trying to add two dist dependencies to docker container:
Copy code
./pants run py/apps/cronos/docker:docker_src
14:27:39.91 [ERROR] 1 Exception encountered:

  Exception: Can only merge Directories with no duplicates, but found 2 duplicate entries in :

`<http://MANIFEST.in|MANIFEST.in>`: 1.) file digest=64958bd91c39a06f78542a71383436bbf0c798138e4fe46780c8d9c08aa91278 size=179:

include <http://README.org|README.org> LICENSE VERSION 
include pybind/cnufft/cnufft.c
include pybind/cnufft/cnufft.h
include pybind/cnufft/libcnufft.so
include pybind/cnufft/libfnufft.so
prune *.pyc

`<http://MANIFEST.in|MANIFEST.in>`: 2.) file digest=892591bf1e4c7e1337cfd470096a467805fa2cb6f46b792fe6e3a6a8b0a72e0a size=366:

include README.rst
include setup.py
include nufft/__init__.py
include nufft/nufft.py
include nufft/nufft.pyf
include src/nufft/dfftpack.f
include src/nufft/dirft1d.f
include src/nufft/dirft2d.f
include src/nufft/dirft3d.f
include src/nufft/next235.f
include src/nufft/nufft1df90.f
include src/nufft/nufft2df90.f
include src/nufft/nufft3df90.f
include pyproject.toml
h
Ooops! Can you share your BUILD files?
s
Sure, first sub project build:
Copy code
poetry_requirements(name="pybind")

python_sources(
    name="pybind0",
)

resource(name="pyproject", source="pyproject.toml")
resources(
    name="pybind_files",
    sources=[
        "README.rst",
        "VERSION",
        "<http://MANIFEST.in|MANIFEST.in>",
        "pybind/cnufft/*",
        "pybind/cnufft/src/*",
    ],
)

python_distribution(
    name="pybind_dist",
    dependencies=[
        ":pyproject",
        ":pybind_files",
        "py/libs/pybind/setup.py:pybind0",
    ],
    provides=python_artifact(
        name="pybind",
        version="0.2.1",
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
    generate_setup=False,
)
second subproj build:
Copy code
poetry_requirements(name="nufft")

python_sources(name="nufft0")

resource(name="pyproject", source="pyproject.toml")
resources(
    name="nufft_files",
    sources=[
        "README.rst",
        "LICENSE",
        "<http://MANIFEST.in|MANIFEST.in>",
        "src/nufft/*",
        "nufft/nufft.pyf",
    ],
)

python_distribution(
    name="nufft_dist",
    dependencies=[
        ":pyproject",
        ":nufft_files",
        "py/libs/nufft/setup.py:nufft0",
    ],
    provides=python_artifact(
        name="nufft",
        version="0.0.3",
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
    generate_setup=False,
)
docker build:
Copy code
docker_image(
    name="docker_src",
    source="Dockerfile.main",
    dependencies=[
        "py/libs/nufft:nufft_dist",
        "py/libs/pybind:pybind_dist",
	":cfiles",
    ],
)

files(
    name="cfiles",
    sources=[
	"../scripts/run.sh",
	"./.tmux.conf",
    ]
)
h
Huh. @curved-television-6568, thoughts?
README.rst and MANIFEST.in are presumably the two file collisions the merge is complaining about, but why are we merging to begin with?
Does this happen with 
./pants package py/apps/cronos/docker:docker_src
too, or just 
run
?
c
Ouch. Sorry about that. Will take a look during the weekend..
s
It happents with 
./pants package ...
also
c
Hmm… which version of Pants is this?
I do not quite reproduce this issue, at least on the latest dev version of Pants (2.13.0.dev5)
I’m curious what your build context contains. If you add a bogus instruction (such as 
COPY foo /
) to make the docker build fail, Pants will print a message describing what files you have in the build context. You may need to temporarily remove the dependency on one of the dists in order to avoid the merge conflict to get that far, though.
Example output (from my repro attempt)
Copy code
$ ./pants package src/docker:test
21:14:45.55 [INFO] Completed: Building docker image test:latest
21:14:45.56 [WARN] Docker build failed for `docker_image` src/docker:test. The src/docker/Dockerfile.test have `COPY` instructions where the source files may not have been found in the Docker build context.

However there are possible matches. Please review the following list of suggested renames:

  * foo => proj_b-2.0.0-py3-none-any.whl

There are additional files in the Docker build context that were not referenced by any `COPY` instruction (this is not an error):

  * proj_a-1.0.0-py3-none-any.whl
  * proj_a-1.0.0.tar.gz
  * proj_b-2.0.0.tar.gz


21:14:45.56 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building docker image test:latest' failed with exit code 1.
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?



Use `--no-process-cleanup` to preserve process chroots for inspection.
I do not quite reproduce this issue, at least on the latest dev version of Pants (2.13.0.dev5)
Also tried on 2.10.0 real quick, I get the same behaviour there as well (i.e. I do still not repro the issue)
@sparse-pharmacist-76603 Please see if I’ve missed anything critical, or see if you can flesh something out based on your files, if you’re able to do that would be great 🙂
s
@curved-television-6568 Please take a look @ this repo: https://github.com/dimitar-petrov/pants_merge_debug
c
Thanks, will take a look later today! 👍
Got stuck on compiler errors for the nufft lib. Can post error message tomorrow (not in front of the computer atm)
s
This image should be able to build the modules:
Copy code
FROM 3.8-bullseye

RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
	curl \
        git \
	unzip \
	gfortran-10 \
 && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
👍 1
Hi @curved-television-6568, pushed another commit to that repo https://github.com/dimitar-petrov/pants_merge_debug with docker-compose.yml and instructions how to reproduce it inside the container.
Let me know if I can provide anything else.
c
Thanks. Will take another stab at this today. I'm on a Mac by default so the container will help :)
OK, so this issue is not related to the Docker backend, but with how 
pex_binary
works. The repo provided by @sparse-pharmacist-76603 reproduces the issue, and also for 
./pants package py/apps/pyapp:venv
.
My hunch is that it has to do with how resources are included in the pex file. @bitter-ability-32190 do you have any insights on that, given your recent investment on how files vs resources work?
b
👀 Gotta get my scroll finger warmed up
It's because of your source root
When we strip the source roots of both 
/py/libs/nufft/LICENSE
and 
/py/libs/pybind/LICENSE
you get 
LICENSE
, and if you ever depend on both in the sandbox 💥
🔥 1
s
@bitter-ability-32190, @curved-television-6568 thanks for looking into it. Is there a workarround I can implement?
c
🤷 I'll have to rely on the others chiming in on this one. Perhaps @happy-kitchen-89482 can find something/someone?
h
I'll take a quick look, I'm still not sure why we're not just building two separate wheels and including them, i.e., which pex is the one they are being merged into
The repro repo will be very helpful, thanks for setting that up
Ah, it's the "local dist" build, of course... No, that works fine if I do, say, 
./pants repl
.
@sparse-pharmacist-76603 Thanks for the bug report. Can you open an issue for this so we can track it? Include all the details you've provided here and link to your very helpful repro? Thanks!
c
Just to ask the question on behalf of Dimitar.. So no workaround currently?
h
Thinking about it...
👌 1
I think the issue is that we naively follow dependencies transitively via deps on python_distribution targets.
That may be too naive
c
Could a dependency exclude work to avoid getting the files included too far, perhaps?
s
👍 1
h
Thanks!
Ah, good idea @curved-television-6568, dependency excludes are a workaround.
@sparse-pharmacist-76603 Add these to the dependencies of the targets that depend on the two distributions:
Copy code
"!!py/libs/nufft/MANIFEST.in:nufft_files",
        "!!py/libs/nufft/README.rst:nufft_files",
        "!!py/libs/nufft/pyproject.toml:pyproject",
        "!!py/libs/nufft/setup.py:nufft0",
        "!!py/libs/pybind/MANIFEST.in:pybind_files",
        "!!py/libs/pybind/README.rst:pybind_files",
        "!!py/libs/pybind/pyproject.toml:pyproject",
        "!!py/libs/pybind/setup.py:pybind0",
E.g.,
Copy code
pex_binary(
    name="venv",
    dependencies=[
        "py/libs/nufft:nufft_dist",
        "py/libs/pybind:pybind_dist",
        "!!py/libs/nufft/MANIFEST.in:nufft_files",
        "!!py/libs/nufft/README.rst:nufft_files",
        "!!py/libs/nufft/pyproject.toml:pyproject",
        "!!py/libs/nufft/setup.py:nufft0",
        "!!py/libs/pybind/MANIFEST.in:pybind_files",
        "!!py/libs/pybind/README.rst:pybind_files",
        "!!py/libs/pybind/pyproject.toml:pyproject",
        "!!py/libs/pybind/setup.py:pybind0",
    ]
)
👍 2
This is far from ideal, but should get you through this until we can fix the underlying issue
b
(@happy-kitchen-89482 I think this touches on my discussion too... https://github.com/pantsbuild/pants/discussions/15847)
h
I think it might
Although this is a bit different, what we're trying to do is coherent, we're just doing it wrong
b
Well it would've been easier to handle it right, I think, if say, we have a "build deps" vs "runtime deps". The LICENSE is definitely build-time dep and therefore wouldn't be considered for transitive operations
h
True!
9 Views
Open in Slack
PreviousNext
Powered by