https://pantsbuild.org/ logo
Powered by
c

curved-microphone-39455

01/18/2023, 5:18 PM
Hey there! I have progressing really nicely to get all my CI/CD merged with Pants but one project is bugging me hard. I have a Django Project and I have been following this https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ to start creating the BUILD file that will generate the Docker image at the end. Currently I am having this error while running 
package
Copy code
#7 [deps 3/3] RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app
#7 sha256:3328d42cc75c5094a8bfd08b0070ed2a8d406a2bba037baaba700942750bf8d2
#7 4.316 /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.bootstrap/pex/dist_metadata.py:397: PEXWarning: Ignoring 1 `Requires` field in /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.deps/django_colorfield-0.8.0-py3-none-any.whl metadata:
#7 4.316 1.) Requires: django (>=2.2)
#7 4.316
#7 4.316 You may have issues using the 'django-colorfield' distribution as a result.
#7 4.316 More information on this workaround can be found here:
#7 4.316   <https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585>
#7 4.316
#7 4.316   pex_warnings.warn(
#7 4.338 /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.bootstrap/pex/dist_metadata.py:397: PEXWarning: Ignoring 1 `Requires` field in /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.deps/django_widget_tweaks-1.4.12-py3-none-any.whl metadata:
#7 4.338 1.) Requires: django (>=2.2)
#7 4.338
#7 4.338 You may have issues using the 'django-widget-tweaks' distribution as a result.
#7 4.338 More information on this workaround can be found here:
#7 4.338   <https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585>
#7 4.338
#7 4.338   pex_warnings.warn(
#7 6.736 Encountered collision building venv at /bin/app from /binary.pex:
#7 6.736 1. /bin/app/lib/python3.10/site-packages/tests/__init__.py was provided by:
#7 6.736        sha1:da39a3ee5e6b4b0d3255bfef95601890afd80709 -> /root/.pex/installed_wheels/5b3bc328e79c857f3d9a59b674e6770097cd73bb8696625b754c3b077c5e2fd3/django_rest_passwordreset-1.3.0-py3-none-any.whl/tests/__init__.py
#7 6.736        sha1:adc83b19e793491b1c6ea0fd8b46cd9f32e592fc -> /root/.pex/installed_wheels/591847b93161828c4e1b0197e3f531e82689d43624e2ff5142ea938514584112/python_quickbooks-0.9.2-py3-none-any.whl/tests/__init__.py
#7 ERROR: executor failed running [/bin/sh -c PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app]: exit code: 1
------
 > [deps 3/3] RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app:
------
executor failed running [/bin/sh -c PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app]: exit code: 1
and this is my BUILD file
Copy code
python_sources(
    dependencies=[":static-files"],
)

resources(name="static-files", sources=["**/*.html", "**/*.css", "**/*.js", "**/*.png"])

poetry_requirements(
    name="poetry",
    module_mapping={
        "django-allauth": ["allauth"],
        "django-decorator-include": ["decorator_include"],
        "django-multifactor": ["multifactor"],
        "python-quickbooks": ["quickbooks", "intuitlib"],
        "djangorestframework-simplejwt": ["rest_framework_simplejwt"],
        "django-recaptcha": ["captcha"],
        "django-invitations": ["invitations"],
    },
)

pex_binary(
    name="binary-deps",
    entry_point="gunicorn",
    dependencies=[":poetry", "src/services/dashboard/dashboard/wsgi.py"],
    layout="packed",
    include_sources=False,
    include_tools=True,
)

pex_binary(
    name="binary-srcs",
    entry_point="gunicorn",
    dependencies=[":poetry", "src/services/dashboard/dashboard/wsgi.py"],
    layout="packed",
    include_requirements=False,
    include_tools=True,
)

pex_binary(
    name="binary-manage",
    entry_point="manage.py",
    dependencies=[":poetry", "src/services/dashboard/manage.py"],
    layout="packed",
    execution_mode="venv",
)

docker_image(
    name="img-deps",
    repository="dashboard",
    registries=["build"],
    image_tags=["deps"],
    skip_push=True,
    instructions=[
        "FROM python:3.10-slim",
        "COPY src.services.dashboard/binary-deps.pex /binary-deps.pex",
        "RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary-deps.pex venv --scope=deps --compile /opt/app",
    ],
)

docker_image(
    name="img-srcs",
    repository="dashboard",
    registries=["build"],
    image_tags=["srcs"],
    skip_push=True,
    instructions=[
        "FROM python:3.10-slim",
        "COPY src.services.dashboard/binary-srcs.pex /binary-srcs.pex",
        "RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary-srcs.pex venv --scope=srcs --compile /opt/app",
    ],
)

docker_image(
    name="img",
    dependencies=[":img-srcs", ":img-deps"],
    registries=["<http://docker.io|docker.io>"],
    repository="repo/dashboard",
    image_tags=["{build_args.RELEASE_VERSION}"],
    instructions=[
        "FROM python:3.10-slim",
        "RUN apt-get update && apt-get install iputils-ping -y",
        'ENTRYPOINT ["/opt/app/pex"]',
        "WORKDIR /opt/app",
        "EXPOSE 80",
        "COPY --from=build/dashboard:deps /opt/app /opt/app",
        "COPY --from=build/dashboard:srcs /opt/app /opt/app",
        "COPY src.services.dashboard/binary-manage.pex /opt/app/manage",
        'CMD ["--worker-class=gevent","--worker-connections=1000", "--workers=1", "-b", "0.0.0.0:80", "dashboard.wsgi:application"]',
    ],
)
I understand that 2 library is trying to write the same thing in the PEX compile, but I am totally lost on how to solve this issue, I have tried with a simple version and with many more configuration. Any idea on how to solve this?
e

enough-analyst-54434

01/18/2023, 5:42 PM
The ever-present 
--help
is a friend you can lean on. I won't leave you hanging, but try 
PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app --help
interactively and see if there is an option that describes your problem.
I will say though, those 2 wheels are janky / bad citizens. You might consider filing bugs with each. They should not be including a 
tests
package.
@curved-microphone-39455 you can cry uncle re 
--help
if you still can't figure it out and I can push you along, but I figure its better to self serve if you can.
c

curved-microphone-39455

01/18/2023, 5:51 PM
@enough-analyst-54434 I always prefer to learn by my self and thanks for the tip, I will try this way, but PEX is really new to me and I was reading all their doc since I was thinking the 
/binary.pex
was already bundle and I was sure the process was coming from the 
pex_binary()
from pants, but if I can debug on runtime in my Dockerfile Step, yeah!
e

enough-analyst-54434

01/18/2023, 5:51 PM
Reading doc last place, --help 1st place! I work that way anyhow.
Basically --help is tied to code, and code is reality.
Doc can drift from reality
c

curved-microphone-39455

01/18/2023, 5:52 PM
Totally agree haha but like I said, sure I had to debug the Pants process that generate the Pex Binary and I was confuse how to run 
--help
mid process of pants 
package
e

enough-analyst-54434

01/18/2023, 5:53 PM
That's not the way to think about it.
You have a Dockerfile RUN step that errors.
Ask for help for that run step 1st.
c

curved-microphone-39455

01/18/2023, 5:54 PM
Make sense, thanks 🙂
I would be really lucky that 
--help
reveal me an option that looks like 
--collisions-ok
so it let me build correctly and allow me to open an issue with those libraries that create the collision since they are not correctly packaged... I would be really lucky 😉
Thanks 😄
e

enough-analyst-54434

01/18/2023, 6:22 PM
Ok, great, You're welcome. Thanks for being game to figure that out.
7 Views
Powered by