https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all, I am struggling with understanding the do...
# general
a
Hey all, I am struggling with understanding the docker builds. I am testing out the environment builds in 2.15.0rc3. When I build the pex locally, I can see the .deps folder contains manylinux wheels. However, when I package a docker image, it is containing macos wheels. I am struggling to understand how the pex is getting pulled in with the wrong wheels. I am only currently supporting the one environment for linux, but am developing on macos.
e
Pants docker support is not fully integrated. It just takes a package but does not influence how the package is built; so you must arrange the package it takes to be built for the right platform. IOW, does your 
pex_binary
target spell out it must be built for Linux in any way?
Do you use 
platforms
or 
complete_platforms
fields?
Oh, by "environment builds" do you mean?: https://www.pantsbuild.org/v2.15/docs/environments If so I apologize and will step away and let others help. I do not know much about the new environments feature.
a
Correct, trying the environments field:
Copy code
pex_binary(
    name="app",
    entry_point="main.py:main",
    layout="packed",
    execution_mode="venv",
    environment="python_3_9"
)
e
Ah, ok. Yeah - someone else will need to chime in. There is alot of activity here and even some fixes in the last few days / weeks - so thanks for trying this out and reporting issues.
But I think I can speak on those other helpers behalf - more data is always better than less when remote debugging! Can you provide the other relevant details? What is the environment definition for "python_3_9", etc.
a
pants.toml
Copy code
[environments-preview.names]
default = "//:python_3.9.16"
local = "//:macos_x86_desktop"
python_3_9 = "//:python_3.9.16"
BUILD:
Copy code
docker_environment(
  name="python_3.9.16",
  platform="linux_x86_64",
  image="python:3.9.16",
  python_bootstrap_search_path=["<PATH>"]
)

local_environment(
  name="macos_x86_desktop",
  compatible_platforms=["macos_x86_64"],
)

local_environment(
  name="macos_arm_desktop",
  compatible_platforms=["macos_arm64"],
)
w
will see if i can repro this.
mm. actually, re-reading this, i think that there is a piece missing from your snippets: you mention that your final output is a 
docker_image
target?
a
here is the additional context with the pex and the docker image:
Copy code
pex_binary(
    name="app",
    entry_point="main.py:main",
    layout="packed",
    execution_mode="venv",
    environment="python_3_9"
)

docker_image(
    name="docker",
    image_tags=["test"],
    repository="ticketing_service",
    instructions=[
        "FROM python:3.9.16",

        "COPY klaviyo.domains.event_processing.ticketing_service.server/app.pex /bin/app.pex",
        ...]',

    ]
)
So if I am understanding, we should add the platform to this and that would help with the correct routing of the deps?
w
no: i’m able to reproduce the issue, and this is a bug. the 
docker_image
target should be respecting the environment set on the 
pex_binary
, but isn’t.
i’ll have a patch out shortly. sorry for the trouble!
a
Just want to confirm that this works for me. I am now able to get run the docker image. Thank you for the speedy fix
Hmm, running just got me this. I think I messed something up in the environment:
Copy code
| => ./pants_from_sources package klaviyo/domains/event_processing/ticketing_service/server:app --keep-sandboxes=on_failure
15:58:20.32 [INFO] Preserving local process execution dir /private/var/folders/4j/6vs2kh994fq5klsq03n786zr0000gp/T/pants-sandbox-Q160Nu for Test binary /Users/tyler.bream/.pyenv/shims/python.
15:58:20.42 [INFO] Preserving local process execution dir /private/var/folders/4j/6vs2kh994fq5klsq03n786zr0000gp/T/pants-sandbox-rTpWNQ for Test binary /Users/tyler.bream/.pyenv/versions/2.7.18/bin/python.
15:58:20.51 [INFO] Preserving local process execution dir /private/var/folders/4j/6vs2kh994fq5klsq03n786zr0000gp/T/pants-sandbox-Kc6iX0 for Test binary /Users/tyler.bream/.pyenv/versions/infrastructure-deployment/bin/python.
15:58:20.57 [INFO] Preserving local process execution dir /private/var/folders/4j/6vs2kh994fq5klsq03n786zr0000gp/T/pants-sandbox-4p04ly for Test binary /Users/tyler.bream/.pyenv/shims/python3.
15:58:24.06 [INFO] Preserving local process execution dir /private/var/folders/4j/6vs2kh994fq5klsq03n786zr0000gp/T/pants-sandbox-l4FYF0 for Building 35 requirements for klaviyo.domains.event_processing.ticketing_service.server/app.pex from the 3rdparty/python/python3-deps_lock.txt resolve: backoff==1.11.1, blinker==1.4, certifi==2022.6.15, charset-normalizer==2.1.0, click==8.0.1, deprecated==1.2.13, dumb-init==1.2.5, flask==2.0.1, googleapis-common-protos==1.56.1, grpcio==1.46.3, gunicorn==20.1.0, idna==3.2, itsdangerous==2.0.1, jinja2==3.0.1, markupsafe==2.0.1, opentelemetry-api==1.12.0, opentelemetry-exporter-otlp-proto-grpc==1.12.0, opentelemetry-exporter-otlp-proto-http==1.12.0, opentelemetry-exporter-otlp==1.12.0, opentelemetry-instrumentation-flask==0.33b0, opentelemetry-instrumentation-wsgi==0.33b0, opentelemetry-instrumentation==0.33b0, opentelemetry-proto==1.12.0, opentelemetry-sdk==1.12.0, opentelemetry-semantic-conventions==0.33b0, opentelemetry-util-http==0.33b0, protobuf==3.20.1, requests==2.28.1, sentry-sdk==0.12.3, six==1.16.0, statsd==3.3.0, typing-extensions==4.3.0, urllib3==1.26.6, werkzeug==2.0.1, wrapt==1.14.1
15:58:24.06 [INFO] Completed: Building 35 requirements for klaviyo.domains.event_processing.ticketing_service.server/app.pex from the 3rdparty/python/python3-deps_lock.txt resolve: backoff==1.11.1, blinker==1.4, certifi==2022.6.15... (807 characters truncated)
15:58:24.06 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building 35 requirements for klaviyo.domains.event_processing.ticketing_service.server/app.pex from the 3rdparty/python/python3-deps_lock.txt resolve: backoff==1.11.1, blinker==1.4, certifi==2022.6.15, charset-normalizer==2.1.0, click==8.0.1, deprecated==1.2.13, dumb-init==1.2.5, flask==2.0.1, googleapis-common-protos==1.56.1, grpcio==1.46.3, gunicorn==20.1.0, idna==3.2, itsdangerous==2.0.1, jinja2==3.0.1, markupsafe==2.0.1, opentelemetry-api==1.12.0, opentelemetry-exporter-otlp-proto-grpc==1.12.0, opentelemetry-exporter-otlp-proto-http==1.12.0, opentelemetry-exporter-otlp==1.12.0, opentelemetry-instrumentation-flask==0.33b0, opentelemetry-instrumentation-wsgi==0.33b0, opentelemetry-instrumentation==0.33b0, opentelemetry-proto==1.12.0, opentelemetry-sdk==1.12.0, opentelemetry-semantic-conventions==0.33b0, opentelemetry-util-http==0.33b0, protobuf==3.20.1, requests==2.28.1, sentry-sdk==0.12.3, six==1.16.0, statsd==3.3.0, typing-extensions==4.3.0, urllib3==1.26.6, werkzeug==2.0.1, wrapt==1.14.1' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 190, in _run_module_as_main
    mod_name, mod_spec, code = _get_main_module_details(_Error)
  File "/usr/local/lib/python3.9/runpy.py", line 239, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr/local/lib/python3.9/runpy.py", line 158, in _get_module_details
    code = loader.get_code(mod_name)
  File "<frozen zipimport>", line 159, in get_code
  File "<frozen zipimport>", line 709, in _get_module_code
  File "<frozen zipimport>", line 536, in _get_data
FileNotFoundError: [Errno 2] No such file or directory: '/pants-sandbox/pants-sandbox-l4FYF0/./pex'
e
@alert-dawn-51425 you may be hitting this: https://github.com/pantsbuild/pants/issues/18162#issuecomment-1421673661 - in short, are you on latest Docker for Mac and if not can you try upgrading?
a
I am on the latest: Docker Desktop 4.16.2 (95914) is currently the newest version available.
e
Ok, that isn't it then. The presentation is very similar to that issue but clearly more / different going on.
a
It does look the same. I am running both the ./pants from the rc3 version, and the branch provided above. It appeared after I ran the ./pants commend, realized i didn't want that, and exited the command and then re-ran the ./pants_from_source
I am consistently getting that at the moment.
e
Alright. I think Stu is offline for the day; so you may not get a paged-in mind back on this until tomorrow.
a
Works for me. Thanks for checking on this.
restarting docker fixed the issue, btw
Any idea when this would be available?
w
i’ve reopened https://github.com/pantsbuild/pants/issues/18162 … it’s potentially hairy, since I don’t know how much we’ll be able to trust Docker’s filesystem consistency moving forward. given that, i might end up doing something that sidesteps that problem entirely (possibly behind a flag, depending on the performance impact)
…but to answer your question: hopefully by the end of the day, maybe tomorrow
a
I am happy to test out if necessary.
w
@alert-dawn-51425: are you able to get your use case fully working on 
2.15.0rc5
? the hypothesis i just posted to the ticket is that the missing file error you experienced shouldn’t happen in 
2.15.x
a
Yes, I have been using it for the last few days with no issues.
🎉 1
Is the 2.15.0rc5 pex available yet for on the releases. I don't see it. I wanted to download and store for a docker image.
e
Looks like that release step was missed. I've fired up a build for that PEX locally - it'll probably be 30 minutes or so.
That said ... why / how do you use this PEX in the container? There are almost certainly better (more performant) ways depending on what you are doing.
a
I want to avoid the CI/CD having to download the pex each time, and want to seed the docker image for running CI with the pex.
e
Yeah - no need for a PEX for that. Have a Docker RUN step or steps that follow the consumer installing Pants instructions. Currently there is no flag to select a Pants version to seed; so you need to create a throw-away pants.toml in the RUN that has the 
pants_version
you want to seal in as cached to the image.
Does that make sense?
It may make sense for me to add a way to seed w/o creating a throw-away pants.toml over in scie-pants. Then you could just say 
PANTS_VERSION=X pants -V
anywhere to seed version X.
Ok, upload started. My bandwidth is poor and it's 209MB, so 10 more minutes or so.
8 Views
Open in Slack
PreviousNext
Powered by