https://pantsbuild.org/ logo
Join Slack
Powered by
I'm trying to run <prefect> inside a Docker contai...
# general
a
I'm trying to run prefect inside a Docker container using Pants. I build my PEX binary with the following BUILD file:
Copy code
pex_binary(
    name="prefect",
    script="prefect",
    dependencies=["3rdparty/python#prefect", "projects/path/to/my/code"],
    layout="loose",
    execution_mode="venv"
)
If I package this PEX and run it from the dist folder the server starts as expected. If I create a Docker image that copies the very same PEX and runs it, it fails with:
Copy code
server-1  | Failed to find compatible interpreter on path /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.
server-1  | 
server-1  | Examined the following interpreters:
server-1  | 1.) /usr/local/bin/python3.11 CPython==3.11.3
server-1  | 
server-1  | No interpreter compatible with the requested constraints was found:
server-1  | 
server-1  |   Failed to resolve requirements from PEX environment @ /bin/prefect.
server-1  |   Needed cp311-cp311-manylinux_2_28_x86_64 compatible dependencies for:
server-1  |    1: pendulum<3.0.0,>=2.1.2
server-1  |       Required by:
server-1  |         prefect 2.14.2
server-1  |       But this pex had no ProjectName(raw='pendulum', normalized='pendulum') distributions.
Can anyone point me in the right direction of why it would work in one scenario and not in the other? I feel like the Pendulum dependency should fail in either case. Based on what I can tell from the following issue Pendulum appears to have an issue with its wheels for 3.11.3. How would I work around this with Pants?
✅ 1
I've explicitly added the Pendulum ^2.1.2 dependency and created an override in my 
python_requirements
file to specify that Prefect depends on it. Using 
pants dependencies --transitive
on the PEX lists pendulum as a dependency.
Hmm, I guess this could be related to https://github.com/pantsbuild/pants/issues/12930
h
Re "If I package this PEX and run it from the dist folder the server starts as expected." - on linux? macos?
I don't think #12930 is related, that is a build time dep issue
And is about build time for distributions, not for pexes
A .pex file is a zip file, so you can use 
zip -t
to look at its manifest and see what wheels are embedded in it.
a
Sorry. The image I'm trying to run it on is based on debian slim-buster, whereas the local machine is Ubuntu 22.04. Looking at the manifest I see that pendulum-2.1.2-cp311-cp311-manylinux_2_35_x86_64.whl is included. Unfortunately even with PEX_VERBOSE set to 9 the output of supported tags for me is truncated.
I'm a bit out of my depth here, but I attempted to set 
platforms
on the PEX to platforms=["linux_x86_64-cp-3.11.3-cp3.11.3"]. This results in:
Copy code
stderr:
No pre-built wheel was available for pendulum 2.1.2.
Successfully built the wheel pendulum-2.1.2-cp39-cp39-manylinux_2_35_x86_64.whl from the sdist pendulum-2.1.2.tar.gz but it is not compatible with the requested foreign target abbreviated platform cp311-cp311cp3.11.3-linux_x86_64.
You'll need to build a wheel from pendulum-2.1.2.tar.gz on the foreign target platform and make it available to Pex via a `--find-links` repo or a custom `--index`.
h
Don't use platforms, use complete_platforms
platforms are about to be deprecated, they don't always work (and you've found one canonical example of that)
👍 1
manylinux version mismatch
a
I used 
pex3 interpreter inspect --markers --tags
to generate a platforms file for me, but get an error saying 
Invalid environment entry provided: pex.pep_508.MarkerEnvironment() argument after ** must be a mapping, not str
. This seems strange. It would be beneficial to have an example file present in the documentation for complete_platforms.
Fixed itself, somehow... Perhaps I had a trailing comma or something. However, when I provide a complete_platforms file Pants insists on picking the wrong interpreter, and falls back to 3.9.16. I've grepped the entire monorepo and there is no reference to 3.9.16 anywhere that should even remotely touch this PEX.
As an example, Benjy:
Copy code
{
  "compatible_tags": [
    "cp311-cp311-manylinux_2_35_x86_64",
    "cp311-cp311-manylinux_2_34_x86_64",
    "cp311-cp311-manylinux_2_33_x86_64",
    ...
  ],
  "marker_environment": {
    "implementation_name": "cpython",
    "implementation_version": "3.11.3",
    "os_name": "posix",
    "platform_machine": "x86_64",
    "platform_python_implementation": "CPython",
    "platform_release": "6.2.0-35-generic",
    "platform_system": "Linux",
    "platform_version": "#35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct  6 10:23:26 UTC 2",
    "python_full_version": "3.11.3",
    "python_version": "3.11",
    "sys_platform": "linux"
  }
}
With this as the platform-file and interpreter constraints globally limited to 3.11.3 OR 3.11.4 I still get the following message:
Copy code
stderr:
No pre-built wheel was available for pendulum 2.1.2.
Successfully built the wheel pendulum-2.1.2-cp39-cp39-manylinux_2_35_x86_64.whl from the sdist pendulum-2.1.2.tar.gz but it is not compatible with the requested foreign target complete platform cp311-cp311-manylinux_2_35_x86_64.
You'll need to build a wheel from pendulum-2.1.2.tar.gz on the foreign target platform and make it available to Pex via a `--find-links` repo or a custom `--index`
I read this as if Pants is trying to use 3.9.X as its intepreter when building the pex?
e
The details aren't fully lining up for me from this thread, but to perhaps shed light on the root issue: Looking at this example complete platform: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/awslambda/python/complete_platform_3.11-x86_64.json you might notice a descending set of 
_2_X_
tags. Those represent the glibc version installed on the machine (
2.26
) and lower. In other words this describes a CPython 3.11 interpreter installed on a machine with glibc 2.26 that is compatible with all platform specific wheels that link glibc 2.26 or older. Now you appear to be using two Linux systems that only share a kernel (this is how containers work). One system has glibc 2.25 installed, one 2.28. If you build a platform specific wheel on the 2.28 system, the 2.25 system will not be able to use it. The other direction should work though (and this is where I can't make the details line up from the discussion above). As to your Python 3.9 complaint, that's a Pants issue, but also a bit of a red herring. What you're seeing there is Pex doing a last ditch build and pray hail mary. If you get lucky, great. If not, you should buckle down and do the obvious thing, build native software on the actual platform it's intended to run on.
👍 1
a
Thanks for the explanation about the Hail Mary. That makes a lot of sense. And yes, as you say I'm building the project on a platform that uses glibc 2.35:
Copy code
ldd --version
ldd (Ubuntu GLIBC 2.35-0ubuntu3.4) 2.35
When we exec into the actual container running the software we find the following:
Copy code
root@890c125fec46:/srv# ldd --version
ldd (Debian GLIBC 2.28-10+deb10u2) 2.28
That's unfortunate. I'll see if I can upgrade the glibc version in the container.
I guess I can also just build the wheel directly in a container and upload it to our internal package index. That's probably cleaner and simpler. Do I understand correctly that the reason I'm first hitting this issue now is that all of my other packages apparently come with pre-built wheels for 2_28?
And perhaps also why this was working when we did Poetry install directly in the containers previously, since this would obivously build the 2_28 wheel!
I guess I can also just build the wheel directly in a container and upload it to our internal package index.
Just did this with great success 🙂
e
@acoustic-library-86413 I think you've basically got it, but not the details part I warned of. A wheel built linking to glibc 2.25 should in fact work on a CPython interpreter with access to glibc 2.28. That interpreter will have a most-specific tag including 
_2_28_
and then consecutive lower glibc versions through 
_2_25_
. So I've given you an obvious failure case: build against 2.28 does not work against 2.25, but, IIUC, your situation is the opposite.
a
No, I think it's correct. The host version is 35 not 25. So I'm building the wheel on 35 and trying to use it in 28, which shouldn't work?
e
Aha - yup. My brain saw 25. Great.
👍 1
5 Views
Open in Slack
PreviousNext
Powered by