https://pantsbuild.org/ logo
Join Slack
Powered by
Hi All, I also recently started using pantsbuild t...
# general
r
Hi All, I also recently started using pantsbuild to setup a new python monorepo. Most of these projects in monorepo will end up on ECS, so one of the goals is to build a docker image with pants. One of these projects uses Flask which has dependency on Markupsafe. So now, when i run pex_binary target on my local(mac), using pants run just fine. But if I copy pex binary to a Dockerfile it gives the error for no compatible package being available for Markupsafe.
Here is the error that I am getting
Copy code
10:33:21.13 [INFO] Completed: Building docker image docker:latest
10:33:21.76 [INFO] Completed: Building docker image docker:latest
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Examined the following interpreters:
1.) /usr/local/bin/python3.11 CPython==3.11.7

No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/636b1a85f473e02d60a7e56c42b3b81a8a15de61.
  Needed cp311-cp311-manylinux_2_36_x86_64 compatible dependencies for:
   1: MarkupSafe>=2.1.1
      Required by:
        Werkzeug 3.0.1
      But this pex had no ProjectName(raw='MarkupSafe', normalized='markupsafe') distributions.
   2: MarkupSafe>=2.0
      Required by:
        Jinja2 3.1.3
      But this pex had no ProjectName(raw='MarkupSafe', normalized='markupsafe') distributions.
Here is my dockerfile
Copy code
FROM python:3.11-slim

COPY src.python.rlg/bin.pex /app.pex
ENTRYPOINT ["/usr/local/bin/python3.11","/app.pex"]
c
hm, it looks like there's also an error about incompatible Interpreter Constraints. Can you check that the value in pants.toml for 
[python].interpreter_constraints
includes the version in the docker image (CPython==3.11.7), or share that?
r
Here is my pants.toml
Copy code
[GLOBAL]
pants_version = "2.19.0"

# read more: <https://www.pantsbuild.org/2.19/docs/using-pants/key-concepts/backends>
backend_packages = [
  "pants.backend.build_files.fmt.black",
  "pants.backend.shell",
  "pants.backend.python",  # core python language support
  "pants.backend.python.lint.black", # python black linter
  "pants.backend.python.lint.isort", # python isort linter
  "pants.backend.docker"
]

[anonymous-telemetry]
enabled = false
repo_id = "00000000-0000-0000-0000-000000000000"


# for details: <https://www.pantsbuild.org/2.19/docs/using-pants/key-concepts/source-roots>
[source]
root_patterns = [
  '/src/python',
  '/test/python',
  # if there are any other roots or other languages added later on, need to be added here
]

[python]
# lock to a single minor version.
interpreter_constraints = ["==3.11.*"]

# Enable the "resolves" mechanism, which turns on lockfiles for user code. See
# <https://www.pantsbuild.org/docs/python-third-party-dependencies>. This also adds the
# `generate-lockfiles` goal for Pants to generate the lockfile for you.
enable_resolves = true

resolves = { python-default = "python-default.lock"}

[python-bootstrap]
# We search for interpreters both on the $PATH and in the `$(pyenv root)/versions` folder.
#  If you're using macOS, you may want to leave off the <PATH> entry to avoid using the
#  problematic system Pythons. See
#  <https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path>.
search_path = ["<PATH>", "<PYENV>"]

[python-infer]
# 2.17 is transitioning to a new, faster parser for dependency inference:
use_rust_parser = true
Basically, for markupsafe cp311-cp311-manylinux_2_*17*_x86_64 whl is available while its looking for cp311-cp311-manylinux_2_*36*_x86_64 .
Here is a except from python-default.lock if that helps
Copy code
{
  "artifacts": [
    {
      "algorithm": "sha256",
      "hash": "3a57fdd7ce31c7ff06cdfbf31dafa96cc533c21e443d57f5b1ecc6cdc668ec7f",
      "url": "<https://files.pythonhosted.org/packages/f8/81/56e567126a2c2bc2684d6391332e357589a96a76cb9f8e5052d85cb0ead8/MarkupSafe-2.1.5-cp311-cp311-musllinux_1_1_x86_64.whl>"
    },
    {
      "algorithm": "sha256",
      "hash": "7502934a33b54030eaf1194c21c692a534196063db72176b0c4028e140f8f32c",
      "url": "<https://files.pythonhosted.org/packages/0c/40/2e73e7d532d030b1e41180807a80d564eda53babaf04d65e15c1cf897e40/MarkupSafe-2.1.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl>"
    },
    {
      "algorithm": "sha256",
      "hash": "629ddd2ca402ae6dbedfceeba9c46d5f7b2a61d9749597d4307f943ef198fc1f",
      "url": "<https://files.pythonhosted.org/packages/11/e7/291e55127bb2ae67c64d66cef01432b5933859dfb7d6949daa721b89d0b3/MarkupSafe-2.1.5-cp311-cp311-macosx_10_9_universal2.whl>"
    },
    {
      "algorithm": "sha256",
      "hash": "0e397ac966fdf721b2c528cf028494e86172b4feba51d65f81ffd65c63798f3f",
      "url": "<https://files.pythonhosted.org/packages/18/46/5dca760547e8c59c5311b332f70605d24c99d1303dd9a6e1fc3ed0d73561/MarkupSafe-2.1.5-cp311-cp311-musllinux_1_1_aarch64.whl>"
    },
    {
      "algorithm": "sha256",
      "hash": "6ec585f69cec0aa07d945b20805be741395e28ac1627333b1c5b0105962ffced",
      "url": "<https://files.pythonhosted.org/packages/1c/cf/35fe557e53709e93feb65575c93927942087e9b97213eabc3fe9d5b25a55/MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl>"
    },
    {
      "algorithm": "sha256",
      "hash": "5b7b716f97b52c5a14bffdf688f971b2d5ef4029127f1ad7a513973cfd818df2",
      "url": "<https://files.pythonhosted.org/packages/6b/cb/aed7a284c00dfa7c0682d14df85ad4955a350a21d2e3b06d8240497359bf/MarkupSafe-2.1.5-cp311-cp311-macosx_10_9_x86_64.whl>"
    },
    {
      "algorithm": "sha256",
      "hash": "c061bb86a71b42465156a3ee7bd58c8c2ceacdbeb95d05a99893e08b8467359a",
      "url": "<https://files.pythonhosted.org/packages/6d/c5/27febe918ac36397919cd4a67d5579cbbfa8da027fa1238af6285bb368ea/MarkupSafe-2.1.5-cp311-cp311-musllinux_1_1_i686.whl>"
    },
    {
      "algorithm": "sha256",
      "hash": "d283d37a890ba4c1ae73ffadf8046435c76e7bc2247bbb63c00bd1a709c6544b",
      "url": "<https://files.pythonhosted.org/packages/87/5b/aae44c6655f3801e81aa3eef09dbbf012431987ba564d7231722f68df02d/MarkupSafe-2.1.5.tar.gz>"
    },
    {
      "algorithm": "sha256",
      "hash": "b91c037585eba9095565a3556f611e3cbfaa42ca1e865f7b8015fe5c7336d5a5",
      "url": "<https://files.pythonhosted.org/packages/97/18/c30da5e7a0e7f4603abfc6780574131221d9148f323752c2755d48abad30/MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>"
    }
  ],
  "project_name": "markupsafe",
  "requires_dists": [],
  "requires_python": ">=3.7",
  "version": "2.1.5"
}
However if I install markup-safe directly inside the container using pip it doesn't complain and installs the same 'cp311-cp311-manylinux_2_*17*_x86_64' whl.
c
I'm not sure, I'm not really familiar with manylinux. the whl name spec is of the form 
manylinux_${GLIBCMAJOR}_${GLIBCMINOR}_${ARCH}
. So it looks like the docker container has a different GLIBC version than the one in the lockfile. I think that could happen if you generated the lockfile on an older OS, since python3.11-slim is Debian Bookworm based (which is using glibc 2.36, which is why it's requesting that type of whl)
r
yes the lockfile was generated on different OS, mac os to be specific.
i also suspected this and in order to mitigate this, I tried to setup a docker_environent for building the project but failed
as I was not exactly sure what to provide under 
[environments-preview.names]
in pants.toml
c
oh, it looks like markupsafe only has manylinux_2_17 whls
r
yups thats also true
but that did not prevent pip from installing the same wheel under the same container in which pex was failing
c
I wonder if it's installing it as an sdist
no, that's not it, pip3 downloads the older whl just fine. I wonder if it's something about how we're using PEX
r
yes, this is what i get by running pip install inside container
Copy code
root@cdc706217258:/# pip install --no-cache markupsafe
Collecting markupsafe
  Obtaining dependency information for markupsafe from <https://files.pythonhosted.org/packages/97/18/c30da5e7a0e7f4603abfc6780574131221d9148f323752c2755d48abad30/MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata>
  Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Installing collected packages: markupsafe
Successfully installed markupsafe-2.1.5
Yes probably, I am very new to pex, started using just a day before yesterday
This is what I am passing to pex_binary target
Copy code
pex_binary(
    name="bin",
    layout="packed",
    include_tools=True,
    execution_mode="venv",
)
and dockerfile where I am executing this is already shared
c
I can't reproduce the problem on a new repository. I'm not sure if the host OS might be playing a factor, I'm on Ubuntu 22.04 and I think you're on a Mac. Some of the other folks here use Mac, although I think the main issue was package availability for the ARM64 architecture, which isn't happening here. I think we've reached the limits of my knowledge on this. There are folks who know more about PEX and platform stuff and Macs, hopefully they'll chime in.
r
no worries. Thanks a lot @careful-address-89803 for looking into this. In my view a more generic approach to solve this problem is using remote build using docker_environment. I tried to do this but was unable to so because I was not exactly sure what to provide under 
[environments-preview.names]
in pants.toml
Can you provide some help / doc / example here so that I can run my builds using a docker container. So there these kind of issues shouldn't popup at least hypothetically.
h
https://www.pantsbuild.org/2.19/docs/using-pants/environments should give you enough to get Pants using environments
Click on the 
BUILD
and 
pants.toml
tabs in the example to see what you need in each
But to test this initially you can simply clone your repo into a docker container with that image and run 
pants generate-lockfiles
manually there, just to ensure that you get the right thing.
r
Hi @happy-kitchen-89482 I have gone through the docs and tried that thing already. It was giving me an error as if it is looking for a local path
Copy code
12:15:31.14 [INFO] waiting for pantsd to start...
12:15:31.96 [INFO] pantsd started
12:15:32.02 [ERROR] 1 Exception encountered:

ResolveError: The address //:local_busybox from the option [environments-preview].names does not exist.

The target name ':local_busybox' is not defined in the directory . Did you mean one of these target names?

  *


Use --print-stacktrace for more error details and/or -ldebug for more logs.
See <https://www.pantsbuild.org/v2.19/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.19/docs/getting-help>
h
Pants is expecting a target named 
local_busybox
in the root-level BUILD file. Do you have such a target?
r
No I don't. What should be the type of target with this name at root level?
This is the detail I see missing in the doc or at least I was not able to understand that from docs.
h
As I mentioned earlier, click on the "BUILD" tab in the example to see what the BUILD file should contain
Screenshot 2024-02-05 at 8.18.58 AM.png
r
Yes I saw that but I added that in a nested build file
sorry for that mistake
h
Try to follow the example exactly, just changing what you need for your case (for example, instead of the busybox image you probably want to use your relevant image)
r
Cool, let me try that and then will get back to you.
Thanks @happy-kitchen-89482 It works now. Sorry for my stupid mistake. Yesterday I spent my whole day doing various permutation and combinations to make this work. But still It's not working as per my expectations: Here are my expectation, let me know if they are valid or not. 1. I expected that once I configure a build environment, all the pants command including / specially 
generate-lockfiles
will run inside that environment. 2. I configure one of pex_binary target with that newly created docker environment and assumed that pex _binary will now run under that environment. But it gives me this warning:
Copy code
19:04:55.46 [WARN] The `run` goal was called with target `src/python/test:bin`, which specifies the environment `local_python3`, which is a `docker_environment`. The `run` goal only runs in the local environment. You may experience unexpected behavior.
I am also using this option to set the new docker environment my default environment for everything in my root build file:
Copy code
__defaults__(all=dict(environment="local_python3"))
9 Views
Open in Slack
PreviousNext
Powered by