https://pantsbuild.org/ logo
Powered by
b

bland-father-19717

01/30/2022, 6:38 AM
I had a problem when I used a pex file to build a docker image. The execution env of pants is 
macos
and the env of the docker image is 
linux
. I think it’s probably due to the difference between these environments, but I haven’t found a solution yet.
Copy code
pex.environment.ResolveError: A distribution for numpy could not be resolved in this environment.Found 1 distribution for numpy that do not apply:
1.) The wheel tags for numpy 1.22.1 are cp38-cp38-macosx_10_9_x86_64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 12)))):
cp38-cp38-manylinux_2_31_x86_64
cp38-cp38-manylinux_2_30_x86_64
cp38-cp38-manylinux_2_29_x86_64
cp38-cp38-manylinux_2_28_x86_64
cp38-cp38-manylinux_2_27_x86_64
...
Copy code
pex_binary(
    name="bin",
    entry_point="main.py",
    dependencies=[
        ":lib",
    ],
    platforms=["linux_x86_64-cp-38-cp38", "current"]
)
I tried the above settings in the BUILD file, but the following error was observed.
Copy code
ERROR: Could not find a version that satisfies the requirement numpy<2.0.0,>=1.22.1
ERROR: No matching distribution found for numpy<2.0.0,>=1.22.1
pid 28121 -> /Users/xxx/.cache/pants/named_caches/pex_root/venvs/7f518a293766277276f42aa896ce105b056e5634/8877b0e2523d139b27e4bbf76b63fbbff4348ce3/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/xxx/.cache/pants/named_caches/pex_root --log /private/var/folders/4b/_8k28n_s6fvcpl8ncyvxjvkcthgjbs/T/process-executionJfqWzB/.tmp/tmpzbxodwib/pip.log download --dest /private/var/folders/4b/_8k28n_s6fvcpl8ncyvxjvkcthgjbs/T/process-executionJfqWzB/.tmp/tmp3omguv22/Users.xxx..pyenv.versions.3.6.8.bin.python3.6 numpy<2.0.0,>=1.22.1 protobuf>=3.12.1 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
f

fast-nail-55400

01/30/2022, 12:00 PM
Known issue. Pants doesn’t currently have the ability to choose a different target platform than the execution platform. Others have seen this issue in the form of Pants building a Go binary for macOS but Docker is using Linux. https://pantsbuild.slack.com/archives/C046T6T9U/p1637606432111100?thread_ts=1637606432.111100
b

bland-father-19717

01/30/2022, 1:44 PM
Thank you for your reply, @fast-nail-55400. https://pantsbuild.slack.com/archives/C046T6T9U/p1637606942112000?thread_ts=1637606432.111100&amp;cid=C046T6T9U
We support platforms for PEX binaries already!
In the discussion you shared, it looks like the 
platforms
argument of 
pex_binary
solves such problems, but doesn’t it?
h

hundreds-father-404

01/30/2022, 2:27 PM
Yeah to clarify, it sounds like you're using a 
pex_binary
target? Can you share the BUILD file for it?
b

bland-father-19717

01/30/2022, 2:35 PM
I created and added source code that will be reproduced in the repository. # 
./pants run src/python/main/main_a/Dockerfile
https://github.com/peachanG/pants-sample/tree/0fce9a8617e3e2e578947a66c4d182594048a12e
Copy code
00:08:06.30 [INFO] Canceled: Building docker image docker:latest
00:08:08.11 [INFO] Completed: Building docker image docker:latest
Traceback (most recent call last):
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/environment.py", line 322, in activate
    self._activated_dists = self._activate()
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/environment.py", line 673, in _activate
    resolved = self.resolve()
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/environment.py", line 506, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/environment.py", line 530, in resolve_dists
    for qualified_req_or_not_found in self._root_requirements_iter(reqs):
  File "/root/.pex/unzipped_pexes/605d10852c333aa60c12bd3bcca26ad08b1b153b/.bootstrap/pex/environment.py", line 469, in _root_requirements_iter
    raise ResolveError(message)
pex.environment.ResolveError: A distribution for numpy could not be resolved in this environment.Found 1 distribution for numpy that do not apply:
1.) The wheel tags for numpy 1.22.1 are cp38-cp38-macosx_10_9_x86_64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 12)))):
cp38-cp38-manylinux_2_31_x86_64
cp38-cp38-manylinux_2_30_x86_64
cp38-cp38-manylinux_2_29_x86_64
# 
./pants run src/python/main/main_a/Dockerfile
https://github.com/peachanG/pants-sample/tree/de3279823175b5cdc9d9f9276b3a387996afa182 Sorry, I couldn’t reproduce it. How can I solve this problem?
Copy code
00:12:01.34 [INFO] Canceled: Building docker image docker:latest
00:12:03.30 [INFO] Completed: Building docker image docker:latest
/usr/bin/env: 'python3.6': No such file or directory
h

happy-kitchen-89482

01/30/2022, 4:12 PM
To clarify, Pants can build multiplatform PEX files, but only if there are wheels available for all transitive requirements for all platforms other than the one the build is running on.
If it has to build native code from an sdist, then, just like pip, poetry and any other tool, it can only do that for the local platform.
Let me look at your specific errors and see what's going on
Ah, thanks for the reproduction repository, that is super helpful!
@bland-father-19717 since Slack is ephemeral, do you mind opening an issue for this at https://github.com/pantsbuild/pants/issues/new/choose ? Then we can track the solution properly, and make it available in the long-term for other users. Thanks!
e

enough-analyst-54434

01/30/2022, 5:05 PM
The platform info is not being passed in that Pex command (starting with --disable-pip-version-check and ending with --timeout 15). So Pants is ignoring the pex_binary platforms field in this scenario somehow.
If you read the --dest argument, that implies the resolve is happening for the local pyenv Python 3.6.8 interpreter. That in turn suggests maybe "current" is being (over?) interpreted by Pants or Pex and a resolve for the local Mac is being attempted in addition to the Linux platform resolve. It's the Mac resolve that is failing.
If you switch from "current" to the platform that matches the Macs in your dev fleet (and matches numpy available wheels), things should work.
b

bland-father-19717

01/31/2022, 1:39 AM
@happy-kitchen-89482 Thank you! I created an issue.https://github.com/pantsbuild/pants/issues/14317
@enough-analyst-54434 I commented on the issue, but let me thank you here as well. Thanks for the help ! I’ll close the issue.
💯 1
❤️ 1
h

happy-kitchen-89482

01/31/2022, 2:59 AM
Glad we were able to help!
❤️ 1
👍 1
8 Views
Powered by