https://pantsbuild.org/ logo
Powered by
#general
Title
# general
a

ancient-lawyer-12557

04/26/2023, 5:07 PM
Hello, I hope this is the right place to ask: today I was setting up our (new, experimental) Pants project on someone else's machine (mac) for the first time, and encountered this surprising error. We build some PEX files in a 
docker_environment
for linux, and we resolved this issue by installing 
go
on their machine. Is it expected that pants depends on go to be able to do this?
Copy code
16:58:44.50 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal
  in Resolve transitive targets
  in Resolve direct dependencies of target - im-ingestion:docker

ProcessExecutionFailure: Process 'Building dockerfile_parser.pex from dockerfile-parser_default.lock' failed with exit code 1.
stdout:

stderr:
pid 27139 -> /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/bin/python -sE /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /Users/x/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /Users/x/.cache/pants/named_caches/pex_root/built_wheels/sdists/dockerfile-3.2.0.tar.gz/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/cp311-cp311-macosx_11_0_arm64.fe97d14875ad4e3bb9c900ebabe3432d /Users/x/.cache/pants/named_caches/pex_root/downloads/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/dockerfile-3.2.0.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  ERROR: Command errored out with exit status 1:
   command: /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/setup.py'"'"'; __file__='"'"'/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-wheel-115d_a4i
       cwd: /Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/
  Complete output (5 lines):
  running bdist_wheel
  running build
  running build_ext
  $ GOPATH=/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpsynsnw6g go get -d
  error: [Errno 2] No such file or directory: 'go'
  ----------------------------------------
  ERROR: Failed building wheel for dockerfile
ERROR: Failed to build one or more wheels
c

curved-television-6568

04/26/2023, 5:09 PM
The dockerfile parser library we use depends on go for building, so looks like there was no wheel available for it in this case..? so failed when attempting to build from sdist, I guess
so in short, no this is not expected
a

ancient-lawyer-12557

04/26/2023, 5:11 PM
Looking at this: https://pypi.org/project/dockerfile/
This project uses setuptools-golang when built from source. To build from source you'll need a go compiler.
Thanks @curved-television-6568 I suppose I need to find out under which circumstances a machine would need to build from source; until then I might need to make sure my team have go to develop in the project
c

curved-television-6568

04/26/2023, 5:11 PM
looking at the logs, seems you are using py3.11? might be the reason here…
Copy code
dockerfile-3.2.0.tar.gz/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/cp311-cp311-macosx_11_0_arm64.fe97d14875ad4e3bb9c900ebabe3432d
a

ancient-lawyer-12557

04/26/2023, 5:13 PM
Ah, indeed we are
c

curved-television-6568

04/26/2023, 5:14 PM
tools may run on other versions though, so if you have an older version of python available (not the system one) making pants use that here could help
might be that the default constraint there is a bit too liberal
e

enough-analyst-54434

04/26/2023, 5:31 PM
Python 3.11 is not the issue, these are abi3 wheels. But MacOS 11 is the issue. The arm wheel is MacOS 12; so it will only work with MacOS 12+ IIUC.
@curved-television-6568 and @ancient-lawyer-12557 to clarify: The wheel that is the candidate here is 
dockerfile-3.2.0-cp38-abi3-macosx_12_0_arm64.whl
That works for CPython 3.8+ due to using the limited abi denoted by the abi3 tag.
👍 1
But the OS is macosx_12_0 and that's where you get burned.
c

curved-television-6568

04/26/2023, 5:34 PM
Ah..
e

enough-analyst-54434

04/26/2023, 5:35 PM
I mean, you know where I think you actually got burned here!
a

ancient-lawyer-12557

04/26/2023, 5:35 PM
Perfect thank you, it's reasonable for my team to upgrade their macs - much more so than having to install go
e

enough-analyst-54434

04/26/2023, 5:35 PM
I'd try 1 upgrade 1st. Its generally unwise to believe me.
a

ancient-lawyer-12557

04/26/2023, 5:35 PM
Thanks for helping me understand this @curved-television-6568 @enough-analyst-54434
e

enough-analyst-54434

04/26/2023, 5:35 PM
I don't use a Mac so can't test what I'm saying is true.
c

curved-television-6568

04/26/2023, 5:36 PM
I do but are generally more confused when it comes to python package distributions
e

enough-analyst-54434

04/26/2023, 5:40 PM
The "simplest" way to answer this sort of thing is to grep the list of tags your interpreter supports because the algorithm is set intersection and nothing more. For example:
Copy code
$ pex3 interpreter inspect --python /usr/bin/python3.9 --tags -i2 | grep cp36-abi3-manylinux2010_x86_64
    "cp36-abi3-manylinux2010_x86_64",
Of course I used a different tag here - I get no hits for the mac tag:
Copy code
$ pex3 interpreter inspect --python /usr/bin/python3.9 --tags -i2 | grep cp38-abi3-macosx_12_0_arm64
$
The hard work of calculating that list is standardized and encapsulated by PyPA in their packaging distribution. All resolvers, Pip, Poetry, Pex, Pdm, etc use that.
🙌 1
c

curved-television-6568

04/26/2023, 5:44 PM
thanks, this will go into my notes in hope I’ll remeber to look it up next time I need it.
My mac have:
Copy code
pex3 interpreter inspect --python `which python` --tags -i2 | grep "cp38-abi3" | grep arm64
    "cp38-abi3-macosx_13_0_arm64",
    "cp38-abi3-macosx_12_0_arm64",
    "cp38-abi3-macosx_11_0_arm64",
e

enough-analyst-54434

04/26/2023, 5:44 PM
Are you MacOS 11?
c

curved-television-6568

04/26/2023, 5:44 PM
which helps answer some uncertanies I had wrgt Mac OS versions here.
no, 13
e

enough-analyst-54434

04/26/2023, 5:44 PM
Hrm, that scuttles my claim then.
c

curved-television-6568

04/26/2023, 5:44 PM
so guess there’s some backwards compat
e

enough-analyst-54434

04/26/2023, 5:45 PM
So, @ancient-lawyer-12557 definitely test. I appear to be a liar.
😅 1
I'm not lying about the algorithm - it is in fact set intersection.
c

curved-television-6568

04/26/2023, 5:45 PM
No, if they are on 11, then they would likely not support 12 and 13
e

enough-analyst-54434

04/26/2023, 5:46 PM
I'm wrong about mac compat apparently though.
Oh - right back compat.
My brain was reversing.
c

curved-television-6568

04/26/2023, 5:46 PM
yea
e

enough-analyst-54434

04/26/2023, 5:47 PM
So, @curved-television-6568 there is only 1 other element to a resolve match, the 
Requires-Python
value in the wheel METADATA file. That's generally displayed on PyPI under project details.
c

curved-television-6568

04/26/2023, 5:48 PM
Copy code
Requires: Python >=3.6.1
e

enough-analyst-54434

04/26/2023, 5:48 PM
Tag intersection + Requires-Python match and you're done.
c

curved-television-6568

04/26/2023, 5:48 PM
in this case
e

enough-analyst-54434

04/26/2023, 5:48 PM
Tags earlier in the list have higher precedence if there are more than 1 matching wheels.
c

curved-television-6568

04/26/2023, 5:48 PM
good to know
e

enough-analyst-54434

04/26/2023, 5:48 PM
Requires: Python >=3.6.1
that is ~different and old
It's 
Requires-Python: >=3.6.1
The 
Requires
thing is long past common use.
c

curved-television-6568

04/26/2023, 5:49 PM
it’s what’s presented on pypi.. no release for quite some time tho
e

enough-analyst-54434

04/26/2023, 5:49 PM
Yeah.
c

curved-television-6568

04/26/2023, 5:50 PM
only no new release with that yet
e

enough-analyst-54434

04/26/2023, 5:54 PM
Requires-Python was added here https://peps.python.org/pep-0345/ in 2009. Requires was added here https://peps.python.org/pep-0314/#requires-multiple-use in 2003. It has this gem that basically explains everything about why Python packaging is a bit of a mess: "... There’s no canonical list of what strings should be used; the Python community is left to choose its own standards."
So that 
Python
in your example is an example of one of those yolo strings not codified anywhere that are supported only by happenstance of the history of tools like setuptools and pip.
c

curved-television-6568

04/26/2023, 5:59 PM
oh boy
2 Views
Powered by