https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, I hope this is the right place to ask: toda...
# general
a
Hello, I hope this is the right place to ask: today I was setting up our (new, experimental) Pants project on someone else's machine (mac) for the first time, and encountered this surprising error. We build some PEX files in a 
docker_environment
for linux, and we resolved this issue by installing 
go
on their machine. Is it expected that pants depends on go to be able to do this?
Copy code
16:58:44.50 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal
  in Resolve transitive targets
  in Resolve direct dependencies of target - im-ingestion:docker

ProcessExecutionFailure: Process 'Building dockerfile_parser.pex from dockerfile-parser_default.lock' failed with exit code 1.
stdout:

stderr:
pid 27139 -> /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/bin/python -sE /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /Users/x/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /Users/x/.cache/pants/named_caches/pex_root/built_wheels/sdists/dockerfile-3.2.0.tar.gz/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/cp311-cp311-macosx_11_0_arm64.fe97d14875ad4e3bb9c900ebabe3432d /Users/x/.cache/pants/named_caches/pex_root/downloads/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/dockerfile-3.2.0.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  ERROR: Command errored out with exit status 1:
   command: /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/setup.py'"'"'; __file__='"'"'/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-wheel-115d_a4i
       cwd: /Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/
  Complete output (5 lines):
  running bdist_wheel
  running build
  running build_ext
  $ GOPATH=/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpsynsnw6g go get -d
  error: [Errno 2] No such file or directory: 'go'
  ----------------------------------------
  ERROR: Failed building wheel for dockerfile
ERROR: Failed to build one or more wheels
c
The dockerfile parser library we use depends on go for building, so looks like there was no wheel available for it in this case..? so failed when attempting to build from sdist, I guess
so in short, no this is not expected
a
Looking at this: https://pypi.org/project/dockerfile/
This project uses setuptools-golang when built from source. To build from source you'll need a go compiler.
Thanks @curved-television-6568 I suppose I need to find out under which circumstances a machine would need to build from source; until then I might need to make sure my team have go to develop in the project
c
looking at the logs, seems you are using py3.11? might be the reason here…
Copy code
dockerfile-3.2.0.tar.gz/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/cp311-cp311-macosx_11_0_arm64.fe97d14875ad4e3bb9c900ebabe3432d
a
Ah, indeed we are
c
tools may run on other versions though, so if you have an older version of python available (not the system one) making pants use that here could help
might be that the default constraint there is a bit too liberal
e
Python 3.11 is not the issue, these are abi3 wheels. But MacOS 11 is the issue. The arm wheel is MacOS 12; so it will only work with MacOS 12+ IIUC.
@curved-television-6568 and @ancient-lawyer-12557 to clarify: The wheel that is the candidate here is 
dockerfile-3.2.0-cp38-abi3-macosx_12_0_arm64.whl
That works for CPython 3.8+ due to using the limited abi denoted by the abi3 tag.
👍 1
But the OS is macosx_12_0 and that's where you get burned.
c
Ah..
e
I mean, you know where I think you actually got burned here!
a
Perfect thank you, it's reasonable for my team to upgrade their macs - much more so than having to install go
e
I'd try 1 upgrade 1st. Its generally unwise to believe me.
a
Thanks for helping me understand this @curved-television-6568 @enough-analyst-54434
e
I don't use a Mac so can't test what I'm saying is true.
c
I do but are generally more confused when it comes to python package distributions
e
The "simplest" way to answer this sort of thing is to grep the list of tags your interpreter supports because the algorithm is set intersection and nothing more. For example:
Copy code
$ pex3 interpreter inspect --python /usr/bin/python3.9 --tags -i2 | grep cp36-abi3-manylinux2010_x86_64
    "cp36-abi3-manylinux2010_x86_64",
Of course I used a different tag here - I get no hits for the mac tag:
Copy code
$ pex3 interpreter inspect --python /usr/bin/python3.9 --tags -i2 | grep cp38-abi3-macosx_12_0_arm64
$
The hard work of calculating that list is standardized and encapsulated by PyPA in their packaging distribution. All resolvers, Pip, Poetry, Pex, Pdm, etc use that.
🙌 1
c
thanks, this will go into my notes in hope I’ll remeber to look it up next time I need it.
My mac have:
Copy code
pex3 interpreter inspect --python `which python` --tags -i2 | grep "cp38-abi3" | grep arm64
    "cp38-abi3-macosx_13_0_arm64",
    "cp38-abi3-macosx_12_0_arm64",
    "cp38-abi3-macosx_11_0_arm64",
e
Are you MacOS 11?
c
which helps answer some uncertanies I had wrgt Mac OS versions here.
no, 13
e
Hrm, that scuttles my claim then.
c
so guess there’s some backwards compat
e
So, @ancient-lawyer-12557 definitely test. I appear to be a liar.
😅 1
I'm not lying about the algorithm - it is in fact set intersection.
c
No, if they are on 11, then they would likely not support 12 and 13
e
I'm wrong about mac compat apparently though.
Oh - right back compat.
My brain was reversing.
c
yea
e
So, @curved-television-6568 there is only 1 other element to a resolve match, the 
Requires-Python
value in the wheel METADATA file. That's generally displayed on PyPI under project details.
c
Copy code
Requires: Python >=3.6.1
e
Tag intersection + Requires-Python match and you're done.
c
in this case
e
Tags earlier in the list have higher precedence if there are more than 1 matching wheels.
c
good to know
e
Requires: Python >=3.6.1
that is ~different and old
It's 
Requires-Python: >=3.6.1
The 
Requires
thing is long past common use.
c
it’s what’s presented on pypi.. no release for quite some time tho
e
Yeah.
c
only no new release with that yet
e
Requires-Python was added here https://peps.python.org/pep-0345/ in 2009. Requires was added here https://peps.python.org/pep-0314/#requires-multiple-use in 2003. It has this gem that basically explains everything about why Python packaging is a bit of a mess: "... There’s no canonical list of what strings should be used; the Python community is left to choose its own standards."
So that 
Python
in your example is an example of one of those yolo strings not codified anywhere that are supported only by happenstance of the history of tools like setuptools and pip.
c
oh boy
2 Views
Open in Slack
PreviousNext
Powered by