https://pantsbuild.org/ logo
Join Slack
Powered by
Hitting an issue trying to set up a docker build o...
# general
l
Hitting an issue trying to set up a docker build on Mac OS ARM (pants 
2.16.0.dev3
):
Copy code
stderr:
There was 1 error downloading required artifacts:
1. dockerfile 3.2 from <https://files.pythonhosted.org/packages/6e/75/1395fac29bd2dcccb51ffae11ba9a19fe159eabf9596bca4cf49c1a2b191/dockerfile-3.2.0.tar.gz>
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
This seems to be coming from bootstrapping the docker backend. Where can I find the logs of this process to keep troubleshooting? I’ve got 
--keep-sandboxes=on_failure
set.
h
Hey, did you find a solution here? Someone is reporting something similar on a different requirement, after upgrading to 2.14
I'm wondering if this is an artifact of upgrading Pants (it shouldn't happen, so it's still a bug if so, but wondering if that is the immediate cause)
Can you run with 
-ldebug
and see if that gives more info?
@enough-analyst-54434, where can we find the pip logs in this case?
e
You need to pass 
--preserve-pip-download-log
to Pex. It will print the path to stderr and retain it.
h
Hmm, Pants does not do this today, so ... damn
l
No solution yet here. Will try to dig into it more today and see if I can get some more helpful logs.
h
At least run with 
-ldebug
. And I wonder if this has to do with Python versions/interpreter constraints
E.g., it's trying to isntall 
dockerfile
on an unsuitable interpreter somehow
l
Here’s what I think is the most relevant excerpt from running with 
-ldebug
and `--print-stacktrace`: https://gist.github.com/jcrumb-u21/7163d47b523c4753c9b8f13a80582ce1
If I have time today I’ll poke at the pants code and add the flag John mentioned to the pex call.
e
Right on the tin it says you need go tooling to build the sdist: https://pypi.org/project/dockerfile/ Do you have that?
l
I do
e
Ok ... does that require env vars to work that are not configured to allow leaking through to Pants actions? GO_ROOTS, etc? It's been a while for me.
l
I don’t believe so, but let me double check I’m able to install an sdist of that package with the same interpreter with 
env -i.
e
I'm pretty sure this is just down to getting a macOS aarch64 wheel built for the dockerfile sdist; and that's trickier in Pants hermeticity shrink wrap.
@#$%! macOS arm
h
Seriously
l
pip install dockerfile
will pull a wheel on macOS ARM, and 
pip install --no-binary :all: dockerfile
worked for me as well. Trying it now under 
env -i
to see if 
GO_ROOT
is the problem.
h
Nothing but trouble from M1s!
e
Whoa - seriously? There exists a pre-built dockerfile M1 wheel? Looking ...
l
dockerfile-3.2.0-cp38-abi3-macosx_12_0_arm64.whl
Let me try with python 3.7 since that’s what pants is picking
e
Yeah - that's the issue
You need 3.8 to get lucky.
since its abi3 though ... hrm, let me see if there is a 
packaging
bug re this.
And @late-keyboard-89314 your on macOS >= 12 ?
l
I did manage an sdist install of 
dockerfile
with the same interpreter pants selected.
Yeah, running ventura
e
Ok, well your idea to use 
--keep-sandboxes=on_failure
was correct. You need to poke at the 
__run.sh
script there and get your hands dirty finding out what's different. Editing that script will allow insertion of things like 
--preserve-pip-download-log
, etc.
Although, wait a sec. Why are you using Pants 2.16.x? This:
Copy code
10:49:30.95 [DEBUG] spawned local process as Some(75453) for Process { argv: ["/Users/jay/.pyenv/versions/3.7.11/bin/python3.7", "./pex", "--tmpdir", ".tmp", "--jobs", "6", "--pip-version", "20.3.4-patched", ...
Says you're not using it for the 
[python] pip_version = "22.3"
feature.
Which ... would be good to try to rule out old 
packaging
, etc.
And https://gist.github.com/jcrumb-u21/7163d47b523c4753c9b8f13a80582ce1#file-pants-debug-L13 definitely does not have 
GO*
in it's env; so ignoring why the abi3 wheel is not getting selected, that explains why the sdist fails to build I think.
But! ... remembering my abi3 better - I think abi3 says any Python 3 >= the one the wheel was built with, which 3.7 is not. Let me check that.
Yeah, an abi3 wheel is compatible with the version of Python it was compiled with and subsequent; so that macOS arm wheel on PyPI is only good for >=3.8
If you really need to be using Python 3.7, you'll need to confront the sdist build / go tooling requirement thicket.
On last thing - when debugging using 
__run.sh
keep in mind that it is not hermetic like the actual Pants process execution is. It will leak through your environment! So if the 
__run.sh
command works, try then blanking out select env vars and seeing if it still works, although this may be complicated by successful caching of wheel builds in 
~/.cache/named_caches/pex_root/
l
Why are you using Pants 2.16.x?
I need a bugfix that hasn’t hit the 2.14/2.15 release branches yet.
If you really need to be using Python 3.7
I don’t need to be using Python 3.7 at all, it just happens to be on my system and is what pants is selecting when installing this tool. This project actually doesn’t use Python at all at the moment, and the backend isn’t enabled:
Copy code
[GLOBAL]
pants_version = "2.16.0.dev3"
backend_packages = [
  "pants.backend.experimental.go",
  "pants.backend.docker",
]
It would be perfectly fine in my case for pants to select python 3.8 and use the wheel for this package.
e
Aha, excellent. Just a sec for steps.
You want the following in `pants.toml`:
Copy code
[dockerfile-parser]
interpreter_constraints = ["CPython>=3.7,<4"]
lockfile = "anywhere/you/want/dockerfile-anyname-really.lock-but-any-extension-or-none
The 2 things to change there are: 1. The ICs: I presented the default, and that's what's killing you. You probably want to change 
<4
to at least `<3.12`but the tighter the better on the upper bound and clearly you want to bump the lower bound. 2. The lockfile path: as should be obvious Then run 
./pants generate-lockfiles --resolve dockerfile-parser
, then try to use Pants again.
As an aside to @happy-kitchen-89482 - this presents an interesting class of cases: Python tooling being used for not-Python backend... should lock to scie-pants shipped Python versions to avoid all this. In the current case that would have been 3.9 and there would have been no issue here.
Even in the current world, this IC should match Pants own and be 
>=3.7,<3.10
since the tool does not care about any user Python code, its just a tool to be used internally by rules.
l
Heyyy, that worked. Tightened the constraint to 
>=3.8
and it worked (presumably because it pulled the wheel). I have such a love-hate thing going with ARM Macs.
Thanks for digging into this with me!
e
You should direct the hate at Apple. The reason things are hard for Apple is they don't allow commodity rackable servers. It all rolls down hill from there. It is expensive in time and $$$ to get OSS projects working for mac arm.
And one way to direct that hate is not buy the damn things!
l
Yeah, I mean it’s a great platform from a performance/battery life perspective when things work but they certainly don’t make it easy for people to get things working.
e
They are simply not developer friendly
Only for developing for their own ecosystem of course.
h
Yet more evidence that we should not ship with loose default interpreter constraints, and force you to pick them explicitly for your repo (and subsequently for tools)
On my tombstone will be engraved "Died of CPython>=3.7,<4"
e
Heh.
Universal locks are a horrible idea. They let you not think too easily.
Open in Slack
PreviousNext
Powered by