https://pantsbuild.org/ logo
Join Slack
Powered by
Hey, I'm having some python/architecture issues wh...
# general
b
Hey, I'm having some python/architecture issues when running pants on an M1 machine. I'll post the stack trace in a thread
āœ… 1
Copy code
Traceback (most recent call last):
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/bin/pants", line 5, in <module>
    from pants.bin.pants_loader import main
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 19, in <module>
    from pants.bin.pants_runner import PantsRunner
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 11, in <module>
    from pants.base.exception_sink import ExceptionSink
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 16, in <module>
    import psutil
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/__init__.py", line 123, in <module>
    from . import _psosx as _psplatform
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psosx.py", line 14, in <module>
    from . import _psutil_osx as cext
ImportError: dlopen(/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so, 0x0002): tried: '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.2HjakR/install/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
Has anyone faced this issue before?
h
Hi, sorry for the trouble.
I did see similar errors (also on psutil) while setting up our new M1 CI runner
This path is already suspect, perhaps: 
/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/bin/pants
That 
x86_64
bit seems wrong, on an ARM64 machine
what do you see when you run
Copy code
file /Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so
b
I was using Rosetta on the terminal, now I'm still getting a similar issue but with 
arm64
instead:
Copy code
Traceback (most recent call last):
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/bin/pants", line 5, in <module>
    from pants.bin.pants_loader import main
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 19, in <module>
    from pants.bin.pants_runner import PantsRunner
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 11, in <module>
    from pants.base.exception_sink import ExceptionSink
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 16, in <module>
    import psutil
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/__init__.py", line 123, in <module>
    from . import _psosx as _psplatform
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psosx.py", line 14, in <module>
    from . import _psutil_osx as cext
ImportError: dlopen(/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so, 0x0002): tried: '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/pants.xHOKd3/install/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
If I run the following command: 
file /Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so
I get this: 
/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64
h
Yeah, so that is an x86_64 binary
I wonder if this is due to running Pants sometimes under rosetta and sometimes not?
Was there any point where you ran Pants with 
arch -x64
?
šŸ‘ 1
Might be worth moving 
~/.cache/pants
and the pip cache 
~/Library/Caches/pip
to one side, and re-bootstrapping by running 
./pants
(without rosetta)
āž• 1
b
Hey I'm just seeing this! As of your q, I ran 
arch -x86_64 ./*pants* version
once. I've removed both 
~/.cache/pants
and 
~/Library/Caches/pip
, and also 
~/Library/Caches/pipenv
just in case. Still no luck. Seeing the same error as before
h
Ah, I was actually recommending moving those and running pants without 
arch
, just running it natively
b
Oh yeah, sorry I wasn't clear enough. As your q was that if at any point I had run pants with the 
arch
bit, I said yeah I had but only once. After that one time, I always ran it without the 
arch
bit and I'm still facing the same issue šŸ˜”
h
Hmmm! This will require a repro to fix I expect. Can you list the exact list of steps that reproduces this on clean state, and I will try and do so on an m1?
Oh, and which Pants version is this?
b
Here are the steps I've followed: ā€¢ Removed these folders 
~/Library/Caches/pipenv ~/Library/Caches/pip ~/.cache/pants/
ā€¢ Ran 
pipenv shell
to start a virtual env ā€¢ Ran 
./pants
And then I get the error. I've also tried cleaning up the virtual environment and running pants rightaway (without the virtual env), but still getting the same error. Python version: 
Python 3.9.10
(CPython) Pants version: 
2.10.0
Pipenv version: 
2022.6.7
I've also tried with pants 
2.11.0
Also tried downloading 
psutil
using this command 
pip install --no-binary :all: psutil
, an arm64 wheel version of it gets downloaded, but when I run 
unzip psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
and then 
file psutil/_psutil_osx.cpython-39-darwin.so
I get this: 
psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64
h
(Parenthetically: you shouldn't need the 
pipenv shell
bit - Pants manages its own virtualenvs)
Something seems funky: The 
--no-binary
flag tells pip to ignore wheels and only use sdists. So that 
pip install --no-binary :all: psutil
command should be building a local wheel from a downloaded sdist.
So assuming that 
psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
you inspected is what was built locally on your machine, then for some reason you're getting an x86_64 binary as a result of a local build
You should see output like this from that pip command:
Copy code
Collecting psutil
  Downloading psutil-5.9.1.tar.gz (479 kB)
     ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā” 479.1/479.1 KB 1.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... done
  Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_12_0_x86_64.whl size=239918 sha256=b7cb70137f8012752d13e1fc6eab0356d2bde2508ef5f188c3a55749fd9a944d
  Stored in directory: /Users/benjyw/Library/Caches/pip/wheels/65/66/89/7ecefa8b8cfa845352eab83061e5f0f93946a306ae25661e03
Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-5.9.1
Is that what you're getting?
b
Let me try that out
h
When I run those steps I get 
psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle arm64
as expected
If you're not then there may be a problem outside of Pants
b
This is the output I get
Copy code
Collecting psutil
  Downloading psutil-5.9.1.tar.gz (479 kB)
     ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā” 479.1/479.1 kB 4.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... done
  Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl size=215962 sha256=743fc90a2b4c1ce322546d8c6d0facf5dfef90b9bb2358082f77a91c348a7e79
  Stored in directory: /Users/nferro/Library/Caches/pip/wheels/65/66/89/7ecefa8b8cfa845352eab83061e5f0f93946a306ae25661e03
Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-5.9.1
And this is the output I get
Copy code
site-packages file psutil/_psutil_osx.cpython-39-darwin.so 
psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64
If you check after you install it with the 
no-binary
bit, your created wheel step looks like this: 
Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_12_0_x86_64.whl
whereas mine has 'arm64' instead of 'x86_64': 
Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
h
Ignore that, that was when I ran it on an x86_64 machine
on an m1 I get: filename=psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
But note that your issue reproduces without Pants in the mix
How was your python3.9 installed?
b
Yeah you're right
If I'm not mistaken I installed it with conda
h
What do you get for 
python3.9 -c "import platform; print(platform.platform())"
?
b
macOS-12.4-arm64-arm-64bit
h
OK, that's good
weird then that when you invoke pip and it invokes setup.py it ends up building an x86 binary
Maybe try installing a python3.9 interpreter using pyenv?
b
Will do that
And see if it fixes the issue
Thank you so much for your help and patience!! I'll let you know if this fixes the issue
Installing it with pyenv did the trick!!!! Thank you so so much @happy-kitchen-89482 šŸ˜„šŸ™ŒšŸ™Œ
h
Great! So evidently the conda-installed interpreter had some platform issue
11 Views
Open in Slack
PreviousNext
Powered by