https://pantsbuild.org/ logo
#general
Title
# general
b

breezy-church-49979

06/14/2022, 10:33 PM
Hey, I'm having some python/architecture issues when running pants on an M1 machine. I'll post the stack trace in a thread
āœ… 1
Copy code
Traceback (most recent call last):
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/bin/pants", line 5, in <module>
    from pants.bin.pants_loader import main
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 19, in <module>
    from pants.bin.pants_runner import PantsRunner
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 11, in <module>
    from pants.base.exception_sink import ExceptionSink
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 16, in <module>
    import psutil
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/__init__.py", line 123, in <module>
    from . import _psosx as _psplatform
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psosx.py", line 14, in <module>
    from . import _psutil_osx as cext
ImportError: dlopen(/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so, 0x0002): tried: '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.2HjakR/install/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
Has anyone faced this issue before?
h

happy-kitchen-89482

06/14/2022, 10:37 PM
Hi, sorry for the trouble.
I did see similar errors (also on psutil) while setting up our new M1 CI runner
This path is already suspect, perhaps:
/Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/bin/pants
That
x86_64
bit seems wrong, on an ARM64 machine
what do you see when you run
Copy code
file /Users/nferro/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so
b

breezy-church-49979

06/14/2022, 10:51 PM
I was using Rosetta on the terminal, now I'm still getting a similar issue but with
arm64
instead:
Copy code
Traceback (most recent call last):
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/bin/pants", line 5, in <module>
    from pants.bin.pants_loader import main
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 19, in <module>
    from pants.bin.pants_runner import PantsRunner
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 11, in <module>
    from pants.base.exception_sink import ExceptionSink
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 16, in <module>
    import psutil
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/__init__.py", line 123, in <module>
    from . import _psosx as _psplatform
  File "/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psosx.py", line 14, in <module>
    from . import _psutil_osx as cext
ImportError: dlopen(/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so, 0x0002): tried: '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/pants.xHOKd3/install/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
If I run the following command:
file /Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so
I get this:
/Users/nferro/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64
h

happy-kitchen-89482

06/15/2022, 2:30 PM
Yeah, so that is an x86_64 binary
I wonder if this is due to running Pants sometimes under rosetta and sometimes not?
Was there any point where you ran Pants with
arch -x64
?
šŸ‘ 1
Might be worth moving
~/.cache/pants
and the pip cache
~/Library/Caches/pip
to one side, and re-bootstrapping by running
./pants
(without rosetta)
āž• 1
b

breezy-church-49979

06/17/2022, 5:40 PM
Hey I'm just seeing this! As of your q, I ran
arch -x86_64 ./*pants* version
once. I've removed both
~/.cache/pants
and
~/Library/Caches/pip
, and also
~/Library/Caches/pipenv
just in case. Still no luck. Seeing the same error as before
h

happy-kitchen-89482

06/17/2022, 9:30 PM
Ah, I was actually recommending moving those and running pants without
arch
, just running it natively
b

breezy-church-49979

06/18/2022, 2:27 AM
Oh yeah, sorry I wasn't clear enough. As your q was that if at any point I had run pants with the
arch
bit, I said yeah I had but only once. After that one time, I always ran it without the
arch
bit and I'm still facing the same issue šŸ˜”
h

happy-kitchen-89482

06/18/2022, 8:58 AM
Hmmm! This will require a repro to fix I expect. Can you list the exact list of steps that reproduces this on clean state, and I will try and do so on an m1?
Oh, and which Pants version is this?
b

breezy-church-49979

06/18/2022, 9:38 AM
Here are the steps I've followed: ā€¢ Removed these folders
~/Library/Caches/pipenv ~/Library/Caches/pip ~/.cache/pants/
ā€¢ Ran
pipenv shell
to start a virtual env ā€¢ Ran
./pants
And then I get the error. I've also tried cleaning up the virtual environment and running pants rightaway (without the virtual env), but still getting the same error. Python version:
Python 3.9.10
(CPython) Pants version:
2.10.0
Pipenv version:
2022.6.7
I've also tried with pants
2.11.0
Also tried downloading
psutil
using this command
pip install --no-binary :all: psutil
, an arm64 wheel version of it gets downloaded, but when I run
unzip psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
and then
file psutil/_psutil_osx.cpython-39-darwin.so
I get this:
psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64
h

happy-kitchen-89482

06/18/2022, 6:27 PM
(Parenthetically: you shouldn't need the
pipenv shell
bit - Pants manages its own virtualenvs)
Something seems funky: The
--no-binary
flag tells pip to ignore wheels and only use sdists. So that
pip install --no-binary :all: psutil
command should be building a local wheel from a downloaded sdist.
So assuming that
psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
you inspected is what was built locally on your machine, then for some reason you're getting an x86_64 binary as a result of a local build
You should see output like this from that pip command:
Copy code
Collecting psutil
  Downloading psutil-5.9.1.tar.gz (479 kB)
     ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā” 479.1/479.1 KB 1.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... done
  Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_12_0_x86_64.whl size=239918 sha256=b7cb70137f8012752d13e1fc6eab0356d2bde2508ef5f188c3a55749fd9a944d
  Stored in directory: /Users/benjyw/Library/Caches/pip/wheels/65/66/89/7ecefa8b8cfa845352eab83061e5f0f93946a306ae25661e03
Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-5.9.1
Is that what you're getting?
b

breezy-church-49979

06/18/2022, 7:40 PM
Let me try that out
h

happy-kitchen-89482

06/18/2022, 7:41 PM
When I run those steps I get
psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle arm64
as expected
If you're not then there may be a problem outside of Pants
b

breezy-church-49979

06/18/2022, 7:46 PM
This is the output I get
Copy code
Collecting psutil
  Downloading psutil-5.9.1.tar.gz (479 kB)
     ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā”ā” 479.1/479.1 kB 4.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... done
  Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl size=215962 sha256=743fc90a2b4c1ce322546d8c6d0facf5dfef90b9bb2358082f77a91c348a7e79
  Stored in directory: /Users/nferro/Library/Caches/pip/wheels/65/66/89/7ecefa8b8cfa845352eab83061e5f0f93946a306ae25661e03
Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-5.9.1
And this is the output I get
Copy code
site-packages file psutil/_psutil_osx.cpython-39-darwin.so 
psutil/_psutil_osx.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64
If you check after you install it with the
no-binary
bit, your created wheel step looks like this:
Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_12_0_x86_64.whl
whereas mine has 'arm64' instead of 'x86_64':
Created wheel for psutil: filename=psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
h

happy-kitchen-89482

06/18/2022, 7:52 PM
Ignore that, that was when I ran it on an x86_64 machine
on an m1 I get: filename=psutil-5.9.1-cp39-cp39-macosx_11_0_arm64.whl
But note that your issue reproduces without Pants in the mix
How was your python3.9 installed?
b

breezy-church-49979

06/18/2022, 7:53 PM
Yeah you're right
If I'm not mistaken I installed it with conda
h

happy-kitchen-89482

06/18/2022, 7:53 PM
What do you get for
python3.9 -c "import platform; print(platform.platform())"
?
b

breezy-church-49979

06/18/2022, 7:54 PM
macOS-12.4-arm64-arm-64bit
h

happy-kitchen-89482

06/18/2022, 7:54 PM
OK, that's good
weird then that when you invoke pip and it invokes setup.py it ends up building an x86 binary
Maybe try installing a python3.9 interpreter using pyenv?
b

breezy-church-49979

06/18/2022, 7:55 PM
Will do that
And see if it fixes the issue
Thank you so much for your help and patience!! I'll let you know if this fixes the issue
Installing it with pyenv did the trick!!!! Thank you so so much @happy-kitchen-89482 šŸ˜„šŸ™ŒšŸ™Œ
h

happy-kitchen-89482

06/19/2022, 11:16 AM
Great! So evidently the conda-installed interpreter had some platform issue
11 Views