https://pantsbuild.org/ logo
Join SlackCommunities
Powered by
Hi guys, pants newbie here. I want to understand w...
# general
f
Hi guys, pants newbie here. I want to understand what is current status of using pants with macos ARM. I started to work on project which uses 1.30.3, but got into problems very quickly with my new M1 macbook.
h
Hello! I use Pants with my M1 but it requires running Pants with Python 3.9 for me, which Pants 2.5 is going to support Was your project already using Pants re why you're on 1.30?
f
already using, I believe it is because of scala/java support
h
Okay. V2 Java/Scala support is being actively worked on. But in the meantime... Which error are you hitting? For me, it was an issue with 
_ctypes
not existing iirc when running Pants with Py37 and Py38. I remember reading that Py39 is the only interpreter to properly work with M1, and indeed running Pants with Py39 fixed it for me. But I think they said they're planning to backport M1 support for Python 3.8. We release 1.30 as Py36, Py37, and Py38 wheels, but not yet Py39. I'm trying to figure out what your error is and if we can possibly get it working with Py38 or if it will be necessary to start releasing 1.30 with Py39
f
the errors I get are also very funny: ā€¢ with python 3.9 (preinstalled on mac sur) I get error that pants 1.30.3 does not support it ā€¢ with python 3.8 (installed via brew) I get this error:
Copy code
Using base prefix '/opt/homebrew/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8'
New python executable in /Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/pants.6QeG1w/install/bin/python3.8
Also creating executable in /Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/pants.6QeG1w/install/bin/python
ERROR: The executable /Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/pants.6QeG1w/install/bin/python3.8 is not functioning
ERROR: It thinks sys.prefix is '/Users/jozefkoval/Work/Projects/esl' (should be '/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/pants.6QeG1w/install')
ERROR: virtualenv is not compatible with this system or executable
./pants: line 276: /Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/1.30.3_py38/bin/python: No such file or directory
ā€¢ with python 3.7 (installed via pyenv) I get following:
Copy code
Using base prefix '/Users/jozefkoval/.pyenv/versions/3.7.10'
New python executable in /Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/pants.WcYtPs/install/bin/python3.7
Also creating executable in /Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/pants.WcYtPs/install/bin/python
Command /Users/jozefkoval/.c...nstall/bin/python3.7 -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command /Users/jozefkoval/.c...nstall/bin/python3.7 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/__init__.py", line 40, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 8, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/main_parser.py", line 12, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/commands/__init__.py", line 6, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/commands/completion.py", line 6, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/base_command.py", line 20, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/download.py", line 37, in <module>
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/utils/glibc.py", line 3, in <module>
  File "/Users/jozefkoval/.pyenv/versions/3.7.10/lib/python3.7/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
ImportError: cannot import name 'main' from 'pip' (/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv_support/pip-19.0.3-py2.py3-none-any.whl/pip/__init__.py)
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv.py", line 2567, in <module>
    main()
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv.py", line 793, in main
    symlink=options.symlink,
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv.py", line 1088, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv.py", line 935, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv.py", line 1025, in _install_wheel_with_search_dir
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
  File "/Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-16.4.3/virtualenv.py", line 886, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command /Users/jozefkoval/.c...nstall/bin/python3.7 - setuptools pip wheel failed with error code 1
./pants: line 276: /Users/jozefkoval/.cache/pants/setup/bootstrap-Darwin-arm64/1.30.3_py37/bin/python: No such file or directory
h
3.7 is the exact error I got. I think that's a non-starter and don't waste time trying to get Pants to run with Py37. You can set the env var 
PYTHON
to force the 
./pants
bash script to use something else, or you can rewrite parts of the Bash script like https://github.com/pantsbuild/setup/blob/cc5094f6546a3045c51b00720348027f5cf1f9f8/pants#L131-L134 3.8 error is surprising to me. Maybe try bumping the VENV_VERSION? https://github.com/pantsbuild/setup/blob/cc5094f6546a3045c51b00720348027f5cf1f9f8/pants#L37. Alternatively you can try using 
python -m venv
rather than virtualenv. Although, actually, before doing that all - try doing 
pip3.8 install pantsbuild.pants==1.30.3
-- It'd be great if we can get this working with Py38, as building wheels with Py39 is non-trivial and would be very difficult to backport to 1.30
f
šŸ¤”
Copy code
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==1.30.3
ERROR: No matching distribution found for pantsbuild.pants==1.30.3
h
Okay I suspect your pip is too old to realize that 
x86_64
does work on M1s. I think they patched pip to do that, but I don't know if it made it into Python 3.8's pip. You could try 
pip3.8 install --upgrade pip
iirc first. Make sure this is pip3.8, as pip3.9 will not have a Py39 wheel https://pypi.org/project/pantsbuild.pants/1.30.3/#files
f
Copy code
> pip3.8 install --upgrade pip
Requirement already satisfied: pip in /opt/homebrew/lib/python3.8/site-packages (21.0.1)
I could not install python 3.8 via pyenv, it might be problem with brew installation.
h
Ah same, 
configure: error: Unexpected output of 'arch' on OSX
. That explains why I don't have Py38 installed on my machine, I had forgotten that So...I fear to get Pants working on your M1 mac, the options would be: 1. Use an alternative environment like a devbox or VM. Wait for the upcoming Java/Scala support so that you can upgrade to Pants 2.x. Ack that that's not satisfying. 2. Contribute a patch to 1.30 to build wheels with Python 3.9. I put some time into trying to do this via Travis with https://github.com/pantsbuild/pants/pull/11462, and the main challenge was a segfault when trying to build in the Docker image šŸ˜± https://travis-ci.com/github/pantsbuild/pants/jobs/472240196#L2098 I gave up on Travis and finished Pants's migration to GitHub actions, although that is very non-trivial to backport to 1.30's CI. Us maintainers don't have enough time to get Py39 wheels building for 1.30, but we can try to help if you're interested like giving instructions - most of the code is already written, it's mainly figuring out that Docker failure
Wait for the upcoming Java/Scala support so that you can upgrade to Pants 2.x.
What language are you primarily working in? If Python, y'all can run Pants in mixed mode where part of the repo is 1.30 and another 2.x. But that only works if you use it primarily for Python and not JVM
f
Yes, the VM would be workaround. Regarding the patch, I am keen to contribute (but have limited knowledge of rust/python). The project is mainly written in Scala so I believe the mixed mode cannot be applied here.
h
Regarding the patch, I am keen to contribute (but have limited knowledge of rust/python)
That would be awesome and very much appreciated. Also helpful to future M1 users who can't yet use v2. I'm happy to try my best to help, only don't have the bandwidth to drive it Would you be able to work on the patch in a VM, or you'd want to use your M1? Asking because compiling the Rust engine only works with newer versions of Rust, and I'd need to first upgrade Rust on the 1.30 branch so that you could compile the engine to work in pantsbuild/pants repo. But if you can use VM, you save that work
f
I will use VM then šŸ˜‰
šŸ™Œ 1
h
Hm I'm typing up instructions and was going to propose that we could switch 1.30 to use GitHub Actions for the release, but still Travis for everything else. I think that would be cherry-pickable. But it means that we would break 1.30 to no longer work with macOS versions before macOS 10.15 Catalina because that's the earliest GH supports šŸ¤” @happy-kitchen-89482 what do you think of maybe us doing a Pants 1.31.0 release that drops support for earlier macOS versions and adds Py39 support? Feels like too breaking of a change to slip into 1.30
h
Yeah, I think we can do that. Sounds like not too hard.
šŸ‘ 1
And gives people a bridge until we have v2 JVM support
h
Sg. @fast-terabyte-66624 I will try to cherry-pick GitHub Actions to 1.30. If I can get it working, I think we'll be in shape. Otherwise we can explore using Travis and I can give some instructions
šŸ‘ 1
h
Alternatively I will take a look at getting python 3.9 wheels building on Travis, that might be simpler than backporting the GHA change.
f
I might get in working via rosetta and another brew/python3.8 installation šŸ¤¦ Will try to compile something and if it is usable I will create PR with detailed description for other people.
okay, compilation succeeded with python 3.8 and jvm build šŸŽ‰
ā— 1
h
oh great!
If you could document that, that would be awesome
f
It there a way to update documentation without creating readme account?
h
I don't think there is. You could type something up on Slack though and we can add it to Readme for you I'm thinking this can go into https://www.pantsbuild.org/docs/troubleshooting under an entry like "Running Pants with M1 Macs / Apple Silicon" Thank you for circling back to this!
f
ah, okay, I found arm mentioned only there: https://www.pantsbuild.org/docs/installation so would update both references
h
That'd be great too! The main entry can go into troubleshooting, and then the Installing Pants section cross-references it (duplication is can be a good thing in docs)
f
okay, tried to describe what I did, feel free to update the cross-reference and also English šŸ˜‰
ā¤ļø 1
h
Thank you Jozef!
h
Thanks!
7 Views
Open in Slack
PreviousNext
Powered by