#21938 pytest failure when running out-of-tree plugin against 2.25.0a0 - macOS wheels include single architecture `native_engine.so` files Issue created by tdyas Problem: Running pytest under Pants v2.25.0a0 is failing on my macOS/x86-64 system. The project under test is a out-of-tree Pants plugin and the test thus imports from the

pantsbuild.pants

distribution, hence why native_engine.so is being loaded in a test. (See the specific error below.) The

native_engine.so

being imported appears to be for arm64 and not for x86-64:

HOST:~/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/pants/engine/internals% file native_engine.so
native_engine.so: Mach-O 64-bit dynamically linked shared library arm64

We seem to be publishing universal2 macOS wheels instead of the previous separate architecture-specific wheels. 2.24.1: • pantsbuild.pants-2.24.1-cp39-cp39-macosx_10_15_x86_64.whl • pantsbuild.pants-2.24.1-cp39-cp39-macosx_11_0_arm64.whl 2.25.0a0: • pantsbuild.pants-2.25.0a0-cp311-cp311-macosx_10_12_universal2.whl • pantsbuild.pants-2.25.0a0-cp311-cp311-macosx_11_0_universal2.whl Unpacking the wheels, we see that the

.so

files in the root of the wheel are multi-architecture. Yet, the

native_engine.so

in the actual package source directory is one architecture only. (x86-64 for the 10.12 wheel and arm64 for the 11.0 wheel.)

./unpacked-wheel-11.0-universal2/pants/engine/internals/native_engine.so:  Mach-O 64-bit dynamically linked shared library arm64

./unpacked-wheel-11.0-universal2/native_engine.cpython-311-darwin.so:      Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64]
./unpacked-wheel-11.0-universal2/native_engine.cpython-311-darwin.so (for architecture x86_64):	Mach-O 64-bit bundle x86_64
./unpacked-wheel-11.0-universal2/native_engine.cpython-311-darwin.so (for architecture arm64):	Mach-O 64-bit bundle arm64

./unpacked-wheel-10.12-universal2/pants/engine/internals/native_engine.so: Mach-O 64-bit dynamically linked shared library x86_64

./unpacked-wheel-10.12-universal2/native_engine.cpython-311-darwin.so:     Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64]
./unpacked-wheel-10.12-universal2/native_engine.cpython-311-darwin.so (for architecture x86_64):	Mach-O 64-bit bundle x86_64
./unpacked-wheel-10.12-universal2/native_engine.cpython-311-darwin.so (for architecture arm64):	Mach-O 64-bit bundle arm64

--- The full error that I saw when trying to run my plugin's test was:

_ ERROR collecting src/python/shoalsoft/FOO/BAR_test.py _
ImportError while importing test module 'src/python/shoalsoft/FOO/BAR_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/ME/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/_pytest/python.py:493: in importtestmodule
    mod = import_path(
/Users/ME/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/_pytest/pathlib.py:587: in import_path
    importlib.import_module(module_name)
/Users/ME/.pyenv/versions/3.11.11/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/Users/ME/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:184: in exec_module
    exec(co, module.__dict__)
src/python/shoalsoft/FOO/BAR_test.py:17: in <module>
    from pants.engine.rules import QueryRule
/Users/ME/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/pants/engine/rules.py:32: in <module>
    from pants.engine.engine_aware import SideEffecting
/Users/ME/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/pants/engine/engine_aware.py:9: in <module>
    from pants.engine.internals import native_engine
E   ImportError: dlopen(/Users/ME/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/pants/engine/internals/native_engine.so, 0x0002): tried: '/Users/ME/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/pants/engine/internals/native_engine.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/tdyas/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/pants/engine/internals/native_engine.so' (no such file), '/Users/tdyas/.cache/pants/named_caches/pex_root/venvs/1/s/79b3222f/venv/lib/python3.11/site-packages/pants/engine/internals/native_engine.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')), '/Users/tdyas/.cache/pants/named_caches/pex_root/venvs/1/9d4bc213af4e92b06fa120abcea29d2f03f0eeef/2982e4bb26bb389cf345ae6b2c01416ccd8d4126/lib/python3.11/site-packages/pants/engine/internals/native_engine.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/tdyas/.cache/pants/named_caches/pex_root/venvs/1/9d4bc213af4e92b06fa120abcea29d2f03f0eeef/2982e4bb26bb389cf345ae6b2c01416ccd8d4126/lib/python3.11/site-packages/pants/engine/internals/native_engine.so' (no such file), '/Users/tdyas/.cache/pants/named_caches/pex_root/venvs/1/9d4bc213af4e92b06fa120abcea29d2f03f0eeef/2982e4bb26bb389cf345ae6b2c01416ccd8d4126/lib/python3.11/site-packages/pants/engine/internals/native_engine.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64'))
- generated xml file: src.python.shoalsoft.FOO.BAR_test.py.tests@@parametrize=pants-2.25.x.xml -
=========================== short test summary info ============================
ERROR src/python/shoalsoft/FOO/BAR_test.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.25s ===============================

pantsbuild/pants