https://pantsbuild.org/ logo
Join Slack
Powered by
<@UB2J9BQA0> About the failing test locally in thi...
# development
r
@hundreds-father-404 About the failing test locally in this pull request What am I missing?
✅ 1
h
I have no idea why native_engine isn't building for you. What platform are you on?
r
macos
h
ARM or x86?
r
x86
pants version = 2.14.0.dev3
h
Okay. I'm not really sure how to debug this, the best idea I have is to first try 
rm -rf ~/.pants/
and 
rm -rf .pants.d src/rust/engine/target src/python/pants/engine/internal/native_engine.so
, which means rebuilding the engine 🐌
😿 1
r
I suppose you mean 
rm -rf ~/.cache/pants
?
➕ 1
h
you could try removing 
src/python/pants/engine/internal/native_engine.so
first to see if that fixes it
r
oops too late 😄
🔥 1
that didn’t help. By the way I have already pushed the test on the branch
h
@enough-analyst-54434 or @happy-kitchen-89482, either of you have ideas?
e
Well, backtrace implicates a Mac (
/Users
) - before thinking too hard is this a damn M1?
r
No
e
Its an intel Mac then?
r
yes
e
Ok...
r
macos 12.5
e
What does 
file src/python/pants/engine/internals/native_engine.so
say?
r
Copy code
src/python/pants/engine/internals/native_engine.so: Mach-O 64-bit dynamically linked shared library x86_64
e
What does this say?: 
nm src/python/pants/engine/internals/native_engine.so | grep Py | grep -v _Z
- Mac may have a different tool for reading names - probably not 
nm
.
`objdump`perhaps?
r
It says lot of things
Copy code
U _PyBaseObject_Type
                 U _PyBool_Type
                 U _PyBuffer_FromContiguous
                 U _PyBuffer_Release
                 U _PyBytes_AsString
                 U _PyBytes_FromStringAndSize
                 U _PyBytes_Size
                 U _PyCMethod_New
                 U _PyDict_GetItem
                 U _PyDict_Items
                 U _PyDict_New
                 U _PyDict_Next
                 U _PyDict_SetItem
                 U _PyDict_Size
                 U _PyErr_Fetch
                 U _PyErr_GivenExceptionMatches
                 U _PyErr_NewExceptionWithDoc
                 U _PyErr_NormalizeException
                 U _PyErr_Print
                 U _PyErr_PrintEx
                 U _PyErr_Restore
                 U _PyEval_EvalCode
                 U _PyEval_RestoreThread
                 U _PyEval_SaveThread
                 U _PyExc_AssertionError
                 U _PyExc_AttributeError
                 U _PyExc_BaseException
                 U _PyExc_BrokenPipeError
                 U _PyExc_BufferError
                 U _PyExc_Exception
                 U _PyExc_IOError
                 U _PyExc_KeyboardInterrupt
                 U _PyExc_OverflowError
                 U _PyExc_RuntimeError
                 U _PyExc_SystemError
                 U _PyExc_TypeError
                 U _PyExc_ValueError
                 U _PyException_GetCause
                 U _PyException_SetCause
                 U _PyFloat_AsDouble
                 U _PyGILState_Ensure
                 U _PyGILState_Release
                 U _PyImport_AddModule
                 U _PyImport_Import
00000000001bab30 T _PyInit_native_engine
                 U _PyInterpreterState_Main
                 U _PyIter_Next
                 U _PyList_Append
                 U _PyList_New
                 U _PyLong_AsLong
                 U _PyLong_AsUnsignedLongLong
                 U _PyLong_FromLong
                 U _PyLong_FromSsize_t
                 U _PyLong_FromUnsignedLongLong
                 U _PyModule_Create2
                 U _PyModule_GetDict
                 U _PyModule_GetName
                 U _PyNumber_Index
                 U _PyObject_Call
                 U _PyObject_DelItem
                 U _PyObject_Free
                 U _PyObject_GenericGetDict
                 U _PyObject_GenericSetDict
                 U _PyObject_GetAttr
                 U _PyObject_GetBuffer
                 U _PyObject_GetItem
                 U _PyObject_GetIter
                 U _PyObject_HasAttr
                 U _PyObject_IsInstance
                 U _PyObject_IsSubclass
                 U _PyObject_IsTrue
                 U _PyObject_Malloc
                 U _PyObject_Repr
                 U _PyObject_RichCompare
                 U _PyObject_SetAttr
                 U _PyObject_SetAttrString
                 U _PyObject_SetItem
                 U _PyObject_Size
                 U _PyObject_Str
                 U _PyObject_VectorcallMethod
                 U _PySequence_Check
                 U _PySequence_Size
                 U _PySet_Type
                 U _PyThreadState_New
                 U _PyTuple_New
                 U _PyTuple_SetItem
                 U _PyType_FromSpec
                 U _PyType_GenericAlloc
                 U _PyType_IsSubtype
                 U _PyType_Type
                 U _PyUnicode_AsEncodedString
                 U _PyUnicode_AsUTF8AndSize
                 U _PyUnicode_EncodeFSDefault
                 U _PyUnicode_FromStringAndSize
                 U _PyUnicode_InternInPlace
                 U _Py_CompileStringExFlags
                 U _Py_IsInitialized
                 U __PySet_NextEntry
                 U __Py_Dealloc
                 U __Py_FalseStruct
                 U __Py_NoneStruct
                 U __Py_NotImplementedStruct
                 U __Py_TrueStruct
e
Importantly it does say 
U _PyCMethod_New
and your error is `E ImportError: dlopen(/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/pants-sandbox-EgsQUA/src/python/pants/engine/internals/native_engine.so, 0x0002): symbol not found in flat namespace (_PyCMethod_New)`so the native_engine.so in the Pants sandbox != to the one in your code tree.
That's just a fact - don't now what to make of it yet.
😄 1
Can you run with --no-process-cleanup so you can run these same commands on the sandbox .so?
👍 1
h
@refined-addition-53644 you're not using remote caching, right?
r
No Eric!
👍 1
So I see the content inside 
__run.sh
Copy code
#!/bin/bash
# This command line should execute the same process as pants did internally.
export HOME=/Users/developer PATH=$'/Users/developer/.cache/pants/pants_dev_deps/Darwin.x86_64.i386.py39.venv/bin:/Users/developer/.sdkman/candidates/spark/current/bin:/Users/developer/.sdkman/candidates/scala/current/bin:/Users/developer/.sdkman/candidates/sbt/current/bin:/Users/developer/.sdkman/candidates/java/current/bin:/Users/developer/.nvm/versions/node/v16.13.0/bin:/usr/local/Cellar/pyenv-virtualenv/1.1.5/shims:/Users/developer/.pyenv/shims:/Users/developer/.pyenv/bin:/usr/local/opt/openssl/bin:/usr/local/opt/ruby/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Users/developer/.cargo/bin:/Users/developer/.fig/bin:/Users/developer/.local/bin:/Users/developer/Library/Application Support/Coursier/bin:/Applications/Visual Studio <http://Code.app/Contents/Resources/app/bin:/Users/developer/.local/bin:/Users/developer/Library/Python/3.9/bin:/Users/developer/Library/Application|Code.app/Contents/Resources/app/bin:/Users/developer/.local/bin:/Users/developer/Library/Python/3.9/bin:/Users/developer/Library/Application> Support/Coursier/bin:/usr/local/lib/ruby/gems/3.0.0/bin:/Users/developer/.gem/ruby/3.0.0/bin:/Users/developer/shantanu/dx/bin:/Users/developer/shantanu/goprojects/bin:/usr/local/opt/go/libexec/bin:/Users/developer/.cargo/bin' PEX_EXTRA_SYS_PATH=$'.:src/python' PYENV_ROOT=/Users/developer/.pyenv PYTEST_ADDOPTS=$'--color=yes --junitxml=src.python.pants.backend.python.lint.flake8.rules_integration_test.py.tests.xml -o junit_family=xunit2' RUST_BACKTRACE=1
cd /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/pants-sandbox-o1zjFN
./pytest_runner.pex_pex_shim.sh --no-header src/python/pants/backend/python/lint/flake8/rules_integration_test.py
I am not sure what else I should look for
So .so does exist inside the sandbox
Copy code
ls -l /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/pants-sandbox-sRoMTb/src/python/pants/engine/internals/native_engine.so
-rwxr-xr-x  1 developer  staff  22847520 Aug 11 18:41 /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/pants-sandbox-sRoMTb/src/python/pants/engine/internals/native_engine.so
e
Ok, the thing to look for is the python version used in 
./pytest_runner.pex_pex_shim.sh
maybe just cat its full contents here.
r
Copy code
platform darwin -- Python 3.7.9, pytest-7.0.1, pluggy-1.0.0
Although when running the script it also throws error
Copy code
./pytest_runner.pex_pex_shim.sh
==================================================================================================================== test session starts =====================================================================================================================
platform darwin -- Python 3.7.9, pytest-7.0.1, pluggy-1.0.0
rootdir: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/pants-sandbox-sRoMTb, configfile: pyproject.toml
plugins: asyncio-0.18.3, metadata-2.0.1, html-3.1.1, icdiff-0.5, cov-3.0.0
asyncio: mode=auto
collected 0 items / 1 error
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Traceback (most recent call last):
  File "/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/pants-sandbox-sRoMTb/.cache/pex_root/venvs/3a41a0a3410ea86da89847adb93d8df66f4d4ff7/b2e7d54b77c9629156f1f0d75d836621e445af8c/pex", line 245, in <module>
    sys.exit(func())
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/_pytest/config/__init__.py", line 188, in console_main
    code = main()
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/_pytest/config/__init__.py", line 166, in main
    config=config
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/_pytest/main.py", line 315, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/_pytest/main.py", line 304, in wrap_session
    session=session, exitstatus=session.exitstatus
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/_pytest/terminal.py", line 792, in pytest_sessionfinish
    outcome.get_result()
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/pants-sandbox-sRoMTb/src/python/pants/conftest.py", line 46, in pytest_sessionfinish
    namespace_init_path.unlink()
  File "/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/pathlib.py", line 1309, in unlink
    self._accessor.unlink(self)
FileNotFoundError: [Errno 2] No such file or directory: 'src/python/pants/__init__.py'
e
I'm sorry, I was not clear. Please 
cat ./pytest_runner.pex_pex_shim.sh
and paste the contents here.
r
Yeah I realized bit late
Copy code
#!/bin/bash
set -euo pipefail

# N.B.: This relies on BASH_SOURCE which has been available since bash-3.0, released in
# 2004. It will either contain the absolute path of the venv script or it will contain
# the relative path from the CWD to the venv script. Either way, we know the venv script
# parent directory is the sandbox root directory.
SANDBOX_ROOT="${BASH_SOURCE%/*}"

function adjust_relative_paths() {
    local value0="$1"
    shift
    if [ "${value0:0:1}" == "/" ]; then
        # Don't relativize absolute paths.
        echo "${value0}" "$@"
    else
        # N.B.: We convert all relative paths to paths relative to the sandbox root so
        # this script works when run with a PWD set somewhere else than the sandbox
        # root.
        #
        # There are two cases to consider. For the purposes of example, assume PWD is
        # `/tmp/sandboxes/abc123/foo/bar`; i.e.: the rule API sets working_directory to
        # `foo/bar`. Also assume `config/tool.yml` is the relative path in question.
        #
        # 1. If our BASH_SOURCE is  `/tmp/sandboxes/abc123/pex_shim.sh`; so our
        #    SANDBOX_ROOT is `/tmp/sandboxes/abc123`, we calculate
        #    `/tmp/sandboxes/abc123/config/tool.yml`.
        # 2. If our BASH_SOURCE is instead `../../pex_shim.sh`; so our SANDBOX_ROOT is
        #    `../..`, we calculate `../../config/tool.yml`.
        echo "${SANDBOX_ROOT}/${value0}" "$@"
    fi
}

export PATH='/Users/developer/.cache/pants/pants_dev_deps/Darwin.x86_64.i386.py39.venv/bin:/Users/developer/.sdkman/candidates/spark/current/bin:/Users/developer/.sdkman/candidates/scala/current/bin:/Users/developer/.sdkman/candidates/sbt/current/bin:/Users/developer/.sdkman/candidates/java/current/bin:/Users/developer/.nvm/versions/node/v16.13.0/bin:/usr/local/Cellar/pyenv-virtualenv/1.1.5/shims:/Users/developer/.pyenv/shims:/Users/developer/.pyenv/bin:/usr/local/opt/openssl/bin:/usr/local/opt/ruby/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Users/developer/.cargo/bin:/Users/developer/.fig/bin:/Users/developer/.local/bin:/Users/developer/Library/Application Support/Coursier/bin:/Applications/Visual Studio <http://Code.app/Contents/Resources/app/bin:/Users/developer/Library/Python/3.9/bin:/usr/local/lib/ruby/gems/3.0.0/bin:/Users/developer/.gem/ruby/3.0.0/bin:/Users/developer/shantanu/dx/bin:/Users/developer/shantanu/goprojects/bin:/usr/local/opt/go/libexec/bin|Code.app/Contents/Resources/app/bin:/Users/developer/Library/Python/3.9/bin:/usr/local/lib/ruby/gems/3.0.0/bin:/Users/developer/.gem/ruby/3.0.0/bin:/Users/developer/shantanu/dx/bin:/Users/developer/shantanu/goprojects/bin:/usr/local/opt/go/libexec/bin>' PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root LC_CTYPE=UTF-8
export PEX_ROOT="$(adjust_relative_paths ${PEX_ROOT})"

execute_pex_args="$(adjust_relative_paths /Users/developer/.pyenv/versions/3.7.9/bin/python3.7) $(adjust_relative_paths pytest_runner.pex)"
target_venv_executable="$(adjust_relative_paths .cache/pex_root/venvs/3a41a0a3410ea86da89847adb93d8df66f4d4ff7/b2e7d54b77c9629156f1f0d75d836621e445af8c/pex)"
venv_dir="$(adjust_relative_paths .cache/pex_root/venvs/3a41a0a3410ea86da89847adb93d8df66f4d4ff7/b2e7d54b77c9629156f1f0d75d836621e445af8c)"

# Let PEX_TOOLS invocations pass through to the original PEX file since venvs don't come
# with tools support.
if [ -n "${PEX_TOOLS:-}" ]; then
  exec ${execute_pex_args} "$@"
fi

# If the seeded venv has been removed from the PEX_ROOT, we re-seed from the original
# `--venv` mode PEX file.
if [ ! -e "${target_venv_executable}" ]; then
    rm -rf "${venv_dir}" || true
    PEX_INTERPRETER=1 ${execute_pex_args} -c ''
fi

exec "${target_venv_executable}" "$@"
e
Ok, great. And now: 
head -1 .cache/pex_root/venvs/3a41a0a3410ea86da89847adb93d8df66f4d4ff7/b2e7d54b77c9629156f1f0d75d836621e445af8c/pex
r
I feel like pants dev dependencies are built using 3.9 but for everything else 3.7 is being used
e
I agree. Looking for proof.
r
Copy code
#!/Users/developer/.cache/pants/named_caches/pex_root/venvs/s/a85f931a/venv/bin/python3.7 -sE
e
Ok, so, as you suspect the tests think Python 3.7 should be used.
Can you edit your 
./pants
script to print out the `PY`it determines?
r
Copy code
PY=/Users/developer/.pyenv/shims/python3.9
e
Ok, so unless there is a fancy attempt at making Pants integration tests robust to this, on the surface your hunch is correct. Pants ootstrap finds 3.9 and builds a 3.9 engine, and then integration tests blindly use that engine file but do not account for setting the Python to use to match it.
@hundreds-father-404 will be more familiar with the state of Pants ITs.
So, @refined-addition-53644 the quick fix is to run 
PY=python3.7 ./pants ...
to do your testing.
But that's not a fix, it's a hack workaround.
This will be slow 1st run as the engine gets re-built.
👍 1
h
Ah yeah let me know if the 
PY
thing works. I've been hardcoding 
PY
and 
[python].interpreter_constraints
the past 1+ year due to M1, so I have no idea if the defaults for the repo still are sensible
Thank you John for the help debugging
r
Thanks John!
Yeah that works!
e
The defaults are sensible fwict, our ITs are just buggy. They do not account for this.
And it looks like we hack around this in CI: https://github.com/pantsbuild/pants/blob/8ec297591cfc584a0644dedfa3691e6939f947c5/.github/workflows/test.yaml#L49-L50 Ideally the fix would be in-repo and the CI hack not needed.
r
hey @hundreds-father-404 could you please approve the workflow again. I had to fix the test. I checked it works locally.
👍 1
2 Views
Open in Slack
PreviousNext
Powered by