About the failing test locally in thi

Question

< hundreds father 404> About the failing test locally in this What am I missing

hundreds-father-404 · Answer

I have no idea why native engine isn t building for you What platform are you on

refined-addition-53644 · Answer

macos

hundreds-father-404 · Answer

ARM or x86

refined-addition-53644 · Answer

x86

refined-addition-53644 · Answer

pants version = 2 14 0 dev3

hundreds-father-404 · Answer

Okay I m not really sure how to debug this the best idea I have is to first try `rm rf ~ pants ` and `rm rf pants d src rust engine target src python pants engine internal native engine so` which means rebuilding the engine snail

refined-addition-53644 · Answer

I suppose you mean `rm rf ~ cache pants`

hundreds-father-404 · Answer

you could try removing `src python pants engine internal native engine so` first to see if that fixes it

refined-addition-53644 · Answer

oops too late smile

refined-addition-53644 · Answer

that didn t help By the way I have already pushed the test on the branch

hundreds-father-404 · Answer

< enough analyst 54434> or < happy kitchen 89482> either of you have ideas

enough-analyst-54434 · Answer

Well backtrace implicates a Mac ` Users` before thinking too hard is this a damn M1

refined-addition-53644 · Answer

No

enough-analyst-54434 · Answer

Its an intel Mac then

refined-addition-53644 · Answer

yes

enough-analyst-54434 · Answer

Ok

refined-addition-53644 · Answer

macos 12 5

enough-analyst-54434 · Answer

What does `file src python pants engine internals native engine so` say

refined-addition-53644 · Answer

```src python pants engine internals native engine so Mach O 64 bit dynamically linked shared library x86 64```

enough-analyst-54434 · Answer

What does this say `nm src python pants engine internals native engine so | grep Py | grep v Z` Mac may have a different tool for reading names probably not `nm`

enough-analyst-54434 · Answer

`objdump`perhaps

refined-addition-53644 · Answer

It says lot of things ``` U PyBaseObject Type U PyBool Type U PyBuffer FromContiguous U PyBuffer Release U PyBytes AsString U PyBytes FromStringAndSize U PyBytes Size U PyCMethod New U PyDict GetItem U PyDict Items U PyDict New U PyDict Next U PyDict SetItem U PyDict Size U PyErr Fetch U PyErr GivenExceptionMatches U PyErr NewExceptionWithDoc U PyErr NormalizeException U PyErr Print U PyErr PrintEx U PyErr Restore U PyEval EvalCode U PyEval RestoreThread U PyEval SaveThread U PyExc AssertionError U PyExc AttributeError U PyExc BaseException U PyExc BrokenPipeError U PyExc BufferError U PyExc Exception U PyExc IOError U PyExc KeyboardInterrupt U PyExc OverflowError U PyExc RuntimeError U PyExc SystemError U PyExc TypeError U PyExc ValueError U PyException GetCause U PyException SetCause U PyFloat AsDouble U PyGILState Ensure U PyGILState Release U PyImport AddModule U PyImport Import 00000000001bab30 T PyInit native engine U PyInterpreterState Main U PyIter Next U PyList Append U PyList New U PyLong AsLong U PyLong AsUnsignedLongLong U PyLong FromLong U PyLong FromSsize t U PyLong FromUnsignedLongLong U PyModule Create2 U PyModule GetDict U PyModule GetName U PyNumber Index U PyObject Call U PyObject DelItem U PyObject Free U PyObject GenericGetDict U PyObject GenericSetDict U PyObject GetAttr U PyObject GetBuffer U PyObject GetItem U PyObject GetIter U PyObject HasAttr U PyObject IsInstance U PyObject IsSubclass U PyObject IsTrue U PyObject Malloc U PyObject Repr U PyObject RichCompare U PyObject SetAttr U PyObject SetAttrString U PyObject SetItem U PyObject Size U PyObject Str U PyObject VectorcallMethod U PySequence Check U PySequence Size U PySet Type U PyThreadState New U PyTuple New U PyTuple SetItem U PyType FromSpec U PyType GenericAlloc U PyType IsSubtype U PyType Type U PyUnicode AsEncodedString U PyUnicode AsUTF8AndSize U PyUnicode EncodeFSDefault U PyUnicode FromStringAndSize U PyUnicode InternInPlace U Py CompileStringExFlags U Py IsInitialized U PySet NextEntry U Py Dealloc U Py FalseStruct U Py NoneStruct U Py NotImplementedStruct U Py TrueStruct```

enough-analyst-54434 · Answer

Importantly it does say `U PyCMethod New` and your error is `E ImportError dlopen private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T pants sandbox EgsQUA src python pants engine internals native engine so 0x0002 symbol not found in flat namespace PyCMethod New `so the native engine so in the Pants sandbox = to the one in your code tree

enough-analyst-54434 · Answer

That s just a fact don t now what to make of it yet

enough-analyst-54434 · Answer

Can you run with no process cleanup so you can run these same commands on the sandbox so

hundreds-father-404 · Answer

< refined addition 53644> you re not using remote caching right

refined-addition-53644 · Answer

No Eric

refined-addition-53644 · Answer

So I see the content inside ` run sh` ``` bin bash This command line should execute the same process as pants did internally export HOME= Users developer PATH=$ Users developer cache pants pants dev deps Darwin x86 64 i386 py39 venv bin Users developer sdkman candidates spark current bin Users developer sdkman candidates scala current bin Users developer sdkman candidates sbt current bin Users developer sdkman candidates java current bin Users developer nvm versions node v16 13 0 bin usr local Cellar pyenv virtualenv 1 1 5 shims Users developer pyenv shims Users developer pyenv bin usr local opt openssl bin usr local opt ruby bin usr local bin usr bin bin usr sbin sbin Library TeX texbin opt X11 bin Users developer cargo bin Users developer fig bin Users developer local bin Users developer Library Application Support Coursier bin Applications Visual Studio <http Code app Contents Resources app bin Users developer local bin Users developer Library Python 3 9 bin Users developer Library Application|Code app Contents Resources app bin Users developer local bin Users developer Library Python 3 9 bin Users developer Library Application> Support Coursier bin usr local lib ruby gems 3 0 0 bin Users developer gem ruby 3 0 0 bin Users developer shantanu dx bin Users developer shantanu goprojects bin usr local opt go libexec bin Users developer cargo bin PEX EXTRA SYS PATH=$ src python PYENV ROOT= Users developer pyenv PYTEST ADDOPTS=$ color=yes junitxml=src python pants backend python lint flake8 rules integration test py tests xml o junit family=xunit2 RUST BACKTRACE=1 cd private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T pants sandbox o1zjFN pytest runner pex pex shim sh no header src python pants backend python lint flake8 rules integration test py```

refined-addition-53644 · Answer

I am not sure what else I should look for

refined-addition-53644 · Answer

So so does exist inside the sandbox ``` ls l private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T pants sandbox sRoMTb src python pants engine internals native engine so rwxr xr x 1 developer staff 22847520 Aug 11 18 41 private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T pants sandbox sRoMTb src python pants engine internals native engine so```

enough-analyst-54434 · Answer

Ok the thing to look for is the python version used in ` pytest runner pex pex shim sh` maybe just cat its full contents here

refined-addition-53644 · Answer

```platform darwin Python 3 7 9 pytest 7 0 1 pluggy 1 0 0``` Although when running the script it also throws error ``` pytest runner pex pex shim sh ==================================================================================================================== test session starts ===================================================================================================================== platform darwin Python 3 7 9 pytest 7 0 1 pluggy 1 0 0 rootdir private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T pants sandbox sRoMTb configfile pyproject toml plugins asyncio 0 18 3 metadata 2 0 1 html 3 1 1 icdiff 0 5 cov 3 0 0 asyncio mode=auto collected 0 items 1 error Interrupted 1 error during collection Traceback most recent call last File private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T pants sandbox sRoMTb cache pex root venvs 3a41a0a3410ea86da89847adb93d8df66f4d4ff7 b2e7d54b77c9629156f1f0d75d836621e445af8c pex line 245 in lt module gt sys exit func File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pytest config init py line 188 in console main code = main File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pytest config init py line 166 in main config=config File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy hooks py line 265 in call return self hookexec self name self get hookimpls kwargs firstresult File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy manager py line 80 in hookexec return self inner hookexec hook name methods kwargs firstresult File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy callers py line 60 in multicall return outcome get result File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy result py line 60 in get result raise ex 1 with traceback ex 2 File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy callers py line 39 in multicall res = hook impl function args File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pytest main py line 315 in pytest cmdline main return wrap session config main File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pytest main py line 304 in wrap session session=session exitstatus=session exitstatus File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy hooks py line 265 in call return self hookexec self name self get hookimpls kwargs firstresult File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy manager py line 80 in hookexec return self inner hookexec hook name methods kwargs firstresult File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy callers py line 55 in multicall gen send outcome File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pytest terminal py line 792 in pytest sessionfinish outcome get result File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy result py line 60 in get result raise ex 1 with traceback ex 2 File Users developer cache pants named caches pex root venvs s a85f931a venv lib python3 7 site packages pluggy callers py line 39 in multicall res = hook impl function args File private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T pants sandbox sRoMTb src python pants conftest py line 46 in pytest sessionfinish namespace init path unlink File Users developer pyenv versions 3 7 9 lib python3 7 pathlib py line 1309 in unlink self accessor unlink self FileNotFoundError Errno 2 No such file or directory src python pants init py ```

enough-analyst-54434 · Answer

I m sorry I was not clear Please `cat pytest runner pex pex shim sh` and paste the contents here

refined-addition-53644 · Answer

Yeah I realized bit late ``` bin bash set euo pipefail N B This relies on BASH SOURCE which has been available since bash 3 0 released in 2004 It will either contain the absolute path of the venv script or it will contain the relative path from the CWD to the venv script Either way we know the venv script parent directory is the sandbox root directory SANDBOX ROOT= $ BASH SOURCE function adjust relative paths local value0= $1 shift if $ value0 0 1 == then Don t relativize absolute paths echo $ value0 $ else N B We convert all relative paths to paths relative to the sandbox root so this script works when run with a PWD set somewhere else than the sandbox root There are two cases to consider For the purposes of example assume PWD is ` tmp sandboxes abc123 foo bar` i e the rule API sets working directory to `foo bar` Also assume `config tool yml` is the relative path in question 1 If our BASH SOURCE is ` tmp sandboxes abc123 pex shim sh` so our SANDBOX ROOT is ` tmp sandboxes abc123` we calculate ` tmp sandboxes abc123 config tool yml` 2 If our BASH SOURCE is instead ` pex shim sh` so our SANDBOX ROOT is ` ` we calculate ` config tool yml` echo $ SANDBOX ROOT $ value0 $ fi export PATH= Users developer cache pants pants dev deps Darwin x86 64 i386 py39 venv bin Users developer sdkman candidates spark current bin Users developer sdkman candidates scala current bin Users developer sdkman candidates sbt current bin Users developer sdkman candidates java current bin Users developer nvm versions node v16 13 0 bin usr local Cellar pyenv virtualenv 1 1 5 shims Users developer pyenv shims Users developer pyenv bin usr local opt openssl bin usr local opt ruby bin usr local bin usr bin bin usr sbin sbin Library TeX texbin opt X11 bin Users developer cargo bin Users developer fig bin Users developer local bin Users developer Library Application Support Coursier bin Applications Visual Studio <http Code app Contents Resources app bin Users developer Library Python 3 9 bin usr local lib ruby gems 3 0 0 bin Users developer gem ruby 3 0 0 bin Users developer shantanu dx bin Users developer shantanu goprojects bin usr local opt go libexec bin|Code app Contents Resources app bin Users developer Library Python 3 9 bin usr local lib ruby gems 3 0 0 bin Users developer gem ruby 3 0 0 bin Users developer shantanu dx bin Users developer shantanu goprojects bin usr local opt go libexec bin> PEX IGNORE RCFILES=true PEX ROOT= cache pex root LC CTYPE=UTF 8 export PEX ROOT= $ adjust relative paths $ PEX ROOT execute pex args= $ adjust relative paths Users developer pyenv versions 3 7 9 bin python3 7 $ adjust relative paths pytest runner pex target venv executable= $ adjust relative paths cache pex root venvs 3a41a0a3410ea86da89847adb93d8df66f4d4ff7 b2e7d54b77c9629156f1f0d75d836621e445af8c pex venv dir= $ adjust relative paths cache pex root venvs 3a41a0a3410ea86da89847adb93d8df66f4d4ff7 b2e7d54b77c9629156f1f0d75d836621e445af8c Let PEX TOOLS invocations pass through to the original PEX file since venvs don t come with tools support if n $ PEX TOOLS then exec $ execute pex args $ fi If the seeded venv has been removed from the PEX ROOT we re seed from the original ` venv` mode PEX file if e $ target venv executable then rm rf $ venv dir || true PEX INTERPRETER=1 $ execute pex args c fi exec $ target venv executable $ ```

enough-analyst-54434 · Answer

Ok great And now `head 1 cache pex root venvs 3a41a0a3410ea86da89847adb93d8df66f4d4ff7 b2e7d54b77c9629156f1f0d75d836621e445af8c pex`

refined-addition-53644 · Answer

I feel like pants dev dependencies are built using 3 9 but for everything else 3 7 is being used

enough-analyst-54434 · Answer

I agree Looking for proof

refined-addition-53644 · Answer

``` Users developer cache pants named caches pex root venvs s a85f931a venv bin python3 7 sE```

enough-analyst-54434 · Answer

Ok so as you suspect the tests think Python 3 7 should be used

enough-analyst-54434 · Answer

Can you edit your ` pants` script to print out the `PY`it determines

refined-addition-53644 · Answer

```PY= Users developer pyenv shims python3 9```

enough-analyst-54434 · Answer

Ok so unless there is a fancy attempt at making Pants integration tests robust to this on the surface your hunch is correct Pants ootstrap finds 3 9 and builds a 3 9 engine and then integration tests blindly use that engine file but do not account for setting the Python to use to match it

enough-analyst-54434 · Answer

< hundreds father 404> will be more familiar with the state of Pants ITs

enough-analyst-54434 · Answer

So < refined addition 53644> the quick fix is to run `PY=python3 7 pants ` to do your testing

enough-analyst-54434 · Answer

But that s not a fix it s a hack workaround

enough-analyst-54434 · Answer

This will be slow 1st run as the engine gets re built

hundreds-father-404 · Answer

Ah yeah let me know if the `PY` thing works I ve been hardcoding `PY` and ` python interpreter constraints` the past 1+ year due to M1 so I have no idea if the defaults for the repo still are sensible

hundreds-father-404 · Answer

Thank you John for the help debugging

refined-addition-53644 · Answer

Thanks John

refined-addition-53644 · Answer

Yeah that works

enough-analyst-54434 · Answer

The defaults are sensible fwict our ITs are just buggy They do not account for this

enough-analyst-54434 · Answer

And it looks like we hack around this in CI <https github com pantsbuild pants blob 8ec297591cfc584a0644dedfa3691e6939f947c5 github workflows test yaml L49 L50> Ideally the fix would be in repo and the CI hack not needed

refined-addition-53644 · Answer

hey < hundreds father 404> could you please approve the workflow again I had to fix the test I checked it works locally