https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, I've recently started using Pants. Everythi...
# general
a
Hello, I've recently started using Pants. Everything worked perfect unitll I upgraded to macOS Monterey. Now when I try to run any pants command I get this:
Copy code
15:46:09.41 [DEBUG] spawned local process as Some(99409) for Process { argv: ["/Users/sinisaderasevic/.pyenv/versions/3.9.7/bin/python", "./pex", "--python-path", "/Users/sinisaderasevic/.pyenv/versions/3.9.7/bin:/Users/sinisaderasevic/.virtualenvs/apollo/bin:/Users/sinisaderasevic/.pyenv/shims:/Users/sinisaderasevic/.pyenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin", "--tmpdir", ".tmp", "--interpreter-constraint", "CPython==3.9.*", "--", "-c", "import hashlib, os, sys\n\npython = os.path.realpath(sys.executable)\nprint(python)\n\nhasher = hashlib.sha256()\nwith open(python, \"rb\") as fp:\n  for chunk in iter(lambda: fp.read(8192), b\"\"):\n      hasher.update(chunk)\nprint(hasher.hexdigest())\n"], env: {"CPPFLAGS": "", "LC_CTYPE": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/Users/sinisaderasevic/.virtualenvs/apollo/bin:/Users/sinisaderasevic/.pyenv/shims:/Users/sinisaderasevic/.pyenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON_PATH": "/Users/sinisaderasevic/.pyenv/versions/3.9.7/bin:/Users/sinisaderasevic/.virtualenvs/apollo/bin:/Users/sinisaderasevic/.pyenv/shims:/Users/sinisaderasevic/.pyenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_files: Digest { hash: Fingerprint<26cf32db81511eab7e4996e09d5a21b162c66212333573648332b89025965ab8>, size_bytes: 158 }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, description: "Find interpreter for constraints: CPython==3.9.*", level: Debug, append_only_caches: {CacheName("pex_root"): CacheDest(".cache/pex_root")}, jdk_home: None, platform_constraint: None, is_nailgunnable: false, cache_scope: PerRestartSuccessful }
15:46:09.80 [DEBUG] Completed: Find interpreter for constraints: CPython==3.9.*
15:46:09.80 [DEBUG] Completed: Scheduling: Find interpreter for constraints: CPython==3.9.*
15:46:09.80 [DEBUG] Completed: Find Python interpreter for constraints
15:46:09.80 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex_component
15:46:09.80 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
15:46:09.80 [DEBUG] Completed: pants.backend.python.lint.black.rules.setup_black
15:46:09.80 [DEBUG] Completed: Format with Black
15:46:09.80 [DEBUG] Completed: `fmt` goal
15:46:09.80 [DEBUG] computed 1 nodes in 0.422410 seconds. there are 1501 total nodes.
15:46:09.80 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 236, in _run_inner
    return self._perform_run(goals)
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 175, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 192, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 538, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 506, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython==3.9.*' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 483, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 539, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 655, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
    return runner()
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/bin/pex.py", line 1149, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/bin/pex.py", line 895, in build_pex
    interpreters = list(
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 139, in iter_compatible_interpreters
    for interpreter_or_error in _iter_interpreters():
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 104, in _iter_interpreters
    for interp in PythonInterpreter.iter_candidates(
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 570, in iter_candidates
    for interpreter in cls._filter(iter_interpreters()):
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 922, in _filter
    for interp in pythons:
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 551, in iter_interpreters
    for candidate in cls._find(
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/jobs.py", line 431, in execute_parallel
    yield spawn_result.spawned_job.await_result()
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/jobs.py", line 220, in await_result
    return result_func(stdout)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 667, in create_interpreter
    raise cls.IdentificationError("Could not establish identity of {}.".format(binary))
pex.interpreter.IdentificationError: Could not establish identity of /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper.
Any help would be appreciated. Thanks
f
what command were you running?
try re-rerunning it with 
--pex-verbosity=3
. this will enable some debug logging from pex. (full trace logging from pex would be 
--pex-verbosity=9
but let’s start with debug logging)
also what version of Pants?
ah 2.7.0 based on the log
my theory is that Pants/Pex has cached information about Python on your system that is not applicable after the upgrade. wiping the Pants cache may solve the issue. `rm -rf ~/.cache/pants/lmdb_store`will wipe that cache.
👍 1
a
wow, that was a quick response, thanks. I did try removing lmdb_store and still get the error. with 
--pex-verbosity=3
I get the following:
Copy code
stdout:

stderr:
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython<4,>=3.6'], Matching Interpreter: /Users/sinisaderasevic/.pyenv/versions/3.9.7/bin/python3.9
pex: Hashing pex
pex: Hashing pex: 28.2ms
pex: Isolating pex
pex: Isolating pex: 0.0ms
pex: Executing: PYTHONPATH=/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/isolated/3a02dad6854f8902a33d01a33e91e9bec363375f /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper -s -c import os
import sys

from pex.common import atomic_directory, safe_open
from pex.interpreter import PythonIdentity


encoded_identity = PythonIdentity.get(binary='/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper').encode()
sys.stdout.write(encoded_identity)
with atomic_directory('/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/interpreters/0da171869419d89cf47b8965046d839e73b6679c/168f8054e40d67e95eb73294da7d466eedc1502a/11d4ff0f20894e4107f3a046978d90be5e64ba51', exclusive=False) as cache_dir:
    if not cache_dir.is_finalized:
        with safe_open(
            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'
        ) as fp:
            fp.write(encoded_identity)

pex: Executing: PYTHONPATH=/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/isolated/3a02dad6854f8902a33d01a33e91e9bec363375f /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper -s -c import os
import sys

from pex.common import atomic_directory, safe_open
from pex.interpreter import PythonIdentity


encoded_identity = PythonIdentity.get(binary='/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper').encode()
sys.stdout.write(encoded_identity)
with atomic_directory('/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/interpreters/0da171869419d89cf47b8965046d839e73b6679c/168f8054e40d67e95eb73294da7d466eedc1502a/11d4ff0f20894e4107f3a046978d90be5e64ba51', exclusive=False) as cache_dir:
    if not cache_dir.is_finalized:
        with safe_open(
            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'
        ) as fp:
            fp.write(encoded_identity)

Traceback (most recent call last):
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 483, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 539, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 655, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
    return runner()
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/bin/pex.py", line 1149, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/bin/pex.py", line 895, in build_pex
    interpreters = list(
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 139, in iter_compatible_interpreters
    for interpreter_or_error in _iter_interpreters():
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 104, in _iter_interpreters
    for interp in PythonInterpreter.iter_candidates(
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 570, in iter_candidates
    for interpreter in cls._filter(iter_interpreters()):
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 922, in _filter
pex: Executing: PYTHONPATH=/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/isolated/3a02dad6854f8902a33d01a33e91e9bec363375f /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper -s -c import os
import sys

from pex.common import atomic_directory, safe_open
from pex.interpreter import PythonIdentity


encoded_identity = PythonIdentity.get(binary='/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper').encode()
sys.stdout.write(encoded_identity)
with atomic_directory('/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/interpreters/0da171869419d89cf47b8965046d839e73b6679c/168f8054e40d67e95eb73294da7d466eedc1502a/11d4ff0f20894e4107f3a046978d90be5e64ba51', exclusive=False) as cache_dir:
    if not cache_dir.is_finalized:
        with safe_open(
            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'
        ) as fp:
            fp.write(encoded_identity)

    for interp in pythons:
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 551, in iter_interpreters
    for candidate in cls._find(
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/jobs.py", line 431, in execute_parallel
    yield spawn_result.spawned_job.await_result()
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/jobs.py", line 220, in await_result
    return result_func(stdout)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/interpreter.py", line 667, in create_interpreter
    raise cls.IdentificationError("Could not establish identity of {}.".format(binary))
pex.interpreter.IdentificationError: Could not establish identity of /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper.
pex: Executing: PYTHONPATH=/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/isolated/3a02dad6854f8902a33d01a33e91e9bec363375f /usr/bin/python3 -s -c import os
import sys

from pex.common import atomic_directory, safe_open
from pex.interpreter import PythonIdentity


encoded_identity = PythonIdentity.get(binary='/usr/bin/python3').encode()
sys.stdout.write(encoded_identity)
with atomic_directory('/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/interpreters/0da171869419d89cf47b8965046d839e73b6679c/5f42128bc6a36bf22b44d4d69e18ee6daff6178b/97d4f4bb286ebcc33dfa013e7072aef619dd9397', exclusive=False) as cache_dir:
    if not cache_dir.is_finalized:
        with safe_open(
            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'
        ) as fp:
            fp.write(encoded_identity)
f
does that 
pythonwrapper
file even still exist? (
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper
)
a
Yeah it does, it's there
f
try wiping the Pants named caches as well: 
rm -rf ~/.cache/pants/named_caches
I’m unfortunately not too familiar with pex and so I am trying to guess where it caches its resolutions.
a
I've googled a bit what this file can be, and apparently /usr/bin/python is a symlink to this
I wanted to create a symlink from usr/bin/python to my python3, but this can't be done. At least not easily
f
you can adjust the paths used to find python in pants.toml, no need to symlink anything
but this seems like pex shouldn’t fail if there is a problem trying to evaluate an interpreter
a
that was it
I set 
interpreter_search_paths
not to incoude /usr/bin/python and it works now
Thanks a lot. I would've spend hours googling this. you saved me a great deal of pain
👍 1
❤️ 1
h
Thank you for reporting it! Now the next step is figuring out how to fix this so that other users aren't confused To clarify, was the fix to delete the named caches directory? Or to set interpreter_search_paths?
a
The fix was to put the following in the pants.toml file:
Copy code
interpreter_search_paths = [
    # This will use all interpreters in `$(pyenv root)/versions`.
    "<PYENV>"
]
Basically what was already explained in the official website on this link.
👍 2
❤️ 1
f
e
@adorable-spoon-41460 if you're still about, what happens when you run 
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper
directly on the command line?
a
I get this:
Copy code
pythonwrapper[3129:20922] pythonwrapper is not supposed to be executed directly. Exiting.
😂 1
e
Aha, thank you.
h
is there a way to set different 
interpreter_search_paths
depending on the os? our CICD fails because of this i believe:
Copy code
Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<3.10,>=3.7' failed with exit code 1.
stdout:

stderr:
Failed to find compatible interpreter on path /usr/local/bin.
h
currently, the best solution is using 
--pantsrc-files
, which has defaults of:
Copy code
default: [
          "/etc/pantsrc",
          "~/.pants.rc"
      ]
Or using an env variable, e.g. loading it with 
direnv
and an 
.envrc.macos
file
🙌 1
e
@adorable-spoon-41460 as far as I can tell, the only way this can happen is if running 
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper
directly produces the output you quoted to STDERR, outputs nothing to STDOUT and exits 0. If you get a chance to confirm, I appreciate it.
I'm pretty darn sure that has to be correct, it leads to a repro anyhow: https://github.com/pantsbuild/pex/pull/1500
a
Sure, just tell me what do you need me to do to check this and I'll be happy to do it. I am not sure if I am overstaying my welcome here by writing the next issue I have now, sorry. I tried solving it on my own. I've deleted all the files related to pants and downloaded a script again, and ever tried playing with pex_bootstrapper.py, but no luck. Well, here it goes. Now I have commands like 
fmt
, 
lint
, 
package
, 
typecheck
working fine, but 
test
is going crazy. When I try to run a couple of tests (a folder containing 4-5 of them), it works fine. But, when I run the folder containing more, it just keeps doing:
Copy code
Filesystem changed during run: retrying ... in 500ms...
and at the end I get this:
Copy code
RuntimeError: The venv for /private/var/folders/_p/m9ryyzks3qs7g0nqd_k4r0t80000gn/T/process-execution3YzrAk/.tmp/tmp8bw5vchj at /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/85b4f4e93ae5ba2da0e0900056335af23364a65c/f09295b5168c89cd3d8efa8947efb26fa9d0b422 has hash collisions with 33 other venvs!
I saw in the code that these collisions happen when symlink to venv for pex runner gets created starting from 8 characters of the generated hash until the end. My guess is that somehow different pex runners generate the same hash for venv. I tried playing with pex .py files trying to do smoe quick fix, but was not able to do it. I am just not knowledgable enough yet. If it's now too much to ask, can I get help with this as well ? Sorry for the Long Post, here’s a Potato 🥔
e
The hash collision may be a red herring or at least secondary. The 
Filesystem changed ...
continually happening is only likely if one of the tests modifies files in the workspace. Do you know if this is the case?
a
Not as far as I know. It used to work without issues before OS upgrade.
e
Ok... lets try drilling in on the 
Filesystem changed ...
1st. Can you do these in order?: 1. 
rm -rf .pants.d
2. 
pkill pantsd
(or however you'd like to kill it) and ensure there is no pantsd process. 3. 
./pants test ...
you should see initial log lines containg 
Initializing scheduler...
and 
Scheduler initialized.
if step 2 worked. If you don't see those two 1st lines, try step two again. 4. report the full contents of 
.pants.d/pants.log
a
No useful info in the log file. However, stdout:
Copy code
22:15:36.87 [INFO] Initializing scheduler...
22:15:37.00 [INFO] Scheduler initialized.
22:15:40.00 [INFO] Completed: Composing 12 requirements to build requirements.pex from repository.pex
22:15:40.70 [INFO] Completed: Building pytest_runner.pex
22:15:40.72 [INFO] Canceled: Building pytest_runner.pex
22:15:40.72 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 236, in _run_inner
    return self._perform_run(goals)
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 175, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 192, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 538, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/sinisaderasevic/.cache/pants/setup/bootstrap-Darwin-arm64/2.7.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 506, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 404, in ensure_venv
    os.symlink(venv_dir, short_venv_path)
FileExistsError: [Errno 17] File exists: '/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b' -> '/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/short/0a22e09b2ac391980be39a125d20dcfc59451c76'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 483, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 539, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 655, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
    return runner()
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/bin/pex.py", line 1165, in main
    seed_info = seed_cache(options, pex, verbose=options.seed == Seed.VERBOSE)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/bin/pex.py", line 1219, in seed_cache
    venv_pex = ensure_venv(pex)
  File "/Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 411, in ensure_venv
    raise RuntimeError(
RuntimeError: The venv for /private/var/folders/_p/m9ryyzks3qs7g0nqd_k4r0t80000gn/T/process-executionRETbSy/.tmp/tmprnzhbipe at /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b has hash collisions with 33 other venvs!
1.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
2.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
3.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
4.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
5.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
6.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
7.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
8.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
9.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
10.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
11.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
12.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
13.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
14.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
15.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
16.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
17.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
18.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
19.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
20.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
21.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
22.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
23.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
24.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
25.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
26.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
27.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
28.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
29.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
30.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
31.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
32.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
33.) /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs/48ceaa27d6c5227e92829223e66a63f923e74b08/7844231537c9dbf01a552110f465f35678a1a22b
e
Hrm. Can you do this and then re-try?:
Copy code
mv /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root/venvs.sav
a
After moving the folder and retrying, the outcome is the same. venvs folder is recreated and again, after a lot of retries due to filesystem changed, I get the same errors at the end.
e
Alright - this is making no sense at all to me yet. The 1-33 list is strange beyond the occurence - the real chances of this are on the vague order of 1 / # atoms in universe - each entry in the collisions list should be different, but all are the same and they == the venv in question; so this seems to strongly indicate a bug in Pex. I'll file something here in a bit and link.
Still noodling on how this could possibly happen: https://github.com/pantsbuild/pex/issues/1504
Alright, there is only one possibility on the Pex side of things. I'll work on a fix. That said, the current error is just masking another error. Pants is failing to to create a venv from a Pex and that is just being masked here by the collision error. Its unclear to me how the 1st 33 failures to create a venv aren't surfacing on stderr for you.
I'm sorry for your continued pain here. Although I have a fix for the consfusing 33 Pex collisions error, once that is used in Pants, it will only help expose the real error. It would be great to shorten up the cycle though and figure out that underlying error sooner. @adorable-spoon-41460 as such, if you could try: 1. 
rm -rf /Users/sinisaderasevic/bull/apollo/.cache/pants/named_caches/pex_root
2. 
./pants --no-pantsd --no-process-execution-local-cache -ldebug --pex-verbosity=9 <rest of your pants command>
And report the full output of step 2, that should reveal the true error.
a
Sorry for not getting back at you sooner. Was out of the office. I just executed the tests with full logging. Here's the log:
log
e
@adorable-spoon-41460 not a problem. Thanks for that log. It looks like you're using Pants 2.7.0. The 1st step would be to upgrade to 2.7.1 which fixes some known issues that may be in-play. With 2.7.1 you still need to exclude the 
pythonwrapper
from your interpreter search path though. If you try 2.7.1rc2 (I just released this, but haven't announced) - that is likely the best bet. If you could try that upgrade and report back, I'd be grateful.
👍 1
a
I've tried with 2.7.1 and 2.7.1rc1 and still get the same error. It says that 2.7.1rc2 is not available.
e
a
I'm on it. I've deleted all the folders that I know are related to pants: • ~/.pex • ~/.cache/pants • project/.cache • project/.pants.d • project/.pids I've set the version to 2.7.2rc1, ran the command and ... The logs started at 215849.65, and now is 222443.47, and it's still going strong with retries due to Filesystem changed. venvs folder is now 300MB and steadily growing.
Wait, it just completed all the tests successfully after half an hour
e
The 1st resolve for a repo will be slow. Presumably no slower than Pip though for the same resolve.
So, it sounds like you're unblocked then?
a
Well, I was using 
pytest
directly instead of 
pants test
to avoid having to wait. This was my workaround. I also tried 2.8.0rc1 to see if I can overcome this, but no luck there as well. I just wanted to contribute, so that if some1 else starts having the same issues, they will be dealt with. No biggie, I will keep using 
pytest
for now. Maybe this is something to do with a new mac OS, I've no idea. I am the only one in my team to upgrade to new Monterey and only one having these issues. No1 else wanted to upgrade because of this 🙂 Let's see if this is fixed with a new version of mac OS
e
For 2.8.x you need 2.8.0rc2 - this one: https://pypi.org/project/pantsbuild.pants/2.8.0rc2/
To be clear, the 2.7.2rc1 and 2.8.0rc2 releases were just done (by me). In part to fix exactly the Monterey issues with 
pythonwrapper
. Thanks for testing.
a
No problem, thanks for dedicating time to try to help me out. I owe you a 🍺
8 Views
Open in Slack
PreviousNext
Powered by