https://pantsbuild.org/ logo
Join Slack
Powered by
FYI ASU issue when installing a lock file. I unfor...
# development
h
FYI ASU issue when installing a lock file. I unfortunately do not have time to investigate this right now. Does anyone else reproduce?
Copy code
❯ ./pants test src/python/pants/backend/python/util_rules/lockfile_metadata_test.py
Copy code
Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.pytest_runner.run_python_test (src/python/pants/backend/python/util_rules/lockfile_metadata_test.py:tests)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/ericarellano/code/pants/src/python/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building 11 requirements for requirements.pex from the 3rdparty/python/user_reqs.lock resolve: PyYAML<7.0,>=6.0, ansicolors==1.1.8, packaging==21.3, pex==2.1.77, pytest<7.1.0,>=6.2.4, setuptools<58.0,>=56.0.0, toml==0.10.2, types-PyYAML==6.0.3, types-setuptools==57.4.7, types-toml==0.10.3, typing-extensions==4.0.1' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/fetcher.py", line 79, in get_body_stream
    yield body_stream
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 67, in save
    for chunk in iter(lambda: stream.read(self._BUFFER_SIZE), b""):
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 67, in <lambda>
    for chunk in iter(lambda: stream.read(self._BUFFER_SIZE), b""):
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 463, in read
    n = self.readinto(b)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 507, in readinto
    n = self.fp.readinto(b)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/ee5e123a2c302f34cbb0831c696b2134df893f39/.bootstrap/pex/pex.py", line 504, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/ee5e123a2c302f34cbb0831c696b2134df893f39/.bootstrap/pex/pex.py", line 409, in _wrap_coverage
    return runner(*args)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/ee5e123a2c302f34cbb0831c696b2134df893f39/.bootstrap/pex/pex.py", line 440, in _wrap_profiling
    return runner(*args)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/ee5e123a2c302f34cbb0831c696b2134df893f39/.bootstrap/pex/pex.py", line 560, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/ee5e123a2c302f34cbb0831c696b2134df893f39/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/ee5e123a2c302f34cbb0831c696b2134df893f39/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/bin/pex.py", line 739, in main
    do_main(
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/bin/pex.py", line 760, in do_main
    pex_builder = build_pex(
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/bin/pex.py", line 609, in build_pex
    resolve_from_lock(
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 305, in resolve_from_lock
    pool.map(
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 163, in download_artifact
    return catch(
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/result.py", line 101, in catch
    return func(*args, **kwargs)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/resolve/lockfile/download_manager.py", line 152, in store
    self.save(
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/installed_wheels/607cd8973e976e388537a962f1c47db79acb93c0c37055a6b47e4fa99c13442a/pex-2.1.77-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 69, in save
    digest.update(chunk)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 169, in __exit__
    raise RuntimeError("generator didn't stop after throw()")
RuntimeError: generator didn't stop after throw()
pn this branch: https://github.com/pantsbuild/pants/pull/15074. Was trying to reproduce the CI failure
e
socket.timeout: The read operation timed out
So retry in CI is fine. I'll file a Pex issue to unmask the real error in these cases.
h
hm in CI i see https://github.com/pantsbuild/pants/runs/5887288678?check_suite_focus=true#step:14:129
Copy code
15:05:49.82 [ERROR] Completed: Run Pytest - src/python/pants/backend/python/util_rules/lockfile_metadata_test.py:tests failed (exit code 2).
130
/home/runner/.cache/pants/named_caches/pex_root/venvs/s/200996e5/venv/bin/python3.7: can't open file '/home/runner/.cache/pants/named_caches/pex_root/venvs/d42841a31cba806a3a8511be281c9bb745f3a197/c0d45e5f0038cae13f87b120ee9e80b473618a0b/pex': [Errno 2] No such file or directory
131
ah, To clarify, the socket timeout is from my local Mac
Two separate issues
e
Not seperate IIUC. The 1st leads to Pex erroring converting the exception to an 
Error
instance: https://github.com/pantsbuild/pex/issues/1715
Oh, gotcha. Well - that CI error - no clue.
Yeah, that's reproducible afaict IFF 
/home/runner/.cache/pants/named_caches/pex_root/venvs/d42841a31cba806a3a8511be281c9bb745f3a197/c0d45e5f0038cae13f87b120ee9e80b473618a0b/
exists but is empty. If you then delete the empty 
/home/runner/.cache/pants/named_caches/pex_root/venvs/d42841a31cba806a3a8511be281c9bb745f3a197/c0d45e5f0038cae13f87b120ee9e80b473618a0b/
dir leaving just the parent dir behind ( 
/home/runner/.cache/pants/named_caches/pex_root/venvs/d42841a31cba806a3a8511be281c9bb745f3a197/
) - everything works and the venv re-builds. Baked in to this whole scheme to save 40ms of overhead from running from the 
--venv
PEX file directly is the assumption that the PEX_ROOT won't be tampered with. It can be missing, and a rebuild happens, but if there is a targeted delete of interior contents all bets are off.
Now, it of course seems very improbable there was a targeted delete run in CI, so that leaves open the question of how that (atomic) directory could have no contents or partial contents.
👍 1
but if there is a targeted delete of interior contents all bets are off.
Because, to check for this would require an operation slower than or roughly taking the same time as the 40ms; thus, negating the whole point.
So ... I'd love to understand what happened there, but I suspect a retry will work - or at least terminate in some other error. There is 0 useful information you'll erase with a retry since we've got just that 1 log line to go on.
h
Also it's awesome that github preserves prior runs. There is no downside to re-running CI, except for the very real problem that it allows you to paper over problems that we should solve (guilty)
Open in Slack
PreviousNext
Powered by