https://pantsbuild.org/ logo
Join Slack
Powered by
Can I get some help making sense of this error? It...
# general
h
Can I get some help making sense of this error? It's coming up after migrating to 2.17.0. The command is just a simple 
pants package <some pex>
Copy code
23:36:29.59 [WARN] Making attempt 1 to backtrack and retry `Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,>=41.0, hvac<1,>=0.11.2, pexpect<5,>=4.4.0, pika<2,>=1.2.1, protobuf==3.19.5, psutil<6,>=5.8.0, pyjwt<3,>=2, pyserial<4,>=3.4, python-can<4,>=3.1.1, pyudev<1,>=0.24.0, pyyaml<6,>=5.4.1, pyzmq<26,>=25.1, requests<3,>=2.28, setuptools<66,>=65.5.1, tabulate<1,>=0.8.1, typeguard<3,>=2.12.1, types-PyYAML<=6.0.12.6,>=6.0.0, types-click<8,>=7.0.0, typing-extensions<5,>=4.1.1, yubihsm[http]<3,>=2.1`, due to missing digest Digest { hash: Fingerprint<2427f1221b498b758272f1493ffcd7750962e363d32f933b3b80cddcd3d33107>, size_bytes: 14208017 }.
23:36:29.59 [INFO] Dirtying @rule(pants.engine.process.fallible_to_exec_result_or_raise())
23:36:29.59 [INFO] Dirtying @rule(pants.backend.python.util_rules.pex.build_pex(astranis.sidekick.automated_remote_io_test/buildbot_robin.pex))
23:36:29.59 [INFO] Dirtying @rule(pants.backend.python.util_rules.pex.create_pex())
23:36:29.59 [INFO] Dirtying @rule(pants.backend.python.goals.package_pex_binary.built_pacakge_for_pex_from_targets_request(astranis/sidekick/automated_remote_io_test:buildbot_robin))
23:36:29.59 [INFO] Dirtying @rule(pants.core.goals.package.environment_aware_package())
23:36:29.59 [INFO] Dirtying @rule(pants.core.goals.package.package_asset())
23:36:29.59 [INFO] Dirtying Package
23:36:29.59 [INFO] Filesystem changed during run: retrying `@rule(pants.core.goals.package.package_asset())` in 500ms...
23:36:30.09 [INFO] Starting: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.09 [INFO] Completed: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.11 [INFO] Starting: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.16 [INFO] Completed: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.18 [WARN] Making attempt 1 to backtrack and retry `Building local_dists.pex`, due to missing digest Digest { hash: Fingerprint<fe8697b96cf36d668c6c41fa27b4f06f1202579e313cac5de6cae1a3040572d4>, size_bytes: 607533 }.
23:36:30.18 [INFO] Dirtying @rule(pants.engine.process.fallible_to_exec_result_or_raise())
23:36:30.18 [INFO] Dirtying @rule(pants.backend.python.util_rules.pex.build_pex(local_dists.pex))
23:36:30.18 [INFO] Dirtying @rule(pants.backend.python.util_rules.pex.create_pex(local_dists.pex))
23:36:30.18 [INFO] Dirtying @rule(pants.backend.python.util_rules.local_dists.build_local_dists())
23:36:30.18 [INFO] Dirtying @rule(pants.backend.python.util_rules.pex_from_targets.create_pex_from_targets())
23:36:30.18 [INFO] Dirtying @rule(pants.backend.python.util_rules.pex.create_pex())
23:36:30.18 [INFO] Dirtying @rule(pants.backend.python.goals.package_pex_binary.built_pacakge_for_pex_from_targets_request(astranis/sidekick/automated_remote_io_test:buildbot_robin))
23:36:30.18 [INFO] Dirtying @rule(pants.core.goals.package.environment_aware_package())
23:36:30.18 [INFO] Dirtying @rule(pants.core.goals.package.package_asset())
23:36:30.18 [INFO] Dirtying Package
23:36:30.18 [INFO] Filesystem changed during run: retrying `Process(Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,>=41.0, hvac<1,>=0.11.2, pexpect<5,>=4.4.0, pika<2,>=1.2.1, protobuf==3.19.5, psutil<6,>=5.8.0, pyjwt<3,>=2, pyserial<4,>=3.4, python-can<4,>=3.1.1, pyudev<1,>=0.24.0, pyyaml<6,>=5.4.1, pyzmq<26,>=25.1, requests<3,>=2.28, setuptools<66,>=65.5.1, tabulate<1,>=0.8.1, typeguard<3,>=2.12.1, types-PyYAML<=6.0.12.6,>=6.0.0, types-click<8,>=7.0.0, typing-extensions<5,>=4.1.1, yubihsm[http]<3,>=2.1)` in 500ms...
23:36:30.69 [INFO] Starting: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.69 [INFO] Completed: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.73 [INFO] Starting: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.73 [INFO] Completed: Building 23 requirements for astranis.sidekick.automated_remote_io_test/buildbot_robin.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, cryptography<42.0,... (393 characters truncated)
23:36:30.74 [INFO] Starting: Building local_dists.pex
23:36:30.74 [INFO] Completed: Building local_dists.pex
23:36:30.75 [INFO] Starting: Building local_dists.pex
23:36:30.75 [INFO] Completed: Building local_dists.pex
23:36:30.76 [INFO] Starting: Building local_dists.pex
23:36:30.76 [INFO] Completed: Building local_dists.pex
23:36:30.76 [INFO] Starting: Building local_dists.pex
23:36:30.76 [INFO] Completed: Building local_dists.pex
23:36:30.77 [INFO] Starting: Building local_dists.pex
23:36:30.77 [INFO] Completed: Building local_dists.pex
23:36:30.79 [INFO] Starting: Building local_dists.pex
23:36:30.79 [INFO] Completed: Building local_dists.pex
23:36:30.80 [INFO] Starting: Building local_dists.pex
23:36:30.80 [INFO] Completed: Building local_dists.pex
23:36:30.81 [INFO] Starting: Building local_dists.pex
23:36:30.81 [INFO] Completed: Building local_dists.pex
23:36:30.82 [ERROR] 1 Exception encountered:
Engine traceback:
  in `package` goal
IntrinsicError: Could not identify a process to backtrack to for: Missing digest: Was not present in the local store: Digest { hash: Fingerprint<faad51a6a108fba9d40b2a10e82a2646fccbaf8c3d9be47818f4bffae02d94b8>, size_bytes: 4098329 }
We switched to 2.17.0 and started using 
pants
over the in-repo script at the same time.
b
I wonder if this might be related to the FSDB work, @bitter-ability-32190? @high-yak-85899 does it work if you run 
PANTS_LOCAL_STORE_DIR=${mktemp -d) pants package ...
, to run with a completely fresh cache? (I'm not suggesting deleting your real cache, because keeping that state might be good to help reproduce...)
h
This was happening in CI, so a little hard to test
b
ah, okay 👍
Given the lack of (useful) response here so far, could you file a bug on github so this doesn't get forgotten in slack? Thanks!
g
Had this happen as well!
👀 1
b
Copy code
Could not identify a process to backtrack to for
Hmm I remember a slew of these a while ago in development. @witty-crayon-22786 can you help me remember when these are issued
h
Surprisingly, even switching to 
pants run
with no 
pex_binary
defined resulted in the same failure.
b
Are there any other secondary owners of that file?
h
pants list
only shows the one file
Presumably meaning it's only owned by the nearest 
python_sources
target
b
Wait, back up. Which failure?
h
This is switching 
pants package <thing>
to 
pants run <thing>
after removing the 
pex_binary
approach.
Untitled
b
Oh the missing digest. OK. I think the warning you saw and this are likely unrelated
h
Should we try to just nuke the cache on this machine? I'm not able to reproduce the package or run failures elsewhere.
Huon, your recommendation seemed to work reliably. We're going to proceed with wiping our cache for now to see if things become more stable. I'll work on getting a ticket up.
oof, we wiped our cache and restarted the system and immediately saw the same issue 😢
😢 1
Bug filed over at https://github.com/pantsbuild/pants/issues/19748. We reverted to 2.16 for the time.
👍 1
b
Thanks! Is there any info in your 
pants.toml
you can share (preferably the whole thing, but at least any cache configuration options)?
h
Oh yeah, I meant to include that detail.
We're using 
fetch
right now for cache content behavior.
b
Ah, so you have a remote cache too?
h
We do. We use 
bazel-remote-cache
for that
👍 1
3 Views
Open in Slack
PreviousNext
Powered by