https://pantsbuild.org/ logo
Join Slack
Powered by
Hey, I've been getting this error relatively frequ...
# general
s
Hey, I've been getting this error relatively frequently (maybe 5 times per day). The fix is pretty simple - find the pantsd process & kill it. Unfortunately unable to reproduce it reliably enough (it kinda just happens) to put together a meaningful github issue. Is there any diagnostics or something I can look at when it happens?
Copy code
IntrinsicError: Could not identify a process to backtrack to for: Missing digest: Was not present in the local store: Digest { hash: Fingerprint<98b10e5952562b7ff6d281b26e78104b1c0b6cdc224f661bb3276aa1db380d6c>, size_bytes: 51 }
Will also add - we're using a microservices setup with Tilt to automatically rebuild & reload our microservices when their python code changes It seems to happen most when I'm editing code shared between many pex targets (and hence Tilt kicks of a flurry of pants commands to rebuild all the pex)
b
i get the same error often when running tests; i delete the .pants.d folder, rebuild docker and everything works
h
Presumably
if you could add your details to that issue (platform, arch, which pants goals you’re running) that would be great
Sorry for the trouble, this is a vexing bug
A way to consistently reproduce it would be gold
s
no worries - will do!
In case anyone else comes across this, I've had luck with a 
pants-wrapper.sh
script that detects this issue, force kills pantsd, and retries the pants command
Copy code
#!/usr/bin/env bash

set -euo pipefail

retry_once=false
log_file=$(mktemp -t pants-wrapper-stderr.XXXXXX)

cleanup() {
  rm -f "$log_file"
}
trap cleanup EXIT

kill_pantsd() {
  # Try pkill first
  if command -v pkill &>/dev/null; then
    pkill -f 'pantsd \[.*kiid\]' || true
  else
    # Fallback using ps + awk
    ps aux | awk '/pantsd \[.*kiid\]/ { print $2 }' | xargs kill 2>/dev/null || true
  fi
}

run_pants() {
  pants "$@" 2> >(tee "$log_file" >&2)
}

run_pants "$@" || {
  if grep -q "Missing digest: Was not present in the local store" "$log_file" && [ "$retry_once" = false ]; then
    echo "Detected missing digest error. Restarting pantsd..." >&2
    kill_pantsd
    retry_once=true
    run_pants "$@"
  else
    echo "Pants failed without recoverable error." >&2
    exit 1
  fi
}
h
Nice (although not nice that you have to do this)
s
I was thinking... Could 
pants
itself do this? If the pants CLI got the error back, it could kill pantsd & re-run the command Maybe it tells pantsd to dump it's state to some file & exit, and prints a helpful message about attaching the file to the GH issue? (and then re-runs the command)
35 Views
Open in Slack
PreviousNext
Powered by