https://pantsbuild.org/ logo
#general
Title
# general
p

plain-river-51682

08/26/2020, 12:18 PM
Hey, after updating from pants 1.25 to 1.29 we see this error sometimes in our CI runs:
Copy code
Build graph construction failed: ExecutionError 1 Exception encountered:
  Exception: Snapshot failed: Throw { val: Error { kind: Io(Os { code: 28, kind: Other, message: "No space left on device" }), paths: [] }, python_traceback: "Traceback (no traceback):\n  <pants native internals>\nException: Error { kind: Io(Os { code: 28, kind: Other, message: \"No space left on device\" }), paths: [] }", engine_traceback: ["Fingerprinting: <SOME SCALA FILE>.scala"] }
pants is invoked in gitlab-ci inside a docker container. The host has enough diskspace available. Where is it trying to write?
j

jolly-midnight-72759

08/26/2020, 2:16 PM
@plain-river-51682 A couple of questions: • Does the job work sometimes on your CI and fails sometimes for the same tests/builds? • What is your setting for
pants_workdir
? • Is the repo mounted as read-only in the docker container?
p

plain-river-51682

08/26/2020, 2:19 PM
Does the job work sometimes on your CI and fails sometimes for the same tests/builds?
yes, when we retry the same job, it suddenly succeeds. Though every retry gets checked out in a new docker container
Is the repo mounted as read-only in the docker container?
the repo and all paths are fully writeable
What is your setting for 
pants_workdir
?
we havent touched this, so I guess it's using the default?
j

jolly-midnight-72759

08/26/2020, 2:20 PM
The default is
${REPO_PATH}/.pants.d
. This shouldn't be an issue if the repo is read-write mounted.
p

plain-river-51682

08/26/2020, 2:21 PM
Ya, this path is writable.
j

jolly-midnight-72759

08/26/2020, 2:22 PM
Under macOS we've had to enter a custom
ulimit
(17000). I see
docker run
has an option to set this for containers (
--ulimit rtprio=<value>
). It is a stab in the dark, but if it is easy to set there is no harm in trying.
Without this set on our laptops we get a very different error message, so I'm not too hopeful that it would be that simple to fix.
w

witty-crayon-22786

08/26/2020, 5:32 PM
somewhere between 1.26 and 1.29 we started using inotify by default
p

plain-river-51682

08/27/2020, 11:43 AM
that might be it, we give it a try!
5 Views