quaint-piano-62770
06/12/2024, 10:31 AM...
return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.
If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). If you keep seeing this error message, try the troubleshooting steps below. If none of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
- Exit other applications, including applications running in the background.
- Set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`).
- Disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.
I know there have been discussions previously about ensuring enough RAM is available and cleaning cache directories but this does not help unfortunately. Directories I’ve cleared
% rm -rf ~/Library/Caches/nce
% rm -rf .pants.d
% rm -rf ~/.cache/pants/
Does anyone have any other suggestions?quaint-piano-62770
06/12/2024, 12:44 PM--no-pantsd
option it says
% pants --no-pantsd fmt ::
18:12:43.27 [INFO] Initializing Nailgun pool for 24 processes...
18:12:51.15 [INFO] Initializing Nailgun pool for 24 processes...
zsh: illegal hardware instruction pants --no-pantsd fmt ::
3. Trying pants --pantsd-max-memory-usage=16GiB fmt ::
does not help too and I get the same error about OOM.wide-midnight-78598
06/12/2024, 1:35 PMpants count-loc ::
)
• What OS/hardware are you running?
• How frequent is the OOM?
• How much available memory do you have when you OOM?
• If you're working in a team, is everyone seeing this or just you?
• Do you run into this on something small, like example-python
or similar?quaint-piano-62770
06/12/2024, 2:11 PMpants count-loc ::
fails with the same error. My project is big in terms of disk space ~10GB but most of that is because of data/model files. The actual source code files are few and small.
2. MacOS, Sonoma 14.5
3. Almost every command at the moment.
4. I have a 32GB machine, and ~20GB free. I have close to 90GB of disk available out of 500GB.
5. It was only me but another colleague complained of the same today.
6. I created an example repo to highlight a different issue and things seem to work there. This is a tiny repo.wide-midnight-78598
06/12/2024, 2:37 PMquaint-piano-62770
06/12/2024, 2:37 PMwide-midnight-78598
06/12/2024, 2:39 PMpants
and is it updated? brew update
? https://github.com/pantsbuild/scie-pants/releases/tag/v0.12.0
• Do you know if those large files are being pulled into your sandbox? pants count-loc -ldebug --keep-sandboxes=always ::
<-- Not sure if that'll work if you're OOMingquaint-piano-62770
06/12/2024, 2:48 PMbrew install pantsbuild/tap/pants
as per this.
2. pants -ldebug --keep-sandboxes=always count-loc ::
-> did print a bunch of stuff, but eventually hit OOM. How do I figure out if large files are being pulled into the sandbox?wide-midnight-78598
06/12/2024, 2:53 PMwide-midnight-78598
06/12/2024, 3:02 PMpants.toml
and in the docker_image
? I feel like that's a problemquaint-piano-62770
06/12/2024, 3:04 PMwide-midnight-78598
06/12/2024, 3:17 PMTryingJust a note that the default is 4GiB, and the recommendation is to try lowering it just to retain less stuff.does not help too and I get the same error about OOMpants --pantsd-max-memory-usage=16GiB fmt ::
wide-midnight-78598
06/12/2024, 3:21 PMquaint-piano-62770
06/12/2024, 3:24 PMpants --pantsd-max-memory-usage=2GiB fmt ::
fails similarly.
Another suggestion from the DM was to try to do it on a subset folder and that works
pants fmt src/<project-a>::
so I am guessing some project is the culprit. I’ll try to narrow it down.wide-midnight-78598
06/12/2024, 3:26 PMlist
and count-loc
are your friends here to see how many targets you've got. I'm curious if there is a single offending target, or if collectively all of your targets are causing "something" (too many nodes, memory usage, etc)
Pants runs on some pretty big projects, which is why this is such a surprise to mequaint-piano-62770
06/12/2024, 3:31 PMwide-midnight-78598
06/12/2024, 3:32 PMquaint-piano-62770
06/12/2024, 3:36 PMwide-midnight-78598
06/12/2024, 3:39 PMquaint-piano-62770
06/12/2024, 3:44 PMwide-midnight-78598
06/12/2024, 3:46 PMwide-midnight-78598
06/12/2024, 3:47 PMwide-midnight-78598
06/12/2024, 3:51 PMCycleException: The dependency graph contained a cycle:
-> u-boot-builder:u-boot-builder-2
u-boot-builder:u-boot-builder
-> u-boot-builder:u-boot-builder-2
quaint-piano-62770
06/12/2024, 3:52 PMwide-midnight-78598
06/12/2024, 3:55 PMquaint-piano-62770
06/12/2024, 3:55 PMquaint-piano-62770
06/12/2024, 4:47 PMwide-midnight-78598
06/12/2024, 4:54 PMwide-midnight-78598
06/12/2024, 4:54 PMquaint-piano-62770
06/12/2024, 4:58 PM