Seeing this issue frequently, ```... return PyNail...
# general
q
Seeing this issue frequently,
Copy code
...
return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). If you keep seeing this error message, try the troubleshooting steps below. If none of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
 - Exit other applications, including applications running in the background.
 - Set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`).
 - Disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.
I know there have been discussions previously about ensuring enough RAM is available and cleaning cache directories but this does not help unfortunately. Directories I’ve cleared
Copy code
% rm -rf ~/Library/Caches/nce 
% rm -rf .pants.d 
% rm -rf ~/.cache/pants/
Does anyone have any other suggestions?
Some more notes: 1. Trying with v2.20.0, v2.21.0 and v2.22.0a0 I get the same error. 2. When I try with
--no-pantsd
option it says
Copy code
% pants --no-pantsd fmt ::  
18:12:43.27 [INFO] Initializing Nailgun pool for 24 processes...
18:12:51.15 [INFO] Initializing Nailgun pool for 24 processes...
zsh: illegal hardware instruction  pants --no-pantsd fmt ::
3. Trying
pants --pantsd-max-memory-usage=16GiB fmt ::
does not help too and I get the same error about OOM.
w
Can you provide some more info? • How big is your project (hard to define, I guess - maybe start with
pants count-loc ::
) • What OS/hardware are you running? • How frequent is the OOM? • How much available memory do you have when you OOM? • If you're working in a team, is everyone seeing this or just you? • Do you run into this on something small, like
example-python
or similar?
q
1.
pants count-loc ::
fails with the same error. My project is big in terms of disk space ~10GB but most of that is because of data/model files. The actual source code files are few and small. 2. MacOS, Sonoma 14.5 3. Almost every command at the moment. 4. I have a 32GB machine, and ~20GB free. I have close to 90GB of disk available out of 500GB. 5. It was only me but another colleague complained of the same today. 6. I created an example repo to highlight a different issue and things seem to work there. This is a tiny repo.
w
Interesting. I'm running a similar, but slightly older version of the machine - and have never run into the error. You're getting an OOM with the example repo???
q
No OOM in the example repo.
w
Phew... Okay. This doesn't necessarily feel like it's about memory then. Couple more things to check: • How did you install
pants
and is it updated?
brew update
? https://github.com/pantsbuild/scie-pants/releases/tag/v0.12.0 • Do you know if those large files are being pulled into your sandbox?
pants count-loc -ldebug --keep-sandboxes=always ::
<-- Not sure if that'll work if you're OOMing
q
1. Installed pants long back using brew,
brew install pantsbuild/tap/pants
as per this. 2.
pants -ldebug --keep-sandboxes=always count-loc ::
-> did print a bunch of stuff, but eventually hit OOM. How do I figure out if large files are being pulled into the sandbox?
w
Good question, I've never had this problem - so I don't know where you got - if you can share the logs? You can also DM me if you don't want to leak anything. If you're able to open a sandbox, you can see if anything is in there - my assumption is that they should not be in there - but always worth a sanity check
Re: Example repo - why do you have the same build_arg on both
pants.toml
and in the
docker_image
? I feel like that's a problem
q
the example repo issue is resolved 😅 I even created a separate issue for it.
👍 1
w
Trying
pants --pantsd-max-memory-usage=16GiB fmt ::
does not help too and I get the same error about OOM
Just a note that the default is 4GiB, and the recommendation is to try lowering it just to retain less stuff.
Looking at the logs (DM) this bails out before sandboxing or anything - just FYI for anyone wondering
q
I tried reducing the max memory usage but
Copy code
pants --pantsd-max-memory-usage=2GiB fmt ::
fails similarly. Another suggestion from the DM was to try to do it on a subset folder and that works
Copy code
pants fmt src/<project-a>::
so I am guessing some project is the culprit. I’ll try to narrow it down.
w
Yeah, if you can narrow down to the offending folder (or folders),
list
and
count-loc
are your friends here to see how many targets you've got. I'm curious if there is a single offending target, or if collectively all of your targets are causing "something" (too many nodes, memory usage, etc) Pants runs on some pretty big projects, which is why this is such a surprise to me
q
I narrowed down the culprit project but I don’t still know why this project is an issue. Are symlinks in my project an issue for Pants? (thats the only major thing I see as different compared to other projects in the repo)
w
Delete it and find out 🏃‍♂️
😅 1
q
Deleting symlinks did not help 😞
w
Well, at least there is somewhere to start. Does that problematic project do anything else differently? Substantially more files? Larger files? Anything incongruent?
q
This project contains multiple smaller python projects (<10), each of these has a separate BUILD file. All my other projects are just individual projects with a single BUILD file. In terms of source files, this project does not have that many files compared to my other projects that work fine.
w
any cyclic deps?
not sure if pants even allows that
Yeah, pants is smart
Copy code
CycleException: The dependency graph contained a cycle:
-> u-boot-builder:u-boot-builder-2
   u-boot-builder:u-boot-builder
-> u-boot-builder:u-boot-builder-2
q
Hmm, nothing stands out as an issue to me. I’ve not had this issue previously with this project. I’ll have to go fishing in commits where this stopped working 😞
w
👍 Whelp, at least it's narrowed down a bit - please keep us informed if you find anything
q
thanks for your time! appreciate it.
👍 1
The issue was with symlinks but not the ones I had created correctly. A rogue incorrectly created symlink, nested deep in the project and that was self referencing led to this. Deleting it and now everything works as expected. Thanks again SJ!
w
Woot!
I wonder if there would be some way to catch that - but yeah, I can imagine that would have been hard to find 🙂
q
Yea, for some reason my IDE (VSCode) did not show this symlink maybe because it was pointing incorrectly. I could only see it in the terminal. Git diff was also not my friend because I had a bunch of changes, some very close to this file so it got lost in them.