Agreed that the first priority should be removing use of `Fi

Question

Agreed that the first priority should be removing use of `FileContent` The thing is I have no idea what is causing that to get persisted I don t see it anywhere in the outputted graph or in our code related to Pytest

average-vr-56795 · Answer

We can try removing your new uses at least D <https github com pantsbuild pants pull 7739>

hundreds-father-404 · Answer

Yay Although I don t think I had added a new use here With the new inject rule we go from Snapshot gt ExecuteProcessRequest gt Digest and avoid needing FilesContent

hundreds-father-404 · Answer

Any ideas for how to figure out why we re serializing the entire files contents given that the graph doesn t include `FileContent` Notably we do serialize the `BUILD` file for ` pants list` just it s always never more than one file Looking at the rule graph I m wondering if `Validate` is happening

hundreds-father-404 · Answer

Hm update Realized I had not been running with the inject init rule PR applied It does stop us from materializing `FileContent` for sources which is great The remaining issue is that we materialize each `BUILD` file involved in the sources

average-vr-56795 · Answer

Yeah that ll be in `parse address family`

average-vr-56795 · Answer

So I guess it comes down to We need the graph to only reference the digest and for the bytes to only be handed to the rule outside the graph somehow

hundreds-father-404 · Answer

gt Yeah that ll be in `parse address family` Is the only fix for this to move that logic to Rust Could we possibly do that Sending materialized files seems very costly

hundreds-father-404 · Answer

gt and for the bytes to only be handed to the rule outside the graph somehow Have Python use `with open ` to grab the bytes itself

average-vr-56795 · Answer

I m not entirely sure how we d do it tbh I think < witty crayon 22786> mentioned the possibility yesterday though

witty-crayon-22786 · Answer

Well materializing them is one thing Keeping them around for long periods is another

witty-crayon-22786 · Answer

If they re nowhere in the Graph it might be that we have Keys for them

witty-crayon-22786 · Answer

but probably not We should only create Keys for things that we are referencing from the Graph

witty-crayon-22786 · Answer

One important thing to differentiate is whether there is a Handle that points directly to the object vs the object being held by another python object

witty-crayon-22786 · Answer

For example if the AddressMapper was holding onto BUILD files after parsing them

hundreds-father-404 · Answer

So we would expect that ` pants list` also should not be keeping the `BUILD` file contents in the handles at the end of its run right If so I ll focus on solving that because it s a simpler rule and would also by extension fix this `O b ` behavior for test runner where `b` is number of BUILD files

witty-crayon-22786 · Answer

I m not sure that one is a blocker though filing a ticket would be good but the total size of all BUILD files in most repos is ~200MB at most

witty-crayon-22786 · Answer

but yea we wouldn t expect the content to be held because that intrinsic doesn t even have an associated Node in the Graph we lift the content and then pass it directly to the rule

hundreds-father-404 · Answer

Good idea to confirm first I ll try a CI run again with init rule applied `O t + b ` is very different than `O t + n ` keeping all the BUILD files in memory vs the entire repo in memory

witty-crayon-22786 · Answer

Can also compare memory usage locally by running with enable pantsd and watching top But can do both in parallel