Running into a weird caching issue when trying to pipe things:

./pants dependees --transitive some/file.py | xargs ./pants filter --target-type=python_tests --granularity=file

If I run this once, it takes a while to compute, which I expect. But, on subsequent tries it looks like Pants usually needs to completely recompute the depedency graph every time I run it. (Well, using

-ldebug

shows that it's hitting cache for each module but it just takes a while to traverse the whole graph, even if it's using cache each time). My question is... why doesn't it just remember the final output of that recent

./pants dependees

command? Does this have something to do with pantsd?

which version of Pants? there was a fix for bad behavior there recentlyish

older, i'm on 2.7.1rc1

https://github.com/pantsbuild/pants/pull/13028 … so it was actually backported i think

yes

hm, yea: that should have been fixed by #13028. are any other options changing between the two runs?

nope, I can typically reproduce this just by pushing ⬆️ and [enter] immediately to just re-run the command

i mean: between the two goals that are piped between one another

yes, pantsd seems to be restarting between these runs

what does the log say?

ahah

15:55:08.28 [WARN] Completed: Find Python interpreter to bootstrap PEX - No bootstrap Python executable could be found from the option `interpreter_search_paths` in the `[python-setup]` scope. Will attempt to run PEXes directly.
15:55:16.62 [ERROR] The scheduler was invalidated: Exception('pantsd process 433454 was using 1026.92 MiB of memory (above the `--pantsd-max-memory-usage` limit of 1024.00 MiB).')
15:55:17.19 [ERROR] service failure for <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7ef308554ee0>.
15:55:17.20 [INFO] Waiting for ongoing runs to complete before exiting...

hm… yea, that’s very surprising for only

dependees

+

filter

…

we have an enormous dep graph, so it would make sense that we would hit this

hm. that’s unexpected. is the machine (vm/container) constrained in any way?

no... I think I'm hitting a pantsd bug tbh, and it seems to be triggered by the filter command

should be fine… in `pantsbuild/pants`:

$ time ./pants list :: | xargs ./pants filter --target-type=python_tests | wc -l
     131

real	0m3.786s
user	0m0.416s
sys	0m0.442s
$ time ./pants list :: | wc -l
    1412

real	0m0.788s
user	0m0.162s
sys	0m0.208s

yeah in looking for reproducers it's specifically passing lots of targets to filter that's triggering the degrading pantsd behavior

does it render a

reinitializing

line…?

that's just the way I've gotten it to work here, the same behavior happens with xargs

if you’re able to privately gist me the

-ldebug

output of running

filter

repeatedly, that would be helpful. there were some minor fixes in this area, and i’m wondering if they’re relevant. but it’s still unexpected 1) to use that much memory, 2) for things to get slower over time

Is there a way to send log output to a file while preserving the interactive output on stderr?

Awesome! Yea, there were some improvements for invalidation.