I still need to catch up with the conversation, but it’s on my radar. Will comment here once I’m up to date 🙂

OK. I’m caught up. So this concept of reducing the latency in pants, and in particular when running pantsd is definitely something I’m interested in; although I haven’t written any code yet to go in that direction, so there is no “competing implementation” to worry about. The document Stu mentioned is a high level proposal I wrote to try and identify some bottlenecks when we run pants to “not do much” (conceptually), which still make it pretty unresponsive. The idea of rewriting the Nailgun client in rust is one of the ways envisaged to improve the situation. Here is the proposal: https://docs.google.com/document/d/1xHWok1wgxupt4V_F1RC-YjZVwzAN0-mvrY2ldFoms8c/edit#heading=h.smiin5t86bqb In terms of profiling, I have used py-spy before and found it helpful; but when I tried more recently I had issues with it. That being said, that was on Mac and I find that profiling is easier on Linux in general, so you may have a better time on Arch. In addition to profiling, which can help you find some bottlenecks, it may be useful to use benchmarks to verify that your solutions do bring the expected time savings. I spent a very small amount of time starting something here: https://github.com/pierrechevalier83/bench_pants_no_ops. Feel free to have a look and use it as a starting point if you find it useful. So I haven’t written any code to write a Nailgun client in rust, but feel free to ask me questions as I love rust and code that doesn’t make me want to kill myself because it takes so long to do anything at all; so I’m definitely interested in collaborating on this with early code reviews if nothing else 😄 God speed

@hundreds-breakfast-49010 ^^

I'll take a look at your document

from looking around the code yesterday, I think I now understand more or less what pants is doing from the time we execute the

pants

shell script in the repo, to where the code enters one of

{Local,Remote}PantsRunner

and there definitely seems to be more complexity there than is necessary, although there's also some useful-looking business logic (e.g. the code that checks the locale)

it sounds like we have reason to think that starting the Nailgun client in python is particularly expensive?

I've been playing with py-spy a bit but I haven't used it in that use case yet

I'm also not particularly good at interpreting flame graphs

I’m also not particularly good at interpreting flame graphs

If you want to schedule a call I can walk you through it (probably not today. Before 6pm UK time would be good for me)

It sounds like we have reason to think that starting the Nailgun client in python is particularly expensive?

Yes. When I did play with py-spy earlier, I noticed that there was a ~ 1 second “python tax” for importing modules that we kind of pay whatever we do in pants. That accounted for half of a no-op.