So now that 2.32.0 is out and happy, who's ready to talk about switching from GIL to free threads? Has anyone looked at the concurrency and if we're counting on the GIL or if the threading handled in the rust side is sufficient locking to allow safe python concurrency?

as far as I know, no one has looked into it yet - as getting to 3.14, period, was the existing plan. In fact, it's probably worth just flipping the switch to no-gil, and seeing what happens in CI. Though, CI passing isn't uniformly a good metric of it working, it's at least interesting. I'm catching up with some dayjob work I've sorely ignored, but I'll be back on some CI work/cleanup after that

pyo3 has been already upgraded to a version supporting free threaded

Yeah, they upgraded a while ago - but in terms of it actually working, I don't think anyone has tested.

We are for sure relying on GIL w.r.t memoization in python.

I just mentioned it from the point of we are ready library-wise to begin reasoning about how to "un-GIL" our code. We definitely will need to reason about how and what the GIL is currenrtly protecting.

Is going GIL free even that big of a performance improvement? I feel like most of the hot paths are already written in Rust, though I've never run pants through a profiler.

Yes. I've done the profiling for our workload.

👍 That's what I would have guessed and sorta remembered.

So my job (for another week, I'm changing jobs) is doing a POC with OpenAI where we get unlimited Codex tokens. This seems like a fun sacrifice at the altar of Moloch...

Is going GIL free even that big of a performance improvement? I feel like most of the hot paths are already written in Rust, though I've never run pants through a profiler.

I have no idea if free threading will be a successful path, but most profiles <https://github.com/pantsbuild/pants/pull/23147> I've looked at point towards there being plenty of hot-Python paths.

In a simple, ~wide scaling test, using my own cobbled-together free-threaded build, I saw Pants regularly using several-X cores to parallelize. >100% CPU was common, with spikes up to 600%, just when I happened to be looking. I can't make any promises about the actual correctness of the cobbling, so could be bunk, but what I saw seemed reasonable.

Cool I got CI working with freethreaded - handful of test failures but not too bad. They mostly appear to be related to changing the interpreter version

Collin and I might pick this up. Do you anticipate adding anything to that initial PR? Or should we just take that and run with it?

Don't think so, I was going to fix the failing tests but I've been super busy these past couple days writing handoff docs and such so no time.

ok. we'll see what the next week brings for my free time. 🙂

once I start my new gig,

Congrats!

github.com/pantsbuild/pants/pull/23477 thanks @acoustic-librarian-29560, I ran with it. Wasn't as scary a change as I feared.

Oooh, what’s the perf improvement look like?

3-4x on my company repo cold, from 1.3 cores to 6

@aloof-tiger-68736 @fast-school-44220 I tacked a bunch of Pants reviewers on that pull request github.com/pantsbuild/pants/pull/23477 - but if you guys want to put eyeballs on it (or even test it out, as you have a very interesting and abnormal use case)

We plan to test it out, today, against some scale-testing builds. Will report back.

Do we have any good load tests of Pants? In switching to free-threaded, I am leaning towards us collecting some amount of evidence of "durability" testing before we land it.

The testing that Collin mentioned should beat it up pretty good. I suppose it won't tell us if two threads get cross-wired somehow, but it should shake out any deadlocks or crashes.

We have a bunch of users who have large repos, but I guess we should narrow down what exactly is expected to be the same, and what use cases will scale up well

It's a lot is the answer. The more nuanced take is that it depends on goal what the % is, but target resolution, field processing etc is all entirely GIL bound, and ~every piece of work depends on it which leads to long waits, idling workers and a sluggish engine

Success: building Pants locally with this FT PR both successfully completed our initial scaling test, and demonstrated a perf improvement. This is very rough, but still gives a general sense: This first scaling test involves a ~63k-dir, ~256k-file tree on NFS, with ~16.5k of what are effectively BUILD files strewn throughout. The scaling test's

@goal_rule

cmdline is pointed to a file containing a list of ~6.5k of those BUILD files, to kick things off, and some subset of the remaining ~10k are dynamically discovered during the build. These BUILD files collectively describe ~200k (trivial) processes that end up getting invoked by

execute_process

. Many of these processes take as inputs the outputs of others, so there's still something approximating a "real build", with hierarchy. Scott focused on RBE runs. I focused on local runs. At the full scale, my runs were doomed to DNF (many, many hours of runtime), without what is now PR #23466 ("permit

vfork

in local `execute_process`"), because they would slow to a crawl early. With that

vfork

PR in place, the full build takes my machine somewhere in the ballpark of ~35m. (From memory, Scott's RBE runs using unmodified Pants were in the 20m-30m ballpark; we ran from different machines in different environments, so I can't speak to the nature of that discrepancy.) With the

vfork

PR and the free-threading PR in place, the full build took my machine ~7m6s, given

--process-execution-local-parallelism=16 --rule-threads-core=16 --rule-threads-max=32

. This is likely not the optimal set of knobs, but I haven't run many tests yet. Halving those configurables bumps the wallclock up to ~9m27s, for example, which is still a significant improvement.

I have a branch on the way annotating all dataclasses with slots=True. Likely to make a real dent in that metric. Or not, seems as if 3.11+ is better at inlining layouts than I thought.

#23481 appears to have had a significant impact reducing memory consumption of my above test. Last week's run was ~7m6s, and ~33.8 GB max RSS (TOT plus free-threading and

vfork

PRs). With TOT from today (which now includes the

vfork

fix and newly includes #23481, and again plus free-threading PR), the same test with the same knobs was a similar ~7m2s, but only 18.7 GB max RSS. This is an excellent improvement and for such a small PR. Thank you!! Hopefully there's some other opportunities lurking elsewhere for us to find as well; our larger-scale tests consume hundreds of GB of memory (before this PR; untested with it), so even "small" memory optimizations could still have fairly significant impact.

A little late to the game here, but I am seeing signs of life on the free threaded build on our real build with REAPI execution. Just running small stuff so far, not full scale. Hopefully that comes in the next few days.

I have an upstream change in pyo3 open for review that'll make more threads ~always show improvements if the machine has the cores. https://github.com/PyO3/pyo3/pull/6200