Has anyone thought about compiling Pants with

mypyc

? 🤔

No no one has 🙂 It would be cool. I wonder if it would be hard to get the native extension working properly, but that's only speculation

I wonder if you could eliminate the roundtrip by the runtime form Rust -> Python runtime -> Python compiled binary and just go from Rust -> compiled code

In progress. :)

I wonder if you could eliminate the roundtrip by the runtime form Rust -> Python runtime -> Python compiled binary and just go from Rust -> compiled code

I suppose this is approximately the overhead of an additional C function call though 🤔

I'm pretty pumped too - if it works half decently, I can start dropping languages like flies. Working on getting a trivial prototype running today (https://github.com/sureshjoshi/pants-plugins/blob/2-mypyc-prototype/hellofib/BUILD.pants#L3-L14)

Godspeed

Mad respect for the effort. So, I had originally thought you were doing a profile and compile mission, then I found out it was the whole-scale approach. Seemed monumental, but would have been dope

A surprising amount of files were compilable. Although I imagine when it came time to execute, the real pitfalls would've exposed themselves (e.g. you can't

inspect

the caller's frame, which is how

collect_rules()

works)

Interestingly, I bet there is some way to auto-discover which files would be compilable - based on a set of rules about what the decorators in Pants do. Actually, I think the best improvement would come from being able to compile all the tooling that we import. Like, all the linters, formatters, etc - get compiled on installation, and then just native Python to bundle all those rules together. Much, much, much harder to do and maintain, but its one of those pipe dreams. I'd also be curious if Pex'ing could be sped up, not that I find it slow - but I haven't ever looked at that codebase, so 🤷

Compiled

pants

uses a compiled

pex

which on installation of

black

compiles it?

Well, not compiled Pants in that case. But a compiled pex runner, with compiled Black, yes

Why not Pants as well? 🤡 Gotta go fast

Lol, if possible, sure - turtles all the way down

I've also thought about the implication on testing. If you are compiling your source, you want to test with the compiled source to ensure the compiled version works as expected (the Py source acts as a fallback + a source of truth for the code)

for sure, testing on intermediate code is whack

How so?

Well, lets say you download an sdist of a tool, which has great testing - then you mypyc it - and now you have a compiled binary that hasn't been tested that you're trying to use

Make it a config setting because YMMV?

Agreed... One day... One day... [stares off into distance]

A step in the right direction would be to: • upgrade

mypy

in repo. Note that newer

mypy

is somewhat stricter • Move towards full typing in-repo and

--strict

Bummer, thank you for exploring this though! +1 to the suggestions for moving forward Re: compiling, Pants runs PEXes in venv mode which has near-native performance. https://www.pantsbuild.org/docs/reference-pex_binary#codeexecution_modecode But one remaining issue is first-party code still has to be compiled into

.pyc

https://github.com/pantsbuild/pants/issues/11339

@hundreds-father-404 "native" meaning, native python speeds? Or native machine code speeds?

The former (from reading the docs).

That's what I would have assumed - and it would line up with my anecdotal testing of pex'd projects

native python speeds

FWIW the test-consuming-compiled-python wouldn't be tooooo hard in Pants, as Pants has cached codegened files 🤔

the real pitfalls would've exposed themselves (e.g. you can't inspect the caller's frame, which is how collect_rules() works)

Would it be helpful to memorialize the concrete issues you encountered in a GitHub issue perhaps? Iirc it was decorators like

@frozen_after_init

and then

collect_rules

? On par, I continue to personally prioritize end-user experience over plugin author experience. (Although of course important to balance that). So I'd be curious to see all the things we'd have to "give up" for better perf To be clear, please don't go on a wild goose chase tho trying to catalog even more issues. Makes sense to not try pursuing this more for now. Rather, to memorialize the work you already did

From what I remember: • All properties must have return type annotations • No inner classes or functions • No dynamic attributes with normal attribute access, gotta use `getattr`/`setattr` • No class decorators or any class funny-business aside from normal ones like

dataclass

• comparison operators must return bools • ...probably much more but didn't inspect the compiler output in each case

Wait, no inner functions?

I think both, but I'm going on memory here

Yeah I'd be surprised if those didn't work, agreed

I'd honestly be surprised either way. If they don't work then how do you do interesting things. If they do work, how do they handle captured variables in C-land?

I mean, I assume they'd just be handled as function pointers, like everything else in C 🙂 I just did a

mypy --strict

test, and using an inner function works fine, but returning an inner function requires some finesses methinks, as it complained about something that I don't think was tru https://mypy.readthedocs.io/en/stable/common_issues.html#narrowing-and-inner-functions

Sorry, to be clear, I meant that

mypyc

may/may not allow inner functions with captured variables. As those can't be converted to native C functions easily

ahh, okay okay.

Another approach to speeding up Pants a la PyOxidizer 😄 https://pantsbuild.slack.com/archives/C046T6TA4/p1646748241557059

Yeah I think the ultimate Pants endgame would be both: • PyOxidized app. Benefits: Single-file installation, Pants-provided Python runtime (embedded via PyOxidizer), latest Python version (for features + speed) ◦ I think another lesser point here is a win of Pants-provided Python installations for the user •

mpypyc

compiled sources. Benefits: super speedy Python code

On my way home just now, I was just thinking about Python and PyOx. There are a few issues - not pants-specific, but workflow specific that I would love to see solved (can you tell I'm about to take on another project that I'm trying to convince the clients to let me use Python for?) • Developer productivity - Python is a no brainer here • Performance - Even with 3.11 optimizations, certain pieces of code might need to be sped up depending on how often they run, so getting

mypyc

and/or Cython up and running would be great. I currently use Cython for running Python on serverless deployments - pays itself off immediately, every call • Single-file deployment to remove Docker as a dependency ◦ PyOx is only sometimes single-file deployment, depends on a bunch of factors - I'm trying to find ways to crack this one - so we can reliably assume single-file deployment, and having a folder of libs is some edge case

Have you tried PyPy btw? I've always wanted to try it with Pants but haven't really dabbled because of Pants's distribution model

Tried it a long time ago, never really stuck with any Py variants

Oh that's an interesting angle. I think you'd want to experiment with

venv

vs

zipapp

https://www.pantsbuild.org/v2.10/docs/reference-pex_binary#codeexecution_modecode

venv

is the way to go methinks, I'm just trying to see if PyOxidizer will even run that as a module, and if I can bundle it as a resource. I'm just not sure, theoretically, if that should or shouldn;t work. Reading through the docs to find out, hoping for something that works in any way

How do you mean "run as a module"?