https://pantsbuild.org/ logo
Join Slack
Powered by
Hi friends! I'm working on a Pycon proposal re: Ru...
# development
h
Hi friends! I'm working on a Pycon proposal re: Rust + Python, how it enables us to have performance with accessibility. Will explain 1) how we use it at a high-level, 2) why Rust, and 3) takeaways/insights for other projects It is not a tutorial, I don't want to do a deep dive into the CPython crate vs. PyOxidizer for example Thoughts on take-aways we have with using Python + Rust? Main one is Rust + Python is awesome, but also maybe 1. Be hesitant to add in Rust. Benchmark. 2. Minimize the FFI boundary. 3. Integration tests are crucial. Proposal at https://docs.google.com/document/d/1D0MBEo5Dw97qN91TqoCHwtZf4R8gEFU3RDm46tePRvQ/edit#. Feedback very much appreciated!
Another way of framing #2...have a strong "separation of concerns", know what Python excels at and what Rust excels at My point is that the FFI boundary is awkward, and it's likely better to do full Python or full Rust if you don't have a nice separation. Pants works because we have this well scoped scheduler that's abstracted away
3 might be boring / kinda obvious? And I'm not really sure that's the right takeaway? I more think it could be insightful to talk about how we test things in pantsbuild/pants, but I'm not sure what the actual takeaway is
cc @average-vr-56795 @red-balloon-89377 @early-needle-54791, really curious what y'all think given your wisdom from the earlier days of the engine
a
I'll give it some thought 🙂 But my initial reaction to your 3 is: A few integration tests are crucial, but a really well-defined/separated API with lots of unit tests is going to make your life so much easier (which hopefully isn't news to anyone!)
đź‘Ť 2
h
I agree. Maybe for #3, what I really am after is "You really need to follow testing best practices"? I don't think we have anything new to say about testing, other than that if you have FFI, it's even more important to do what everyone is telling you to do
(FYI this talk is targeted for intermediate.)
a
I feel like a large part of that is really pointing at a lack of type analysis across the boundary?
đź’Ż 2
🔥 1
h
from the perspective of Rust, the type of every python object is 
PyObject
(ignoring the places we're still using 
Value
)
đź‘Ť 1
I wonder if there are intersting things to say about the rust build problem of creating the 
libengine.so
file and getting python to understand it correctly
this is a corner of the codebase I don't understand as well as some other people; and I remember when we were running into issues with this lib being big enough to cause problems with test sandboxing
h
I wonder if there are intersting things to say about the rust build problem of creating the libengine.so file and getting python to understand it correctly
I don't think I'll want to spend a ton of time on this part because using an 
.so
file is pretty common via Cython I think in the "How we use it" part, I'd include like 30 seconds explaining we compile an .so executable via Rust, then load it simply by ensuring the resource file is in the right place and importing like normal
a
The switch between "avoid linking libpython" and "ok we'll link libpython" is probably worth mentioning (I think Stu is your expert there?)
đź‘Ť 1
w
and then also the switch from the “reduced” PEP-384 API to the full API, which required targeting particular versions
https://pantsbuild.slack.com/archives/C0D7TNJHL/p1612389789060100?thread_ts=1612389223.057800&cid=C0D7TNJHL this isn’t actually true… you can include types in the the FFI method signatures, and 
cpython
handles the conversion.
via the 
FromPythonObject
and 
ToPythonObject
traits… which we could implement for more things
`PyObject ~= Value`… but you don’t need to use 
PyObject
in your method signatures.
BUT, mypy doesn’t know about those. which is why we also have https://github.com/pantsbuild/pants/blob/95ed385f71feec341f02ef5476bf95ade1a0b4dc/src/python/pants/engine/internals/native_engine.pyi, unfortunately. maybe there is a better way there.
đź‘Ť 1
…could totally generate that file too.
đź’Ż 1
đź‘€ 1
one minor correction: the relevant “x vs y” here is 
cpython
crate vs 
py03
crate… we used 
cpython
at the time because 
py03
required nightly rust. you don’t need to go any deeper than that though… afaik, 
py03
started as a fork of 
cpython
, and they’re relatively similar in design still.
b
Hi to folks I don't know yet! I'm the head of developer relations over at Toolchain, where I get to work with Eric, Greg, Stu, Benjy, John, Asher, etc. I'm new to Pants, so the below feedback is general:
Re "unoriginal" points: Key is to decide whether a point is unoriginal in this unusual context + unoriginal to this particular audience. The audience will be Python intermediates, but presumably have little or no Rust background. So a point that seems unoriginal from a Pants 2 contributor's POV may be very valuable for this audience to hear —for possibly the first time, or as reminder/context within an uncommon situation for an intermediate Pythonist. At the other end of the spectrum, things that are axiomatic within Python community can be powerful for calling out how a point of yours reflects community values and community best practices. So, sometimes an "unoriginal" point is worth giving specialness too. Re tests: As a listener, I also appreciate hearing not just that I need good tests but what kind of surprises might be around that corner. Are there things that I wouldn't usually think to test for in Python that need more probing when writing tests for Rust? Are there any things I get to take for granted, e.g. from stdlib, that free me from things that I'd expect to test in Python? Re overall: You've gotten to dig into deep and fascinating corners of both languages. Are there corners of Python or Rust that it's uncommon to explore as much as happened on developing Pants 2? These are fascinating to hear about, and bonus great if hearing about them now can help someone in that audience avoid pain. If you see any interesting opportunities for Pythonistas to piggyback on the code that Pants community has written here, where a 3rd party would be able to a leg up for their own thing unrelated to Pants, these are great to hear about those. Maybe it unlocks someone else's imagination...
đź‘Ť 2
w
yea, very good points. and you have to expect that ~90% of your audience will never have used Rust, so worth touching on the “can’t get it wrong” stuff that Rust brings for all code, not just FFI. maybe also worth contrasting to how you could get it wrong in C (“forgot to release the GIL”, “segfaulted because deleted object”, etc)
đź’Ż 2
a
One thing I'd possibly think about bringing up (or not, it's maybe kind of niche, but also, kind of Pantsy) is the dependency story... Rust has a really nice single package manager, and dependencies are generally really easy to pick up, and that's designed into the language - the difference between 0, 1, or 10 dependencies is trivial. Python has a somewhat jankier dependency story. Publishing is weird, there are wheels and eggs, native deps aren't always clear or easy to work with, and the difference between 0 and 1 dependency is huge. (Pants tries to make the Python world feel a bit more like the Rust world here, yay Pants, good opportunity for a plug)
h
First draft! Thanks for the feedback everyone 🙂 https://docs.google.com/document/d/1D0MBEo5Dw97qN91TqoCHwtZf4R8gEFU3RDm46tePRvQ/edit# Feedback welcomed
f
I guess this is a bit late to be suggesting a new topic altogether...but is there any attempt to address a question like "why use rust to extend python rather than C/C++?". Do you have any experience developing python extensions with those things?
đź‘Ť 1
❤️ 1
w
i had just added a comment to that effect, heh. coke
it’s an important point.
Rust is as fast as C or C++, but much safer. it might have a higher learning curve, but in terms of new contributors actually successfully landing new code, the hypothesis is that it is easier.
âž• 1
🎯 1
@hundreds-father-404: and that reminds me: here is the original motivating document. https://docs.google.com/document/d/1C64MreDeVoZAl3HrqtWUVE-qnj3MyWW0NQ52xejjaWw/edit
f
i mean this is specific nit of mine too, because I have done a bit of C++ extension dev and I hate it
w
yea, i don’t know what the state of art for C++ extension development is, but the 
cpython
crate is really, really solid.
f
the big argument on the science/research side of python for C/C++ is the larger community around it, and the established practice providing C++ SDKs for hardware, be it GPUs for number crunching, or just drivers for things you get from vendors
w
yep, good point.
f
i haven't used rust enough to know its story for interacting with those things
w
it has a ways to go before you can do it natively, admittedly. you’d have to generate rust bindings for the C/C++ library in some cases, and then use rust to interact with python.
f
i get that you're supposed to be able to write wrappers at the FFI boundaries
w
but as an anecdote: Pants’ rust code embeds a few different C libraries, and has embedded C++ code in the past. and the rust packaging ecosystem mostly hides that from you successfully.
🤔 1
f
well a lot of C++ libs have such bad APIs that the work necessary to build an idiomatic rust wrapper might actually make it easier to work with than working with it natively in C++
đź’Ż 1
idk, just anything you can do to get people to stop using C++ haha
đź‘Ť 2
đź’Ż 2
i don't think the hardcore C++ people are reachable, but i think teams that might be considering doing extension work for the first time might be an easier sell
h
"why use rust to extend python rather than C/C++?"
cokeCarina and I were talking about this at the exact time you sent that suggestion, @flat-zoo-31952! We realized that is really what I wanted to make this talk about, and it was getting a bit muddied into making Pants the subject of the talk, rather than only something that guides the talk I'll be rewriting this before the deadline next week
b
h
I don't think Rust even has a higher learning curve than C++
đź‘Ť 1
🙏🏻 1
C++ has been around longer, so there's a larger pool of people who are familiar with it
and in general I think the software industry should be trying to move away from C/C++
b
Stu answers the big question of "Why would I, a Python dev, want to hear -- let alone pursue -- this?" It also shifts away from subtext that we need Rust because Python is lacking. The core topic is that Rust can brings the wins for Python that most people look to C/C++ for. You're presenting an alternative, as not only legit but with less friction and more benefit.
đź‘Ť 1
Open in Slack
PreviousNext
Powered by