Hi! I'm preparing a talk for the engineering team ...
# general
c
Hi! I'm preparing a talk for the engineering team at my company regarding pants and how we are using it so far. I'd like to include an overview of how pants works on the inside, could you point me to something like this?
🔥 2
a
yes! here are three links: (1) v1 task development guide: https://www.pantsbuild.org/dev_tasks.html. v1 tasks are now deprecated and this is less about "how it works" (2) v1 vs v2 comparison: https://pants.readme.io/docs/pants-v1-vs-v2 (3) the pants v2 engine readme (~6 months out of date): https://github.com/pantsbuild/pants/blob/master/src/python/pants/engine/README.md
please let me know if there are any points you'd like clarification on! we probably are going to move the v2 engine readme into the new docsite "soon"
c
Thanks! I was thinking of something slightly different. I have already included info on how to use pants and imprison of v1 vs v2, but now I'd like to include some info on how pants for example constructs the dependency graph or something on that line
👍 1
a
that’s exactly what i was worried you’d ask. @witty-crayon-22786 do you know of any places where we describe the recursive invocations of the graph hydration rule to hydrate the v2 dependency graph?
🤞 1
if not, we could probably outline it. the way the engine creates the v2 dependency graph (adding to an append-only structure to avoid copying) involves a v2 rule invoking itself which is explicitly something bazel won’t ever be able to do unless they expose that capability from the skyframe execution engine which is very unlikely
this docstring summarizes the above and is conveniently located within the short function that performs this recursion: https://github.com/pantsbuild/pants/blob/5bf9c45e2e35ac09ca102cd8a49f9c00f6cc8df3/src/python/pants/engine/legacy/graph.py#L430-L436
👍 1
it's recursion through multiple separate intervening methods now instead of calling itself directly
c
Thanks! Yes, this is the kind of things I had in mind. Which are all the methods involved in the recursion?
a
i think it may just be https://github.com/pantsbuild/pants/blob/5bf9c45e2e35ac09ca102cd8a49f9c00f6cc8df3/src/python/pants/engine/legacy/graph.py#L479-L483 as well, but we would have to have implemented an understanding of the similarity between
Address
and
Addresses
, which would be great and i think may have come across my desk at some point oops
oh no! there's also the whole bunch of methods that hook up
Address -> HydratedTarget
. that part is more complex only because there's more to do and involves snapshotting source file contents and converting parsed BUILD files into v2
Target
subclasses (at
src/python/pants/engine/target.py
, not
build_graph/target.py
)
p
h
That’s wonderful! Here are slides for the talk linked to by Asher right above: https://docs.google.com/presentation/d/1gxsdCcrA96TghoEz_50hMW1p-_6VUnCEE69vjImRhaw/edit?usp=sharing Also, I’m very likely picking up our plugin documentation this week. When is the talk?
c
Thanks this is all brilliant!! The talk is this Thursday Eric, sorry I know it was last minute question for the extras!
h
Cool. I’m unlikely to have the docs ready for helping with prepping the talk, but hopefully they’ll be ready for after if people are curious to learn more 🙂 Let us know if we can help at all with the talk! E.g. if it would help for us to look over the slides, or if you have any questions. @happy-kitchen-89482 has given a few talks at companies about Pants so he might be a good resource too
w
apologies that we don’t have updated architecture docs anywhere… if there is any portion of the talk that Eric linked that you think would be interesting to expand on, i’d be happy to write up some details
👍 2
c
Thank so much again :) these are the slides if you want to leave m your thoughts (it's aimed to be a 30min presentation) https://docs.google.com/presentation/d/1tbgactMmAo63OFb1KR6_8RP1sxT-f-7VB7fiZ0e_l34/edit?usp=sharing
w
will do before your tomorrow!
c
It's a basic talk but maybe tomorrow I add some slides about some of the internals
h
Great slides! Doesn't look like I have commenting privileges?
Had a couple of suggestions
c
Oh let me check the privileges
Updated, sorry for that Ben
h
Very true about the name being hard to google 😅 fun fact, Pants’s name comes from originally being short for Python Ants, from the Apache Ant tool. Pants has deviated very very far from Ant, but the name stuck for better or worse
c
Update: talk went great and some people are really excited to get their projects on pants too 🙌 thank you everyone 🙏
h
That’s wonderful! Thanks for the update! One thing we’re working on is trying to make it much easier to onboard. In 1.30, we added dependency inference, where pants will read your python code and infer your dependencies for you so that you can leave them off in BUILD files: https://pants.readme.io/v2.0/docs/release-notes-1-30#experimental-dependency-inference-feature The next step I’m working on is target inference. We’ll dynamically create a target for every python file you have, meaning you don’t need BUILD files in most cases. Our motivation is to reduce the amount of boilerplate you need to use Pants.