https://pantsbuild.org/ logo
Join Slack
Powered by
Hey there. Where can I read about the mechanism Pa...
# general
w
Hey there. Where can I read about the mechanism Pants uses to detect which targets changed? Is it something like: 1. Scan all subfolders for build files 2. Hash all folders with build files 3. Compare to previous hashes ? What happens when the folder hierarchy becomes large and there are many build files? Does the scanning part remain fast?
e
The reading would start here: https://github.com/pantsbuild/pants/blob/master/src/python/pants/init/target_roots_calculator.py#L182 In short, the process goes: 1. consult git to get the list of changed files 2. get the set of unique directories containing all of 1. 3. parse the BUILD files for all those directories and their parent up to the build toor dir 4. Ask all the targets parsed from 3 if they own one of the changed files 5. operate on all targets that answered yes to 4
👍🏻 1
The inefficient bit is the fact that an owning target can live in any directory at or above the file in question. This is - in part - due to pants rules for targets and file ownership.
w
Thanks. So part of the caching for a target’s output is the last git revision in which it was built?
e
No
Consultation of git is only done for the changed case, it has nothing to do with day to day target result caching by Pants tasks and rules
h
I there may be a crossed wire here? What John is referring to is the “changed” mechanism, which in Pants specifically refers to the ability to run tasks only on targets whose sources have changed in the SCM. This is done via flags like 
--changed-parent
etc. See https://github.com/pantsbuild/pants/blob/master/src/python/pants/scm/subsystems/changed.py#L27 for details.
This is one of three mutually exclusive ways in which Pants can select a list of targets to operate on (the other two being “explicitly name them on the command line” and “--owner-of”)
But I have a feeling Guy is asking about the Pants invalidation mechanism, i.e., regardless of how a list of targets was selected, how does Pants decide which of those have changed and therefore require re-compilation or whatever?
Or did John get you right and I get you wrong @wonderful-winter-39732?
w
if you use 
pantsd
the scanning part can remain fast.
w
Actually @happy-kitchen-89482 I was asking about how slow things will become when the repo gets larger. I think in my question I assumed I’m building the entire repo (many build files) while @enough-analyst-54434 assumes I’m building a specific target and therefore only needs to know about that target and the build files above it in the folder hierarchy. Maybe it’s leftovers from my Maven days that I assume I need to build everything 🙂
But the answers here gave me more questions (thanks @happy-kitchen-89482)
@enough-analyst-54434 I don’t understand why you need to query Git at all? If I change a file locally Pants can tell and will then recompile the relevant targets. What difference does it make if the change originates from some local change or from a change pulled from a remote repo?
h
You don’t necessarily need to query Git. If you 
./pants compile ::
then Pants will construct a build graph, then fingerprint every source file in all the specified targets (in your case all targets) to see which have changed and act on those. This will be fast if you use 
pantsd
, as @witty-crayon-22786 mentioned. But if you do 
./pants compile --changed-parent=<treeish>
then it consults git to find out which targets might have changed, as @enough-analyst-54434 described above, and then fingerprint only those and continue from there, which may be faster (certainly prior to 
pantsd
).
Think of the 
--changed-parent
(operate on all targets whose sources are different from their state at this sha) as a replacement for 
::
(operate on all targets)
But then, for every such target operated on, pants fingerprints the sources and decides if it needs to do anything.
For example, it’s possible that nothing needs to be done even if SCM state has changed. E.g.: you compile target T at state A, then change T’s source to state B, then commit state B, creating sha SHA, then revert the source back to state A, then compile again with 
--compile-parent=SHA
. Now Pants will conclude that it needs to operate on target T, because its source (which is in state A) is in a different state from the one at SHA (state B). But when Pants actually operates on that target it discovers that it doesn’t need to do anything, because the last compile was also at state A.
My point with this example is to demonstrate that consulting git is for generating a set of candidate targets to act on, not how those targets are invalidated.
w
OK sounds logical - if I know what changed then why make pants search for it?
This option will be very useful for us I think
h
Not sure what you mean by “if you know what changed”? You of course can always run pants just on the targets you currently care about: 
./pants compile path/to/target
instead of on the entire repo.
w
I think I phrased it wrong, it wasn’t really a question 🙂 What I meant to say was - since there’s already a list of changed files in the GIT log it’s a good idea to use it to deduce which targets should be built. Hope this makes more sense
h
Ah yes, exactly, although sometimes you may know better than git about what’s relevant to the task at hand
w
so yes, 
pantsd
is relevant here
twitter has begun keeping 
pantsd
warm in CI run-over-run (https://github.com/pantsbuild/pants/pull/6059 is very relevant there)
it can be a very significant speedup. and will get more significant as more is moved into the daemon
it's probably almost time to enable pantsd by default... can follow https://github.com/pantsbuild/pants/issues/4438
5 Views
Open in Slack
PreviousNext
Powered by