hm. yea, worth investigating further.

one of the primary advantages of having targets broken up like that is to allow for distributed caching though. so worth accounting for the speedup of very rarely/never having to run a clean compile locally

(that was with more memory, or no?)

I agree. the cache will help a huge amount. I’m just worried about developers changing a file on their own machine that triggers a lot of dependencies, then having this be very slow for them.

I’ve been trying different combinations of workers and heap. I’ll try to get some more concrete numbers next week about what works best.

Is it idiomatic to declare a fingerprint strategy as a parameter to

Task.invalidated

? Or is there some better way to tie a fingerprint strategy to a particular type of target?

(Or is there a better way to adjust the invalidation calculation, in general, than creating a new fingerprint strategy?)

If the fingerprint strategy truly applies to the target always, no matter what verb (task) is being performed against it, then you might just populate the target payload fields appropriately to form the combined fingerprint that makes sense: https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/jvm/targets/jar_library.py#L33-L45

Otherwise, if either the fingerprint is reall specific to the task-target pair, or the fingerprint is not easily composed of target fields in a straightforward way, yeah, pass the FPStrategy

Example: a java_library has no truly global fp. A javadoc task only really needs the doc comment hash, not the code hash. Likewise, a javac task only needs the code, not the doc comments.

It does have a global fp I guess, including all source, but the point is its overkill and may thrash the cache for a given verb (task) needlessly. Hopefully that makes sense.

@enough-analyst-54434 It does, thanks. When you pass a fingerprint strategy, does that strategy need to apply only to the specific targets passed in the list before it, or all targets in the transitive closure?

(looking at examples in the code, I'm thinking only those targets. But somehow in my testing I'm getting something from deeper in the dep graph, and my code crashes because it's trying to grab something out of the target that's not there)

It only needs to handle the targets passed to the

invalidated

call.

It could assert

... um the transitive closure of the targets you hand it

The

sort_targets

call here basically: https://github.com/pantsbuild/pants/blob/master/src/python/pants/invalidation/cache_manager.py#L341

If that is used, you'll get a closure ... I think

Yeah, that's what I was worried about. 🙂 Those are targets of different types, and I don't necessarily want to write a fingerprint strategy that handles all those types, too. I suppose as long as previous tasks takes care of those targets, I can just route around them with if isinstance?

Yes, you can always do that - return None IIRC for taargets that do not contribute to your tasks FP

👍

If its by type, great. You could also construct the FP Strategy with the exact set of targets to fp and set compare, or really do anything to decide which to use

@straight-waiter-86090 did you ever make your custom task unpacked_jars-aware? Any chance that's something you could share?

Disclaimer: Pants newb. I'm interested in using pants to pull a zip of IntelliJ jars from JetBrains, unpack them, and use them as a jar_library. (Contrast with the Pants IntelliJ plugin which currently does all of this w/ a shell script, which gets the job done, but is less dog-foody.). I've got jar_library downloading their zip, then unpacked_jars getting the files I care about, but seems like the only places to go from here are: remote_sources (feels "right", but can't produce a jar_library), looking "under the covers" in ./.pants.d/unpack-jars/unpack-jars/, or writing a custom plugin/target/whatever.

remote_sources

is iirc, deprecated (in favor of

unpacked_jars

)

what are you trying to do with the jars?

if you just want them on a classpath, should just add a dep on the

jar_library

if they contain sources, then

unpacked_jars

should work

is there a way to execute some code before/after

test

goal? we currently use sbt and for example start/stop docker container when running integration-test suite ..

you can use a

prep_command