https://pantsbuild.org/ logo
#development
Title
# development
b

bitter-ability-32190

06/06/2022, 3:07 PM
@ancient-vegetable-10556 (or anyone else familiar) Can someone help me understand what
src/python/pants/jvm/non_jvm_dependencies.py
is/does?
1
a

ancient-vegetable-10556

06/06/2022, 3:08 PM
One moment
b

bitter-ability-32190

06/06/2022, 3:08 PM
(take your time, not urgent at all, jus tpinged you because of the file history)
a

ancient-vegetable-10556

06/06/2022, 3:09 PM
It’s for dependencies of JVM-related targets that do not produce JVM symbols
Because JVM targets have to produce build artifacts that get smooshed together into a JAR, but unrelated file dependencies should not.
b

bitter-ability-32190

06/06/2022, 3:10 PM
Can you explain it to me a bit dumber? I haven't done Java since High School, and even then I only marginally knew programming in terms of coding (not the ecosystem at all). I'm asking because it's a bit "special" in the world of asset handling in Pants codebase
Why does
NoopClasspathEntryRequest
need field sets at all if they aren't used? 🤔
a

ancient-vegetable-10556

06/06/2022, 3:12 PM
one moment
I have to re-orient myself with the rest of the JVM code, it’s been a month or two
1
🙏 1
b

bitter-ability-32190

06/06/2022, 3:13 PM
Much appreciated ❤️ Trying to orient myself for all the possible cases for the
asset
unification proposal
a

ancient-vegetable-10556

06/06/2022, 3:16 PM
OK, so the trick is in
jvm/compile.py
— there’s a method
ClasspathEntryRequestFactory.for_targets
. This is used to find an implementation that fulfils a JVM-compatible dependency. It’s necessary because there are multiple implementations for fulfilling JVM dependencies (java compiler, scala compiler, 3rd-party package resolution, etc). It figured out which implementation to use based on the fieldsets on the dependency target, and each implementation specifies which fieldsets it’s compatible with
b

bitter-ability-32190

06/06/2022, 3:17 PM
And the reason this exists for only for possible file fieldsets and not resource, is because resources are actually consumed by the compiler (e.g. "embedded") and files are tossed away?
a

ancient-vegetable-10556

06/06/2022, 3:17 PM
The problem was that if you had a
file
dependency, there’d be no implementation to fulfil its dependency, so the compile would fail. We made the decision at the time that it would be better to handle a no-op compile in a generic and extensible way than to special-cases
Yes, resource dependencies do get embedded in the final compile result.
b

bitter-ability-32190

06/06/2022, 3:17 PM
Making more sense 🙂
a

ancient-vegetable-10556

06/06/2022, 3:17 PM
see
resources.py:assemble_resources_jar
b

bitter-ability-32190

06/06/2022, 3:18 PM
Secondly, why did we need to support the "generator" sources? I'm assuming in the cases where someone depended on the generator directly (same fo rthe relocated files tgt)?
a

ancient-vegetable-10556

06/06/2022, 3:18 PM
yup
it’s true of every
ClasspathEntryRequest
implementation
and
JvmResourcesRequest
is the special-case implementation for resource, jfyi
b

bitter-ability-32190

06/06/2022, 3:19 PM
😕 That is... unfortunate. It doesn't leave much room for plugin-implemented asset generators, does it? 😕
a

ancient-vegetable-10556

06/06/2022, 3:20 PM
It does, it’s completely pluggable! (one moment, again)
See
compile.py:calculate_jvm_request_types
— each
ClasspathEntryRequest
subclass needs some way of generating a
.jar
(actually just a zip with a specific directory structure). In JVM land, generating a
.jar
is unavoidable*, so you’ll need some way to do that anyway
(*not technically true, but more than true enough for our purposes)
b

bitter-ability-32190

06/06/2022, 3:24 PM
I see! OK I think I have what I need. I really appreciate the time ❤️
a

ancient-vegetable-10556

06/06/2022, 3:25 PM
Yup! The thing you originally found was just the easiest way to make a no-op that fit within the existing plugin structure. If our design is inflexible, do let me know, we don’t want to get in the way of things being useful
b

bitter-ability-32190

06/06/2022, 3:26 PM
I think in this case, that code will disappear because all assets that end up as a dependency will become resources (within reason). If the user wants to opt-out, they use dependency excludes. One question I'll have in the future will be if this PR will be useful for that resource collection: https://github.com/pantsbuild/pants/pull/15749
Also feel free to check out the asset proposal and leave comments if you got 'em: https://docs.google.com/document/d/1gdJjhfAiVymTTcV6sRmsHTvQrzgaFmXdY8-fkdFaqxc/edit?usp=sharing
a

ancient-vegetable-10556

06/06/2022, 3:27 PM
Nope! We need support for file dependencies for JVM tests.
So that’s why there’s a distinction between
file
and
resource
dependencies already
b

bitter-ability-32190

06/06/2022, 3:28 PM
Nope! We need support for file dependencies for JVM tests.
The proposal tosses (or "unifies" depending on how you wanna look at it) the distinction between
file
and
resource
and leaves the user with only one asset type:
asset
.
a

ancient-vegetable-10556

06/06/2022, 3:48 PM
I will need to stew on that proposal for a while; the semantic difference between
file
and
resource
targets is extremely significant in JVM land.
which mostly boils down to how you access a
resource
from inside a
.jar
at runtime vs how you access a file from the filesystem at runtime
I’d be happy to talk ideas with you at some point, but the assumption that resources and files are interchangeable really doesn’t hold in JVM land
b

bitter-ability-32190

06/06/2022, 3:53 PM
The one-liner takeaway is "put assets in every relevant sandbox" it's up to what's running on/in the sandbox which determines what to do with the assets
a

ancient-vegetable-10556

06/06/2022, 4:13 PM
Yeah, I don’t think that works, because both can be necessary, and they are not semantically comparable in JVM-land
to be clear: I don’t think there is a need for two different target types, per se, but there will need to be some sort of way to distinguish between the two in
BUILD
files (possibly in how you link them to the target that needs them)
I think most of the confusion that you highlight in the doc arises from most languages not having a significant semantic difference between files and resources, but it is significant in JVM land
(I suspect the reason we have both
file
and
resource
in the first place is a holdover from V1 when Pants was much more JVM focused)
b

bitter-ability-32190

06/06/2022, 4:20 PM
Yeah, the current behavior is "I, the user, am giving you a metadata hint about how I expect this to be consumed" but unfortunately that really isn't great at the "node" metadata source. As you point out, it'd be better to model this at the consumption site, however even that can mostly be avoided if we simply continually manifest assets in their relevant sandboxes (the compilation sandbox, the test sandbox, etc...)
We currently support this "transitive hint" (but not well, lol) where a
foo_source
depending on a
file
is a hint to the
foo_test
to maybe materialize it as loose in the test sandbox, but don't include it as a resource when packaging. Instead, let's avoid these little hints and treat everyone equally.
a

ancient-vegetable-10556

06/06/2022, 4:25 PM
Yeah, I agree entirely there. I think the difference is that a
resource
sometimes needs to be considered as a true dependency
I’m imagining the case where you provide an “asset” to a JVM test — the question is whether the code under test should be opening the asset as a capital-R
Resource
, or whether it should be opening it as an item from the filesystem, or if both could be necessary
I don’t think that’s something you can correctly figure out from context
and it’s definitely not correct to include every filesystem file as a JVM resource
b

bitter-ability-32190

06/06/2022, 4:28 PM
So forgive my Java ignorance. Are tests are also compiled before running? A la any other compiled language
a

ancient-vegetable-10556

06/06/2022, 4:32 PM
Yes. Tests are compiled before running. As part of that, the entire dependency tree is also compiled, and the
Resource
targets are specified as part of the java classpath (i.e. the packaged items available to the JVM). The
File
targets are made available for opening using filesystem commands.
JVM compilers need “all” of the dependencies present at compile time, and
resource
targets are targets that are semantically significant to the compiler, the packager and to the runtime. Files are just things that code may need to be aware of or able to access at some point.
b

bitter-ability-32190

06/06/2022, 4:35 PM
In this case, as it stands (and we can mold the targets to adjust if needed) all assets will act both like resources (in that they will be packaged) and files (in that they will be loose on the filesystem int he sandbox
a

ancient-vegetable-10556

06/06/2022, 4:41 PM
Right, so it is frequently not appropriate to include files as resources — tests are the first case that comes to mind — and that is where the consumption issue becomes relevant. To be clear, I’m not against most of the proposal, just highlighting that the conceptual modelling that Pants already has is based on how the JVM sees the world, and that doesn’t hold for basically every other language. It clearly creates confusion for implementers as well as users. However, because the modelling is already good for the JVM, trying to bias towards being sensible for everything that’s not the JVM is likely to make the JVM usage less clear
b

bitter-ability-32190

06/06/2022, 4:42 PM
Right, so it is frequently not appropriate to include files as resources
Why is that?
a

ancient-vegetable-10556

06/06/2022, 4:45 PM
One use case is that if you’re inside a corporation releasing a proprietary JAR for external consumption, and you have sample data files for use with
pants test
or
pants run
which contain internal data, then you definitely don’t want that included as part of the JAR
b

bitter-ability-32190

06/06/2022, 4:46 PM
In JVM-land are all resources packaged? No metadata or anything in the middle to self-select which dependencies get packaged?
(E.g. in Python you might have
<http://MANIFEST.in|MANIFEST.in>
)
a

ancient-vegetable-10556

06/06/2022, 4:47 PM
Yes. Resources are intrinsically things that are part of the final packaged product.
And they are accessible in JVM code by asking the JVM for a handle to a resource — they are treated intrinsically differently to files.
b

bitter-ability-32190

06/06/2022, 4:49 PM
Ah, then I suspect we'd need to add a field to the
java_source
target to allow for such selection. @witty-crayon-22786 sound about right? One olive branch to extend here is if consuming the resource is relatively rigid in the code, we can add JVM "asset inference" like we do in python, which will automatically determine resource v not resource
w

witty-crayon-22786

06/06/2022, 4:50 PM
Ah, then I suspect we’d need to add a field to the
java_source
target to allow for such selection. @witty-crayon-22786 sound about right?
i think where we had left it in the last discussion was that we needed a
data_dependencies
field on
*_tests
to signal that you wanted loose files, but we hadn’t decided whether it needed to be transitive. haven’t caught up on the design doc, sorry
which… doesn’t make sense in this case probably. what would it mean for it to be transitive. so yea, you’d still need to be able to mark it on a
java_source
so… yea, probably.
a

ancient-vegetable-10556

06/06/2022, 4:52 PM
I don’t think it needs to be transitive per se
and being able to mark a data dependency on any jvm source file would probably solve the problem (for tests or otherwise)
w

witty-crayon-22786

06/06/2022, 4:53 PM
… yea, sorry. i guess the “is it transitive” question is instead: “should
java_sources
, which cannot themselves consume files, be able to declare a dependency on files… or is that only `*_tests`”
b

bitter-ability-32190

06/06/2022, 4:54 PM
...and piggy-backing off Stu's. The reaosn we "allow" transitive file deps is the convenience of that metadata being applied to all tests of my source. Right?
w

witty-crayon-22786

06/06/2022, 4:54 PM
yea.
a

ancient-vegetable-10556

06/06/2022, 4:54 PM
There’s probably a want of describing what files might be accessed by a given
jar
when you
pants run
it, but that’s hard to track, and would make more sense to be declared at the source
and yes, you’re right about tests
b

bitter-ability-32190

06/06/2022, 4:55 PM
I think JVM is special here, so I'm going to call it out in the doc. But yeah I think we want the field on
java_source
and lets call a spade a spade.
data_dependencies
is too vague. Depending on whether we think files or resources are more prevalent it'd either be (semi-bikeshed)
package-asset-dependencies
(a list of asset dependencies to package) or the converse:
dont-package-asset-dependencies
(treat these as we did "files" before)
a

ancient-vegetable-10556

06/06/2022, 4:55 PM
describing what files are consumed by the implementing code makes a lot more sense than describing on the test that may not directly consume it
w

witty-crayon-22786

06/06/2022, 4:55 PM
@bitter-ability-32190: it’s the same as Go and Python (under PyOxidizer or zipapp)
“embedded” resources vs loose files
b

bitter-ability-32190

06/06/2022, 4:56 PM
@witty-crayon-22786 I think in other ecosystems there's a middleman picking and choosing which resources to embed out of the sandbox or not. We only need this for places there isn't that middleman
w

witty-crayon-22786

06/06/2022, 4:58 PM
python doesn’t have that middleman either, in that we don’t require a setup.py
b

bitter-ability-32190

06/06/2022, 4:58 PM
describing what files are consumed by the implementing code makes a lot more sense than describing on the test that may not directly consume it
Yes and no. The source code isn't actually consuming it though, it's just expecting it to exist in the environment its being run in. Another way of putting it is the source code has a dependency on the runtime environment, which Pants can't control. It's just convenient to declare the dependency on the source as a hint to downstream environments we ourselves create.
python doesn’t have that middleman either, in that we don’t require a setup.py
Not for PEX, but for
python_distrubution
we do*
w

witty-crayon-22786

06/06/2022, 5:01 PM
and being able to mark a data dependency on any jvm source file would probably solve the problem (for tests or otherwise)
relatedly, Alonso responded with his use case: https://github.com/pantsbuild/pants/issues/15500#issuecomment-1144003041 … haven’t correlated it with the design yet.
this line:
The normal
file
behaviour was enough for that (couldn’t use
resources
because you can not run binaries from inside an archive file).
refers to the fact that a resource starts out embedded in a jar, so you need to extract it somewhere to use it as a file… that’s a fairly rare case, because 90% of the time you can open a stream and ignore the fact that it isn’t a file
so… test helper code with loose file dependencies