https://pantsbuild.org/ logo
Powered by
b

bitter-ability-32190

05/16/2022, 3:49 PM
Picking up from the above ^. What would it look like if we unified 
files
and 
resources
into one target type: 
asset
? Are there languages/ecosystems where the distinction is important (from a Build tooling perspective)?
Eric points out the source root stripping behavior. Could we not make it conditional? Strip if under a source root, don't strip otherwise? It's literally the distinction I make when explaining to my peers. "Use 
resource
if it's under 
path
, use 
file
otherwise"
w

witty-crayon-22786

05/16/2022, 4:53 PM
yes, it matters in: go, python, java, scala, kotlin, etc
For 
test
the loose files exist either way. There’s no discernible difference in the 
test
sandbox. So a user can use either, and they can also use 
open
and 
pkgutil
.
resources have source roots applied: i.e., they live in packages, and have a prefix stripped… files do not, and will be at their absolute path in the repo
it matters in: go, python, java, scala, kotlin, etc
it matters in every language which has a way of loading resources “from your binary” as opposed to loose files in the filesystem.
b

bitter-ability-32190

05/16/2022, 4:54 PM
Right, but I can 
open
a 
resource
or 
pkgutil
load a 
file
if I get the right info in my test. I'm not verifying I "did the right thing as if I was packaged".
e

enough-analyst-54434

05/16/2022, 4:54 PM
Nack on latter
Resources need to be part of Python packages
You may have files that are not.
b

bitter-ability-32190

05/16/2022, 5:01 PM
right but if they were rooted I could 
pkgutil
load them at test-time
(and don't forget what I'm proposing still has source root stripped if they need to be)
e

enough-analyst-54434

05/16/2022, 5:07 PM
You could not if they were 2.7 and had no 
__init__.py
in parent dirs - I'm pretty sure. For example.
b

bitter-ability-32190

05/16/2022, 5:09 PM
Also, we're all talking about from a resource-loading perspective. Again, I'm asking from a build-tooling perspective
e

enough-analyst-54434

05/16/2022, 5:11 PM
I'm a bit lost, but from a build tooling perspective there is only the stripping of some prefix path that is different.
➕ 1
b

bitter-ability-32190

05/16/2022, 5:11 PM
That's kind of my point. Could that not just happen for you?
e

enough-analyst-54434

05/16/2022, 5:12 PM
I think! Targets are just horrible noun crutches. The real answer lies in the universe of verbs. I know of no Python or Java verb that would treat things differently except for prefix stripping.
✅ 1
b

bitter-ability-32190

05/16/2022, 5:13 PM
(I'd be happy to PoC this too)
e

enough-analyst-54434

05/16/2022, 5:15 PM
So, forget running tests. I can have a pex_binary that includes some html and css files it needs to serve. To do that it needs to hand a directory to a webserver - the web root. That's a case where the file access API that will be used is the file API and the meaning of the web root has 0 to do with the PYTHONPATH. That's an example case that I think Pants tells you you're not allowed to do in a natural way today with 
file
targets. For further complication, assume the html / css is shared amongst several projects and langs stacks. There is some corporate branding in there that should be fixed.
So if you can hit that target, excellent.
w

witty-crayon-22786

05/16/2022, 6:20 PM
I’m a bit lost, but from a build tooling perspective there is only the stripping of some prefix path that is different.
…
That’s kind of my point. Could that not just happen for you?
in Python maybe: but not on the JVM. on the JVM, 
resources
are ~always in JARs, while 
files
aren’t.
That’s an example case that I think Pants tells you you’re not allowed to do in a natural way today with 
file
targets. For further complication, assume the html / css is shared amongst several projects and langs stacks. There is some corporate branding in there that should be fixed.
well… the answer right now would be to put both the 
pex_binary
and the 
files
in an 
archive
target, and ship the archive to production
b

bitter-ability-32190

05/16/2022, 6:24 PM
@witty-crayon-22786 for JVM how does the sandbox look for tests? Are resources loose files?
w

witty-crayon-22786

05/16/2022, 6:24 PM
no: they’re inside of jars on the CLASSPATH… similar to the PYTHONPATH
(which is also a thing for Python: a zipfile PYTHONPATH entry, although i don’t think that it has a name other than “installed zipped wheel”)
b

bitter-ability-32190

05/16/2022, 6:25 PM
Well at least in the JVM case the tests can tell the difference, so that's a plus
e

enough-analyst-54434

05/16/2022, 6:39 PM
in Python maybe: but not on the JVM. on the JVM, 
resources
are ~always in JARs, while 
files
aren’t. (edited)
Resources are always on the classpath. They may be loose on the fs.
w

witty-crayon-22786

05/16/2022, 6:40 PM
yea. that was more a statement about the current implementation… jars are very good for loading perf on the JVM, so they’re likely here to stay
e

enough-analyst-54434

05/16/2022, 6:43 PM
Ok. And as another implementation detail like that, there is currently (for a while now) no such thing as a zipped Python resource for end user code since all PEXes now self-extract to either an unzipped dir or a venv. It used to be you could have a PEX where the user (non-3rdparty dep) code ran from inside the zip, but those days are long past.
Its only (a subset of) Pex bootstrap code that truly has to worry about running from within a zip.
b

bitter-ability-32190

05/16/2022, 6:46 PM
Fair so 
zipapp
arguments are out for the most part. It's mostly about correctness and "dont have 2 ways of doing one thing"
e

enough-analyst-54434

05/16/2022, 6:47 PM
You mean on the target modelling side. On the runtime code side its perfectly good to have 2 ways to load a file, resource APIs or FS.
b

bitter-ability-32190

05/16/2022, 6:47 PM
Yes.
e

enough-analyst-54434

05/16/2022, 6:49 PM
So, on asset(some arg) vs file & resource - I think they amount to same thing different spelling. Only the user will know which is correct for my pex_binary css example. Afaict there is absolutely no way to guess what stripping to use.
If that's right, the only reason to move from files & resources to assets(...) is for some idea of "simpler" / "consistent", still same amount of work for BUILD file writer IIUC.
b

bitter-ability-32190

05/16/2022, 6:51 PM
My argument is to always strip src root if the asset it rooted, otherwise dont. So the file is always relative to its neighbors in the sandbox or in the deployed thing.
still same amount of work for BUILD file writer IIUC
I would argue less work -> less cognitive burden of choosing one or the other
e

enough-analyst-54434

05/16/2022, 6:51 PM
So if I place a file under a source root I'm foced to use it source-root-stripped?
I guess that seems fine, but that's exactly where my creativity of user use cases always fails me.
b

bitter-ability-32190

05/16/2022, 6:56 PM
So if I place a file under a source root I'm foced to use it source-root-stripped?
Yes but IMO that follows the element of least surprise. • In the sandbox it isn't in some other path (wtf did my file go?!) • How you load it during test time is how you'll load it in the deployed env • You can relocate your code and your code doesn't break (depending on how you wrote your value. No hope today if you're hardcoding source root)
The big wins are that: • (all the above about element of least surprise) • Less cognitive burden on users on which to use • Tons of duplicitous code in Pants gone (having to handle either file or resource or both) ◦ A good example is 
experimental_shell_command
. How can this work for both 
file
and 
resource
? Two codegenerators? Two targets? • No more possibility of getting this wrong for new code (we will continue to get this wrong, it's just the nature of bugs) ◦ Again 
experimental_shell_command
chose files. Then I ran into an issue, what if it was a non-maintainer that ran into this? I can debug and then hack together a plugin, but most people can't
w

witty-crayon-22786

05/16/2022, 8:24 PM
Yes but IMO that follows the element of least surprise.
i don’t think that i agree on “least surprise”… doing something differently in a context-specific way rather than due to an explicit use of one target type vs another is magical. for example: what is currently an error (using 
resources
outside of a source root) would become a silent missing file. i agree that it would simplify the code though.
b

bitter-ability-32190

05/16/2022, 8:26 PM
would become a silent missing file.
Missing when? In a packaged thing? Couldn't it still work when packaged?
In that case, we'd still be able to warn, no? Convert 
resources
to 
files
, then warn if an unrooted file is trying to be packaged.
w

witty-crayon-22786

05/16/2022, 8:34 PM
both cases are valid for a test, and there is no signal of which you intended
yes, for package you could warn/error
b

bitter-ability-32190

05/16/2022, 8:36 PM
That's... my point? Right now there's 2 ways and the distinction is easy to get wrong. And worse, sometimes Pants makes the decision for you and you're SOL
w

witty-crayon-22786

05/16/2022, 8:37 PM
i think that you’re conflating 
experimental_shell_command
not giving you a choice with `resources`/`files` targets, which are very explicitly a choice
i don’t agree that the distinction is easy to get wrong. https://www.pantsbuild.org/docs/assets#when-to-use-each-asset-target-type
b

bitter-ability-32190

05/16/2022, 8:38 PM
Well, it's both. So long as there's 
resources
and 
files
there will be code which has to choose. And therefore choose wrong
w

witty-crayon-22786

05/16/2022, 8:40 PM
there will be code which has to choose. And therefore choose wrong
the code shouldn’t be choosing though… it should be pushing the choice to the user. as you said: that makes the code more complicated, and we need a solution for 
experimental_shell_command
.
@bitter-ability-32190: i don’t know. some of my response is “this is how its always been”ism. but i know that there are some traps here. a design doc that addressed the permutations of `file`/`resource` and the various consumers (tests vs archives vs packaegs) in a clean way might be convincing.
✅ 1
b

bitter-ability-32190

05/16/2022, 8:44 PM
That's kind of what I was fishing for, so I'm glad you said it. "is this worth a design doc"?
w

witty-crayon-22786

05/16/2022, 8:44 PM
yea, i think so.
🔥 1
b

bitter-ability-32190

05/16/2022, 8:45 PM
Migration/deprecation will be... fun 😂
I do think the end-result will be cleaner both for Pants code and for users. But it's up to me to prove it 😈
w

witty-crayon-22786

05/16/2022, 8:46 PM
not as a “formality on the way to approval” though: rather, to ensure that the edge cases are encountered for, because i continue to suspect that trying to do it automatically will be too magical
✅ 1
b

bitter-ability-32190

05/16/2022, 8:46 PM
FWIW I'm also tainted by the worldview of having no source root (or mine is 
/
) so the distinction gets really arbitrary
w

witty-crayon-22786

05/16/2022, 8:46 PM
yea, source roots are important in multi-project layouts.
b

bitter-ability-32190

05/16/2022, 8:48 PM
Putting on my source-roots hats. I think the argument still holds. Especially considering sandbox inspection
In the sandbox it isn't in some other path (wtf did my file go?!)
w

witty-crayon-22786

05/16/2022, 8:49 PM
i’m not sure about that. if someone was expecting something at a repo-stripped path and it is elsewhere…
anywho.
😂 1
Powered by