Proposal Elide `file` and just use `resource` everywhere Goo

Question

Proposal Elide `file` and just use `resource` everywhere Google Doc <https docs google com document d 1pGib2ZJiyltfij0aZMm81S2XbATa8s6 ghuLnf2Xr k edit usp=sharing>

bitter-ability-32190 · Answer

Shoutout to < hundreds father 404> for the help

bitter-ability-32190 · Answer

I m working on the code changes now to serve as a PoC

bitter-ability-32190 · Answer

The PoC is yielding all sorts of fruit Lots of improvements in behavior

hundreds-father-404 · Answer

< happy kitchen 89482> < witty crayon 22786> this has my unequivocal blessing and Josh has a proof of concept working But needs to be split up into prefactors to derisk review Before doing that would be helpful to get your blessing when either of you get a chance

witty-crayon-22786 · Answer

yea will take a look this afternoon

witty-crayon-22786 · Answer

thank you

happy-kitchen-89482 · Answer

Looking now

bitter-ability-32190 · Answer

I think < enough analyst 54434> had thoughts from our thread as well

bitter-ability-32190 · Answer

OK I think I m going to revamp this one into a V2 to better capture the intent and change the proposal a smidge Prose isn t my strong suit so thanks for indulging me slightly smiling face

bitter-ability-32190 · Answer

OK V2 <https docs google com document d 1gdJjhfAiVymTTcV6sRmsHTvQrzgaFmXdY8 fkdFaqxc> Now primarily focused less on the technical difficulties with 2 assets types and focused on the core of the issue there s only one type of asset assets Trying to force a choice between two types based on runtime behavior of code has failed us and our users Also both `file` and `resource` will be deprecated for a single unambiguous type `asset`

bitter-ability-32190 · Answer

< witty crayon 22786> < happy kitchen 89482> < hundreds father 404> < enough analyst 54434> please take another look Also < curved television 6568> you might have an opinion too wink

hundreds-father-404 · Answer

Thanks Proposal looks good to me I like the edit to call it `asset` and clarify what s going in this is about not forcing specific functionality into a target type

bitter-ability-32190 · Answer

I actually have a real world use case of relocating a `file` sitting outside our main code tree into our main code tree as a `resource` It s impossible today but would be trivially supported by `relocated assets` in the propsal

curved-television-6568 · Answer

Oh boy I feel like a single `assets` target ought to be able to cover the use cases for `files` and `resources` but I see all the nuances presented in the comments in that google doc it makes my granted somewhat tired yawning face head spin stuck out tongue

witty-crayon-22786 · Answer

talked with Eric briefly on another topic and one thing that occurred to us was that one potential option would be to say we don t support distinguishing between a resource pkg data Classloader loaded file and a loose file you ll always get both

hundreds-father-404 · Answer

Concrete example with a `go package` depending on an `asset` at compilation time we include the asset in the sandbox Go will need it if the user is using `embed` Otherwise it will just be ignored by the compiler at test time we include the asset in the sandbox The user might access it via Filesystem APIs Otherwise ignored

bitter-ability-32190 · Answer

That s kind of another way of slicing the proposal An asset is simply a declaration of a thing and you specify dependencies on that thing How that thing gets packaged or copied or materialized is in the details of the action taking place not in the details of how it is declared

witty-crayon-22786 · Answer

looses flexibility for test targets i can t avoid both packing a huge file into my app and placing it in the sandbox but probably wouldn t have any non performance implications

hundreds-father-404 · Answer

Interesting point w performance of huge assets Would immutable digests make any difference

witty-crayon-22786 · Answer

i don t think so if you re testing a pyoxidizer or go binary then you would be embedding an X GB file into a binary

bitter-ability-32190 · Answer

gt i can t avoid both packing a huge file into my app and placing it in the sandbox Is that true would the asset not be a dependency of the test and therefore be place din the sandbox

witty-crayon-22786 · Answer

< bitter ability 32190> if you don t have a distinction between types and always place it in both places via then we wouldn t know to only put it in the sandbox

bitter-ability-32190 · Answer

WHy would it be in both though if it isn t a dependency of the foo source Just the test

witty-crayon-22786 · Answer

either way probably depends whether a test pulls in transitive or only direct dependencies but can trigger the case either way

witty-crayon-22786 · Answer

but i guess you re saying that a test would only be able to get a direct dependency on an `asset` as a loose file i e a test target depending on an `asset` would always get it as loose

hundreds-father-404 · Answer

gt foo source Go only has a single target type `go package` So when you add a dep to it it might be because the source code uses the embed resources API at compilation time Or it could be because tests are reading it as a loose file w Filesystem API By getting rid of `file` vs `resource` Pants loses the ability to distinguish intent Other than the performance concern I think That Is Just Fine though

bitter-ability-32190 · Answer

gt Or it could be because tests are reading it as a loose file w Filesystem API Help me understand why it d be a dependency of the `go package` if it is a test time dependency

hundreds-father-404 · Answer

Because the `go package` includes both the source files amp test files There is no `go tests` it is just `go package`

witty-crayon-22786 · Answer

can embed up to 2gb in a go binary <https groups google com g golang nuts c jFKGLbTv2XQ>

bitter-ability-32190 · Answer

Ahhh ok so there IS a case where one asset can be eithe rpackaged or sandbox filed In that case I think both proposals make sense 1 Only have one asset type assets Because to hit the point home we shouldn t be defining how an asset is used at runtime by forcing the user to classify the asset at metadata time 2 In places where the usage is ambiguous should hopefully be be rare I can only think of tests which themselves must be compiled We employ Stu s edge metadata idea Allow users to choose which asset is loose vs packaged in the test binary

bitter-ability-32190 · Answer

Which we can tackle 2 separately

witty-crayon-22786 · Answer

with the difference from bazel being that it would only be in cases where it was ambiguous

bitter-ability-32190 · Answer

coke

witty-crayon-22786 · Answer

part of the reason why i think that bazel might have recursive `data` dependencies is for requests like this <https github com pantsbuild pants issues 15500> i need to better understand why Alonso wanted transitive `file` dependencies there gt The only alternative now is making the `file` dependencies being direct which is not optimal as it forces downstream targets knowing about all of their upstream potential `file` dependencies

witty-crayon-22786 · Answer

will comment

bitter-ability-32190 · Answer

Yeah transitive assets is an interesting case Perhaps its the case that we dont inherit transitive assets in tests as those should be packaged with their sources in the case of Python we inherit them in the sandbox but not the target

bitter-ability-32190 · Answer

For that ticket it seems like a cheating shortcut to include a dependency on the source so that all test environments inherit the asset But thats my 2c

witty-crayon-22786 · Answer

gt For that ticket it seems like a cheating shortcut to include a dependency on the source so that all test environments inherit the asset i mean that s what the distinction between `resources` and `files` is

witty-crayon-22786 · Answer

but i don t think i understand what syntax you re thinking of there

witty-crayon-22786 · Answer

bitter-ability-32190 · Answer

It s arguably very nuanced to say all environments you test me in requires this specific loose file in the sandbox therefore I ll add it to the source and expect the test environments to inherit the dep transitvely

witty-crayon-22786 · Answer

it can be a thing if its library code rather than the very top level of your app which loads the loose file although libraries expecting files in particular places is smelly

bitter-ability-32190 · Answer

Exactly Pants has some control over the environment we test your code in but no control over the environment your code is deployed in Having metadata provide weak hints is confusing and misleading Then we end up throwing away your dependency when packaged or not knowing how to transitively consume it because your weak hint doesn t jive with the current action

witty-crayon-22786 · Answer

sure but the symmetry is supposed to be between a test and prod it s just not between `deploy jar` `go binary` or `pex binary` and a test rather between an `archive` `docker image` and a test but i think that that is orthogonal to the transitive or not discussion

bitter-ability-32190 · Answer

Ohh you re saying that if we have an archive which depends on a `pex binary` when we build the archive we throw away the files for the `pex binary` but then transitvely inherit them back for the archive itself Talk about nuanced dizzy face

bitter-ability-32190 · Answer

And now I think I understand how you see the world I do think transitive dependencies is root of the evil here and is extremely relevant to the discussion The distinction between the two asset types breaks down when you think about a singular target because most singular targets can only consume one asset type But when you blend in transitive dependencies it gets messy because a target which can only logically consume asset type A might be depended on by another which supports B or A amp B Then what We need a hint to know what the expectation is

witty-crayon-22786 · Answer

transitive dependencies are one aspect yea the other is some consuming targets actually caring about the distinction between the types

witty-crayon-22786 · Answer

so those seem like the two big open questions Alonso is in UTC i think so should get more info on that one tomorrow

witty-crayon-22786 · Answer

oh related to gt the other is some consuming targets actually caring about the distinction between the types amp amp gt In places where the usage is ambiguous should hopefully be be rare I can only think of tests which themselves must be compiled we have <https www pantsbuild org docs reference python tests coderuntime package dependenciescode> already as an edge metadata case

witty-crayon-22786 · Answer

which basically amounts to a give me `files` dependency

bitter-ability-32190 · Answer

The world is clearer I think one asset type is ideal as it is a better representation of the intent of the user here is an asset how it is consumed will be called out at the consumption site as trying to tell you that here is not always ideal or possible Then the problem is truly how do we let the user tell us their intent This also simplifies Pants code a lot because on the asset generation side there s no duplication On the asset consumption side theres no ambiguity on what the user desired because they told us We no longer have to guess or make decisions for them

witty-crayon-22786 · Answer

to be clear gt On the asset consumption side theres no ambiguity on what the user desired because they told us is already supposed to be the case to the extent that we currently have unclear docs on this subject we probably still will but perhaps being contextualized on a target by target basis would help

bitter-ability-32190 · Answer

I think it s ambiguous because being told a `python source` depends on a `file` is meaningless especially when packaging unless you consider a `python test` or an `archive` which is depended on by a `pex binary`

witty-crayon-22786 · Answer

maybe depends whether the `data dependency` equivalent is transitive

hundreds-father-404 · Answer

this convo might be better as a video conference btw so much nuance here

bitter-ability-32190 · Answer

Well for the most part I got what I needed stuck out tongue which was to understand better where that nuance is and why it isn t accounted for by using one asset type

bitter-ability-32190 · Answer

How we solve that going forward might be worth a call though wink

bitter-ability-32190 · Answer

OK I think we can roll this forward with Eric s way of slicing it it s always in the sandbox It s a re lensing of the proposal so I ll work with them on re phrasing the docs as such Hopefully the nuance of `runtime dependencies` can be avoided and we can enjoy a simple target harmony I have thoughts on our rollout deprecation too so stay tuned

bitter-ability-32190 · Answer

OK I ve responded to comments highlighting that there s some misconceptions about how files get consumed in docker image and archive added a section about another way of thinking about this being that the asset simply is in all sandboxes where file resource were I also changed the deprecation rollout plan to us adding asset as a new type and not touching file resource It s gonna be painful for a release to support 3 asset types but we can pull the plug soon after This ensures we don t knock over the current house of cards and makes incremental PRs easy I also moved it into our shared folder

bitter-ability-32190 · Answer

Gahh I keep forgetting about JVM s Pants metadata is the only metadata case We ll maybe wanna have a short meeting to find which idea we like best or hate least

hundreds-father-404 · Answer

Stu was open to the idea of a call together

witty-crayon-22786 · Answer

yea still think this might be worthwhile can get a Doodle out to schedule it if you re interested

witty-crayon-22786 · Answer

sounds like folks are i ll do that

bitter-ability-32190 · Answer

Much appreciated

witty-crayon-22786 · Answer

< hundreds father 404> < happy kitchen 89482> < ancient vegetable 10556> < bitter ability 32190> and anyone else who is interested

witty-crayon-22786 · Answer

this is potentially related to scoped dependencies <https github com pantsbuild pants issues 12794>

witty-crayon-22786 · Answer

i had forgotten that that was the last time producer decides vs consumer decides aka node vs edge metadata had come up

happy-kitchen-89482 · Answer

It might be good to have < enough analyst 54434> participate

witty-crayon-22786 · Answer

and < curved television 6568> no pressure to attend but you had had thoughts on the scoped dependencies ticket

witty-crayon-22786 · Answer

booked this for tomorrow at noon PT let me know if you d like an invite

hundreds-father-404 · Answer

Any chance we could do it earlier like 11 am PT I would like to join but will have to leave for the airport 15 minutes in I can always do voice call from Uber

witty-crayon-22786 · Answer

i can move my other meeting if an hour earlier works for < happy kitchen 89482> and < bitter ability 32190>

bitter-ability-32190 · Answer

+1

happy-kitchen-89482 · Answer

WFM

bitter-ability-32190 · Answer

Gonna be a smidge late Also sorry ahead of time gonna be slurping on lunch joy