hey @hundreds-father-404 on this ticket I just want to make sure I understand it right. The current

python_app

target when using

bundle

it can get files from a different folder by using

rel_path

right? I mean, that’s basically what I’m doing… my BUILD file is inside

projects/projectA

and I need to have files under

libs/3rdparty/mydata

to be packaged together.

as long as the new target archive can reproduce the files structure.

Yes, this is the default behavior. It would copy the path exactly as it is.

libs/3rdparty/mydata

would keep that same path That PR is about a feature to allow you to relocate the path at runtime, so that

libs/3rdparty/mydata

can become something you want like

my_custom_prefix/mydata

. This is helpful if you, for example, want to convert

src/python/myorg/myteam/project/pyproject.toml

to simply be

pyproject.toml

in your final

archive

or in your test

Oh so that’s the tricky part (at least to me) it would recreate the full path? I sort of expected it to only recreate the path FROM where my BUILD file lives.

one other thing that I already asked and it still gives me a hard time is the difference between files and resources and when to use on over the other

Sorry about the confusion. I started drafting a page explaining how to use both features, but was waiting to finish this new PR until then For

files

, you will always have the full path as it is in your actual project layout. In contrast,

resources

are loaded like you would load normal code in your language. If you have the file

src/python/myorg/f.py

, then you would say

import myorg.f

, not

import src.python.myorg.f

. It’s the same with

resources

, you would say

pkg_resources.get_data("my_<http://org.my|org.my>_data")

, rather than

pkg_resources.get_data("<http://src.python.my|src.python.my>_<http://org.my|org.my>_data")

oh haha yeah I got that part lol

Yes, precisely. That you want to rewrite

src/python/myorg/myteam/project/pyproject.toml

to

pyproject.toml

, but have Pants do it for you, rather than needing to do it in your test code or modifying the

archive

that Pants had created for you

haha knowing me if it’s there I’ll probably use it but yeah that sounds like a feature that should be there

Still can’t Rust

Oh, about 80% of our contributions are in typed Python 3 btw! We only use Rust when changing the internals of the engine, like optimizing how the engine reads from the filesystem. That all gets abstracted away though; where all the logic of Pants gets written is in Python

so forgive me

And not at all, this type of feedback is extremely useful. There are lots of ways of contributing to an open source project

typed Python 3

that I can do

To recap, there are three proposals out there right now: 1.

files()

target has a

sources_prefix_mapping

field. This is the current PR. But an issue is that it means if someone unintentionally updates that one

files()

target, they may end up breaking a bunch of consumers without realizing it. 2. The information lives entirely at consumers, such as your

archive()

target or your

python_tests

target. This likely has the benefit of being easier to understand because all the magical remapping is centralized to one place. But it could result in lots of duplication; it’s not composable. You can’t define a remapping and then reuse it somewhere else. 3. Keep

files()

the same as before, and add a new target type called

relocated_files

. You could have multiple

relocated_files()

targets describing the same

files()

target, but mapping them in different ways. Your

archive

and

python_tests

targets could either depend on the original

files()

, or on the

relocated_files()

target

oh! #1 I can totally see that happening on my team 😨 folks are trying to add/remove stuff from BUILD files and aren’t even testing to see if it’s working before committing it.

sidenote: the “break a bunch of consumers” is another reason to want alignment between

test

and deploy: to minimize the amount of custom code that can’t be tested, because they can each expect files to live in the same place

Are we cool with a

relocated_files

target mapping 1-1 with a

files

target? Vs. allowing

relocated_files

to aggregate multiple targets and having complex branching mapping, like “if this prefix, use this; otherwise, rewrite like this” That is,

relocated_files(
  name="tgt1",
  files_target="//:json",
  from="original_prefix",
  to="new_prefix",
)

relocated_files(
  name="tgt2",
  files_target="3rdparty/json",
  from="3rdparty/json",
  to="json",
)

vs.

relocated_files(
  files_targets=["//:json", "3rdparty/json"],
  mapping={"": "json", "3rdparty": ""},
)

Even if 1-1 results in much more verbosity, I think it’s going to be worth it.

files

and

resources

already confuses people. This could be a very confusing new feature as well. Keeping it simple will be valuable, even it results in some duplication

if the simple thing can grow into a more complicated thing, that should be fine.

if the simple thing can grow into a more complicated thing

What do you mean?

what i had in mind was

relocated_files(files_target=..)

(note, singular) growing into

relocated_files(files_targets=..)

(note, plural)… that’s an easy deprecation

I think I’m fine with starting with it being plural, so long as we only one mapping expressed in the target, via the

from

and

to

field. It’s the complex mapping in a single target that I want to avoid, e.g. the dictionary

the issue with your

tgt1

and

tgt2

sketch is that that also requires two

files

targets behind them, so four total. not a huge deal, but the mapping dict is more growable, because you can eventually remove the restriction(s)

but the mapping dict is more scalable

Yeah, but harder to reason about

is it? maybe, yea.

I think so. In my example with

{"": "json", "3rdparty": ""}

, what do you think it would do? Note that

""

matches both files, but

3rdparty

only matches one file. Should both transformations be applied? Only one of them?

the question of which patterns apply is valid regardless i think.

Not if we constrain you to only be able to have one

from

and one

to

by them being proper fields. For a given

relocated_files

target, there can only be a single mapping defined. And that gets applied to every single file

the only question (afaict) that is potentially introduced by having multiple mappings in one dict is: “are they applied in some order, or is exactly one pattern applied to a file”

I was thinking it will eagerly fail if it doesn’t match

my guess is that the dict will be easier to use in practice.

Gr. We can’t use

relocated_files(..., from=..)

. Python claims it as a keyword Any ideas for alternatives to

from

and

to

?

before

and

after

perhaps?

src/dst ?

Isn't

files

and

relocated_files

targets that are OUTSIDE the python backend. And

resources

is really

python_resources

since it focuses on packaging the files similar to how

setup.py

does it?

Yeah, this is a good way of thinking about it Raúl! In fact,

resources

used to be a field on the

python_library

target. And I think we used to have a field for

jvm_library

etc I’m going to try to write some docs on this today or tomorrow. It’s definitely a common point of confusion, understandably so.

@jolly-midnight-72759: good points, mostly yes. but the

resources

concept applies to a few different languages which support packaging files to be loaded in a runtime-specific way

ahh. good point