Ergonomic, hermetic, user-friendly software build system for Python, Go, Java, Kotlin, Protobuf, Scala, Shell. Pants lets you fearlessly scale up your codebase!

Pants

I really like `./pants tailor`. it is a clothing-related pun, and it more or less gets at what we actually want this tool to do

And if we add a GUI we already have official emojis deployed :sewing_needle:

Only thing about `tailor` is that it's not an imperative (that is, command voice like "test", "lint", "package" etc.)

I actually interpreted it as the verb `tailor`

I think folks will typo `tailer` sometimes, but our "Did you mean?" message will help

Doctor (from brew?) got me thinking noun, but the verb is more apropos.

we should probably make `tailer` alias to the same command

`tailer` is like tailing logs or something

You need a tailor if you grow taller or a tail, er.

I don't think there's a US/UK english spelling difference here

Just realized that `tailor` is also a verb

OK, my grammar nerdery has been satisfied

I always segfault when I make calls to my inner grammar nerd.

Pants newbie here :wave: I'm trying to implement Pants on our Python repo and things are going well, but now I've gotten to typechecking. Most of our code is typechecked by mypy, but one library seems to be better handled by pytype. Is anyone aware of a pytype plugin for Pants? Barring that, is there a way that I can configure Pants such that when I run `./pants typecheck ::` I can exclude the library that needs pytype? Thanks.

Screen Shot 2021-02-08 at 8.40.55 AM.png

you could add tags to a target, this will look something like this:

and then when running: `./pants --tag=-nolint &lt;goal&gt; ::` that target will be excluded from the run.

AFAIK there is no pytype plugin for pants.

<https://www.pantsbuild.org/docs/advanced-target-selection#tags-annotating-targets>

Indeed, no pytype plugin, but we'd be happy to help you if you wanted to add one!

Appreciated... we're going to see if we can't get our lone library onto mypy, though :sweat_smile: In the meantime, I think the tagging approach should work for us.

And to be clear, although it's probably obvious, "nolint" can be an arbitrary string of your choosing.

My repository has a bunch of separate plotly dash applications. Each application can have an assets folder with JavaScript and CSS.  When running the application, any files in assets are automatically served up by the application. I'd like to share some common CSS with all applications. So I'd have an assets folder in the root of the repository with some CSS. Wondering if there is a way that I could combine the root assets with the assets in the application when calling pants run.

--/repo
  |-- assets
    |-- common.css
  |--myapp
    |--assets
        |-- application.css

So assets/common.css would become part of myapp/assets at runtime.

I think relocated_files (<https://www.pantsbuild.org/docs/reference-relocated_files>) might be what you need.

That sounds like what I need but I must be doing something wrong. I have the following in my root BUILD file:

`*files*(`
    `name = "common_assets",`
    `sources = [`
        `"assets/**",`
    `],`
`)`

In my application folder (/dash_campaign_status_report/BUILD) I have:

`*resources*(`
    `name = "assets",`
    `sources = [`
        `"assets/**",`
    `],`
`)`

`*relocated_files*(`
    `name="common",`
    `files_targets=['//:common_assets'],`
    `src='',`
    `dest='dash_campaign_status_report',`
`)`

`*pex_binary*(`
    `name = "bin",`
    `dependencies = [`
        `":assets",`
        `":common",`
    `],`
    `entry_point = "<http://dash_campaign_status_report.app|dash_campaign_status_report.app>",`
`)`

When I package :bin I see the assets from the application folder but not the common ones.

A bit more information.  When I run ./pants dependencies dash_campaign_status_report:bin, I get these entries related to assets:

`dash_campaign_status_report/assets/dash-campaign-status-report.css:../assets`
`dash_campaign_status_report/assets/dash.css:../assets`
`dash_campaign_status_report:common_assets`

So the assets in the application folder get expanded out to the two CSS files I have in the 'dash_campaign_status_report/assets' folder but the CSS I have in the root assets (assets/common.css) is not there.

I'm running pants 2.3.0.dev3 but I went back and tried in 2.2.x and 2.1.x

Hm I think this could possibly be weirdness with using files targets with a pex_binary? That's a hunch, not certain

Could you please try using the `archive` type? <https://www.pantsbuild.org/docs/resources#archive-create-a-zip-or-tar-file>

You'd keep the `pex_binary` still, and only be adding the new `archive` target. You don't necessarily need to use `archive` at the end of the day, now the goal is solely to verify those files are showing up where you expect in the archive. It will allow us to narrow down the problem

I tried that with the following archive target:

`*archive*(`
    `name = "archive",`
    `files = [`
        `":assets",`
        `":common_assets",`
    `],`
    `format="tar",`
`)`

I get the opposite from this. The tar file has the common.css but not the other ones.

Hmm, could you possibly send a zip file with the setup you're describing? You can strip all content - we only care here about directory structure, along with BUILD file content

Sure.  FYI - when I changed the assets target from resource to files then I got all the CSS in the archive.  I then tried to package the pex_binary and I got no CSS files

sample.tar.gz

K I think I see some things go on:

--

When you have this archive

```archive(
    name = "archive",
    files = [
        ":assets",
        ":common_assets",
    ],
    format="tar",
)```
You can get all the CSS files to show up correctly in the same folder iff `:assets` is `files`, and `:common_assets` is `relocated_files`. This is expected. Note that the `files` field expects file targets, not resources targets. So the resources targets get ignored. That's pretty subtle - we should add a warning

--

For the `pex_binary`, you're right that I can't get the `relocated_files` to show up. I can get `:assets` if it's a `resources` target, but not if it's a `files` target. So there's something going on there.

It looks like this is actually intentional. And now I'm remembering that it has to do with source roots. In a PEX, we strip source roots, so `src/python/myorg/app.py` becomes `myorg/app.py` in the final binary. But source roots are never stripped with a `files`/`relocated_files()` target, and it would be incorrect to put `src/assets/data.json` with that prefix iiuc

Instead, this is why the `archive` target type exists. You'll create an archive with a PEX file in it, and then the loose CSS files also accompanying the PEX. Then, when you run your PEX, it can access those loose files

It will look something like this:

```relocated_files(
    name="common_assets",
    files_targets=['assets:common'],
    src='',
    dest='dash_campaign_status_report',
)

files(
    name = "assets",
    sources = [
        "assets/**",
    ],
)

archive(
    name = "archive",
    files = [
        ":assets",
        ":common_assets",
    ],
    packages=[":bin"],
    format="tar",
)


pex_binary(
    name = "bin",
    entry_point = "main.py",
)```
Does that make sense?

Hm, no you cannot. Tricky..

Semi related, what does `./pants roots` say?

You can package arbitrary files into a `pex` no? Just by having them as `resource` dependencies

the source root would get stripped off, but in this case the source root is `.`, so that would be a noop

We should probably make this all clearer in the docs and also issue appropriate CLI warnings

Thanks for the help tonight.  Looks like I'll probably have to think of another way to make this work. Your last suggestion doesn't quite give me what I need.  This is the repository layout:

`.`
 `|── BUILD`
 `|── assets`
 `|   └── common.css`
 `└── dash_campaign_status_report`
     `├── BUILD`
     `└── assets`
         `├── dash-campaign-status-report.css`
         `└── dash.css`

If I add a resource target to the root BUILD like this:

`resources(`
    `name = "common_assets",`
    `sources = [`
        `"assets/common.css",`
    `],`
`)`

And then I add the resources to the pex_binary target in dash_campaign_status_report/BUILD like this:

`pex_binary(`
    `name = "bin",`
    `dependencies = [`
        `...`
        `"//:common_assets",`
        `...`
    `],`
    `entry_point = ...,`
`)`

I get a pex layout that looks more or less the same as the repository layout:

`.`
 `|── assets`
 `|   └── common.css`
 `└── dash_campaign_status_report`
     `└── assets`
         `├── dash-campaign-status-report.css`
         `└── dash.css`

What I want is:

`.`
 `└── dash_campaign_status_report`
     `└── assets`
         `├── common.css`
         `├── dash-campaign-status-report.css`
         `└── dash.css`

Hmm, I wonder if we should change `pex_binary` to work with files targets <@U051221NF>?

I'm actually not certain why it would be problematic to have an unstripped file like `src/assets/img.png` in the PEX? I think it wouldn't work to load via `pgkutil.get_data()`, but maybe `open()` would still work? If you don't recall, I can play around with this

Okay, I'll play with it later today to see where the limitations are. <@U01HLQBNHC6> are you able to workaround this in the meantime? Or put this particular thread on hold till the next dev release?

it would be on the coder to know how to open the file, you coulg use `pkgutil` if you happened to know that the file fell into a package, and `open()` would only work if you were running unzipped

Yeah that's what I'm thinking. We could maybe encourage `resources` because it Just Works more easily, but agreed.

And better than the confusing alternative of adding `relocated_resources`

I'm good for now. I created some symbolic links to make it work. This will get messy as things grow though so I'll stay tuned.

Didn't get to this today, setting a reminder to look tomorrow

Okay indeed this works! <https://github.com/pantsbuild/pants/pull/11551>

Should go out into 2.3.0 dev release tomorrow. We won't backport to 2.2.0 because this is technically a new feature

--

Thanks for asking about this! Hopefully will avoid lots of unnecessary confusion for future Pants users :slightly_smiling_face:

Nvm, I spoke too soon. `files()` does not play nice with PEX.

I'm thinking we do actually want `relocated_resources`. <@U051221NF> thoughts? <https://github.com/pantsbuild/pants/pull/11551#issuecomment-777926798>

It's easy code to add. Only hesitation is that we're adding yet more complexity to <https://www.pantsbuild.org/docs/resources>

<@U01HLQBNHC6> how are you opening these CSS files? Something like `with open()` or `pkgutil.get_data()`?

<https://github.com/pantsbuild/pants/pull/11551#issuecomment-778388738>

Cc <@U06A03HV1> <@U04S45AHA> for the context on why I've been looking into files and resources

I'm not opening the files. If a plotly dash application has an assets folder, it gets served by the application automatically - <https://dash.plotly.com/dash-enterprise/static-assets>

ok… then the relevant question is probably whether Plotly Dash supports “zipapps”, where the files are embedded as resources and must be loaded via `pkg_resources`… it’s not clear from that page

…maybe … <https://github.com/plotly/dash/blob/dev/dash/dash.py#L718-L755> ?

It looks like yeah, it does use resources. Greg's motivation for relocated_resources is that he has a common.css he only wants to define once, but to include in multiple distinct packages

That sounds sensible to me. It's advanced for sure, but reasonable

Certainly this new target type could be tricky for users to get right because of reasoning about source roots, but I suspect they'd know that and would rather have the possibility

Hey <@U01HLQBNHC6> could you please try out my PR and see if it solves your problem? You'll copy that new target type and the rules registration into your current plugin. Register the new target type and rules in your register.py

<https://github.com/pantsbuild/pants/pull/11559|https://github.com/pantsbuild/pants/pull/11559>

I developed against 2.3, but I think it should work with 2.2 too 

And it needs to be at a different relative path in each app? If not, can just define it once using resources without relocating

It needs to be merged with other CSS files. He has the structure &lt;project&gt;/assets. All the CSS needs to go in there. He wants to add in common.css into that pre-existing folder

That should be fine if the source root is correct: `src/resources/assets`

His source root is the build root. The final path should be &lt;project&gt;/assets/common.css

Right now, it's something like ./common_assets/common.css

I'm current using 2.3 and I will give your PR a try.  I can't do it right now but I will try a little later.  Thanks!

traditionally this has been accomplished by defining more source roots, similar to defining more PYTHONPATH entries, which all end up merged.

Even though the files are being loaded as resources by dash (IIUC), can't they be packaged using relocated files? Or are the existing relocation primitives not expressive enough to do this?

<@U04S45AHA> That was Greg's original attempt, which worked great with ./pants run. But it doesn't work with ./pants package because we don't include files in pex_binary. Hence my PR to "fix" that

But I'm not comfortable landing my files() PR, given how ./pants run and ./pants package would diverge when source roots are used. It would fix Greg's case because his source root is the repo root, but it wouldn't work for others

Hence, me thinking relocated_resources is the way to go

and to be clear: the final location on the pythonpath is supposed to be `${project}/assets/common.css` , rather than `assets/common.css` …?

Yes, that is correct. Where he has &gt;3 distinct projects 

so he’d have to relocate the one file differently for each consumer, i guess… not actually possible to do with source roots.

Yeah, one relocated_resources target per project. Which is totally sensible, we expected that to be the case when designing relocated_files

`run` and `package` diverging seems like a separate (related) problem, perhaps.

There, I think the proper "fix" would be no longer copying loose files into the chroot for run, but that doesn't seem ideal. The issue is because we strip source roots when packaging a binary, but I think we need to keep doing that iiuc

Instead, I plan to address that discrepancy through a warning when you package a pex_binary with files in the closure that those files won't be included, and to use either archive or resources instead 

Hey <@U01HLQBNHC6>, did you have a chance to try out the PR? <https://github.com/pantsbuild/pants/pull/11559>

Sorry no. Following the discussion after you had asked I wasn't sure if you still wanted me too.  I will give it a try now.

If you could, that'd be great. Thanks!

Imo, we should land `relocated_resources` - Dash is loading these things as resources, not as files. And the use case you have is an advanced one easy to mess up, but it's a legit use case

Any Pants-related reason not to upgrade a Macbook to Big Sur?

Yep works fine for me. I think there were some sdists that didn't recognize the new versioning scheme but that's been fixed 