https://pantsbuild.org/ logo
n

nice-florist-55958

09/24/2022, 6:40 PM
How hard would it be to support absolute paths for resource targets’ sources? The issue originates from the following dilemma: our dev environment assumes access to a globally replicated network file system, where lots of resources and binaries are installed, but bringing those resources into an application at runtime is a practice we’d prefer to avoid (having it documented in a BUILD file is much more maintainable and easier to track dependencies from) and is not possible anyway when the dist target is Docker or some other environment that does not have such access. On the other hand, bringing those resources under the project requires us to manually keep them up to date ourselves, not to mention potentially resulting in lots of things you’d rather not have in source control. From our perspective, allowing Pants to bundle resources and data into binaries and distros from network paths is a good solution that balances these issues.
h

happy-kitchen-89482

09/25/2022, 3:03 PM
One issue that comes to mind is file watching. How would Pants know when those external files have changed so it can invalidate builds that include them?
Do they change extremely rarely, or never?
n

nice-florist-55958

09/25/2022, 4:17 PM
In this case they are immutable (unless you refer to a release (sym)link) as everything is installed on that file system via a specific project release.
So X.rc always lives under an extra sub-folder with a specific version name, eg, /afs/global/<proj>/<ver>/etc/X.rc
h

happy-kitchen-89482

09/26/2022, 1:52 AM
Ah, ok, so something could be made to work in practice perhaps
n

nice-florist-55958

09/29/2022, 4:23 PM
@happy-kitchen-89482 What’s your impression on how hard this would be? Is it on the same level as when I implemented a plug-in for “relocatable resource” (a la file) and doable by a newbie or are we getting into the land of rust/engine internals?
h

happy-kitchen-89482

09/29/2022, 5:36 PM
We do already support immutable sources fetched via HTTP: https://github.com/pantsbuild/pants/pull/15457
So if there’s a way to interface with these sources via HTTP then you already have a solution
Otherwise, could extend this to support file:// URLs
1
h

hundreds-father-404

09/29/2022, 5:38 PM
Maybe that already works?
n

nice-florist-55958

10/03/2022, 5:44 PM
Indeed file:// already works (we use this in pants.toml for any binaries it needs from the outside world like pex or scc). Interestingly, just prepending file:// to an invalid file path in the source field shows Pants does indeed try to find it (raising invalid field error reporting 0 files found instead of raising a no abs path error). Changing to a valid file path gets us further: it does try to package, but it gets stuck in an infinite loop of retries.
h

happy-kitchen-89482

10/03/2022, 6:24 PM
huh
would not have expected that to work
is that in an
http_source
?
n

nice-florist-55958

10/03/2022, 6:25 PM
Regular resource target.
h

happy-kitchen-89482

10/03/2022, 6:25 PM
Even more surprised
n

nice-florist-55958

10/03/2022, 6:55 PM
Idk, wouldn’t think too much of it since now I can’t repeat the behavior of it continuously trying, but restarting from “file system changed” — only the “0 files found when expecting 1” error now. The bit about using file:// in the pants.toml config is still a thing though!
I just read about http_source, let me try that.
With http_source everything works as expected, just use file instead of http proto :) Unfortunately it really doesn’t like it when you try to use envvar interpolation. But I guess that’s dangerous anyway re: cache. Does pants cache the downloaded file and rely on http_source input args not changing? I wonder if the same concept could be applied to envvar interpolation?
h

happy-kitchen-89482

10/03/2022, 7:20 PM
envvar interpolation into the path of the file?
n

nice-florist-55958

10/03/2022, 7:30 PM
Yeah
h

happy-kitchen-89482

10/03/2022, 11:45 PM
That would be a feature to add I guess
I think it would be fine wrt caching, since the content is checked vs the sha256 anyway
4 Views