https://pantsbuild.org/ logo
Join Slack
Powered by
<https://github.com/pantsbuild/pants/issues/19054>...
# development
w
https://github.com/pantsbuild/pants/issues/19054 I think this is a must have to not have massive IO in the JS backend, but I'm either missing something obvious or in need of other ideas how to achieve it.
b
So we have a plan where immutable digests are going away, to be replaced with an includelist. The only thing I think blocking it is we don't support directories for the new style, only files. CC @witty-crayon-22786 I think we can add symlink support to the large file store, but it is only used for includelisted digests. I can file a ticket
f
What if the processor executor supported symlinking to "in place" large files? I.e., treat node_modules in the repo as the place to symlink to.
The file watch support could be used to know when to recompute the digest.
b
I think that'd be much more difficult and would break the existing model?
f
probably would unless it was implemented via a read-only bind mount
so scratch "symlinking" in my prior comment and replace with bind mount.
although probably only relevant on linux
w
CC @witty-crayon-22786 I think we can add symlink support to the large file store, but it is only used for includelisted digests. I can file a ticket
the original ticket is still open: https://github.com/pantsbuild/pants/issues/17712
@worried-painter-31382: but yea. either marking them explicitly as immutable_inputs, or helping with the implementation of that one. even if we didn’t actually change the implementation of immutable_inputs to be persistent (as Josh mentions), and instead just changed to using a heuristic to decide which things to make immutable, that would be sufficient.
oh… the original original was here: https://github.com/pantsbuild/pants/issues/17282
could probably re-open that one actually.
done.
w
Interesting. As far as I can grok this (sorry, only skimmed through the discussion), a heuristic would not solve my problem because the node resolution algorithm would fail as long as there isn't a top-level folder actually named node_modules, symlinks pointing to other locations would cause the resolution to fail like I observed in my tests.
w
yea, see the comment I left on the ticket: might require a wrapper script
👍 1
w
Indeed, a wrapper script should work with current immutable_digests. Is it (near) future proof though? If immutable digests as I know them are going away, what I'd like to see from https://github.com/pantsbuild/pants/issues/17282 is there some way to construct an immutable directory as a "unit" instead of letting a heuristic decide, or flattening the directory structure. Neither of them would work here OOTB, and neither (I think) could be scripted around as easily as the current immutable_digests.
b
We use the huerstic for files. For directories it'd always be opt-in
(and files are also opt-in-able)
w
It should be future proof to copy the contents of a directory-which-might-be-a-symlink into a new directory that you are certain is a directory
But I suppose it depends what other hidden requirements the node directory structure has
But also, the include/exclude list approach should allow you to mark a directory as being mutable while all of its contents are immutable which I think would also solve this?
(depends on the implementation I suppose.)
w
All files must (eventually) actually hardlink to a folder called 
node_modules
.node doesnt preserve symlinks, it follows them and then attempts to resolve imports from that location, I think is the crux of the issue
Open in Slack
PreviousNext
Powered by