hello! I think maybe related to this thread https://pantsbuild.slack.com/archives/C01CQHVDMMW/p1650560808502339?thread_ts=1650558518.648039&cid=C01CQHVDMMW: I'm running into an issue where I'd like to preserve some symlinks in an output digest. In a plugin I am running

npm install

and then

npm run build

in two separate processes. I currently have the

node_modules

set as the output_digest for the first process, but when I access it all of the symlinks in the

./node_modules/.bin/

directory are materialized to real files. but since they all reference files relative to the linked location it doesn't work! is there anything I can do there? I'd like to have the process results cached since installing the dependencies can take a while and doesn't change?

but when I access it all of the symlinks in the

./node_modules/.bin/

directory are materialized to real files. but since they all reference files relative to the linked location it doesn’t work!

not sure what you mean here… the captured files should be “real files” with content, so i’m not sure what you mean by the linked location not existing… there shouldn’t be any links left

sorry! they are executable javascript files - and they load javascript files relative to the original location of the file (what the symlink pointed to originally)... does that make sense?

What you're describing here is currently why I have to do this dirty little hack https://github.com/pantsbuild/pants/blob/9b1e5f45592db0b0876f569ff614b78c180d7fba/src/python/pants/backend/javascript/subsystems/nodejs.py#L82-L87 I played around with editing the

Process

path, but couldn't find the right magic to get this running.

and they load javascript files relative to the original location of the file (what the symlink pointed to originally)... does that make sense?

yea, it does. but doesn’t that mean that you need to capture more files from the sandbox? afaict, it’s not to do with the symlink itself… just ensuring that you’re capturing “enough” of the sandbox

oooo kinda big yikes - so does it install everything globally? (I don't really know anything about node but one of my colleagues wanted to mess around with it and now I am stuck supporting it for our build tools) currently my plugin uses the system node so I don't really want to mess around with those paths! uh... right now what I'm doing is making the npm run-scripts entries just... call node directly with the actual file (and installing the modules with

npm install --no-bin-links

so it doesn't like.... accidentally work sometimes) @witty-crayon-22786 so I'm including all those files - but for example

node_modules/.bin/nuxt

has something like

const suffix = require('../package.json').name.includes('-edge') ? '-edge' : ''

in the original process - the nuxt script is a symlink to

node_modules/nuxt/bin/nuxt

(or something) and so from that directory

../package.json

resolves, but once I get the output_digest that symlink is a real file and

../package.json

is looking in a different place where it doesn't exist!

ahhh, got it.

(so for example I can have the entry in my

scripts

object in `package.json`:

"pants:build": "node ./node_modules/nuxt/bin/nuxt.js build"

which is equivalent (as far as I can tell) to:

"pants:build":  "nuxt build"

But is not like... what people are used to doing

@average-australia-85137 If you're using system node, then you should be in a better position, because at least all node-centric stuff will align, so I'm actually a bit surprised that what you're doing isn't working. After Node's all good to go, pointing at node_modules should generally be enough

@wide-midnight-78598 wrong Nate

... I hate slack name resolution

@wide-midnight-78598 - I'm not sure I'm actually in a better position? (unless you can see how I can get out of my jam above?)

@average-australia-85137: regarding the symlink issue: it would be worth commenting on https://github.com/pantsbuild/pants/pull/15211, as this is a concrete motivation for supporting symlink capture* that i hadn’t considered.

@average-australia-85137 I'm going to be messing around with the Javascript plugin tonight to add some caching. If you had a trivial sample repo, I could test it out and see if I could solve it?

bleh right now I don't have a good test case (cause our real case is in the private repo and i haven't pushed up my changes to our plugin that exposed this anyways! here's the plugin repo: https://github.com/compyman/pants-node and I think... changing the package.json to include nuxt & run nuxt build in the script would expose the problem (it... also wouldn't work because there isn't a nuxt project set up) but I think that would repo the issue

Okay, I'll take a crack at installing and running something, anyways -

@witty-crayon-22786 just commented! & thanks @wide-midnight-78598

Alright, I tried a few things out - and no luck. NPM is kinda super dependent on symlinks, at its own detriment I believe. I've had issues like this outside of pants. For your specific use case, it may be worth looking into something like

npx

to run in lieu? Doesn't solve your problem, but it's what i ended up doing for the Prettier plugin

womp yeah - I was able to work around if by directly referencing the file in the scripts object in package.json (i.e.

nuxt build

doesn't work but

node node_modules/nuxt/bin/nuxt.js build

does work!)

Definitely interested - did you get the symlink problem solved? Or worked around?

uh the work around right now is to run things like in the previous message!

Oh, hahaha, alright 🙂

haha I really don't know very much about how node works! but if you have any suggestions I'd be willing to try them out

Unfortunately, without symlinks - they're mostly hacks and slashes. So, I lean into that hack for now 🙂 Will look more into it in the next week or so, when I add some more PRs for a couple backends