https://pantsbuild.org/ logo
#development
Title
# development
a

ancient-vegetable-10556

03/27/2023, 5:35 PM
Is there anything trivial on 2.17 that needs to be done to make the new JS backend not throw errors when doing `--changed-since`/anything else that scans the repo?
b

bitter-ability-32190

03/27/2023, 5:36 PM
Does it require some tool locally?
go
backend had/has this problem 😞 I just disable it in my local pants config for the repo
a

ancient-vegetable-10556

03/27/2023, 5:36 PM
can’t, there are
javascript_sources
in BUILD files
b

bitter-ability-32190

03/27/2023, 5:36 PM
🫠
a

ancient-vegetable-10556

03/27/2023, 5:36 PM
If it’s just an environment thing on my machine, I can fix it, otherwise, I’d rather see the changes backed out until Pants works cleanly with the backend enabled
@worried-painter-31382 ^ping
w

worried-painter-31382

03/27/2023, 5:40 PM
What's the error sorry?
a

ancient-vegetable-10556

03/27/2023, 5:40 PM
Filing a ticket, and I’ll link momentarily
👍 1
w

worried-painter-31382

03/27/2023, 5:42 PM
stu found weirdness w.r.t there being a package.json file in some directory I cannot see (I think it was in the doc directory?), I have https://github.com/pantsbuild/pants/pull/18523 for that particular error
a

ancient-vegetable-10556

03/27/2023, 5:43 PM
w

worried-painter-31382

03/27/2023, 5:43 PM
hmm
a

ancient-vegetable-10556

03/27/2023, 5:44 PM
Happy to debug whatever is necessary on my machine, but generally speaking I’d prefer to see the backend be able to be turned off if our general development instructions can’t be followed
w

worried-painter-31382

03/27/2023, 5:47 PM
There's very little that can actually go wrong here (in theory), I'll have to check what the silicon tarballs contain for the node install
a

ancient-vegetable-10556

03/27/2023, 5:51 PM
let me know if I can look at anything for you; my strong preference is that this works 🙂
w

worried-painter-31382

03/27/2023, 6:27 PM
The tarball looks the same to me as intel linux, except the exe (for obvious reasons...) Can you preserve the sandbox? Is the file (
../lib/cli.js
) in there? Do you have read access in those folders? Have you brew installed npm or node (should not be required, just wondering because I notice that for some reason
/bin
is set in the process PATH)?
a

ancient-vegetable-10556

03/27/2023, 6:27 PM
OK, working from backwards: I have npm and node on my machine
let’s see about preserving the sandbox
Copy code
chrisjrn@chrisjrns-MacBook-Pro pants-sandbox-03GF3M % ls -la
total 72
drwxr-xr-x    8 chrisjrn  staff    256 Mar 27 11:28 .
drwx------@ 508 chrisjrn  staff  16256 Mar 27 11:28 ..
lrwxr-xr-x    1 chrisjrn  staff     45 Mar 27 11:28 ._npm -> /Users/chrisjrn/.cache/pants/named_caches/npm
lrwxr-xr-x    1 chrisjrn  staff    155 Mar 27 11:28 __node -> /private/var/folders/0m/h2n902qn38b8555xq8z0w_h40000gn/T/immutable_inputs9SG4Ex/.tmpXga02S/a97a3f6ba08afdca934b08117b3f19f04975937860fa883bf39cc26c20ec8406
-rwxr-xr-x    1 chrisjrn  staff    357 Mar 27 11:28 __run.sh
-rw-r--r--    1 chrisjrn  staff  23261 Mar 27 11:28 package-lock.json
-rw-r--r--    1 chrisjrn  staff    528 Mar 27 11:28 package.json
-rw-r--r--    1 chrisjrn  staff   1452 Mar 27 11:28 script.cjs
w

worried-painter-31382

03/27/2023, 6:29 PM
right, so the
__node
is the place where I'd expect the hermetic node installation be
and for the
lib/cli.js
(it's an node/npm internal script) to be
Would be interested to see what
node
your
__run.sh
eventually picks up. I'm not very good with mac but I see at least
/usr/bin/env bash
and
usr/bin/env node
shebangs in the code path to where you hit the error, so maybe the other installs are polluting the sandbox somehow? Still doesn't really explain why a node install is missing lib/cli.js, but maybe that is just a symptom of something else?
a

ancient-vegetable-10556

03/27/2023, 6:50 PM
Yeah, if you’re seeing calls to
env
, things are probably going to go wrong 🙃
unless you’re being very specific about constructing $PATH
w

worried-painter-31382

03/27/2023, 6:50 PM
I guess adding
/bin
to it isn't considered specific 😅
a

ancient-vegetable-10556

03/27/2023, 6:51 PM
OHGODNO
w

worried-painter-31382

03/27/2023, 6:51 PM
hehe, that was heritage, I didn't dare touch that when I started this JS journey
a

ancient-vegetable-10556

03/27/2023, 6:53 PM
Copy code
chrisjrn@chrisjrns-MacBook-Pro pants-sandbox-03GF3M % which node
/opt/homebrew/bin/node
chrisjrn@chrisjrns-MacBook-Pro pants-sandbox-03GF3M % /opt/homebrew/bin/node --version
v19.7.0
so there’s only one
node
on my system
where are you constructing those processes?
Let’s take a look at how the process environment is constructed
w

worried-painter-31382

03/27/2023, 6:56 PM
node is installed via a binary download and sent into the process via an immutable digest
then it is exposed by adding that installed-to-dir to PATH
a

ancient-vegetable-10556

03/27/2023, 6:56 PM
where in the codebase?
w

worried-painter-31382

03/27/2023, 6:57 PM
src/python/pants/backend/javascript/subsystems/nodejs.py:237 for the env
a

ancient-vegetable-10556

03/27/2023, 6:57 PM
OK, let me take a look
OK, well
/bin/bash
is ancient, and
/usr/bin/bash
is not.
So
PATH
is just
/bin
and wherever the self-contained SDK manifests?
w

worried-painter-31382

03/27/2023, 7:01 PM
yarp
a

ancient-vegetable-10556

03/27/2023, 7:03 PM
I’d probably want to use
BinaryShimsRequest
to get
bash
etc on the
PATH
rather than using
/bin
👍 1
Let’s see if the error goes away if it calls a modern
bash
w

worried-painter-31382

03/27/2023, 7:03 PM
I think you need
sh
as well, just a heads up
a

ancient-vegetable-10556

03/27/2023, 7:04 PM
Copy code
"PATH": f"/usr/bin:/bin:{self.binary_directory}",
doesn’t change the output afaict
😕 1
w

worried-painter-31382

03/27/2023, 7:08 PM
Another thing to test I guess is setting
PANTS_NODEJS_KNOWN_VERSIONS
to nothing and set the version to your local install, it should find any node you have on PATH, which I guess should work if the hypothesis is what I think it is
Wouldn't really tell us much about why the downloaded install is being prickly though...
a

ancient-vegetable-10556

03/27/2023, 7:10 PM
Copy code
version, platform, sha256, filesize = [x.strip() for x in version_str.split(“|”)]
ValueError: not enough values to unpack (expected 4, got 1)
hang on
overriding
PANTS_NODEJS_VERSION=19.7.0
worked
w

worried-painter-31382

03/27/2023, 7:11 PM
ah, right
PANTS_NODEJS_KNOWN_VERSIONS
wouldn't match that, of course
(I think the other error was an empty string as list-only element instead of empty list thing)
a

ancient-vegetable-10556

03/27/2023, 7:12 PM
aha
w

worried-painter-31382

03/27/2023, 7:17 PM
Soo when pointing to the brew installed node it "all works"? Or just the option parsing?
a

ancient-vegetable-10556

03/27/2023, 7:18 PM
The rules that were getting picked up did not fail
so it eventually got through to running the tests I wanted to run
I note that our CI isn’t running against mac os
arm64
, perhaps (annoyingly) this is relevant
w

worried-painter-31382

03/27/2023, 7:28 PM
Mm, I just tried installing npm via
apt
(ubuntu 22), which I guess is the closest equivalent we have here, and it is still well behaved on
main
running
Copy code
./pants --no-local-cache test src/python/pants/backend/javascript::
which npm
/usr/bin/npm
a

ancient-vegetable-10556

03/27/2023, 7:29 PM
let me check that invocation on my machine
that exact invocation passes
w

worried-painter-31382

03/27/2023, 7:30 PM
oh
a

ancient-vegetable-10556

03/27/2023, 7:30 PM
Now checking --changed-since with local cache switched off
(so slow)
that also passes
and switching the local cache back on fails
well that’s just weird
so that leads me to believe this is a caching bug on my machine. Let’s see what happens if I move the pants cache out of the way for a moment
w

worried-painter-31382

03/27/2023, 7:53 PM
the tar contains symlinks, is there a gotcha there with immutable digests? I'm well out of my depth on pants intrinsics 😬
a

ancient-vegetable-10556

03/27/2023, 7:54 PM
The experts on symlinks are probably @bitter-ability-32190 and @witty-crayon-22786
(that said, blowing away my pants cache and starting again seems to have fixed the issue)
w

worried-painter-31382

03/27/2023, 7:54 PM
Im just postulating based on the error, it would makes sense that a symlink that somehow got treated as a file would not find relative paths
The
BinaryShim
comment is probably still relevant though I assume? Should I create an issue for that?
w

witty-crayon-22786

03/27/2023, 7:57 PM
symlinks should be correctly handled when a tar file is extracted/captured/splatted later
a

ancient-vegetable-10556

03/27/2023, 7:58 PM
@worried-painter-31382
The
BinaryShim
comment is probably still relevant though I assume? Should I create an issue for that? (edited)
Yeah, you should be explicitly putting binary dependencies on your path (possibly with version-matching tests if any binaries are particularly version-sensitive; POSIX tools aren’t)
w

worried-painter-31382

03/28/2023, 7:40 AM
https://github.com/pantsbuild/pants/issues/18609 please add the javascript label 🙂