@ancient-vegetable-10556 Regarding this awesome

adhoc_tool

workflow (https://github.com/pantsbuild/example-adhoc/blob/main/javascript/BUILD), how should a tool like

pnpm

be handled, which places all installed deps into a global content store, and then hardlinks them into the local repo? https://pnpm.io/faq#why-does-my-node_modules-folder-use-disk-space-if-packages-are-stored-in-a-global-store

Hrrrrrm. How does it figure out where the global store lives?

And yet more specifically:

ProcessExecutionFailure: Process 'the `adhoc_tool` at frontend/web:install' failed with exit code 1.
stdout:
Lockfile is up to date, resolution step is skipped
Progress: resolved 1, reused 0, downloaded 0, added 0
Packages: +627
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are copied from the content-addressable store to the virtual store.
  Content-addressable store is at: /Users/sj/Library/pnpm/store/v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 627, reused 112, downloaded 0, added 101
Progress: resolved 627, reused 230, downloaded 0, added 231
Progress: resolved 627, reused 391, downloaded 0, added 390
Progress: resolved 627, reused 411, downloaded 0, added 409
Progress: resolved 627, reused 477, downloaded 0, added 471
Progress: resolved 627, reused 564, downloaded 0, added 560
.../node_modules/@sveltejs/kit postinstall$ node postinstall.js
 ELIFECYCLE  Command failed with exit code -2.
Progress: resolved 627, reused 626, downloaded 0, added 627, done
.../node_modules/@sveltejs/kit postinstall: Failed

However, when I open the sandbox and run it manually:

⏺ T/pants-sandbox-y2xLyl % /bin/bash -c $'cd frontend/web && /usr/local/bin/pnpm install'                                                                                                                                                                                               ⎇ 
Lockfile is up to date, resolution step is skipped
Packages: +627
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Progress: resolved 627, reused 626, downloaded 0, added 0, done
node_modules/.pnpm/@sveltejs+kit@1.10.0_svelte@3.55.1+vite@4.1.4/node_modules/@sveltejs/kit: Running postinstall script, done in 1s
Done in 3.4s

this seems fraught.

Yeah, gonna fall back to npm shortly, just wanted to try the existing wrokflow

You would need to reconfigure the global store to be somewhere inside the sandbox, otherwise there’ll be shenanigans around caching.

Yeah, that's one question mark - but the post-install flow seems to still be a problem.

stderr:
npm WARN deprecated stable@0.1.8: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility>
npm WARN deprecated @types/sass@1.45.0: This is a stub types definition. sass provides its own type definitions, so you do not need this installed.
npm ERR! code ENOENT
npm ERR! syscall spawn sh
npm ERR! path /private/var/folders/18/q1r7phps28nc9rx5j_0t3jmm0000gp/T/pants-sandbox-96EvqO/frontend/web/node_modules/@sveltejs/kit
npm ERR! errno -2
npm ERR! enoent spawn sh ENOENT
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/sj/.npm/_logs/2023-03-13T19_08_14_541Z-debug-0.log

npm ERR! syscall spawn sh

what’s this about?

https://docs.npmjs.com/cli/v6/using-npm/scripts https://github.com/sveltejs/kit/blob/master/packages/kit/postinstall.js

"postinstall": "node postinstall.js"

I guess I’m trying to figure out what it’s trying to spawn, and wether it’s expecting something to live on the

PATH

.

Yeah, I'm trying to get that too - it almost looks like it's complaining about

sh

which doesn't make sense to me

That’s exactly what it’s doing

Ahhhh right, I was just coming to that realization - the path is wiped out

so try making a

system_binary

for

sh

, and adding it as a

runnable_dependency

on the

adhoc_tool

Completely forgot we wipe out the PATH 🤦‍♂️

hooray for reproducibility?

Lol, my mind has been blanking on everything these past few months. Can barely remember how to compile code

I feel that 🙂

I've been fighting with how node does stuff for YEARS. Like, the happy path is perfectly fine, but anything straying a hair away is brutal

Cool! Anyway, I’d definitely suggest trying to make

pnpm

not use the global store, otherwise you are going to get weird cacheability results

Yep, I think that's a fallback - but my policy is always "run EXACTLY what I do in dev, and see what 💥 "

I think “rules skipping due to cacheability” is likely to be an error that presents subtly down the track

Yep, agreed. I think this is already a good workflow/policy to ensure in the new JS backend as well - especially if we use corepack to install installers

with backends, we’re able to make opinionated choices about tools that match Pants’ worldview. With

adhoc_tool

I can sit in a corner, visibly sobbing, saying “please don’t do it that way” while the world ignores me 😄

workdir

?

Not really… but you can use

f"{build_file_dir()}/blah

if you really need it.

I'm just looking at this error:

npm ERR! enoent ENOENT: no such file or directory, open '/Users/sj/Developer/mydir/package.json'

Where the BUILD is located in

/Users/sj/Developer/mydir/frontend/web

run_shell_command(
    name="build",
    command="npm run build",
    execution_dependencies=[":install",],
)

I would have thought it was relative to the BUILD root

run_shell_command

is not executed in the sandbox

So, it always takes the working dir of where

pants

was executed? Or project root?

you may need to put a

cd {chroot} &&

in front of the command; or move files into a useful location

Just tried it in a subdir, it looks in the project root still. Okay cool - I think I have all the info I need, I'll report back when I have this thing compiling (hopefully)

enjoy

Yayyyyy

Run npm run preview to preview your production build locally.

> Using @sveltejs/adapter-static
  Wrote site to "build"
  ✔ done

woo

run_shell_command

doesn't cache, but the

adhoc_tool

does, correct?

Right

👍 Trying to do a no-op when I run twice in a row, but npm install seems to run

Everything in the dependencies should no-op correctly

Here's a an example of using

adhoc_tool

and

run_shell_command

to build a Typescript + SvelteKit example (basically,

npm install

and then call package.json commands)

# Grabbing build/bundle-centric files, as well as eslint/prettier files
files(
    name="build-meta",
    sources=[
        "*eslint*", 
        "*prettier*", 
        "*npm*", 
        "*.json", 
        "*.cjs", 
        "*.js", 
        "*.ts", 
        "*.yaml",
    ]
)

# The source files themselves - TODO: exclude the *.test.ts files
# Make sure to grab `svelte-kit` which is part of a post-install
files(
    name="sources",
    sources=[
        ".svelte-kit/**/*", 
        "src/**/*", 
        "static/**/*",
    ]
)

# Node/SvelteKit's post-install needs the host's `sh` to be in the environment
system_binary(
    name="sh",
    binary_name="sh",
    fingerprint_args=["--version"],
)

# Pull the host's `node` into the environment
system_binary(
    name="node",
    binary_name="node",
    fingerprint=r"v(19|[2-9][0-9])\..*\..*",
    fingerprint_args=["--version"],
)

# npm, yarn, etc... The tool should attempt to install to a local folder for better caching (i.e. not a global store ala pnpm?)
system_binary(
    name="npm",
    binary_name="npm",
    fingerprint_args=["--version"],
    fingerprint_dependencies=[":node"],
)

# Perform and cache the `npm install` step, `:build-meta` is required for this step, but the `:source` should be passed through
adhoc_tool(
    name="install",
    runnable=":npm", 
    runnable_dependencies=[":sh", ":node"],
    args=["install"],
    output_dependencies=[":build-meta", ":sources"],
    execution_dependencies=[":build-meta"],
    output_directories=["node_modules"],
    timeout=300,
)

# Run build directly in the appropriate working directory - this should be an `adhoc_tool` if it 
# will be output to the local working directory
run_shell_command(
    name="build",
    command="cd {chroot}/frontend/web && npm run build",
    execution_dependencies=[":install",],
)

# Try running my tests
run_shell_command(
    name="test",
    command="cd {chroot}/frontend/web && npm run test",
    execution_dependencies=[":install",],
)

It's so good! For tests, are you aware of https://www.pantsbuild.org/v2.16/docs/reference-experimental_test_shell_command ? Additionally, one pattern we've found quite successful for adhoc builds is a cacheable

shell_command

/

adhoc_tool

invocation to actually do the build, and then

run_shell_command

that does whatever non-cached follow-up is required (e.g. copying things back to the repo, if integrating with existing dev workflows)

Thanks for the info! I'd seen the test command but hadn't thought to use it yet. I have a lint:check, typecheck, and then a "test" for this repo - so looking at how to do that using Pant's goals. This is my first time using that set of commands for this use case, so I'm slowly learning my way around. In this specific case above, the "build" isnt' a cacheable build, it's a confirmation that the build succeeds, so to keep it semantically cleaner for my team - I put that one in a run_shell_command instead, and I would watch for an error. For my "actual" build process, whenever it's setup in a couple weeks - I'd be copying the example-adhoc repo, which does use

adhoc_tool

for the build process - and that makes much more sense. The next thing to see if if I can use this for setting up my ansible deployment to be a bit cleaner. There is a plugin in the works, but there are so many usecases to think about - that right now, starting with this set of commands for my immediate deployment is great.

So there is not good support for

lint

and

check

yet. I had a half-baked design that I might get to later this month, or I can talk it through with someone.

👍 No worries, I didn't think there was yet. In my case, a

run

with a fail is all it needs to do today

There is

experimental_test_shell_command

, but that has not kept up with `shell_command`/`adhoc_tool` features

Ahh, no no, we already have a javascript formatter (prettier)

Cool!

Neat! It would be really easy to add a target type with default sources, or even a macro, to de-boilerplate that

files()

target.

are you finding

run_shell_command

to be sufficient for your needs, or would you prefer something that matches

adhoc_tool

a bit better?

For most of them, we'd like something like https://github.com/pantsbuild/pants/discussions/18235, because we're basically doing codegen that gets checked-in, and having pants manage that natively would be much better than having to remember to manually run each of the codegen commands in sequence, as appropriate. For the other one, we're running

open

on a docs HTML file (after copying to a fixed location to avoid a proliferation of browser tabs), so

run_shell_command

is fine for that.