https://pantsbuild.org/ logo
#welcome
Title
# welcome
l

lemon-noon-33245

04/26/2023, 3:04 PM
Hey everyone, I’m a founder at Trinio, a startup from Brazil working on a fast checkout for e-commerces. Currently, our codebase is all in Node and we had been using turborepo to manage our monorepo. We’re slowly switching some services to Golang as we migrate them to AWS lambdas, which prompted the need for a new tool. Since the engineering team is pretty heads down working on the product, I volunteered as a tribute to run some tests with some build tools. At this point I tried Bazel, Please, both of which had a few dealbreakers for us, and now Pants, which seems promising so far. I echo Tony in that I’d love to contribute at some point too!
b

busy-vase-39202

04/26/2023, 3:28 PM
Haha, love a good Hunger Games reference. Do hit us up with as many skeptical questions as you need to feel confident that you've finally landed on a good fit. If it's not a good fit, we'll definitely tell you. We'd rather that devs get their needs met well, regardless of whether it's by Pants or something else. By the way, have you seen @worried-painter-31382's ongoing work on JS/TS support? I know he'd really appreciate feedback from organizations that would be likely to use it, and guidance on which choices would make it most useful. So you've arrived at a great time to have impact right away, just by giving thoughtful feedback.
👍 1
l

lemon-noon-33245

04/26/2023, 3:31 PM
Totally have! I’ve seen the effort on Github and checked all opened issues on the matter. That was what made me decide to try it. For now I’ve been implementing using the new
adhoc_tool
but the plan was to switch to the JS/TS native backend once that’s ready. In particular we need
pnpm
or
yarn
support since we’re heavily relying on workspaces.
b

busy-vase-39202

04/26/2023, 3:38 PM
Ah cool! You're way ahead of me. Excellent. I know he wants to make sure that effort can prove useful for more people than just himself, so communicating about what you'd find most useful would be very appreciated and would probably help move things forward more.
l

lemon-noon-33245

04/26/2023, 3:40 PM
Sounds good! @worried-painter-31382 hit me up whenever, glad to share what my ideal pants usage would look like!
b

busy-vase-39202

04/26/2023, 3:42 PM
In the meantime, any feedback you can offer on those issues would probably be useful if you've got time to try stuff out and or suggest approaches.
l

lemon-noon-33245

04/26/2023, 3:44 PM
Happy to try stuff out as soon as pnpm or yarn works. I can’t really have a real life use case for the tool with plain npm. AFAIK that development is still in progress, right?
w

worried-painter-31382

04/26/2023, 3:47 PM
pnpm support pr will probably be open this weekend :)
Initial support, that is
b

busy-vase-39202

04/26/2023, 3:47 PM
I love when someone asks a question at the perfect moment like that.
l

lemon-noon-33245

04/26/2023, 3:48 PM
Amazing, I’ll keep an eye on it and test as soon as it’s open!
b

busy-vase-39202

04/26/2023, 3:55 PM
💃
Thanks! I hope this works out well for both of you. 🙂
w

worried-painter-31382

04/30/2023, 5:32 PM
🚀 2
❤️ 2
🔥 2
l

lemon-noon-33245

05/02/2023, 1:59 PM
Awesome, I’ll give it a try! @worried-painter-31382 is there an easy way to test this PR with scie-pants or should I just clone the repo and pull the branch?
Nevermind, just checked the docs. Cloned your PR and using PANTS_SOURCE to point to it. I’ll give it a shot today or tomorrow and keep you posted on the progress
w

worried-painter-31382

05/02/2023, 5:19 PM
Sorry, I was on release duty (not for pants, though) 🙂 Thank you for testing, I think there's a lot of rough edges needs ironing out, but hard to find them without getting something out the door!
l

lemon-noon-33245

05/02/2023, 5:23 PM
Hey @worried-painter-31382, pardon me if I end up asking stupid questions! I’m going through the code to try to understand what are the targets and goals that you implemented and how to use them but since I’m new to the code base I might be understanding things wrong. I added a
package_json
target to the root of our pnpm workspace folder and to start testing things out just ran
pants package //node:package
. It starts installing node modules in the sandbox but then I get an error:
Copy code
Failed to digest inputs: "Failed to open \"/private/var/folders/rz/n_vkc61s0_90pc18wy28l4mw0000gn/T/pants-sandbox-9xQzu3/node/node_modules/.pnpm/@typescript-eslint+typescript-estree@5.56.0_typesc
ript@4.9.3/node_modules/@typescript-eslint/typescript-estree/dist/node-utils.js\": Too many open files (os error 24)"
I tried setting a higher ulimit with
ulimit -n 100000
but still got the same error.
Ha! I was missing a 0
10000 won’t do but a 100000 will
w

worried-painter-31382

05/02/2023, 5:28 PM
So
pants package <package_json_target>
will produce a an output the same as
pnpm pack
would do. If you're looking to produce a transpiled output, a
node_build_script
is probably more similar to what you'd do in turborepo or similar
(but Im guessing, of course)
E.g if you have a package.json
Copy code
{
  "scripts": {
    "build": "tsc ./src"
  }
}
You'd have to tell pants about it in your
BUILD
via
Copy code
package_json(scripts[node_build_script(entry_point="build"]))
Andreas has made some awesome headway with help with symbols, so
pants help node_build_script
(probably) works to at least get some idea what it expects for inputs and outputs
The output artifacts that are generated can be inspected by running the
pants export-codegen
goal
l

lemon-noon-33245

05/02/2023, 6:05 PM
Gotcha! At first I thought that node_build_script was a target. Is there a way I can set system binaries that need to exist for the build script to run? In my case, we’re building a Next.js application and I’m getting two errors at the moment. 1.
uname: command not found
- I believe probably bc PATH hasn’t been set in the sandbox so Next can’t find uname. 2.
Couldn't find a pages directory
. I believe probably bc I haven’t specified what are the sources that should be moved to the sandbox. Tried settings the
_sources
argument in
node_build_script
but something tells me that this looks like a private field, lol
Found
dependencies
on package_json now
w

worried-painter-31382

05/02/2023, 6:26 PM
Hmm, indeed node_build_script:s are not a target per se, more similar to
adhoc_tool
if you're familiar. 1. No, not yet. I want to introduce feature parity with
adhoc_tool
here, since it seems nodejs ecosystem has some amount of assumptions about what binaries that are ~always available. You probably can work around it by assigning
PATH=
in the build script 2.
dependencies
yes! If you have javascript sources, pants can infer this for you, if you dont, you'd need a `resources`target and set the dependencies in the
BUILD
file
I have implemented support for pants to inspect these properties for the package.json https://nodejs.org/api/packages.html#package-entry-points and https://docs.npmjs.com/cli/v9/configuring-npm/package-json#browser for inference in the sandbox, but they only work with javascript sources, for now.
If that doesn't fit a particular tool (I can't recall what Next.js sets up right now), you can use
dependencies=["src/index.js"]
etc.
l

lemon-noon-33245

05/02/2023, 6:31 PM
Got it! They might not work for now then since it’s mostly tsx files. I had to add the other workspaces that this package depends on as well. For instance, we have a
types
lib workspace that this project imports and it’s marked as
workspace:^types
in
package.json
. Pants couldn’t infer that dependency right away (maybe because it’s also a TS project?), so I had to add it to
dependencies
w

worried-painter-31382

05/02/2023, 6:33 PM
Did you map the other lib to a
package_json
?
l

lemon-noon-33245

05/02/2023, 6:33 PM
Not yet!
w

worried-painter-31382

05/02/2023, 6:33 PM
That explains why pants couldn't find it
l

lemon-noon-33245

05/02/2023, 6:33 PM
lol… yep! 😄
I’ll try that next. But so far so good, @worried-painter-31382! Amazing job on this. Just built and exported the files with
export-codegen
!!!
🎉 1
One minor “bug”. If a package.json doesn’t have a “version” attribute, pants fails with KeyError: ‘version’
For publishing that might make sense but for internal use in a monorepo, it shouldn’t be mandatory in my opinion
w

worried-painter-31382

05/02/2023, 6:43 PM
I think I agree, I was just hesitant to make it optional because npm does not have workspace protocol support (but does support workspaces). Omitting versions have some side effects, pants-codewise. Feel free to open an issue for it.
l

lemon-noon-33245

05/02/2023, 6:56 PM
Just mapped the other libs to a package_json but the source files aren’t being copied to the sandbox
Any idea why? Since they are typescript files, I’m adding them in
dependencies
but no go
w

worried-painter-31382

05/02/2023, 7:02 PM
Which sandbox?
If you're looking to include a dependants transpiled code, I think you'd need to use the
prepare
lifecycle script and make sure it does ~the same thing as the
build
l

lemon-noon-33245

05/02/2023, 7:05 PM
pkg1
depends on the
types
source files (not transpiled).
types
are in
pkg1
package.json dependencies as
workspace:^types
. I added
package_json
to the
types
BUILD file as you mentioned before to see if pants would infer the dependency and copy the source files but it didn’t. So I added the source files to
package_json
in the
dependencies
parameter. No success as well.
w

worried-painter-31382

05/02/2023, 7:07 PM
Hmm, the last bit surprises me.
l

lemon-noon-33245

05/02/2023, 7:07 PM
If I explicitly add
types:files
to the
pkg1
dependencies it works. What I can’t get working is the implicit resolution of dependencies in the workspace
w

worried-painter-31382

05/02/2023, 7:09 PM
What does
pants peek <path/to/types/package.json>
say?
l

lemon-noon-33245

05/02/2023, 7:12 PM
It only shows the third party dependencies listed in
package.json
. The dependencies I set up explicitly also don’t show up, FIY
w

worried-painter-31382

05/02/2023, 7:12 PM
Copy code
{
  "dependencies": {
    "types": "workspace:^"
  }
}
This is the dependencies in pkg1?
l

lemon-noon-33245

05/02/2023, 7:13 PM
Yep!
None of the workspace dependencies are being listed as well
w

worried-painter-31382

05/02/2023, 7:14 PM
Ok, just checking because if it was "workspace:^types" or some other form it will not work (it being inference)
l

lemon-noon-33245

05/02/2023, 7:14 PM
Yeah, that was a typo on my side, sorry!
👍 1
What’s interesting is that even in the case when it’s working (when I set the dependencies manually) it still doesn’t list the dependencies when I do a
pants peek
w

worried-painter-31382

05/02/2023, 7:17 PM
That is interesting, if you run
pants list
you'll get a list of targets under a directory
l

lemon-noon-33245

05/02/2023, 7:19 PM
List shows the files
Copy code
node/libs/types#__techdocs/cli
node/libs/types#__trinio/types
node/libs/types#jest
node/libs/types#node/libs/types/package.json
node/libs/types#ts-jest
node/libs/types#tslib
node/libs/types#typescript
node/libs/types#vite
node/libs/types#vite-dts
node/libs/types#yarn.lock
node/libs/types/.eslintrc.json:files
node/libs/types/BUILD:files
node/libs/types/README.md:files
node/libs/types/jest.config.js:files
node/libs/types/package.json:files
node/libs/types/tsconfig.json:files
node/libs/types/tsconfig.lib.json:files
node/libs/types/tsconfig.spec.json:files
Works as well in the Next.js package folder
w

worried-painter-31382

05/02/2023, 7:20 PM
Is your org called @trinio? 🙂 I think I might know what is failing
l

lemon-noon-33245

05/02/2023, 7:20 PM
It is!
w

worried-painter-31382

05/02/2023, 7:20 PM
The inference isn't smart enough to elide the @org part of the package names
(because I had no idea you could omit them)
l

lemon-noon-33245

05/02/2023, 7:22 PM
Just so you know, I’m not omitting in my files. I was just omitting them here on slack to be more succint
w

worried-painter-31382

05/02/2023, 7:23 PM
Ah, swing and a miss
😂 1
l

lemon-noon-33245

05/02/2023, 7:23 PM
Meaning the package.json still lists
Copy code
{
  "dependencies": {
    "@trinio/types": "workspace:^"
  }
}
✔️ 1
w

worried-painter-31382

05/02/2023, 7:53 PM
Hmm, and to double check, your
BUILD
for the
types
package is something like
Copy code
package_json(dependencies=[":sources"])
resources(name="sources", sources="*.tsx")
and
pkg1
Copy code
package_json(dependencies=[":sources", "lib/types:sources" , "lib/types/package.json"])
resources(name="sources", sources="*.tsx")
?
l

lemon-noon-33245

05/02/2023, 7:54 PM
Yes, but instead of using
resources
I was using
files
is that a problem?
w

worried-painter-31382

05/02/2023, 7:55 PM
No, should not matter
But this setup works (w.r.t what files end up in the sandbox) correct? It is if you omit the "lib/types:sources" that it fails?
l

lemon-noon-33245

05/02/2023, 8:04 PM
Not really. It only works if I add
Copy code
package_json(dependencies=["//lib/types:sources"])
to
pkg1
Sorry, just saw that you had that on your example
w

worried-painter-31382

05/02/2023, 8:05 PM
Right, but essentially the only way it will work is if the package depends on the sources in the dependency, not the package itself
I think it is expected, if a bit unfortunate. The current algorithm assumes it can use
source
(so
files
, or
javascript_source
for example) dependencies to find owning
package_json
, and from there populate sandboxes.
package_json
files can depend on javascript files, they don't depend on eachother currently, strangely enough. This why
pants peek
was confounding... What this means is that current dependency inference will not guess, even though you've listed a first party dep on your package, since there's no dependency between some .tsx file in pkg1 to a .tsx file in the other package. I did not reflect on this because I've worked solely with javascript sources which ties this together because an import in a package will then resolve this.
l

lemon-noon-33245

05/02/2023, 8:07 PM
Ah, makes sense!
So once we add support to typescript this should also work as the dependency graph will be built from the “imports”
w

worried-painter-31382

05/02/2023, 8:11 PM
Yes, because then pants could see that e.g
lib/pkg1/file.tsx
has
import { MyType } from "types/MyType"
, pants will know
lib/types/MyTypes.tsx
is a dep, pants will see that
lib/types/package.json
is the owner, it will be installed, all is well
Now what happens is there is no link between lib/pkg1/file.tsx and lib/types/MyTypes.tsx, so the inference is snipped right in the middle
👍 1
l

lemon-noon-33245

05/02/2023, 8:42 PM
@worried-painter-31382 If I want to package a docker_image using the
package_json
output, theoretically I just need to add
:package-json
as a dependency, correct?
Since my
docker_image
depends on
package_json
it’s trying to package the
package_json
target and getting the following error:
Copy code
16:43:03.51 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Packaging .tgz archive for @trinio/checkout@1.0.0' failed with exit code 1.
stdout:
 ERROR   ERROR  Unknown option: 'recursive'
w

worried-painter-31382

05/03/2023, 6:55 AM
Mm, docker integration is not something I've had time to work on. What usecase does your org have for docker + nodejs? Do you use it for the runtime, building frontend, both?
b

busy-vase-39202

05/03/2023, 12:31 PM
@curved-television-6568 created Pants' docker support and might have some insight to lend here too.
c

curved-television-6568

05/03/2023, 12:38 PM
I’m not sure where this
recursive
option is coming from…
l

lemon-noon-33245

05/03/2023, 2:05 PM
It seems to be a pnpm error. When I run
pants package //node:docker
it is first trying to package the
package_json
dependency. For that it runs
pnpm pack
and apparently
recursive
is being passed as an option, which is invalid.
What usecase does your org have for docker + nodejs? Do you use it for the runtime, building frontend, both?
Running the frontend build with pants and then building a docker image to push to our Kubernetes cluster.
@worried-painter-31382 it is indeed a pnpm bug: https://github.com/pnpm/pnpm/issues/4351
But my question is why is
pants
requiring
package_json
to package as well since all we needed in this case is a build?
w

worried-painter-31382

05/03/2023, 2:26 PM
I think it is a side effect of your dependency graph stating
docker_image
depends on the
package_json
. I think what you want to do is be more specific in your dependencies here. Is pants in charge of the package:able artifact that will run the server?
l

lemon-noon-33245

05/03/2023, 2:31 PM
Is pants in charge of the package:able artifact that will run the server?
That’s what I was hoping to do.
I think it is a side effect of your dependency graph stating
docker_image
depends on the
package_json
. I think what you want to do is be more specific in your dependencies here.
Is there a way to be more specific here? I tried depending on
package-json#build
but it still asked to package.
w

worried-painter-31382

05/03/2023, 2:32 PM
Is there a way to be more specific here? I tried depending on
package-json#build
but it still asked to package.
I dont have the code in front of me, but it could be that there's a sneaky dependency between
package_json#build
and the
node_package
target (which is what is "packed"). You can use the
!!
operator to force the target out of your dependencies in the
docker_image
I would consider this a bug if that is the case, though
l

lemon-noon-33245

05/03/2023, 3:08 PM
I can’t find the node_package target. Is it supposed to be “generated” (sorry, don’t know the name of these # “targets”) by
package_json
as well?
w

worried-painter-31382

05/03/2023, 3:08 PM
Indeed, it is a generated target. I would expect it to have name akin to
__trinio
(target names are not allowed to have the @ character)
l

lemon-noon-33245

05/03/2023, 3:43 PM
The name of the package is @trinio/checkout. When I run
pants list
I can see it as
package-json#__trinio/checkout
is that the one you’re referring to that I should remove with
!!
as a dependency?
w

worried-painter-31382

05/03/2023, 3:44 PM
I think so
I'm guessing a lot here, but the end goal should be to not let pants detect a dependency between the docker_image and the package:able target that fails.
l

lemon-noon-33245

05/03/2023, 3:49 PM
If I deny that dependency it doesn’t copy the built files to the docker_image sandbox 😞
Just tried creating an intermediary target to see if I could fool the graph dependency into not packaging stuff but you guys did too much of a good job with it to be easily fooled… haha
w

worried-painter-31382

05/03/2023, 4:04 PM
In general I think docker_image only works with packageable targets and
file
, so the
#build
target needs to be bundled into a packageable target (an archive, a pex_binary or similar) for it to eventually show up in the docker build context, maybe you already have that set up, but just fyi
l

lemon-noon-33245

05/03/2023, 4:12 PM
With
adhoc_tool
I was using it directly. With
package_json
I was hoping I could use directly too. I tried adding an intermediary
files
target that would collect the output from
package-json#build
and then use that as a dependency to
docker_image
but it still tries packaging
@trinio/checkout
.
w

worried-painter-31382

05/03/2023, 4:17 PM
adhoc_tool
produces
file
,
node_build_script
produces
resource
because a likely envisioned scenario was to produce some fronted bundle via the js backend and package it in a pex.
I tried adding an intermediary
files
target that would collect the output from
package-json#build
I don't think that is a possibility with pants?
That said, I think your use case is valid and the backend needs work w.r.t docker compat. Would you mind creating a minimal reproduction so that I can write an issue for this?
l

lemon-noon-33245

05/03/2023, 4:35 PM
Would you mind creating a minimal reproduction so that I can write an issue for this?
Sure, I’ll work on it now. I think this is a pretty common use case, tbh. People will either want to build a docker image with the output or package it to deploy to an AWS Lambda function, Heroku or whatever.
As a kind of a separate issue, if packaging is not working with
pnpm pack
should we provide a different way to package? Specially since it is not possible to convert a resource into something that the dependency graph won’t mark as a packageable dependency. Another use case we have is to package the output of a webpack build of backend services so we can use them in Lambdas.
So even if it worked, it seems to me that the way you’re using pnpm pack right now will not pack the dist but it will pack the whole package before running the build script. We need to be able to access the output after a build somehow.
w

worried-painter-31382

05/03/2023, 5:20 PM
The intention is for
pants package
targeting a
package_json
to produce a npm(-like) distributable package (think library authors). The closest equivalent is the python
python_distribution
target. There is no equivalent to the
pex_binary
in the nodejs backend yet, but I guess that could be a way forward. . The
node_build_script
producing
resource
and only
resource
can be considered MVP-pains. It is part of the inital "wants" of the js backend, that a
pex_binary
can depend on a js target that produces
resource
s. Your
docker_image
target triggering
pnpm pack
isn't expected. As you noticed the export-codgen goal did not run
pnpm pack
. This is some side effect of the inferred dependencies of the backend which might be broken, I don't know. I am however not surprised the current docker backend didnt "just work" at any rate, it's current implementation is... limited. It works well with
pex_binary
and
deploy_jar
(and
adhoc_tool
, I guess? haven't tried.)
It might be as simple as
node_build_script
needs to be both a package:able target and a
export-codegen
compatible target, although I don't know if that is feasible.
l

lemon-noon-33245

05/03/2023, 5:38 PM
Yeah, it makes sense!
@worried-painter-31382 I’m setting up the minimal repro for you at https://github.com/theoribeiro/pants-pnpm . For some reason I’m getting a
ValueError: empty pattern
whenever I run pants on this repo. It doesn’t happen if I comment
package_json()
in the root BUILD file. No idea why bc everything is very similar to my other repo where this works.
@worried-painter-31382 tried a few other things but can’t for the life of me figure out this
empty pattern
error. Have you ever seen it caused by
package_json
?
Unrelated to the repro, should the node_test_script be picked up by the test goal?
w

worried-painter-31382

05/05/2023, 3:54 PM
Im going to spend some devtime on this this weekend, might have better awnsers after. node_test_script only works together with javascript_test, presently
1
https://github.com/pantsbuild/pants/issues/18918 https://github.com/pantsbuild/pants/issues/18920 https://github.com/pantsbuild/pants/issues/18922 https://github.com/pantsbuild/pants/issues/18923 Thank you so much for testing this out @lemon-noon-33245. I've reported & corrected some of the bugs mentioned and spent some time trying to design a target setup that fits. 1. I think making
node_build_script
package:able is the right way forward to make webpacked backend avail for docker, which I hope https://github.com/pantsbuild/pants/issues/18923 addresses fully. 2. I think https://github.com/pantsbuild/pants/issues/18922 addresses the fact that a lot of time users have no intention of creating a tarball-dist from a package.json. I suspect that use case is slim for pants users, so opt-in with explicit target is better, and hopefully makes 1. easier to use with docker backend. The reasoning is; a
npm_distribution
will probably be a leaf in the transitive dependency graph, rather than a connected node which
node_package
is likely to become in a workspace setup.
I think the test-goal only working with javascript_test is a side-effect of there not being any support for transpiled languages yet, I touched on the topic here: https://github.com/pantsbuild/pants/discussions/17357#discussioncomment-5624433
l

lemon-noon-33245

05/08/2023, 4:27 PM
@worried-painter-31382 thanks for following up! Seems like these are great solutions to the issues and use cases I tested. As for the
pnpm pack --filter
issue , as a suggestion you could
cd
into the workspace folder (or use it as a working dir) and run
pnpm pack
without
--filter
. This seems to work and would solve that issue for when people are using
npm_distribution
. Excited to test these solutions when they get merged!
I think the test-goal only working with javascript_test is a side-effect of there not being any support for transpiled languages yet, I touched on the topic here: https://github.com/pantsbuild/pants/discussions/17357#discussioncomment-5624433
That makes sense. Don’t know when you’re thinking about tackling this but glad to test whenever you have something working. Can you think of any workarounds to testing our typescript projects? I could run it with
adhoc_tool
and export the coverage as an output file/directory but then I’d have to install the dependencies again just for that, without taking full advantage of
package_json
w

worried-painter-31382

05/08/2023, 5:53 PM
Not that I know of, short of maybe "tricking" pants into treating typescript files as javascript