https://pantsbuild.org/ logo
#general
Title
# general
f

fresh-continent-76371

11/09/2022, 9:48 PM
In the repo, we have some tasks where antlr is used to generate Python lexers, from the antlr G4. the process is simply to run the req'd antr JAR with with the jvm. eg.
Copy code
java -Xmx500M -cp "/usr/local/lib/antlr-4.9.3-complete.jar:.:/usr/local/lib/antlr-4.9.3-complete.jar" org.antlr.v4.Tool
At a simple level this is
experimental_shell_command
, am I correct in thinking i can get Pants to "supply" the JVM, and the antlr dependency. adding the jvm backend, gives me the
jvm_artifact(...)
to pull antlr, How do i make the "jvm" a tool / dependency to the
experimental_shell_command
or is there a "java run" type target instead ?
b

bitter-ability-32190

11/09/2022, 10:00 PM
I don't have your answer, but your question is well timed! https://github.com/pantsbuild/pants/issues/17405 is currently on our minds
f

fresh-continent-76371

11/09/2022, 10:06 PM
Yea, that’s it :-) I’m mocking up right now this part ; expand that issue to ensure it’s not python centric, but can do tool dependencies of Java, go, python, npm, etc
Also it would be wise to drop (alias) the experimental_shell Name also. A suggestion would be shell_run And shell_run_generate — which expects output
b

bitter-ability-32190

11/09/2022, 10:09 PM
"Experimental" is our delineator for "likely to change, sooooo"
And the shell part is also a misnomer while we're at it 😛
😄 1
f

fresh-continent-76371

11/09/2022, 10:21 PM
hmm, what does a
jvm_artifact
name look like, that I am able to add it to the
shell_command( ..., dependencies["whatgoeshere"], ...)
I have my artifact as
Copy code
jvm_artifact(
    group="org.antlr",
    artifact="antlr-complete",
    version="4.9.3",
)
w

witty-crayon-22786

11/09/2022, 10:50 PM
it doesn’t have a guaranteed name until it’s actually been packaged into a particular format
in this case, you’d probably need to define a
deploy_jar
target which depended on the
jvm_artifact
(explicitly), and then have your
experimental_shell_command
depend on the
deploy_jar
then you’d invoke the
deploy_jar
using its name, which is well defined: https://www.pantsbuild.org/docs/reference-deploy_jar#codeoutput_pathcode
java -Xmx500M -jar $my_deploy_jars_name
and yea: as Josh said: “put a JVM in this shell command” is definitely a topic for the linked ticket
f

fresh-continent-76371

11/09/2022, 11:12 PM
does pants automatically generate a jvm "default.lock" file _ it does not seem to be, and i cannot find what the format is ..
my pants.toml is thus far
Copy code
[jvm]
jdk = "temurin:1.17"

[jvm.resolves]
jvm-default = "lib/3rdparty/jvm/default.lock"
and now
Copy code
10:11:07.92 [ERROR] 1 Exception encountered:

  KeyError: 'entries'
which is • not a great error mesaage 🙂
w

witty-crayon-22786

11/09/2022, 11:13 PM
yes, it does
f

fresh-continent-76371

11/09/2022, 11:14 PM
and only came up after i added the path to an empty lock file.
w

witty-crayon-22786

11/09/2022, 11:14 PM
rather than adding an empty file, you should use
./pants generate-lockfiles
to generate the file
f

fresh-continent-76371

11/09/2022, 11:14 PM
ah
w

witty-crayon-22786

11/09/2022, 11:15 PM
and if it is not present, pants should actually print an error suggesting that
b

bitter-ability-32190

11/09/2022, 11:15 PM
(yeah we should maybe schema-check dict options)
@witty-crayon-22786 do we have a ticket for that?
w

witty-crayon-22786

11/09/2022, 11:15 PM
try removing the file, and see if you get a warning / error?
f

fresh-continent-76371

11/09/2022, 11:16 PM
yes it does error when it is not there, but it also did not say what to do about that 🙂
Copy code
10:14:46.00 [ERROR] 1 Exception encountered:

  Exception: Unmatched glob from The resolve `jvm-default` from `[jvm].resolves`: "lib/3rdparty/jvm/default.lock"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/v2.14/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>.
w

witty-crayon-22786

11/09/2022, 11:16 PM
ah. yea, tricky.
f

fresh-continent-76371

11/09/2022, 11:16 PM
generating worked -
./pants generate-lockfiles
w

witty-crayon-22786

11/09/2022, 11:16 PM
i’ll file an issue about that: thanks.
👍 1
f

fresh-continent-76371

11/09/2022, 11:17 PM
(my goal right now is to get the antlr jar, original, or re-packaged, into the sub-process sandbox of
experimental_shell_command
w

witty-crayon-22786

11/09/2022, 11:23 PM
which command did you run to render the “Unmatched glob” error?
it looks like we render a higher level error in some cases, but not the one you encountered
maybe you can add more details here: https://github.com/pantsbuild/pants/issues/17521
👍 1
f

fresh-continent-76371

11/09/2022, 11:25 PM
okay - that was unexpected ,,, pants file ...
Copy code
jvm_artifact(
    name="antlr",
    group="org.antlr",
    artifact="antlr4",
    version="4.11.1",
)

deploy_jar(
   name="pkg",
   main="org.antlr.v4.Tool",
   dependencies=[":antlr"],
)

#
# just a shell to see what it going on, inside
#
experimental_shell_command(
     name="asm_parser",
     command="tree",
     outputs=[ "./" ],
     dependencies=[":pkg"], # ":org.antlrantlr-complete"],
     tools=["tree"]
)
Copy code
./pants run path/to/generator 
10:22:03.95 [INFO] Completed: Assemble combined JAR file
ANTLR Parser Generator  Version 4.11.1
 -o ___              specify output directory where all output is generated
 -lib ___            specify location of grammars, tokens files
 -atn                generate rule augmented transition network diagrams
 -encoding ___       specify grammar file encoding; e.g., euc-jp
 -message-format ___ specify output style for messages in antlr, gnu, vs2005
 -long-messages      show exception details when available for errors and warnings
 -listener           generate parse tree listener (default)
 -no-listener        don't generate parse tree listener
 -visitor            generate parse tree visitor
 -no-visitor         don't generate parse tree visitor (default)
 -package ___        specify a package/namespace for the generated code
 -depend             generate file dependencies
 -D<option>=value    set/override a grammar-level option
 -Werror             treat warnings as errors
 -XdbgST             launch StringTemplate visualizer on generated code
 -XdbgSTWait         wait for STViz to close before continuing
 -Xforce-atn         use the ATN simulator for all predictions
 -Xlog               dump lots of logging info to antlr-timestamp.log
 -Xexact-output-dir  all output goes into -o dir regardless of paths/package
I assume that, whilst run did run the jar, that was not running the "experimental_shell_command" - but rather the "deploy_jar"
w

witty-crayon-22786

11/09/2022, 11:28 PM
yea… to run the
experimental_shell_command
, you’ll want to pass its name:
./pants run path/to/generator:asm_parser
i am… not entirely sure why
pkg
ended up selected there. @bitter-ability-32190 might know better.
b

bitter-ability-32190

11/09/2022, 11:33 PM
Is deploy jar runnable? 🤔
w

witty-crayon-22786

11/09/2022, 11:33 PM
yes
b

bitter-ability-32190

11/09/2022, 11:34 PM
I suspect then this is in a directory named pkg 🧐
w

witty-crayon-22786

11/09/2022, 11:35 PM
ah, yea, maybe… i guess i didn’t think about the command maybe being psuedocode
@fresh-continent-76371: is the directory named
pkg
?
pants will default to selecting the target with the same name as the directory, which might explain this
f

fresh-continent-76371

11/09/2022, 11:37 PM
no, but (and i recall this)
experimental_shell_command
is not runnable anyway . looks like it just picked the "first" off the list of possibles.
Copy code
❯ ./pants run  apps/generator:asm_parser
10:33:45.90 [ERROR] 1 Exception encountered:

  NoApplicableTargetsException: No applicable files or targets matched. The `run` goal works with these target types:

  * deploy_jar
  * docker_image
  * experimental_run_shell_command
  * pex_binary
  * python_source
  * python_test

However, you only specified target arguments with these target types:

  * experimental_shell_command

Please specify relevant file and/or target arguments. Run `./pants --filter-target-type=deploy_jar,docker_image,experimental_run_shell_command,pex_binary,python_source,python_test list ::` to find all applicable targets in your project, or run `./pants --filter-target-type=deploy_jar,docker_image,experimental_run_shell_command,pex_binary,python_source,python_test filedeps ::` to find all applicable files.
w

witty-crayon-22786

11/09/2022, 11:38 PM
ahh, got it. yea, you’ll probably want to define a
shunit_test
that depends on the
experimental_shell_command
to test its outputs
f

fresh-continent-76371

11/09/2022, 11:39 PM
hmm now I am confused 🙂 seems
package
is not for
experimental_shell_command
b

bitter-ability-32190

11/09/2022, 11:39 PM
FWIW I use an `experimental_run_shell_,command`for easy sandbox shenanigans
w

witty-crayon-22786

11/09/2022, 11:41 PM
experimental_shell_command
just produces loose files, which aren’t really packaged in any particular way… if you want to emit files as an
archive
, you could add an
archive
target that depends on your
experimental_shell_command
, and then
./pants package $my-archive
would put them under
$buildroot/dist
or a
shunit_test
for
./pants test
, etc
or a
python_test
, for that matter
h

happy-kitchen-89482

11/10/2022, 1:13 AM
As an aside, this takes me back! ANTLR support was the very first thing I added to Pants v1, 12 years ago, and is how I got involved in all this in the first place! cc @enough-analyst-54434
e

enough-analyst-54434

11/10/2022, 2:31 AM
Heh. Indeed!
I just made the Terence Parr sign at you!
f

fresh-continent-76371

11/10/2022, 3:38 AM
🙂
am i correct in understanding that the
experimental_run_command
cannot by "executed" directly from the command line, by a goal
e

enough-analyst-54434

11/10/2022, 4:01 AM
I think that's right. For debugging in general though, you can always add
--keep-sandboxes=always
. You should get Pants log lines noting the various sandbox directories left laying around. You can CD into the one you care about (hopefully which one is easy enough to figure out) and you can look at
__run.sh
to see what Pants runs and play around with things.
f

fresh-continent-76371

11/10/2022, 4:03 AM
This would be a good addition to the documentation, for every "target" method., have a section that describes under which phase (or none) it is expected to be operated with.
(phase == goal)
Okay - part way. so i was able to use experimental_shell_command to run antlr. but as you will see it is not perfect. And I can see it may be better to write a dedicated AntlrPlugin 1. it would be good if experimental_shell_command could just dump the files in dist. They are ready to be consumed 🙂 2. having to use an intermediate "deploy_jar" after jvm_artifact could be improved/ 3. i can see with maybe some massaging I can take the output from
experimental_shell_command
and feed it to
python_distribution
- which then means maybe I should just write a codegen antlr plugin 4. It would be good to be given variables to the dependencies (path or file), by name, 5. also a variable of the "ROOT" of the sanbox, It is brittle to be depending on the path (depth) and full paths to dependencies generated.
Copy code
jvm_artifact(
    name="antlr-dependency",
    group="org.antlr",
    artifact="antlr4",
    version="4.11.1",
)

deploy_jar(
   name="antlr-complete",
   main="org.antlr.v4.Tool",
   dependencies=[":antlr-dependency"],
)


resource(
    name="my_antlr_grammar",
    source="folder/my_grammar.g4"
)

experimental_shell_command(
     name="asm_parser",
     command="""
         mkdir out && \
         java -jar $(find ../../../../ -name antlr-complete.jar ) \
                -o out/ \
                -visitor \
                -Dlanguage=Python3 folder/my_grammar.g4
         
     """,
     outputs=[ "out/" ],
     dependencies=[":antlr-complete", ":qps_asm_antlr_grammar"],
     tools=["tree", "java", "find"],
)


archive(
    name="qps_asm_lexer",
    files=[":asm_parser"],
    format="zip"
)
g

gorgeous-winter-99296

11/10/2022, 3:03 PM
You can have them dumped to dist. All you need to do is enable a codegen backend - for some reason,
pants.backend.docker
is the one I end up using. That gives you
export-codegen
, which does that. I'm using that for my codegen-adventures. 🙂 I.,e in your case you'd
pants export-codegen //some/path:asm-parser
and you can skip the archive step if you don't need it.
Unfortunately that still doesn't put them into the workspace proper (dist is ephemeral in my view) but I believe wrapping it all up in
experimental_run_shell_command
will work for that - my plan is to use that to generate goldens for CI but using the shell command output directly in all build steps.
h

happy-kitchen-89482

11/10/2022, 4:57 PM
Thanks for this very helpful feedback! We (by which I mean @ancient-vegetable-10556 mostly) are looking at improvements to the shell-command functionality at the moment, so this is super useful.
For example, having a more straightforward ability to push stuff out to
dist
sounds necessary? Could be
export
, or
package
, both of which do that today in their respective areas
a

ancient-vegetable-10556

11/10/2022, 5:00 PM
At the very least, it seems like
experimental_shell_command
and friends would benefit from having a way to export files
at the moment, the “obvious” way to do it is to create an
experimental_run_shell_command
with a command of
cp {chroot]/artifact.txt *PATH_IN_WORKSPACE*
and dependencies of the step that you want to run ion the sandbox
f

fresh-continent-76371

11/11/2022, 2:40 AM
at the moment, the “obvious” way to do it is to create an
experimental_run_shell_command
with a command of
cp {chroot]/artifact.txt *PATH_IN_WORKSPACE*
and dependencies of the step that you want to run ion the sandbox
so to make that work the best, injecting the
PATH_IN_WORKSPACE
value to the
...shell...
is the need, as this full path varies across CI, and user workstations.
which leads to a question: are their default variables available in the build context at all ?, i.e. ROOT_OF_REPO
what I am attempting is to provide the full path of the repo. injecting into the
experimental_shell_command
like
Copy code
experimental_shell_command(
     ...
     extra_env_vars = [ "BUILDROOT=" + GetBuildRoot(), "BUILD_FILE_PATH=" + GetBuildFilePath() ],
     ...
)
a

ancient-vegetable-10556

11/15/2022, 5:13 PM
Generally speaking, we don’t like to expose the actual filesystem to processes that are managed by Pants, because that makes processes less likely to be cacheable
Generally speaking, you want to explicitly provide the files as
dependencies
, so that they get copied into the sandbox.
f

fresh-continent-76371

01/27/2023, 8:44 PM
This looks good and helpful
a

ancient-vegetable-10556

01/27/2023, 9:37 PM
Great! Let me know if you get the opportunity to try it out soon 🙂
3 Views