https://pantsbuild.org/ logo
p

purple-van-78680

05/28/2020, 2:34 PM
I have run into another issue:
Copy code
09:31:56 00:13       [create-monolithic-jar]
09:31:56 00:13         [add-internal-classes]
09:31:56 00:13         [add-dependency-jars]
                       ==== stdout ====

                       ==== stderr ====
                       java.lang.IllegalAccessError: class org.pantsbuild.tools.jar.JarBuilder$7 tried to access method 'com.google.common.collect.TreeTraverser com.google.common.io.Files.fileTreeTraverser()' (org.pantsbuild.tools.jar.JarBuilder$7 and com.google.common.io.Files are in unnamed module of loader 'app')
                       	at org.pantsbuild.tools.jar.JarBuilder$7.execute(JarBuilder.java:743)
                       	at org.pantsbuild.tools.jar.JarBuilder.getAdditions(JarBuilder.java:1188)
                       	at org.pantsbuild.tools.jar.JarBuilder.getEntries(JarBuilder.java:1093)
                       	at org.pantsbuild.tools.jar.JarBuilder.write(JarBuilder.java:1009)
                       	at org.pantsbuild.tools.jar.Main.doRun(Main.java:401)
                       	at org.pantsbuild.tools.jar.Main.run(Main.java:367)
                       	at org.pantsbuild.tools.jar.Main.main(Main.java:481)
                       	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                       	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                       	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                       	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                       	at com.martiansoftware.nailgun.NGSession.run(NGSession.java:280)

FAILURE: jar-tool failed
This happens for all cases when I try to bundle/jar a JVM/Java target. I have recently upgraded from 1.17 to 1.26 (migrating py2 to py3). Hopefully there is something simple that I need to change.
h

hundreds-father-404

05/28/2020, 2:35 PM
Hm. I’m not quite sure how the JVM backend works. @fancy-queen-20734 or @witty-crayon-22786 does this look familiar?
e

enough-analyst-54434

05/28/2020, 2:44 PM
That smells like old Pants picked an old JDK pre-modules support and new Pants picks a newer JDK that supports modules via: "...and com.google.common.io.Files are in unnamed module of loader 'app'".
p

purple-van-78680

05/28/2020, 2:46 PM
I believe I have update the
pants
script, but that may have been for an earlier version. We are using java 11 in the platform settings. I can download the latest python script
e

enough-analyst-54434

05/28/2020, 2:46 PM
OK - so you were previously using JDK11 fine pre-upgrade?
p

purple-van-78680

05/28/2020, 2:47 PM
Yes, that is correct. We are using the following settings in pants.ini:
Copy code
[jvm-platform]
default_platform: java11

platforms: {
    'java11': {'source': '11', 'target': '11', 'args': [] },
  }
I have also run
./pants ng-killall
in case there was an old nailgun process
I downloaded the latest pants script and removed caches to try to rebuild everything ad still get the error
e

enough-analyst-54434

05/28/2020, 2:53 PM
So that stacktrace is suspicious because the missing guava is not shadowed and we have a general facility for shadowing jvm tools....
What version of pants did this definitely work in?
p

purple-van-78680

05/28/2020, 2:54 PM
We were running v1.17 due to python2 dependencies
e

enough-analyst-54434

05/28/2020, 2:54 PM
s/shadowing/shading/
OK, let me check there - thanks.
πŸ‘ 1
p

purple-van-78680

05/28/2020, 2:58 PM
We are using guava libraries in our
3rdparty/jvm
dependencies if that helps (version
28.2-jre
)
e

enough-analyst-54434

05/28/2020, 3:02 PM
Thanks.
Besides a jar-tool version bump from 0.0.10 to 0.0.17 between those Pants versions (which both use guava 18.0) - I see no relevant changes. It looks like jar-tool has never been shaded.
If you're adventurous I'll give you a diff to apply against 1 python file in ~/.cache/pants/setup/... that adds shading.
p

purple-van-78680

05/28/2020, 3:05 PM
I can definitely try that πŸ™‚
e

enough-analyst-54434

05/28/2020, 3:05 PM
OK - a few minutes to prepare some coherent instructions.
Ah - and lets do this against the Pants upgrade ... which is version 1.26.0 ?
p

purple-van-78680

05/28/2020, 3:06 PM
That is correct, the version I am using is 1.26.0
e

enough-analyst-54434

05/28/2020, 3:10 PM
Here's the diff:
Copy code
diff --git a/src/python/pants/backend/jvm/subsystems/jar_tool.py b/src/python/pants/backend/jvm/subsystems/jar_tool.py
index bd02ef3c5..c6ad8d12d 100644
--- a/src/python/pants/backend/jvm/subsystems/jar_tool.py
+++ b/src/python/pants/backend/jvm/subsystems/jar_tool.py
@@ -10,12 +10,15 @@ from pants.subsystem.subsystem import Subsystem
 class JarTool(JvmToolMixin, Subsystem):
     options_scope = "jar-tool"
 
+    _MAIN = "org.pantsbuild.tools.jar.Main"
+
     @classmethod
     def register_options(cls, register):
         super().register_options(register)
         cls.register_jvm_tool(
             register,
             "jar-tool",
+            main=cls._MAIN,
             classpath=[JarDependency(org="org.pantsbuild", name="jar-tool", rev="0.0.16"),],
         )
 
@@ -24,7 +27,7 @@ class JarTool(JvmToolMixin, Subsystem):
             self.tool_classpath_from_products(
                 context.products, "jar-tool", scope=self.options_scope
             ),
-            "org.pantsbuild.tools.jar.Main",
+            cls._MAIN,
             jvm_options=self.get_options().jvm_options,
             args=args,
             workunit_name="jar-tool",
p

purple-van-78680

05/28/2020, 3:15 PM
applied and attempting to bundle a jar
e

enough-analyst-54434

05/28/2020, 3:18 PM
Ah - sorry - you beat me to instructions. Where did you apply this? Should be:
Copy code
$ PATCH_TARGET=$(find ~/.cache/pants/setup/*/1.26.0_*/lib/*/site-packages/pants/backend/jvm/subsystems/jar_tool.py)
Which for me is:
Copy code
$ echo $PATCH_TARGET
/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py36/lib/python3.6/site-packages/pants/backend/jvm/subsystems/jar_tool.py
p

purple-van-78680

05/28/2020, 3:22 PM
Ah, I did a find and found a different one. I patched
/Users/jay/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/backend/jvm/subsystems/jar_tool.py
and the command you just me gave has
/Users/jay/.cache/pants/setup/bootstrap-Darwin-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/backend/jvm/subsystems/jar_tool.py
however, they are both the same file
e

enough-analyst-54434

05/28/2020, 3:29 PM
I don't see a difference in those two paths.
... maybe the symlink? These are the nice symlinked paths.
p

purple-van-78680

05/28/2020, 3:30 PM
It was a symlink. Sorry, was a copy/past error. I was originally patching
/bootstrap-Darwin-x86_64/pants.ymwVoV/install/lib/python3.7/site-packages/pants/backend/jvm/subsystems/jar_tool.py
which is just a symlink
e

enough-analyst-54434

05/28/2020, 3:31 PM
At any rate - let me know when you get results ... if there is still a stacktrace it should have mangled names prefixed with the package
__pants_shaded__
IIRC.
p

purple-van-78680

05/28/2020, 3:31 PM
got it. Running now
e

enough-analyst-54434

05/28/2020, 3:34 PM
Ah, its
__shaded_by_pants__
... this is a walk down memory lane.
p

purple-van-78680

05/28/2020, 3:35 PM
πŸ‘ It seems that the jvm target is hanging on [cache]:
Copy code
10:32:16 00:03       [create-monolithic-jar]
10:32:16 00:03         [add-internal-classes]
10:32:16 00:03         [add-dependency-jars]
10:32:16 00:03         [cache]
                     No cached artifacts for 1 target.
                     Invalidated 1 target.
10:32:16 00:03         [cache].
                     Using cached artifacts for 1 target.
e

enough-analyst-54434

05/28/2020, 3:37 PM
OK - I did all this cowboy style. Slowing down to apply the patch myself and try building a binary jar...
πŸ‘ 2
This will take a while - switching between source versions is pretty slow (full Rust version changes and rebuilds) - but I'll report back in an hour or so.
p

purple-van-78680

05/28/2020, 4:28 PM
No worries πŸ™‚ Thanks for helping! Let me know if there is anything I can provide in the meantime (I have some meetings in the afternoon in case I don't immediately respond)
f

fancy-queen-20734

05/28/2020, 4:43 PM
If that doesn’t work, you could try adding some
--add-opens
to jvm options for the jar tool. That might not be very scalable.
p

purple-van-78680

05/28/2020, 5:08 PM
will look at that option
e

enough-analyst-54434

05/28/2020, 5:09 PM
OK - my patch introduces a deadlock in the jvm stack that I need to dig into further. You can try to niggle along with Nora's suggestion in the meantime, but I'll dig a bit more a bit later today. I'll probably file an issue to track this shading and its current failure.
coke
p

purple-van-78680

05/28/2020, 5:09 PM
Thanks @enough-analyst-54434!
Still can't figure out that problem, but I am just now getting back to it.
πŸ‘ 1
e

enough-analyst-54434

05/29/2020, 4:22 PM
I have not yet had time to circle back to the deadlock issue when shading. I have not forgotten about you though.
p

purple-van-78680

05/29/2020, 4:23 PM
Thanks John! No rush πŸ˜„
e

enough-analyst-54434

06/01/2020, 6:20 PM
Just dropping in to note I did not get time to circle back here. I likely won't have time until next week unfortunately. I'm hoping @fancy-queen-20734's suggestion to use --add-opens as needed can unblock you in the meantime.
p

purple-van-78680

06/01/2020, 6:56 PM
Thanks for keeping me updated. I will try the
--add-opens
sugggestion soon. This is sort of a migration task to a new version, so we can still work using the older pants versions until we can figure out this issue. One note, we use guava as a jvm depency, and have it include in most of our java targets, however, I did notice we have one target (at least one), that was able to bundle as it did not depend on the guava 3rd party dep.
πŸ‘ 1