https://pantsbuild.org/ logo
#general
Title
# general
b

bored-energy-25252

01/13/2023, 5:13 AM
In my open source project (using Python and Scala) using Pants: https://github.com/liga-ai/liga I want to migrate the Scala part from sbt to pants, but last time I tried
deploy_jar
https://github.com/pantsbuild/pants/issues/17790 , I failed to exclude Apache Spark. Am I doing something wrong or it is not supported in Pants?What is the
deploy_jar
equiv of
sbt package
and
sbt assembly
?
Recently, there are three pull requests on the JVM backend: • https://github.com/pantsbuild/pants/pull/17251https://github.com/pantsbuild/pants/pull/17663https://github.com/pantsbuild/pants/pull/17419 cc @witty-family-13337 @ancient-vegetable-10556 Need help from you!
https://github.com/liga-ai/liga-vision is also using Scala and Python
h

happy-kitchen-89482

01/13/2023, 11:37 AM
@ancient-vegetable-10556 is on vacation. @witty-family-13337 may be around though!
Re https://github.com/pantsbuild/pants/issues/17790, it looks like your BUILD filet is trying to shade Apache Spark, not exclude it?
Are you trying to remove it from the deploy jar?
w

witty-family-13337

01/13/2023, 11:46 AM
Excluding Spark is not yet possible. We need to support scopes or something similar (not fully aware on any progress on that). This is because Spark is still needed to compile the project, we just don’t want it to be part of the final JAR.
h

happy-kitchen-89482

01/13/2023, 11:48 AM
And we have no way of making it "provided" or anything like that. Yeah.
w

witty-family-13337

01/13/2023, 11:48 AM
shading it with the zap rule looks like a workaround attempt to achieve the same, but the final jar will still contain other transitive dependencies from Spark
h

happy-kitchen-89482

01/13/2023, 11:48 AM
So shading it is the wrong thing to do, it sounds like
because the client code will use the shaded version, not the provided version, and so any runtime classname magic will fail
w

witty-family-13337

01/13/2023, 11:49 AM
well, the zap rule will remove any classes under the given pattern
h

happy-kitchen-89482

01/13/2023, 11:50 AM
yes, but won't it rewrite the callers to use the shaded version?
w

witty-family-13337

01/13/2023, 11:51 AM
that’s a good point, and in case of zap, there isn’t a shaded version as it was removed
b

bored-energy-25252

01/13/2023, 11:59 AM
Are you trying to remove it from the deploy jar?
Yes.
Well,
sbt package
and
sbt assembly
seems to be the blocking functionality of Pants JVM (Scala) support
h

happy-kitchen-89482

01/13/2023, 12:01 PM
Or am I misunderstanding how zap works
w

witty-family-13337

01/13/2023, 12:14 PM
well, those sbt goals are supported with
pants package …
applied to a
deploy_jar
. Just that the functionality is incomplete due to lack dependency scopes
b

bored-energy-25252

01/13/2023, 12:24 PM
any plan to solve the dep scopes problem? is it hard?
h

happy-kitchen-89482

01/13/2023, 12:27 PM
I don't think it's crazy hard, just not done yet
The implementation wouldn't be hard, I think it's more around design. How we want to represent the concept of dependency scopes in pants.
E.g., what would a BUILD file look like that specifies scopes...
b

bored-energy-25252

01/13/2023, 12:47 PM
Looking forward to the non-experimental Scala support!
5 Views