#22294 Pants does not support spark scala test Issue created by tomascsantos Describe the bug This minimum code which creates a spark dataframe then shows it: class SparkToPandasSpec extends AnyFlatSpec with Matchers { "Spark DataFrame" should "convert to Pandas and print" in { // Create Spark session val spark = SparkSession.builder() .appName("SparkToPandasTest") .master("local") .config("spark.driver.bindAddress", "127.0.0.1") .getOrCreate() try { // Create sample data val data = Seq(("Alice", 25), ("Bob", 30), ("Cathy", 28)) val columns = Seq("Name", "Age") // Create Spark DataFrame val df = spark.createDataFrame(data).toDF(columns: _*) // Convert to Pandas DataFrame val pandasDF = df.show() // Print Pandas DataFrame println(pandasDF) } finally { spark.stop() } } } Fails with this error indicating the scala version has a mismatch. The scala version is supposed to be handled by pants globally, but it seems like the resolved packages have something compiled with scala 2.13 instead of 2.12.

❯ pants test ::
16:25:25.26 [ERROR] Completed: Run Scalatest - my-spark/src/test/scala/com/foobar/myspark/BaseSpec.scala - failed (exit code 1).
Discovery starting.
Discovery completed in 128 milliseconds.
Run starting. Expected test count is: 1
SparkToPandasSpec:
Spark DataFrame
- should convert to Pandas and print *** FAILED ***
  java.lang.RuntimeException: error reading Scala signature of scala.collection.immutable.package: Scala signature package has wrong version
 expected: 5.0
 found: 5.2 in scala.collection.immutable.package
  at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:51)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:660)
  at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.$anonfun$complete$2(SymbolLoaders.scala:37)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:333)
  at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:34)
  at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1551)
  at scala.reflect.internal.Symbols$<http://Symbol.info|Symbol.info>(Symbols.scala:1514)
  at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$13.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:221)
  at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.$anonfun$info$1(SynchronizedSymbols.scala:158)
  ...
Run completed in 1 second, 328 milliseconds.
Total number of tests run: 1
Suites: completed 2, aborted 0
Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

Pants version 2.26 OS MacOS Additional info Here is the full minimum viable example: https://gist.github.com/tomascsantos/d9685ee8e33431c5a76933fca99cea1c pantsbuild/pants