Find a way forward for JLine updates

[lrytz]

It looks like the JLine version used by the REPL (scala-compiler.jar) and the one used by the sbt shell need to be "compatible enough".

It's not just that the JLine jar from sbt may end up being used by the compiler, but it looks like when running the REPL through sbt we can end up with a mix of classes loaded from either jar (sbt/sbt#8294).

In 2.13.17, we reverted JLine from 3.30 back to 3.29 for that reason (scala/scala#11135), the nightly REPL crashed when launched via current sbt 1.11.6. Similar situations are probably common when mixing older sbt versions with new Scala releases (e.g., sbt 1.9.7 and Scala 2.13.16).

JLine development is quite active lately and we don't want sbt and Scala blocking each other from updating. jline/jline3#1374 mentions JLine 4.0.

I assume Scala 3 has the same issue - wanted to try it with a locally published build but failed wrestling with the build.

[SethTisue]

maybe it's as simple as needing sbt to update first, before Scala 2 updates?

[lrytz]

I cannot convince myself why this would be safe. Some classes are loaded from a recent JLine jar while starting the sbt shell, then more classes are loaded from an old JLine jar while starting the REPL. I don't think backwards binary compatibility between JLine releases is enough here, there can be tighter binary dependencies between classes within the same jar/package.

[bjornregnell]

They need then to run in different JVM instances, no?

[lrytz]

I was hoping it's possible to isolate things (better) with classloaders... sbt does it for the compiler: there's the 2.12 compiler for sbt itself, but it can run any 2.10.x - 3.x compiler in the same process.

But there seem two be two difficulties (sbt/sbt#8294 (comment))

1. When launching the REPL, sbt provides a jline terminal instance to the REPL instead of letting it create its own. (The reason is to prevent issues that arise when both want to control the terminal.) According to the linked comment, sbt attempts to enforce the use of sbt's JLine jar only in that case, but that seems not to work as desired. Maybe that can be fixed.

2. If we were isolating the two JLine versions with classloaders, we still have a problem with JLine's use of native libraries.

   jline3#1374 mentions there are 4 "terminal provider" options in JLine: JNI, FFM, JNA and Jansi, it says JNI is JDK 11+ (not sure why), FFM is 22+.

   JNI, JNA and Jansi all use a native library that's loaded by the JVM, and that cannot be isolated, it's one per process. Looking at JLine 3.27 and 3.30 jars, both have the same file org/jline/nativ/Linux/x86_64/libjlinenative.so used for JNI. But maybe the JNI part of JLine evolves more slowly so that it would be OK in most cases in reality?

   Apparently, with FFM there's no more native library needed, but that's only possible when running on Java 22+.

[lrytz]

That said... yes, what would be the downside of running a separate process? Can it be done today? I tried setting fork := true in various ways but that didn't help (with the test case from sbt/sbt#8294). Maybe that's not supported currently, but it could be. Again, there could be issues due to two processes attemting to take control over the same terminal.

[bjornregnell]

A pitty that we cant connect many many terminals to the same mainframe CPU like in the old days :)
https://en.wikipedia.org/wiki/IBM_3270