ClassNotFoundException MemoryAddress with jdk-19

[metasoarous]

I'm trying out libpython-clj 2.018 on JDK 19, and I'm getting the following error, both the first time I try to initialize, and when attempting to require libpython-clj2.require:

java.lang.ClassNotFoundException: jdk.incubator.foreign.MemoryAddress

I saw issue #180, which suggested adding :jvm-opts ["--add-modules" "jdk.incubator.foreign" "--enable-native-access=ALL-UNNAMED"] to my deps.edn for JVM versions 17 and up. I may tinker around with the version a bit, but it will require rebuilding docker images, so it would be helpful to know if anyone has any experience using libpython-clj 2 with JDK 19.

Thanks

[jjtolton]

Pinging @cnuernber on this one.

[cnuernber]

So JDK-18, 19 aren't at this time supported as they both have incompatible changes to the pathway from an integer to a direct buffer.

[cnuernber]

Specifically 16, 17, 18, 19 all contain incompatible changes there so I have been avoiding upgrading dtype-next to those versions. At this time only JDK-17 is supported beyond JDK-11.

[metasoarous]

Got it 👍 Thanks @cnuernber!

I don't have a strong reason for using any particular JDK version, so happy to stick with JDK 17 for now. It's too bad that we're having to fuss over platform "changes" though. In theory, if someone were motivated, could newer versions be supported?

Thanks!

[cnuernber]

Of course. Its just work - scanning the api changes and fixing dtype-next.

[metasoarous]

I switched to JDK 17, and am seeing the same behavior as before.

For context, I'm pretty sure libpythonclj is finding the correct python installation. And functions like py/->python works. But others do not (such as py/as-python and py/->jvm).

I have a pretty minimal reproduction of this in a Docker container at https://github.com/metasoarous/libpython-require-error. Please let me know if you see anything obviously wrong there, or have hints about what I should be looking for.

Thanks again

[jjtolton]

@metasoarous I see your issue -- and raise you an issue! metasoarous/libpython-require-error#1

[metasoarous]

Thanks @jjtolton; That should be resolved now. Please let me know if it gets you to up the issue stack :-)

[jjtolton]

Much appreciated. I'll dive in shortly!

[cnuernber]

One thing I want to note is that to my knowledge :jvm-opts only works in an alias. A top-level :jvm-opts is silently ignored. So I think the example deps.edn needs an alias with it in it and the dockerfile's cmd needs to be updated to include that alias.

As noted in the other issue if you don't see exactly this warning:

WARNING: Using incubator modules: jdk.incubator.foreign

Then you have not loaded the module.

[jjtolton]

@metasoarous , I reconfigured your deps.edn per @cnuernber 's instructions.

Current file:

  {:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        techascent/tech.ml.dataset {:mvn/version "6.093"}
        ;techascent/tech.ml.dataset {:mvn/version "6.086"
                                    ;:exclusions [org.slf4j/slf4j-api]}
        org.bytedeco/mkl-platform-redist {:mvn/version "2020.1-1.5.3"}
        clj-python/libpython-clj {:mvn/version "2.018"}
        cider/cider-nrepl {:mvn/version "0.28.5"}
        com.taoensso/timbre {:mvn/version "5.2.1"}
        ch.qos.logback/logback-classic {:mvn/version "1.1.3"}}
 :aliases {:opts {:jvm-opts ["--add-modules" "jdk.incubator.foreign"
                       "--enable-native-access=ALL-UNNAMED"]}}}

I now get a different error, but still not working. Note:

root@802025a3b951:/app# java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)

New error:

root@802025a3b951:/app# clj -A:opts -m libpython-test.test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
WARNING: Using incubator modules: jdk.incubator.foreign
WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/abs
WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/infinite?
WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/abs
WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/infinite?
20:15:39.461 [main] INFO  libpython-clj2.python.info - Detecting startup info
20:15:39.502 [main] INFO  libpython-clj2.python - Startup info {:lib-version "3.9", :java-library-path-addendum "/usr/lib", :exec-prefix "/usr", :executable "/usr/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/usr", :base-prefix "/usr", :libname "python3.9m", :base-exec-prefix "/usr", :python-home "/usr", :version [3 9 2], :platform "linux"}
20:15:39.502 [main] INFO  libpython-clj2.python - Prefixing java library path: /usr/lib
20:15:39.577 [main] INFO  libpython-clj2.python - Loading python library: python3.9
20:15:40.095 [main] DEBUG libpython-clj2.python.ffi - Initializing Python C Layer
20:15:40.098 [tech.resource.gc ref thread] INFO  tech.v3.resource.gc - Reference thread starting
20:15:40.100 [main] DEBUG libpython-clj2.python.ffi - Python Home: /usr
20:15:40.124 [main] INFO  tech.v3.datatype.nio-buffer - Unable to find direct buffer constructor -
falling back to jdk16 memory model.
:OK
Execution error (NullPointerException) at clojure.main/main (main.java:40).
Cannot invoke "clojure.lang.IFn.applyTo(clojure.lang.ISeq)" because "f" is null

Full report at:
/tmp/clojure-7296416279702110182.edn

[jjtolton]

Hopefully this is a more tractable error?

[jjtolton]

I think to rule out installation issues, I will try to merge the example Dockerfile with the libpython-clj example Dockerfile to try to rule out installation/version issues.

[jjtolton]

@metasoarous not sure if this solves the initial problem or not, but I put in a PR to your repo so that at least you have a minimal working setup: metasoarous/libpython-require-error#2