OutOfMemory errors

[jaley]

There have been a couple of reports of cloc failing to launch with an OOM error. This is a bit surprising, as it really shouldn't be using much memory at all. Needs profiling.

[robert-stuttaford]

I get a whole bunch of

Could not generate documentation for <namespace in our project>

Presumably because there's no docstrings there.

Then it gives me this, with no extra stacktrace info:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

[jaley]

Hmm... so I ran a profiler over cloc and I'm not seeing any obvious explanations. Here's what happens:

Cloc runs two subprocesses. The first is configured to use exactly your project configuration + cloc as a dependency. It runs some code that loads every namespace in your project and every dependency in turn to pull out the docstring metadata (via codox). This is where you see some logging prints about some namespaces failing, presumably because for whatever reason they don't read/eval in isolation like this.

For me, with a fairly large project, this grew the heap to a maximum of 128MB, which I don't think will exceed the default limits? This phase ends by writing the documentation data out to target/cloc.edn, which on the same (reasonably big) project is about 600k for me.

The second process is a simple Jetty webserver, which loads and indexes the documentation edn data with Lucene and serves it through the web app. That was using about 30MB of heap for me.

Is it possible that the act of loading all the namespaces in your project and its dependencies causes the process to OOM? Does the problem go away if you add a :jvm-opts ["-Xmx1g"] to your project.clj?

[joefreeman]

Increasing the JVM memory didn't help in my case, but I found that removing my Midje tests (temporarily) fixed the OutOfMemoryError problem. Incidentally, I sometimes get the same OutOfMemoryError after running Midje for a while with :autotest enabled.