OutOfMemoryError

[chb0github]

MyBatis version

3.3.1

Database vendor and version

mysql:5.6

Test case or example project

https://drive.google.com/open?id=1xzcb58xn_YwUmrHgMokRjglRAR5thj7T

project space is too big for github

Steps to reproduce

docker run -d --rm -p 3306:3306 -e MYSQL_ROOT_PASSWORD=my-secret-pw mysql:5.6
sleep 15
migrate up

Expected result

success

Actual result

========== Applying: 00003_second_migration.sql ================================

-- MyBatis Migrations FAILURE
-- Total time: 1s
-- Finished at: Fri Dec 15 16:59:35 PST 2017
-- Final Memory: 133M/464M

ERROR: Error executing command. Cause: java.lang.OutOfMemoryError: Java heap space

---

Before you tell me to increase my heapsize, keep in mind that this might need to execute on a small virt and isn't really an option

Addressing issue #1158 as I suggest would allow this problem to at least be addressed by swapping out implementations. Though, the current implementation really should be fixed

[pjweisberg]

You have send_full_script=false in the properties file, though, so those lines shouldn't be executed. The OOM would have to come from somewhere else.

[chb0github]

Ok, thanks for the catch. I have updated the ticket to not be so precise.

[chb0github]

Is anyone watching?

[harawata]

Hi @chb0github ,

You need to correct the delimiter syntax first (e.g. DELIMITER || -> -- @DELIMITER ||).

Then please test it with the latest snapshots.
https://oss.sonatype.org/content/repositories/snapshots/org/mybatis/mybatis-migrations/3.3.2-SNAPSHOT/
The latest Migrations snapshot bundle (.zip) includes MyBatis 3.4.6-SNAPSHOT which is required to use multi-char delimiter.
And mybatis/migrations#101 should resolve OOM where it's possible.

[chb0github]

@harawata

I am not sure why this particular repo was incorrectly referring to an old problem. Perhaps you forgot: I was the one who submitted the original PR to support multi-character delimiters.

Unfortunately, github won't allow me to upload the problematic file. So, I am including the whole repo as a public zip and will update the original issue description with the link. For reference, this is the link

[harawata]

I am not sure why this particular repo was incorrectly referring to an old problem.

Because you stated 'Version = 3.3.1' in the issue description and reverted my PR that corrected DELIMITER syntax in your example repo. :)

As ScriptRunner ultimately has to call java.sql.Statement#execute(String statement), it requires a lot of memory when a single statement is very long no matter how efficiently the file is parsed, I think.
A typical example of such statement is the multi-row INSERT statement generated by mysqldump.
I can see some of these INSERT statements in your script, so it probably is the cause of OOM.

If increasing the heap size is not an option and these INSERTs are the only long lines in the script, disabling --extended-insert should resolve the problem.

[billbarni]

I get the exception when trying to load an 132Mb *.sql script.

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine(ScriptRunner.java:148)
	at org.apache.ibatis.jdbc.ScriptRunner.runScript(ScriptRunner.java:110)

ScriptRunner sr = null;
            try {
                Class.forName(driver);
                sr = new ScriptRunner(DriverManager.getConnection(url, user, pass));

                sr.setSendFullScript(false);
                sr.setDelimiter(";");
                sr.setFullLineDelimiter(true);
                sr.setAutoCommit(true);
                sr.setStopOnError(true);

                sr.setEscapeProcessing(true);
                sr.setRemoveCRs(true);
            } catch (Exception e) {
                e.printStackTrace();
                System.out.println("conn ex");
                return;
            }

           try {
                BufferedReader bufferedReader = new BufferedReader(new FileReader(file), 1024*1024);

                sr.runScript(bufferedReader);
           } catch (Exception e) {
               e.printStackTrace();
               System.out.println("file read ex");
                return;
           }

I already tried many combinantions of configs... setting a bigger buffer... sending or not the whole script and other stuff. And I always get the same heap size exception.

[harawata]

Hi @billbarni ,

This report was actually about OutOfMemoryError that occurred when using ScriptRunner from Migrations.

Looking at the code you posted, you should try setting false to fullLineDelimiter if you haven't.
Note that, as I explained in my previous comment, it is difficult to avoid OutOfMemorryError in some cases.
If you have further question, please use the mailing list.