BIE hub outage

[joe-lipson]

There was a BIE outage on July 19th from around 3:30pm to 4:00pm. Tomcat log showed an out of memory error

Jul 19 15:57:26 aws-bie-2021-1 tomcat9[676]:     at java.base/java.lang.Thread.run(Thread.java:829)
Jul 19 15:57:26 aws-bie-2021-1 tomcat9[676]: Caused by: java.lang.OutOfMemoryError: Java heap space
Jul 19 15:57:42 aws-bie-2021-1 tomcat9[676]: Servlet.service() for servlet [grailsDispatcherServlet] threw exception
Jul 19 15:57:42 aws-bie-2021-1 tomcat9[676]: java.lang.OutOfMemoryError: Java heap space
Jul 19 15:57:43 aws-bie-2021-1 tomcat9[676]: Servlet.service() for servlet [grailsDispatcherServlet] threw exception
Jul 19 15:57:43 aws-bie-2021-1 tomcat9[676]: java.lang.OutOfMemoryError: Java heap space
Jul 19 15:57:46 aws-bie-2021-1 tomcat9[676]: Servlet.service() for servlet [grailsDispatcherServlet] threw exception
Jul 19 15:57:46 aws-bie-2021-1 tomcat9[676]: java.lang.OutOfMemoryError: Java heap space
Jul 19 15:57:47 aws-bie-2021-1 tomcat9[676]: Servlet.service() for servlet [grailsDispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.grails.gsp.GroovyPagesExcept>
Jul 19 15:57:47 aws-bie-2021-1 tomcat9[676]: java.lang.OutOfMemoryError: Java heap space

Investigate to find out what had caused this and put a fix in place. There weren't any out of the ordinary traffic patterns at the time

[joe-lipson]

There was another BIE hubs outage today with the same OutOfMemoryError.

The Java heap has been increased from 3GB to 6GB. This should at least give us more time between outages.

This will need some code level investigation to determine the root cause. In the meantime these actions can be done, they may or may not help!

• Upgrade Grails from 6.0.0 to latest 6.x
• Upgrade to latest current major version for Tomcat and Java
• Set up tomcat to restart every night at 4am