outofmemory exception in tomcat

[developerinlondon]

I am getting the following error throwing up occassionally -


May 5, 2006 3:44:02 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet MultiFetcher threw exception
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at com.flightfile.main.MultiFetcher.processRequest(MultiFetcher.java:159)
at com.flightfile.main.MultiFetcher.doGet(MultiFetcher.java:209)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:868)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:663)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)

this happens on average once every few days, even when the server isnt busy... any ideas how i can find out whats going on?

 

[sedj]

I exepect that in MultiFetcher.java, on line 159, you are creating an object that holds so much data, that you are blowing the max amount of RAM assigned to the JVM.

Alter Tomcat's startup JVM parameters to assign more RAM.

Eg, to assign 512 Mb to the JVM :

-Xms512m -Xmx512m .

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

where in tomcat do i pass the "-Xms512m -Xmx512m ."? I know its in the catalina.sh somewhere...?

 

[sedj]

which tomcat version ?

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

5.5

 

[sedj]

In catalina.sh, just after all the file comments (about line 45), add the below :

JAVA_OPTS=" -Xms512m -Xmx512m $JAVA_OPTS"

This will fire the JVM with 512 Mb assigned

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

thanks. tried upping the value. but the problem seems to be lying somewhere else.

it fell over again this saturday. this time on the catalina log i noticed a lot of errors as follows -
[NewConnection] Cannot connect to DB (SQLException)... jdbc:mysql://10.2.0.245:3306/xml_availability?autoReconnect - java.sql.SQLException: Too many conn
ections
java.lang.NullPointerException
at uk.org.primrose.pool.core.PoolConnection.isClosed(PoolConnection.java:209)
at uk.org.primrose.pool.core.Pool.fill(Pool.java:115)
at uk.org.primrose.pool.core.Pool.get(Pool.java:94)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:479)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
at uk.org.primrose.pool.jmx.Queue.getConnectionWait(Queue.java:496)
...

I am using primrose database pool and have it configured as follows -
1) 5 connections with 1 connection extra.
2) killactiveconnectionsoverage = 120000
3) cycleconnections = 1000

i checked the code over and over again and i cant see anywhere the connection not being closed. each time it gets used it also gets closed. only in one of them i dont have a finally clause... but this shouldnt make a diff?

any ideas?

 

[sedj]

Are you on the latest primrose version ? There was a serious bug found, which 2.7.0 fixes - but it may not fix your error.

It looks like an overloaded pool from that stack trace - ie you have too many clients asking for too few connections OR there is a massive connection leak, and everything is just queueing to get connections.

By rights you should really make sure connections are closed in the finally clause - its just safer.

You should also monitor your database connections on the actual db - and see what machines have been allocated, what would appear to be, too many connections (see the SQLException "too many connections") - something, somewhere is leaking connections without a doubt.

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

i will try upgrading primrose. have just contacted them to see where i can get an upgrade that i can download from headless.

 

[developerinlondon]

if it is a matter of overwriting the jar files and updating the properties file then I just have done it and it seems to be running. time will tell if it fixed the problem.

 

[sedj]

Thats pretty much it ... but as I said before, I doubt it is primrose that is causing this. You need to monitor your db to see what other applications are grabbing excessive connections. If your webapp s the only app using the db, then you need to watch the active connections (via the web management page) in the primrose pool to see if any being left open.

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk