Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Register Log in
  • Congratulations gkittelson on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Can Not Keep Tomcat Up and running ~30 mins

Status
Not open for further replies.
rocnet

rocnet

Programmer
Dec 9, 2001
3
US
Ok our fairly active website was moved to new servers
about a month ago and all is well. In the past on
about a monthly basis the tomcat would fail
and be rebooted, cause not found.

This is the first failure in under a month on the new servers....
So I rebooted:
catalina.sh stop -force
catalina.sh start

Runs for about 30 to 60 mins and fails AGAIN...
I suspect some type of memory leak OR OR....
I look forward to some help as to what is going
on, and why NOW...
We've done nothing different, no software changes etc.

Ok
Basics:

OS RHEL 3.3
Kernel 2.4.21-15.ELsmp
MySql 4.0.23
Tomcat 5.0.28
JDK 1.4.2-07
---------------

Here is the page a user gets when tomcat finally fails:

Code: 
HTTP Status 500 -
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: Unable to get connection, DataSource invalid: "org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted, cause: Timeout waiting for idle object"
	org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:825)
	org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:758)
       {MORE NOT SHOWN}
root cause
javax.servlet.jsp.JspException: Unable to get connection, DataSource invalid: "org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted, cause: Timeout waiting for idle object"
	org.apache.taglibs.standard.tag.common.sql.QueryTagSupport.getConnection(QueryTagSupport.java:276)
	org.apache.taglibs.standard.tag.common.sql.QueryTagSupport.doStartTag(QueryTagSupport.java:159)
       {MORE NOT SHOWN}
note The full stack trace of the root cause is available in the Apache Tomcat/5.0.28 logs.
--------------

"top" when tomcat no longer serving ERRORED OUT...

Code: 
 17:17:00  up 55 days,  7:03,  3 users,  load average: 0.79, 0.36, 0.28
158 processes: 153 sleeping, 5 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    9.5%    0.0%    2.1%   0.0%     0.0%    0.0%   88.2%
Mem:  1020432k av,  999688k used,   20744k free,       0k shrd,   94620k buff
                    679016k actv,  126424k in_d,   16468k in_c
Swap: 2048276k av,   72604k used, 1975672k free                  594428k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
28383 app       25   0 11444  11M  5940 R     5.3  1.1   0:00   0 java
28387 app       25   0 10044 9.8M  5400 R     4.1  0.9   0:00   0 java
25359 root      25   0  105M 105M 14192 S     0.5 10.5   1:48   0 java
28358 root      15   0  1300 1300   904 R     0.3  0.1   0:00   0 top

Notice Low Memory.

--------------

"top" when catalina stopped

Code: 
 17:18:59  up 55 days,  7:05,  3 users,  load average: 0.22, 0.29, 0.26
125 processes: 123 sleeping, 2 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.1%   0.0%     0.0%    0.0%   99.8%
Mem:  1020432k av,  894192k used,  126240k free,       0k shrd,   94620k buff
                    572492k actv,  126932k in_d,   17648k in_c
Swap: 2048276k av,   72604k used, 1975672k free                  593848k cached
--------------

"top" when catalina restarted and running 5 minutes.

Code: 
17:25:45  up 55 days,  7:12,  3 users,  load average: 0.53, 0.82, 0.53
124 processes: 122 sleeping, 2 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.2%   0.0%     0.0%    0.2%   99.6%
Mem:  1020432k av,  950468k used,   69964k free,       0k shrd,   94620k buff
                    632696k actv,  124124k in_d,   16996k in_c
Swap: 2048276k av,   72604k used, 1975672k free                  597208k cached
--------------

Here is the page user gets in browser when tomcat finally fails:

Code: 
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: Unable to get connection, DataSource invalid: "org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted, cause: Timeout waiting for idle object"
	org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:825)
	org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:758)
	org.apache.jsp.forumRedirector_jsp._jspService(forumRedirector_jsp.java:183)
       {MORE NOT SHOWN}
root cause
javax.servlet.jsp.JspException: Unable to get connection, DataSource invalid: "org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted, cause: Timeout waiting for idle object"
	org.apache.taglibs.standard.tag.common.sql.QueryTagSupport.getConnection(QueryTagSupport.java:276)
	org.apache.taglibs.standard.tag.common.sql.QueryTagSupport.doStartTag(QueryTagSupport.java:159)
	org.apache.jsp.tag.web.core.country_tag._jspx_meth_sql_query_0(country_tag.java:272)
       {MORE NOT SHOWN}
--------------

Here is a bit from this log:
/home/app/tomcat/logs/...host_log.txt

Code: 
2005-03-24 17:18:02 ApplicationDispatcher[]: Servlet jsp is currently unavailable
2005-03-24 17:18:04 StandardWrapperValve[jsp]: Servlet.service() for servlet jsp threw exception
java.sql.SQLException: Communication link failure: java.net.SocketException, underlying cause: Connection timed out

** BEGIN NESTED EXCEPTION ** 

java.net.SocketException
MESSAGE: Connection timed out
STACKTRACE:
java.net.SocketException: Connection timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
--------------

And from this log:

Code: 
/opt/tomcat/logs/catalina.out

DEBUG [95]:Resetting menu
DEBUG [95]:Photos: [true] [true]
DEBUG [95]:Library: [true] [true]
DEBUG [95]:Intentions: [true] [true]
DEBUG [95]:Blog: [true] [true]

Fine until.... bunches of  errors pages:

DEBUG [99]:Cookie User:[] login/out:[/] pkey:[] URI: [/errors/exception.htm]
DEBUG [99]:Cookie User:[] login/out:[/] pkey:[] URI: [/errors/exception.htm]
DEBUG [99]:Cookie User:[] login/out:[/] pkey:[] URI: [/errors/exception.htm]

Then the stopping catalina and re starting.

--------------

Suggestions are MOST welcome
and TIA!

-Bob OConnor
 
Sort by date Sort by votes
If I were you I would want to know why you are exausting the conn pool - are you sure you are closing all connections correctly ?

--------------------------------------------------
Free Database Connection Pooling Software
 
Upvote 0 Downvote
Thanks sedj,

Shortly after I wrote my initial post, the server STAYED
up... and has been up for about 20 hours now....

Can you give me some Ideas on how to be sure all
the connections are closed?

I presume you mean after I
catalina.sh stop -force
and before restart.

TIA.
-Bob O

 
Upvote 0 Downvote
I mean programmatically - just make sure that every time you get a new JDBC Connection object, you close it (even if a SQLException is thrown during a transaction - always use the "finally" clause in the try/catch statement).

--------------------------------------------------
Free Database Connection Pooling Software
 
Upvote 0 Downvote
Status
Not open for further replies.

Similar threads

ankitbarmagajma
  • Locked
  • Question
What is a SAN, and how is it used?
Replies
1
Views
10K
wudtecker
wudtecker
PBT2708
  • Locked
  • Question
connectionCacheProperties (InitialLimit & MinLimit) not working in Tomcat 8 for OracleDB 11g
Replies
0
Views
10K
PBT2708
PBT2708
Sanjoy Roy
  • Locked
  • Question
Apache Tomcat Crashed Automatically !!!!
Replies
0
Views
10K
Sanjoy Roy
Sanjoy Roy
everdelightedone
  • Locked
  • Question
tomcat 8.0x java.util.logging.Logger.log to file
Replies
2
Views
10K
everdelightedone
everdelightedone
sri12
  • Locked
  • Question
Tomcat 8 URL rewrite
Replies
0
Views
854
sri12
sri12

Part and Inventory Search

Sponsor

Back
Top