Tomcat hanging with "servlet status75"

[gadeiros]

Hello everybody.

I need help urgently !

I have a severe problem with a productive system, running Tomcat 4.1.29 and Java JDK 1.4.2_01-b06 on a Red Hat Linux system (Kernel 2.4.9-e.27smp).

Regularly, the tomcat service is hanging, i.e. the pages are not accessible any more and in catalina out I receive the following error error messages, see below.

There seem to be several problems. The main one seems to be with the number of threads. As if 75 would not be enough. This indicates, that sessions are not closed or reused or whatever, because there are only 10 to 20 workstations accessing the system.

There are some others with log file directories not existing and timeformats not set correctly, but those might not be related to the main problem.

Can anybody point me to what may be wrong here ?

I can send the configuration file for examination if needed, of course.


Thanks for any help,
Harald Henkel
GS automation GmbH

From catalina.out:
-------------------
log4j:ERROR Either Filename or DatePattern options are not set for [logFile].
17.06.2004 08:36:36 org.apache.tomcat.util.threads.ThreadPool logFull
SCHWERWIEGEND: All threads are busy, waiting. Please increase maxThreads or check the servlet status75 75
Stopping service Tomcat-Standalone
17.06.2004 09:57:21 org.apache.coyote.http11.Http11Protocol destroy
INFO: Stoping http11 protocol on 80 Catalina:type=ThreadPool,name=http80
17.06.2004 09:57:21 org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SCHWERWIEGEND: Caught exception executing org.apache.tomcat.util.net.TcpWorkerThread@1bfcfda, terminating thread
java.lang.IllegalStateException
at org.apache.tomcat.util.threads.ThreadPool.findControlRunnable(ThreadPool.java:351)
at org.apache.tomcat.util.threads.ThreadPool.runIt(ThreadPool.java:304)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:666)
at java.lang.Thread.run(Thread.java:534)
17.06.2004 09:58:30 org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on port 80
Starting service Tomcat-Standalone
Apache Tomcat/4.1.29
17.06.2004 09:58:31 org.apache.struts.util.PropertyMessageResources <init>
INFO: Initializing, config='org.apache.struts.util.LocalStrings', returnNull=true
17.06.2004 09:58:31 org.apache.struts.util.PropertyMessageResources <init>
INFO: Initializing, config='org.apache.struts.action.ActionResources', returnNull=true
17.06.2004 09:58:32 org.apache.struts.util.PropertyMessageResources <init>
INFO: Initializing, config='org.apache.webapp.admin.ApplicationResources', returnNull=true
17.06.2004 09:58:34 org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on port 80
17.06.2004 09:58:34 org.apache.jk.common.ChannelSocket init
INFO: JK2: ajp13 listening on /0.0.0.0:8009
17.06.2004 09:58:34 org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=6/45 config=/usr/jakarta-tomcat-4.1.29/conf/jk2.properties
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: logs/click.log (No such file or directory)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:272)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:151)
at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:204)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:247)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:123)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:87)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:645)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:603)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:500)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:406)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:432)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:456)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:145)
at org.apache.log4j.Logger.getLogger(Logger.java:85)
at browser.client.jsp.tag.Log.<clinit>(Log.java:69)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:141)
at org.apache.jsp.RFclient_jsp.class$(RFclient_jsp.java:130)
at org.apache.jsp.RFclient_jsp._jspx_meth_cb_log_0(RFclient_jsp.java:130)
at org.apache.jsp.RFclient_jsp._jspService(RFclient_jsp.java:64)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:210)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:193)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:781)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:589)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:666)
at java.lang.Thread.run(Thread.java:534)
log4j:ERROR Either Filename or DatePattern options are not set for [file].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: logs/click_session.log (No such file or directory)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
[...]

From click_log.2004-06-17.txt: 69 instances ...
-----------------
2004-06-17 09:57:24 StandardWrapper[/Click:jsp]: Waiting for 69 instance(s) to be deallocated
2004-06-17 09:58:31 WebappLoader[/Click]: Deploying class repositories to work directory /usr/jakarta-tomcat-4.1.29/work/Standalone/localhost/Click
2004-06-17 09:58:31 WebappLoader[/Click]: Deploy class files /WEB-INF/classes to /fts/click/web/WEB-INF/classes
[...]

 

[sedj]

What happends when you increase the number of sessions allowed (try 125) ?

 

[gadeiros]

I don't see, how this could help.
There shouldn't be 75 sessions, so if those are not enough 125 will probably only increase a bit the time until the next "break".

Up to now, they didn't use the system very much (still testing), so this problem only occurred every couple of weeks or so.

But soon they will go productive, and the system has to run 24/7 because it's used to control driverless trucks in a production environment.

Of course I could do this change, but I don't see, how it could help.

BTW, is there a chance to get in contact with some Tomcat developers ? I didn't find any email or forum etc. on the Jakarta Tomcat homepage - besides a mailing list.
Would I have to subscribe to this ?

 

[sedj]

Increasing the number of threads would allow us to determine whether the problem is your webapp locking tomcat's internal http threading process, or whether it is some other error. Thats how it would help.

This line worries me :

"All threads are busy, waiting."

Waiting for what ? What are you doing in the webapp ? Are you creating your own threads and experiencing locking ? You need to debug the application part by part. Ascertain what code is the offending lock.

You will not be able to contact Tomcat's developers personally - how many people do you think want to do that ?!

Yes, you will need to join the Tomcat mailing list if you wish to ask questions to other users of tomcat on this mailing list.




--------------------------------------------
Database Connection Pooling Software

http://www.primrose.org.uk

 

[gadeiros]

I'm searching the
tomcat-user@jakarta.apache.org
archives and found several threads regarding the same problem.

But no real solution yet. Somewhere it was suggested to be a problem in Tomcat prior to 4.1.12 - 4.1.27 and an upgrade to 4.1.29 was suggested.

Interestingly there is now more message in the archives since 2004, and somewhere in that time 4.2.30 was released...

I just subscribed to the tomcat-user mailing list and will ask there, if there was a solution to the problem.

BTW, the application uses JSP to access an oracle database.
Unfortunately most of the "framework" is a black hole to me.
I'm just using it to create a front end for Radio Frequence terminals to our Oracle application.

 

[sedj]

Can you try tomcat 5 to see if it is tomcat4 or not ?

Tomcat 4 is now obsolete, and stopped being developed at version 4.1.30.

We use tomcat 4 for our main production servers, which have thousands upon thousands of requests every day, and have never experienced this problem.

You REALLY need to find out if it is your application or not causing the hang - and you should probably do this by trying tomcat5.

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[gadeiros]

In the tomcat mailing list there was a similar question as mine and somebody replied, that this might be due to the RedHat Linux 2.4 kernel using the (new) NPTL (Native Posix Threads Library) backported from 2.5/6 kernels and the Java JDK not being compatible with that.

By setting some env var LD_KERNEL_ASSUME/LD_ASSUME_KERNEL to 2.4 Java should work with it.

I'll have yet to see, if this does the trick.

 

[js1904]

Hi Guys,
I know this is quite an old thread, but I have the same problem at the moment. Did you find any solutions using tomcat 4.1.29?

Best regards
Jacob