CFMX7 + Apache problem -- excessive pids hanging website

[jdavenport]

Hello,
We are running cfmx7 with apache on red hat linux. A problem started 2 days ago
causing our website to hang, where coldfusion and apache do not seem to be
communicating properly with each other. We will restart the coldfusionmx7
service (with or without restarting httpd service), and everything will be
fine for about 30 mins, then all .cfm pages will hang, and if we check the
status of httpd service it shows tons of pids, ex:

[root@online connectors]# service httpd status
httpd (pid 8592 8591 8590 8589 8588 8587 8586 8585 8584 8583 8582 8581 8574 8573
:
8368 8363 8362 8360 8353 8352 8350 8346 8334 8326 8241 8231 8189 7718 7677 7279
7278 7277 7276 7275 7274 7273 7272 7269) is running...

(ie, about 150 pids ... normally we only see 20 or so pids). It's like some
requests just keep firing over and over but something is not connecting.
We don't have any cf server or apache experts around, just a couple of developers,
and we're at a loss.

We did find a technote from Adobe about the cfmx-connectors.sh file not being
renamed to cfmx-connectors-run.sh properly upon startup, so we did rename that,
but it did not help. Before we did that, when we would restart coldfusion,
we'd get an error about apache connector wizard ("There was an error while
running the connector wizard. Connector installation was not successful") but
CF would still restart. This error has gone away since renaming the
cfmx-connector.sh, but the problem remains.


These messages are appearing repeatedly in the /var/log/httpd/error_log:

"JRun is busy. [15] dropped while sending command"
"Couldn't initialize from remote server, JRun server(s) probably down."
" JRun will not accept request. Check JRun web server configuration and JRun mappings on JRun server."


This message is appearing repeatedly in /opt/coldfusionmx7/logs/cfserver.log:

"java.lang.RuntimeException: Request timed out waiting for an available thread to run.
You may want to consider increasing the number of active threads in the thread pool."


I don't know if these are causes or symptoms. Apologize for the lengthy post ... any suggestions?
thanks much,
Julie Davenport
CF Developer

 

[jdavenport]

update on this:

we ended up taking the .cfm pages off of the site temporarily and leaving only the html pages, then we stopped coldfusion and left it stopped.
the problem ceased. the pids remained at a normal amount, did not accumulate, and the site did not hang. this is not a solution since we need those CF pages, but it seems to point to CF as the problem, as opposed to apache. however, it could still be an issue with the way CF and apache are interacting.

at this point we are out of ideas other than to reinstall CF.

Thanks,
Julie

 

[jdavenport]

Oh yes, I forgot to mention this interesting point:

if we test it by starting up the CF service again, and wait 30-45 minutes, the pids begin to accumulate again and the website hangs again ... even though there are no longer any .cfm pages on the website for users to hit. so just by running CF is making this happen. very strange.....

thanks again,
Julie