java.net.SocketException: Too many open files

[developerinlondon]

for some strange reason my catalina seems to fall over with the above error. heres the full stack trace in catalina.out:

Jun 7, 2005 4:37:29 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8082] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:406)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Jun 7, 2005 4:37:29 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket
Jun 7, 2005 4:37:29 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8082] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:406)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)

any ideas what might be wrong?

thanks in advance!

 

[sedj]

This error comes from the OS, when there are too many open file or socket descriptors (usually more than 1024 on a linux standard kernel).

Check that any socket or file objects are being closed correctly.

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

well the OS has a limitation of 403000+

# less /proc/sys/fs/file-max 
403504

my code has been running for months without any problems.

i read somewhere (

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4189011)

that this can get caused by the JVM limitation, I tried running the example on this link on my machine and it failed wth (Too many open files) error. According to that post it should have been fixed in JVM1.5 but maybe its still an issue with JVM for linux?

 

[sedj]

file-max describes the number of file descriptors for the OS.
I am talking about the limit of file descriptors for a SINLGE PROCESS - which I expect is about 1024.

You try running this C code :

int main(int argc, char* argv[]) {
  int iOpenFilesCnt;
  char szFilename[256];
  sprintf(szFileName, "my_file_%d.txt", iOpenFilesCnt);
  printf("Opening file name : %s, count is : %d", szFileName, iOpenFilesCnt);
  FILE* pFile = fopen(szFileName, "wb");
  if (pFile == NULL) {
    perror("Error opening file :");
    return 1;
  }
  printf(" - done\n");
  // do not close file descriptor !
  return 0;
}

I bet you don't get anyway near 400,000 open files before it dies !

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[sedj]

oops, put that in a loop !

int main(int argc, char* argv[]) {
    while (1) {
	int iOpenFilesCnt;
	char szFilename[256];
	sprintf(szFileName, "my_file_%d.txt", iOpenFilesCnt);
	printf("Opening file name : %s, count is : %d", szFileName, iOpenFilesCnt);
	FILE* pFile = fopen(szFileName, "wb");
	if (pFile == NULL) {
	perror("Error opening file :");
	return 1;
	}
	printf(" - done\n");
	// do not close file descriptor !
    }
	return 0;
}

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

my C is quite rusty but I am getting the following error:

# cc test.c -o test
test.c: In function `main':
test.c:5: error: `szFileName' undeclared (first use in this function)
test.c:5: error: (Each undeclared identifier is reported only once
test.c:5: error: for each function it appears in.)
test.c:7: error: `FILE' undeclared (first use in this function)
test.c:7: error: `pFile' undeclared (first use in this function)
test.c:8: error: `NULL' undeclared (first use in this function)

so assuming you are right, where/how would I change the maxfiles per process? if I run 'ulimit' it shows 'unlimited'?

 

[sedj]

Ah, OK, I didn't bother compiling it :

BTW, I wouldn't bother trying to change the limit - 1024 open descriptors should be enough for any process - and if it isn't ... well perhaps a redesign.

I think you need to work out why and where you are leaking file descriptors. Upping the limit is not really fixing the error is it ? You'll just blow the limit further down the road until the OS dies !

#include <stdlib.h>
#include <stdio.h>
#include <error.h>

int main(int argc, char* argv[]) {
    int iOpenFilesCnt = 0;
    
    while (1) {
    	char szFileName[256];
    	sprintf(szFileName, "my_file_%d.txt", iOpenFilesCnt);
    	printf("Opening file name : %s, count is : %d", szFileName, iOpenFilesCnt);
    	FILE* pFile = fopen(szFileName, "wb");
    	if (pFile == NULL) {
    		perror("Error opening file :");
    		return 1;
    	}
    	printf(" - done\n");
		iOpenFilesCnt++;
    	
    	// do not close file descriptor !
    
    }
    
    return 0;
}

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

you are actually right, the java example also fails when it reaches 1024 simultaneous open files.

so how/where do I up it?

 

[developerinlondon]

well each time a servlet gets invoked it opens some 10+ simultaneous connections, each uses some 10+ database connections, and at peak times we get about 200 of these servlets being invoked (and rising each day as site gets busier). doesnt network connections/database connections open temporary files?

 

[developerinlondon]

I am saying because the original exception was first caused by a Thread creation problem due to 'too many files', then it was followed by a servlet creation problem due to 'too many open files', so i am assuming creating threads/servlets/using commons libraries/logger etc all opens files.

 

[sedj]

Well if you think the max limit per process is legitimately being reached, and you need to up it, then use :

ulimit -n2048

to raise the number to 2048.

However, I would be extrememly surprised if you really had over 1000 socket/file descriptors - and I suspect you are not closing some sockets/files somewhere.

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk