Weblogic6.1sp2 server crash java.lang.OutOfMemory error 1

[dhirendra]

we have developed our application and deployed it on weblogic6.1sp2 and our databaseis Oracle9i both on Tru64 unix. we have used the following tech in our App.

1. JSP for client presentation used with Struts framework.
2. Stateful Session beans which has BusinessLogic and Entity beans for Data operations.
3. MDBs for Asycn. Operation.

Ours is a highly data oriented project which is a for of Product Catalog.

Yhe Problem we are facing is the after running for few hours server is crashing with Java.lang.OutOfMemory Error but when we do VM Stat there is lot of memory left on system and also the weblogic performance graph shows normal GCs and enough memory.


Can any one help me !!!


I am attachng the two errors .
Error 1 :

OCI-21503: program terminated by fatal error
OCI-04030: out of process memory when trying to allocate 2056 bytes (Alloc environm,eoj_malloc)
Errors in file :
OCI-21503: program terminated by fatal error
OCI-04030: out of process memory when trying to allocate 2056 bytes (Alloc environm,eoj_malloc)



----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------

----- Argument/Register Address Dump -----




----- End of Call Stack Trace -----

Error 2:
java.lang.OutOfMemoryError: ilog/rules/webui/IlrWCompiler$CDeleteLine:
at java.lang.ClassLoader.defineClass0 (ClassLoader.java)
at java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) (ClassLoader.java:498) (pc 87)

at java.security.SecureClassLoader.defineClass (SecureClassLoader.java:111) (pc 27)
at java.net.URLClassLoader.defineClass (URLClassLoader.java:248) (pc 171)
at java.net.URLClassLoader.access$100 (URLClassLoader.java:56) (pc 3)
at java.net.URLClassLoader$1.run (URLClassLoader.java:195) (pc 43)
at java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,java.security.AccessControlContext) (AccessController.java)

at java.net.URLClassLoader.findClass (URLClassLoader.java:188) (pc 13)
at java.lang.ClassLoader.loadClass(java.lang.String,boolean) (ClassLoader.java:309) (pc 43)
at sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:286) (pc 36)
at java.lang.ClassLoader.loadClass(java.lang.String) (ClassLoader.java:265) (pc 3)
at ilog.rules.webui.IlrWRenderer.<init> (pc 10)
at ilog.rules.webui.IlrWSyntacticEditor.fzc477 (pc 21)
at ilog.rules.webui.IlrWSyntacticEditor.printComponent (pc 13)
at ilog.webui.dhtml.IlxWComponent.try (pc 73)
at ilog.webui.dhtml.IlxWPort.if(ilog.webui.dhtml.IlxWComponent) (pc 13)
at ilog.webui.dhtml.IlxWComponent.print (pc 2)
at com.dstsystems.concorde.framework.rulesAPI.web.builder.BrmElementPanel$SyntacticView.printComponent (pc 5)
at ilog.webui.dhtml.IlxWComponent.try (pc 73)
at ilog.webui.dhtml.IlxWPort.if(ilog.webui.dhtml.IlxWComponent) (pc 13)
at ilog.webui.dhtml.IlxWComponent.print (pc 2)
at com.dstsystems.concorde.framework.rulesAPI.web.builder.BrmElementPanel.printComponent (pc 35)
at ilog.webui.dhtml.IlxWComponent.try (pc 73)
at ilog.webui.dhtml.IlxWPort.if(ilog.webui.dhtml.IlxWComponent) (pc 13)
at ilog.webui.dhtml.IlxWComponent.print (pc 2)
at ilog.webui.dhtml.components.IlxWTabbedPane$TabCellComponent.printComponent (pc 16)
at ilog.webui.dhtml.components.IlxWTabbedPane$4.printComponent (pc 25)
at ilog.webui.dhtml.IlxWTableElement.try (pc 43)
at ilog.webui.dhtml.IlxWPort.if(ilog.webui.dhtml.IlxWComponent) (pc 13)
at ilog.webui.dhtml.IlxWComponent.print (pc 2)
at ilog.webui.dhtml.components.IlxWTabbedPane.n (pc 111)
at ilog.webui.dhtml.components.IlxWTabbedPane.printComponent (pc 118)
at ilog.webui.dhtml.IlxWComponent.try (pc 73)
at ilog.webui.dhtml.IlxWPort.if(ilog.webui.dhtml.IlxWComponent) (pc 13)
at ilog.webui.dhtml.IlxWComponent.print (pc 2)
at com.dstsystems.concorde.framework.rulesAPI.web.builder.CenterPanel.printComponent (pc 21)
at ilog.webui.dhtml.IlxWComponent.try (pc 73)
at ilog.webui.dhtml.IlxWComponent.a(ilog.webui.dhtml.IlxWScriptWriter,ilog.webui.dhtml.IlxWScriptWriter,ilog.webui.dhtml.IlxWPort) (pc 78)

at ilog.webui.dhtml.IlxWPort.a(ilog.webui.dhtml.IlxWComponent,ilog.webui.dhtml.IlxWScriptWriter,ilog.webui.dhtml.IlxWScriptWriter) (pc 30)

at ilog.webui.dhtml.IlxWUpdateManager.a(ilog.webui.dhtml.IlxWScriptWriter,ilog.webui.dhtml.IlxWScriptWriter,ilog.webui.dhtml.IlxWPort) (pc 124)

at ilog.webui.dhtml.IlxWPort.a(ilog.webui.dhtml.IlxWScriptWriter,ilog.webui.dhtml.IlxWScriptWriter) (pc 10)
at ilog.webui.dhtml.IlxWController.service (pc 584)
at javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest,javax.servlet.ServletResponse) (HttpServlet.java:853) (pc 29)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(javax.servlet.ServletRequest,javax.servlet.ServletResponse,weblogic.servlet.internal.FilterChainImpl) (ServletStubImpl.java:265) (pc 212)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(javax.servlet.ServletRequest,javax.servlet.ServletResponse) (ServletStubImpl.java:200) (pc 4)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet(weblogic.servlet.internal.ServletRequestImpl,weblogic.servlet.internal.ServletResponseImpl) (WebAppServletContext.java:2495) (pc 264)

at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2204) (pc 9)
at weblogic.kernel.ExecuteThread.execute (ExecuteThread.java:139) (pc 12)
at weblogic.kernel.ExecuteThread.run (ExecuteThread.java:120) (pc 62)

 

[MrTom]

what options are you giving the JVM in your startWeblogic script? (also how much memory does the hardware have?)

of course you may have a memory leak in your java

 

[dhirendra]

Thanks for responding Tom.

Its a production system and it has around 2GB RAM. We are starting the server with ms mx of 512M and we have also tried -XNewms option also.

 

[MrTom]

If there are any performance packs loaded you could try disabling them as i've seen people blame them for some problems.
The other step could be to upgrade to sp3. I'm still using sp2 but then need to build a new sever made me look at the sp3 changes and there seem to be quite a few. One which might interest you......

075553 Fixed a memory leak that was occurring because the core
classloader was being stored unnecessarily.

 

[dhirendra]

Can you Please send me the link where u found this information. We have tried with sp3 on solaris where the server is giving Hotspot Error after series of Java.lang.OutOfMemory error. But the OCI-04030 is not reproducable on solaris. The HotSpot error is ....

#
# HotSpot Virtual Machine Error, Internal Error
# Please report this error at
#

http://java.sun.com/cgi-bin/bugreport.cgi

#
# Error ID: 4349254E560E43505000EC 01
#
# Problematic Thread: prio=5 tid=0xb87f8 nid=0xb runnable

 

[MrTom]

i can't find the changes now but i think you need to log in (i don't have my pw with me)

http://websupport.beasys.com/custsupp

as for the Solaris hotspot error have a look at the Solaris patch 108827-26 (search for hotspot). adding LD_LIBRARY_PATH=/usr/lib/lwp
to the start script gets around this. (

http://java.sun.com/docs/hotspot/threads/threads.html)

- apologies if you know all about this already.

 

Hey Tom,
I tried using the LD_LIBRARY_PATH=/usr/lib/lwp option in solaris and I found that the server hangs while compiling JSP. Can you please tell me if there is any formulae to find out the optimal value for MaxPermSize NewSize and other such JVM options for a given system with certain amount of RAM and Number of processors.

Thnks
Dhirendra

 

Hey Tom,
I tried using the LD_LIBRARY_PATH=/usr/lib/lwp option in solaris and I found that the server hangs while compiling JSP. Can you please tell me if there is any formulae to find out the optimal value for MaxPermSize NewSize and other such JVM options for a given system with certain amount of RAM and Number of processors.

Thnks
Dhirendra

 

[dhirendra]

Hey Tom,
I tried using the LD_LIBRARY_PATH=/usr/lib/lwp option in solaris and I found that the server hangs while compiling JSP. Can you please tell me if there is any formulae to find out the optimal value for MaxPermSize NewSize and other such JVM options for a given system with certain amount of RAM and Number of processors.

Thnks
Dhirendra

 

[MrTom]

Hmmmm, shame it hangs. I was assuming you were using Solaris 8 so it may be different for other versions. If you are using 9 then you definately don't need to add the LD_LIBRARY_PATH (as it is now the deafult anyway).

my Java options are JAVA_OPTIONS=&quot;-Xconcurrentio -Xoptimize -XX:NewSize=128m -XX:MaxNewSize=128m -Xms512m -Xmx512m&quot;

and I also changed the -hotspot to -server (near the bottom of the start script). I can't guarantee these are sane/safe options. Made the decisions after reading

http://developer.java.sun.com/developer/technicalArticles/Programming/JVMPerf/

i don't have any multiprocessor systems to play with so i don't know how this affects things. On a system with 2GB RAM i tried using a big heap size of >1GB but Solaris seemed to swap a lot out to disk but i never really measured the performance since the 512MB heap i use now is more than enough for the app.

i also use -vebosegc and -verbose which prints out GC statistics some other class info.

 

[jtnoronha]

We were having the same problem

#
# HotSpot Virtual Machine Error, Internal Error
# Please report this error at
#

http://java.sun.com/cgi-bin/bugreport.cgi

#
# Error ID: 4349254E560E43505000EC 01
#
# Problematic Thread: prio=5 tid=0xb87f8 nid=0xb runnable


with WL6.1SP2 and WL6.1SP3 without the -server flag. Now it seems not to happen, but it is too soon to be sure. Does anyone have any new pointers regarding this situation?

 

[jtnoronha]

I forgot to tell that we started using the &quot;-server&quot; flag.

 

[sekharkosuri]

Hi,

I am also facing the same out of memory problem in solaris environment. Can i know how you people solved this problem??

Thanks
Raj..

 

[svadlenka]

Maybe there is problem with Oracle or oracle client driver, as OCI means Oracle Common Interface.

 

[sekharkosuri]

Can anybody tell which version of JRE has -server option on windows platform.

 

[weblogicShanti]

hi ..
i am uploading 1.2 GB data file in to oracle using java prg.
prg. logic is, it reads the data line by line and then based on the first three char of the line it determines the table to which it will insert data and rest of the data come in that line goes to the row of the table.

small file are uploaded correctly but when i try large file i get
Oct 28, 2002 11:09:28 PM GMT+08:00> <Error> <HTTP> <[WebAppServletContext(3943
07,DefaultWebApp_myserver)] Servlet failed with Exception
java.lang.OutOfMemoryError
<<no stack trace available>>

i am using weblogic 6.0sp2, jdk1.3, 512 mbram

how to give mx and ms option on starting the weblogic ?

in startWebLogic.cmd i have
&quot;%JAVA_HOME%\bin\java&quot; -hotspot -ms512m -mx512m -classpath %CLASSPATH% -Dweblogic.Domain=misasia -Dweblogic.Name=myserver &quot;-Dbea.home=C:\bea&quot; &quot;-Djava.security.policy==C:\bea\wlserver6.0/lib/weblogic.policy&quot; -Dweblogic.management.password=netreach weblogic.Server

please its urgent..

Shanti

 

[sekharkosuri]

Hi,

1. First of all, check your program whether it has any memory leaks. If you can share the piece of code we can identify memory leaks.

2. Use -server option instead of -hotspot

 

[texasmustang]

We're seeing the same hotspot error on BEA 7.0, JDK 1.3.1.03, Oracle 8.1.7 OCI driver & Windows 2000 Server. We're running with -hotspot. The &quot;error managing connection&quot; lines are debug statements in some of our framework code.

error managing connection: java.lang.NullPointerException
error managing connection: java.lang.NullPointerException
error managing connection: java.lang.NullPointerException
#
# HotSpot Virtual Machine Error, Internal Error
# Please report this error at
#

http://java.sun.com/cgi-bin/bugreport.cgi

#
# Error ID: 4349254E560E43505000EC
#
# Problematic Thread: prio=5 tid=0x825490 nid=0xa78 runnable

 

[sanctus]

We have the same type of error with Weblogic 7.0 on Windows and RedHat. We're using DB2 7.2, Struts, Stateless session and entity beans.

#
# HotSpot Virtual Machine Error, Internal Error
# Please report this error at
#

http://java.sun.com/cgi-bin/bugreport.cgi

#
# Error ID: 4349254E560E43505000EC
#
# Problematic Thread: prio=5 tid=0x23f050 nid=0x39c runnable
#

We'll try the -server option now.

 

[wlslearner]

JDK131_03 Sucks. Install Sun Recommended J2se patches there has been lot of bug fixes. Trying upgrading to JDK131_06 or downgrade to jdk131_02. I personally perfer to upgrade.