Memory Leak / Netshield

[marvhuffaker]

I have numerous servers running either netware 5.1, 6.0, or 6.5. Also running different versions of Netshield. I don't have any problems except with one server.

On one server, which has NetWare 6.0 + SP3 + nss patches and Netshield 4.61, the server memory drains to where nothing is left and I start losing resources and dropping connections. When I look at my resources, I"m down to 8% cache buffers. It's mostly used for file services but also hosts GroupWise messenger.

Has anyone else run into this same thing? It also had leaks with previous versions (Netshield 4.60a and 4.60). If you look at the amounts of memory it is requesting, it is completely insane. No other server I've ever seen has had this problem.

Looking for ideas on how to resolve. It has 1.5GB RAM and about 80GB storage, all NSS. Compaq ML370 (generation 1), Dual 1GHz Processor. Here is what I see in my syslog.


12-21-2003 2:54:31 am: SERVER-5.60-2323
Severity = 1 Locus = 19 Class = 2
Cache memory allocator exceeded the minimum cache buffer limit.

12-21-2003 2:54:31 am: SERVER-5.60-0
Severity = 5 Locus = 1 Class = 1
Short term memory allocator is out of memory.
1 attempts to get more memory failed.
request size in bytes 340123792 from Module NETSHLD.NLM

12-21-2003 2:57:07 am: SERVER-5.60-2323
Severity = 1 Locus = 19 Class = 2
Cache memory allocator exceeded the minimum cache buffer limit.

12-21-2003 2:57:07 am: SERVER-5.60-0
Severity = 5 Locus = 1 Class = 1
Short term memory allocator is out of memory.
2 attempts to get more memory failed.
request size in bytes 212375680 from Module NETSHLD.NLM

12-21-2003 3:04:41 am: SERVER-5.60-0
Severity = 5 Locus = 19 Class = 2
Cache memory allocator out of available memory.

12-21-2003 3:04:41 am: SERVER-5.60-0
Severity = 5 Locus = 1 Class = 1
Short term memory allocator is out of memory.
3 attempts to get more memory failed.
request size in bytes 255116048 from Module NETSHLD.NLM

12-28-2003 2:46:24 am: SERVER-5.60-0
Severity = 5 Locus = 19 Class = 2
Cache memory allocator out of available memory.

12-28-2003 2:46:24 am: SERVER-5.60-0
Severity = 5 Locus = 1 Class = 1
Short term memory allocator is out of memory.
4 attempts to get more memory failed.
request size in bytes 92662944 from Module NETSHLD.NLM

12-28-2003 2:52:39 am: SERVER-5.60-2323
Severity = 1 Locus = 19 Class = 2
Cache memory allocator exceeded the minimum cache buffer limit.

12-28-2003 2:52:39 am: SERVER-5.60-0
Severity = 5 Locus = 1 Class = 1
Short term memory allocator is out of memory.
5 attempts to get more memory failed.
request size in bytes 340123792 from Module NETSHLD.NLM

12-28-2003 2:54:37 am: SERVER-5.60-2323
Severity = 1 Locus = 19 Class = 2
Cache memory allocator exceeded the minimum cache buffer limit.

12-28-2003 2:54:37 am: SERVER-5.60-0
Severity = 5 Locus = 1 Class = 1
Short term memory allocator is out of memory.
6 attempts to get more memory failed.
request size in bytes 168933840 from Module NETSHLD.NLM

12-28-2003 2:58:20 am: SERVER-5.60-2323
Severity = 1 Locus = 19 Class = 2
Cache memory allocator exceeded the minimum cache buffer limit.

12-28-2003 2:58:20 am: SERVER-5.60-0
Severity = 5 Locus = 1 Class = 1
Short term memory allocator is out of memory.
8 attempts to get more memory failed.
request size in bytes 88268256 from Module NETSHLD.NLM

Marvin Huffaker MCNE, CNE
Marvin Huffaker Consulting

http://www.redjuju.com

 

[terry712]

never seen this and most of our servers have been in this configuration at some point

dont have messenger running though - although i did on a test system and never had this

assume you have reinstalled netshld - but again cant see that making a difference

is netshield spending a lot of time scanning a particular directory or something like that - although again if it was something like that you would imagine it would give you a cpu hog abend

all i can think of is try excluding stuff and then building up to see if it's a pth it doesnt like or something

 

[1stITMAN]

Have u installed latest definitions that have caused this problems?

I.T Systems Support Engineer
Bsc. (Hons).

 

[marvhuffaker]

Terry, I'll try to see what I can figure out on those lines.... I have upgraded the netshield programs but don't remember if I just copied over existing files or installed clean. I might try that.

1stitman, this problem has been going on for quite some time, regardless of the definitions in use.

There are about 100k files on one of the volumes, but that's not any different than other servers that aren't having problems. nss pools are clean, only thing I can think of is maybe if there are corrupt files somewhere but haven't found any just doing day to day stuff.



Marvin Huffaker MCNE, CNE
Marvin Huffaker Consulting

http://www.redjuju.com

 

[Lou0686]

Hi Marv,

I'm not so sure that Netshield is causing the memory leak. The low memory errors above are being issued because Netshield is requesting more than available. This does not neccessarily mean that Netshield is the culprate.

Did you check Alloc Memory for the module using the most resources. Is it Netshield or is it a different module?

Lou

 

[jclinebe]

We has a similar problem but only on two servers. tid 2966734 has new server.exe for netware 6 before sp4 and netware 5.1

 

[marvhuffaker]

Lou, the wierd thing is that nothing in the alloc memory shows anything abnormal. The highest is NSS at 78MB.

jclinebe, I looked at that TID and the symptoms are very similar. In fact, I was looking at my error logs and most of the problems happen once a week at approximately 2:00am.. This is about when the netshield is set to update its engine. My guess is that you're right on.

I will try that patch and see if it helps.
Thanks.

Marv


Marvin Huffaker MCNE, CNE
Marvin Huffaker Consulting

http://www.redjuju.com

 

[marvhuffaker]

Well, I applied the patch about 2 weeks ago and waited to see what would happen. Today I noticed that my memory had drained back down to 8,300 Cache Buffers (Originally around 392,000).

As I've looked into it more, through Remote Manager and the graphs and stats listed there, I have found that the memory usage is as follows:

Total System Memory 1,610,201,088 1535 MB
Original Cache Memory 1,606,873,088 1532 MB
Current Cache Memory 36,241,408 34 MB
File System Memory 46,886,912 44 MB
Reserved Swap Memory 28,024,832 26 MB
Swap File Size 29,360,128 28 MB
NLM Memory 289,554,432 276 MB
Virtual Memory Pages 1,234,108,416 1176 MB

As you can see, the Virtual Memory pages are what is consuming all of the memory. However, I do know know how to find out what module or process is causing the Virtual Memory pages to hog that much RAM. It appears as if it is from SERVER.NLM.

I will continue to pursue a solution and post anything I find. Thanks for all the helpful ideas.

Marvin




Marvin Huffaker MCNE, CNE
Marvin Huffaker Consulting

http://www.redjuju.com

 

[deejay33]

Hello Marv,

Did you ever figure this one out. We have a NW 6.5 server that seems to be leaking memory. Looking at the statistical trend graph it actually started the day we upgraded from 6.0 to 6.5 sp 1.1. I wish I had seen this post before we rebooted. I would have like to see what our virtual memory was at before the reboot. The main thing I noticed on our is nss.nlm was using about 148mb before the reboot and then only 34mb after. Is that normal? Also, the reboot got me back to 600MB (we have 1gb total) available memory from 215mb. Since I only gained 114mb from nss.nlm the other 271mb must have come from some where else. I do have the printouts of all the Alloc memory from before and after and the total difference is about 130mb so that doesn't account for it. Any ideas? Did you get this solved for you?

Thanks for the help,

Daniel Joaquin

 

[marvhuffaker]

Daniel, I never resolved it on the NW6.0 platform. ALthough I recently upgraded the server to NW6.5 and also got rid of McAfee and replaced it with Symantec Antivirus Corporate edition. It hasn't leaked yet. Right now, I'm only using 2% of the virtual memory (that was where the problem was before).

I still believe that it was a leak in McAfee. The memory always seemed to drain during/after an automatic definition update. But I couldn't ever pinpoint it exactly. Mcafee modules did not exhibit any abnormal trend stats. Also, as crazy as it sounds, I think it may have also been related to running GroupWise Messenger along with McAfee. My servers without Messenger (but other stuff the same), never had this problem.

If you don't mind, could you please start a new post and include more details about what is running on your server, hardware configs, some of the stats, behaviors, etc.



Marvin Huffaker MCNE, CNE
Marvin Huffaker Consulting

http://www.redjuju.com