IP500V2 Memory Leak on 11.0.4 SP 6? 11.0.4.6 2

[rgunther]

Hey guys... So we have an IP500V2 with about 150 IP & SIP extensions. We recently were on release 11.0.4 SP4, and noticed that our memory card showed memory utilization high messages and was as high as approx 95% utilized. We were told this was a known issue in 11.0.4 SP4 and that we needed to install 11.0.4 SP6, in which we did. After doing so, our memory usage went down to about 44%. Well about 2 weeks later now we are at 66%, and i'm afraid its going to continue going up again... Any idea's or suggestions?

 

[tac84]

Have you tried the old format & rebuild SD card through Manager trick?

Thanks, Tim
Adelaide, Australia

 

[sizbut]

Its pretty normal for the old 4GB SD cards to be 60% plus used. Always best to arrange a swap to an 8GB card.

Stuck in a never ending cycle of file copying.

 

[rgunther]

This IP500v2 already has an 8GB card. I have not done the format and re-build SD card option yet as this system is 150 miles from me. If I had a spare avaya 8GB card ready would I need to transfer PLDS licensing & IPOSS to this new sd card?

 

[derfloh]

Only the PLDS Licensing.

IP Office remote service
IP Office certificate check
CLI based call blocking
SCN fallback over PSTN

 

[rgunther]

Thanks guys. If i decide to just format the existing SD card & re-create, then what other files to do i need to put back on the sd card after I use the re-create option? I am assuming the IPO system will be unusable during the time the SD card is removed?

 

[budbyrd]

The card can be removed for two hours before it complains if I remember right. You would need the config.cfg and PLDSkeys.xml files.

Dermis and feline can be divorced by manifold methods.*
*(Disclaimer for all advise given)--'Version Dependent'

 

[critchey]

By chance are there still the old ADI licenses loaded? I was told at one point, dont even remember by who anymore, that if you leave ADI licensing on a newer PLDS system it can cause a memory leak.

The truth is just an excuse for lack of imagination.

 

[rgunther]

Good thought - I also read that on previous posts, and have confirmed that we do not have any remaining ADI licenses on the system.

 

[rgunther]

Hey guys -

So earlier this week (monday night) we pulled the system SD card, performed a format, and re-build. We then put the config file and license file back on, rebooted the system and everything loaded back up normally. However it doesnt appear to have fixed the memory leak issue. After the format/re-build we were sitting at 38% usage. It has continuously been going up about 2% every day, and today (4 days later) we are now sitting at 54%. I am afraid that in another week or two we are going to be right back up to the 90% levels and the system will end up rebooting. Any suggestions?​

 

[qtelcom]

Am I missing something here? what does a SD card have to do with memory leak issue? Are you talking about the drive size filling up on the SD Card? Are you using voicemail pro or the SD card for voicemail?

 

[rgunther]

We are using Voicemail Pro... so its not filling up with Voicemail. Its the memory usage that keeps filling up. It up to 57% percent now, and keeps increasing daily until the system reboots. After the system reboots we are back down to 30-40%, and then it will start filling back up again.

 

[rgunther]

Here is a screenshot..

 

[qtelcom]

looks like no one read your post correctly. The SD card has nothing to do with the internal memory usage on the IP500v2. SP7 should be coming out soon, with that said, if you have a real memory leak and it's causing your system to lock up or reboot you'll need to start a ticket with Avaya support to resolve.

 

[rgunther]

Does anyone think a swapping the IP500 chassis would be something to try? I see now 3 days later we are up to 69%. Im sure by th end of the week we will be up in the 90s. What memory is this - is it the System RAM? See attached...

 

[rgunther]

Hey guys - so we recently decided to swap the IP500 to try and remdedy the issue, but unfortunately still no-go. Therefor we have
-Upgraded from 11.0.4.4 to 11.0.4.6
-Formatted & re-created SD card
-Replaced SD Card
-Replaced IP500v2 Chassis

Not sure what else to even look for here. Our memory usage in system status continues to climb every day (approx 5% daily) until it reaches the "memory utilization high" errors and the system ends up rebooting. I have tried to open a monitor trace to see if anything shows up in there - but not quite sure what options i need to enable to find this sort of isssue. I do see where is shows in the previous logs where it was rasing the memory use warnings -

17:34:00 497953689mS PRN: +++ START OF ALARM LOG DUMP +++
17:34:00 497953689mS PRN: ALARM: 04/10/2021 21:57:40 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.1% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:00:13 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.8% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:01:14 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.2% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:01:35 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.3% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:03:26 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.10% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:05:18 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.3% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:10:13 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.11% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:11:14 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.5% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953690mS PRN: ALARM: 04/10/2021 22:14:37 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.1% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:15:18 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.6% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:17:41 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.6% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:18:22 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.1% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:18:53 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.1% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:19:44 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.1% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:20:04 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.1% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:21:25 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.2% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953691mS PRN: ALARM: 04/10/2021 22:22:05 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.4% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953692mS PRN: ALARM: 04/10/2021 22:22:36 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.4% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953692mS PRN: ALARM: 04/10/2021 22:22:56 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.2% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953692mS PRN: ALARM: 04/10/2021 22:23:16 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.2% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953692mS PRN: ALARM: 04/10/2021 22:23:58 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.2% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953692mS PRN: ALARM: 04/10/2021 22:24:38 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.2% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953692mS PRN: ALARM: 04/10/2021 22:24:59 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.2% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953692mS PRN: ALARM: 04/10/2021 22:26:21 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.3% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953693mS PRN: ALARM: 04/10/2021 22:27:01 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.4% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953693mS PRN: ALARM: 04/10/2021 22:27:42 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.4% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953693mS PRN: ALARM: 04/10/2021 22:28:13 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.4% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953693mS PRN: ALARM: 04/10/2021 22:28:53 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.4% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953693mS PRN: ALARM: 04/10/2021 22:30:05 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.12% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953693mS PRN: ALARM: 04/10/2021 22:31:25 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.7% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953693mS PRN: ALARM: 04/10/2021 22:32:06 IP 500 V2 11.0.4.6.0 build 18 <Memory use warning at 90.8% > MAJOR RAISED addr=00000000 d=0 pc=00000000 ffffffff ffffffff ffffffff ffffffff ffffffff
17:34:00 497953694mS PRN: ALARM: 05/10/2021 06:38:13 IP 500 V2 11.0.4.6.0 build 18 <OSMemPool::Allocate Out of Memory: 3276> CRIT RAISED addr=f02cae7c d=5 pc=f02614cc f0265ce4 f0c3af84 f0c03ab4 f1205748 f0b18d50

Any idea's on what could be going on here? Thanks,

 

[Johnhyde]

Why not downgrade to 11.0.4.4 or 11.0.4.5?

 

[rgunther]

The issue started when we were on 11.0.4.4, so we upgraded to 11.0.4.6 in hopes there was a fix in the latest SP.

 

[tac84]

@rgunther - so is this still an issue or have you managed to resolve it? Curious as to what Avaya have to say about it...

Thanks, Tim
Adelaide, Australia

 

[rgunther]

Unfortunately yes still an ongoing issue. Avays has had me try replacing SD Card, replacing IP500V2 chassis.... still no go. I am starting to believe it is related to our SIP Trunk, as I have connected up a test IP500V2 and configured a sip trunk on it identically the same as our working environment and it appears the test chassis has started doing the same thing. Next step today I'm going to delete the SIP trunk in the test unit and see if the memory utilization stops climbing once it's deleted. I will update when I find out completely.