Hi all, I have an issue onsite that I am struggling to understand let alone fix !!
A brief overview of the issue is thus.
NIM4.1 compliance solution
2 x VOIP loggers in London office - HPGen8 servers running Win2008 R2 Standard SP1 [We have other voip and TDM loggers in the solution]
Both loggers have non-paged memory utilisation increase of 3% per day - when the utilisation reaches around 75% the logger freezes and we have to reboot the server - to mitigate this we do a controlled reboot of the loggers when they near the 60% mark.
We initially thought the issue to be hardware related and after many weeks of Nice 3rd line support we were advised to flatten the server, re install the OS and logger software etc. We happened to have two spare Gen9 servers so we flattened them, reinstalled the OS, had Nice install the logger software and integrate them into the NIM4.1 environment.
Guess what .... we still have the same issue!
Further local testing has now proven the issue to be related to the Span port of the Logger:
Over the Easter 4 day break we removed the Span port from one of the Production voip loggers, that logger's memory utilisation remained constant over the 4 day break, the other logger continued to rise by 3% per day.
We currently have the old Gen8 servers still patched to the production lan, they have their Nice services running and are patched to the Span port aggregation switches - their memory is still rising by 3% per day [the older servers have a blank Hosts file so there is no integration back into the prod NIM4.1 solution] It gives us the opportunity to try and diagnose the issue using non prod loggers etc.
I did a survey of all the access switches that had uplinks to the span port aggregation switches and one switch "stuck out" as having 829 million bits per second in the 5 minute input rate field. Whereas the average figure in this field for the other (28) access switches was around 100,000 bits per second. The access switches are not managed by me but as luck would have it there were no recorded devices patched to this "suspicious" access switch so I shut the ports on the aggregation switch. The memory on the loggers continues to grow at 3% per day with these ports shut down.
[there was a programming issue on the "suspect" access switch that has now been rectified and the ports opened back up, I suspect that the port was spanning everything rather than just the voice vlan]
I am now at a loss as what to do next, anyone experience something similar, or have something for me to try next ?
Thanks in advance
BigGeoff
A brief overview of the issue is thus.
NIM4.1 compliance solution
2 x VOIP loggers in London office - HPGen8 servers running Win2008 R2 Standard SP1 [We have other voip and TDM loggers in the solution]
Both loggers have non-paged memory utilisation increase of 3% per day - when the utilisation reaches around 75% the logger freezes and we have to reboot the server - to mitigate this we do a controlled reboot of the loggers when they near the 60% mark.
We initially thought the issue to be hardware related and after many weeks of Nice 3rd line support we were advised to flatten the server, re install the OS and logger software etc. We happened to have two spare Gen9 servers so we flattened them, reinstalled the OS, had Nice install the logger software and integrate them into the NIM4.1 environment.
Guess what .... we still have the same issue!
Further local testing has now proven the issue to be related to the Span port of the Logger:
Over the Easter 4 day break we removed the Span port from one of the Production voip loggers, that logger's memory utilisation remained constant over the 4 day break, the other logger continued to rise by 3% per day.
We currently have the old Gen8 servers still patched to the production lan, they have their Nice services running and are patched to the Span port aggregation switches - their memory is still rising by 3% per day [the older servers have a blank Hosts file so there is no integration back into the prod NIM4.1 solution] It gives us the opportunity to try and diagnose the issue using non prod loggers etc.
I did a survey of all the access switches that had uplinks to the span port aggregation switches and one switch "stuck out" as having 829 million bits per second in the 5 minute input rate field. Whereas the average figure in this field for the other (28) access switches was around 100,000 bits per second. The access switches are not managed by me but as luck would have it there were no recorded devices patched to this "suspicious" access switch so I shut the ports on the aggregation switch. The memory on the loggers continues to grow at 3% per day with these ports shut down.
[there was a programming issue on the "suspect" access switch that has now been rectified and the ports opened back up, I suspect that the port was spanning everything rather than just the voice vlan]
I am now at a loss as what to do next, anyone experience something similar, or have something for me to try next ?
Thanks in advance
BigGeoff