WAN Issue causing IPSI Sanity loss and reset

[justinsm2]

My configuration is a S8710 with 9 Port Networks with 2 G650's in each PN. Also there are 4 G700's located at additional remote sites with S8300 LSP's. I have duplicated IPSI's in each port network for redundancy along with duplicated CLAN's and Medpro 320's. 2 Port Networks are in remote sites and receive signaling over our EVPN MPLS.
My issue is that what appeared to be a WAN issue where a reroute happened in the cloud causing a momentary loss of link in the WAN and subsuqent latency caused the IPSI to lose sanity resulting in them resetting along with the port networks.
Is there a way to mitigate this so that a IPSI losing sanity won't force the rest to reset with it? Not really seeing why a Remote G650 would be able to cause the internal LAN connected to also reset.

 

[nohuhu]

justinsm2,

ipsi are independent of each other so resetting one of them doesn't affect others. if you have local pns resetting at the same time with remote ones, look hard for network problems. i mean, local network problems. check speed and duplex, network settings, qos, routing, recheck it and then have someone to check everything for you again. it always turns out to be a problem in the network.

 

[pat1960]

I just had 5 Port Network using G650s with duplicated IPSIs go down (2 separate locations). We are in a dual-connect environment (Ipsis and CSS at HQ and G650s connected IP remotely). The G650s are remote from the 8700s, plenty of bandwidth (each on its own DS3) with all network logs showing no issues. The Cisco switch logs show the all IP devices in the G650s (IPSI, VAl, Clan, and Medpro) at both locations lost electrical connection (thus I am thinking that the port networks reset). I went through the logs and can find no evidence that anything was wrong prior to the reset.

The local cabinets connected through IPSIs had no issues (they use a different control network, the remote IPSIs are on Control Network C). What would cause the remote Port Networks to appear to reboot? We also have remote G700s which were fine, they are using the same circuits as the G650s.

One other question, the server list in the IPSIs contains the 2 primary 8700 servers and the 8500 ESS. Is the order of the servers important in the list? In other words if the first server in the list fails will it look to the second server? I noticed that some of the IPSIs have the ESS as the second server in the list, while some have the ESS as the third and last server.

Thanks ahead for any help.

 

[justinsm2]

pat1960,

What version of Communication Manager are you on?

 

[pat1960]

3.1.2 is the version.

 

[nohuhu]

pat1960,

look for 'sanity' errors in logmanager debug trace prior to resets. this log is accessible through maintenance web pages.

p.s. could you please contact me at dwalin [at] dwalin [dot] ru? i have a private question for you, if you don't mind.