CLANs Going Out of Service, Congestion Test Fails

[CChambleeJr]

I have an unusual issue going on, it's been going on for quite some time now, but has recently increased in frequency. According to STATUS CLAN-ALL, we are seeing a lot of our CLAN boards going out of service periodically, sometimes generating server alarms. While there are CLAN boards in the OOS state, if I look at the messages log file on the server I see one or more extensions hammering away at it's registration CLAN, apparently trying to recover its registration. If you status the extension, it shows "In-Service/Idle", but it's connection state is "On Demand". The phone cannot make or receive calls, however. Busy/Release of the phone causes the phone to disconnect, but the problem returns a few seconds later. The only way we've been able to clear the symptom is to take over the extension by logging it in on another device. This stops the recovery cycling in the log, and puts the CLAN back in service. If you test the CLAN while it is showing OOS, it fails test 600 (congestion). See a sample of the registration messages from the log file...

Apr 5 08:03:20 avaya_cm_001 logmanager: IPEVT IPT_REG board=34B02 ip=192.168.1.160 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49306 net_reg= 14 reason=recovery
Apr 5 08:03:20 avaya_cm_001 logmanager: IPEVT IPT_REG board=30D01 ip=192.168.4.135 net_reg= 5 ext= 4509178 ip= 192.26.161.75;49307 net_reg= 14 reason=recovery
Apr 5 08:03:22 avaya_cm_001 logmanager: IPEVT IPT_REG board=04B02 ip=192.168.1.140 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49307 net_reg= 14 reason=recovery
Apr 5 08:03:23 avaya_cm_001 logmanager: IPEVT IPT_REG board=PROCR ip=192.168.1.181 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49308 net_reg= 14 reason=recovery
Apr 5 08:03:24 avaya_cm_001 logmanager: IPEVT IPT_REG board=36B02 ip=192.168.1.164 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49308 net_reg= 14 reason=recovery
Apr 5 08:03:25 avaya_cm_001 logmanager: IPEVT IPT_REG board=34A02 ip=192.168.1.159 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49309 net_reg= 14 reason=recovery
Apr 5 08:03:26 avaya_cm_001 logmanager: IPEVT IPT_REG board=35A02 ip=192.168.1.161 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49309 net_reg= 14 reason=recovery
Apr 5 08:03:27 avaya_cm_001 logmanager: IPEVT IPT_REG board=34B02 ip=192.168.1.160 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49300 net_reg= 14 reason=recovery
Apr 5 08:03:28 avaya_cm_001 logmanager: IPEVT IPT_REG board=36A02 ip=192.168.1.163 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49300 net_reg= 14 reason=recovery
Apr 5 08:03:29 avaya_cm_001 logmanager: IPEVT IPT_REG board=04B02 ip=192.168.1.140 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49301 net_reg= 14 reason=recovery
Apr 5 08:03:31 avaya_cm_001 logmanager: IPEVT IPT_REG board=35B02 ip=192.168.1.162 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49301 net_reg= 14 reason=recovery
Apr 5 08:03:31 avaya_cm_001 logmanager: IPEVT IPT_REG board=21C01 ip=192.168.2.136 net_reg= 4 ext= 4509178 ip= 192.26.161.75;49302 net_reg= 14 reason=recovery
Apr 5 08:03:33 avaya_cm_001 logmanager: IPEVT IPT_REG board=01B02 ip=192.168.1.173 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49302 net_reg= 14 reason=recovery
Apr 5 08:03:34 avaya_cm_001 logmanager: IPEVT IPT_REG board=24C01 ip=192.168.2.141 net_reg= 4 ext= 4509178 ip= 192.26.161.75;49303 net_reg= 14 reason=recovery
Apr 5 08:03:36 avaya_cm_001 logmanager: IPEVT IPT_REG board=02B02 ip=192.168.1.136 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49303 net_reg= 14 reason=recovery
Apr 5 08:03:36 avaya_cm_001 logmanager: IPEVT IPT_REG board=23A02 ip=192.168.2.139 net_reg= 4 ext= 4509178 ip= 192.26.161.75;49304 net_reg= 14 reason=recovery
Apr 5 08:03:37 avaya_cm_001 logmanager: IPEVT IPT_REG board=06A02 ip=192.168.1.143 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49304 net_reg= 14 reason=recovery
Apr 5 08:03:38 avaya_cm_001 logmanager: IPEVT IPT_REG board=23C01 ip=192.168.2.140 net_reg= 4 ext= 4509178 ip= 192.26.161.75;49305 net_reg= 14 reason=recovery
Apr 5 08:03:39 avaya_cm_001 logmanager: IPEVT IPT_REG board=06B02 ip=192.168.1.144 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49305 net_reg= 14 reason=recovery
Apr 5 08:03:40 avaya_cm_001 logmanager: IPEVT IPT_REG board=PROCR ip=192.168.1.181 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49306 net_reg= 14 reason=recovery
Apr 5 08:03:41 avaya_cm_001 logmanager: IPEVT IPT_REG board=01A02 ip=192.168.1.172 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49306 net_reg= 14 reason=recovery
Apr 5 08:03:42 avaya_cm_001 logmanager: IPEVT IPT_REG board=34A02 ip=192.168.1.159 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49307 net_reg= 14 reason=recovery
Apr 5 08:03:43 avaya_cm_001 logmanager: IPEVT IPT_REG board=04A02 ip=192.168.1.139 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49307 net_reg= 14 reason=recovery
Apr 5 08:03:44 avaya_cm_001 logmanager: IPEVT IPT_REG board=34B02 ip=192.168.1.160 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49308 net_reg= 14 reason=recovery
Apr 5 08:03:45 avaya_cm_001 logmanager: IPEVT IPT_REG board=08A02 ip=192.168.1.147 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49308 net_reg= 14 reason=recovery
Apr 5 08:03:46 avaya_cm_001 logmanager: IPEVT IPT_REG board=04B02 ip=192.168.1.140 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49309 net_reg= 14 reason=recovery
Apr 5 08:03:48 avaya_cm_001 logmanager: IPEVT IPT_REG board=02A02 ip=192.168.1.135 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49309 net_reg= 14 reason=recovery
Apr 5 08:03:49 avaya_cm_001 logmanager: IPEVT IPT_REG board=36B02 ip=192.168.1.164 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49300 net_reg= 14 reason=recovery
Apr 5 08:03:50 avaya_cm_001 logmanager: IPEVT IPT_REG board=03B02 ip=192.168.1.138 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49300 net_reg= 14 reason=recovery
Apr 5 08:03:51 avaya_cm_001 logmanager: IPEVT IPT_REG board=21C01 ip=192.168.2.136 net_reg= 4 ext= 4509178 ip= 192.26.161.75;49301 net_reg= 14 reason=recovery
Apr 5 08:03:52 avaya_cm_001 logmanager: IPEVT IPT_REG board=30D01 ip=192.168.4.135 net_reg= 5 ext= 3136957 ip= 192.25.92.41;49301 net_reg= 14 reason=recovery
Apr 5 08:03:53 avaya_cm_001 logmanager: IPEVT IPT_REG board=24C01 ip=192.168.2.141 net_reg= 4 ext= 4509178 ip= 192.26.161.75;49302 net_reg= 14 reason=recovery
Apr 5 08:03:53 avaya_cm_001 logmanager: IPEVT IPT_REG board=PROCR ip=192.168.1.181 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49302 net_reg= 14 reason=recovery
Apr 5 08:03:55 avaya_cm_001 logmanager: IPEVT IPT_REG board=34A02 ip=192.168.1.159 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49303 net_reg= 14 reason=recovery
Apr 5 08:03:56 avaya_cm_001 logmanager: IPEVT IPT_REG board=23A02 ip=192.168.2.139 net_reg= 4 ext= 4509178 ip= 192.26.161.75;49303 net_reg= 14 reason=recovery
Apr 5 08:03:57 avaya_cm_001 logmanager: IPEVT IPT_REG board=34B02 ip=192.168.1.160 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49304 net_reg= 14 reason=recovery
Apr 5 08:03:58 avaya_cm_001 logmanager: IPEVT IPT_REG board=30D01 ip=192.168.4.135 net_reg= 5 ext= 4509178 ip= 192.26.161.75;49304 net_reg= 14 reason=recovery
Apr 5 08:03:59 avaya_cm_001 logmanager: IPEVT IPT_REG board=04B02 ip=192.168.1.140 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49305 net_reg= 14 reason=recovery
Apr 5 08:04:00 avaya_cm_001 logmanager: IPEVT IPT_REG board=PROCR ip=192.168.1.181 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49305 net_reg= 14 reason=recovery
Apr 5 08:04:01 avaya_cm_001 logmanager: IPEVT IPT_REG board=36B02 ip=192.168.1.164 net_reg= 1 ext= 3136957 ip= 192.25.92.41;49306 net_reg= 14 reason=recovery
Apr 5 08:04:03 avaya_cm_001 logmanager: IPEVT IPT_REG board=34A02 ip=192.168.1.159 net_reg= 1 ext= 4509178 ip= 192.26.161.75;49306 net_reg= 14 reason=recovery
Apr 5 08:04:03 avaya_cm_001 logmanager: IPEVT IPT_REG board=21C01 ip=192.168.2.136 net_reg= 4 ext= 3136957 ip= 192.25.92.41;49307 net_reg= 14 reason=recovery

Our phones are OneX Deskphones, running version 6.3.0.37 firmware. CM is 6.0.1. Could this be a firmware issue with the OneX desk phones?

===
Chuck Chamblee
ACSS (CM) | CTP | A+ ce | Security+ ce

 

[joe2938]

Sounds like you have a flaky clan board, may need to replace

 

[wpetilli]

run command status socket and see how many connections your CLAN's have on them. You may need to add or replace if this board is flaky. Any reason why you aren't using the PE for the registrations?

 

[phoneguy55]

^this. move to pe. no reason to use clan's for anything anymore.

 

[CChambleeJr]

I'm not sure that this is a defective CLAN issue. I've narrowed the scope down to just one extension for the purpose of clarity...

Apr 6 09:15:05 cm_server_01 logmanager: IPEVT IPT_REG board=PROCR ip=192.168.1.181 net_reg= 1 ext= 9559057 ip= 192.168.92.9;49300 net_reg= 14 reason=recovery
Apr 6 09:15:07 cm_server_01 logmanager: IPEVT IPT_REG board=34B02 ip=192.168.1.160 net_reg= 1 ext= 9559057 ip= 192.168.92.9;49301 net_reg= 14 reason=recovery
Apr 6 09:15:09 cm_server_01 logmanager: IPEVT IPT_REG board=34A02 ip=192.168.1.159 net_reg= 1 ext= 9559057 ip= 192.168.92.9;49302 net_reg= 14 reason=recovery
Apr 6 09:15:11 cm_server_01 logmanager: IPEVT IPT_REG board=36B02 ip=192.168.1.164 net_reg= 1 ext= 9559057 ip= 192.168.92.9;49303 net_reg= 14 reason=recovery
Apr 6 09:15:13 cm_server_01 logmanager: IPEVT IPT_REG board=35A02 ip=192.168.1.161 net_reg= 1 ext= 9559057 ip= 192.168.92.9;49304 net_reg= 14 reason=recovery
Apr 6 09:15:15 cm_server_01 logmanager: IPEVT IPT_REG board=23A02 ip=192.168.2.139 net_reg= 4 ext= 9559057 ip= 192.168.92.9;49305 net_reg= 14 reason=recovery
Apr 6 09:15:17 cm_server_01 logmanager: IPEVT IPT_REG board=29C01 ip=192.168.4.154 net_reg= 5 ext= 9559057 ip= 192.168.92.9;49306 net_reg= 14 reason=recovery

This extension is apparently trying to register to several CLANs, and to PE, all unsuccessfully. If I take over this extension from my working desk phone by logging into it and thereby forcing it off of it's current device, the issue will clear-up. Wondering if this is a firmware issue? I hear your points on moving to PE for all registrations, but I can't convince my upper-command chain to allow us to reconfigure for that, since Avaya designed the system config originally.


===
Chuck Chamblee
ACSS (CM) | CTP | A+ ce | Security+ ce

 

[wpetilli]

Have you tried swapping the physical phone?

 

[CChambleeJr]

No, there's too many of them doing this. The above capture is just for one of the phones with the issue. The issue seems to affect hundreds of them, at different random intervals.

===
Chuck Chamblee
ACSS (CM) | CTP | A+ ce | Security+ ce

 

[wpetilli]

If you have others that can handle the load, can you disable that clan board and see if the problem follows?