Cisco Default ARP timeout is 240 minutes. If you are re-arping the network at 5 minute intervals, this can cause (and has caused) the exact issue you are experiencing.
We had this exact problem 2½ years ago and it doggon near drove us nuts. Problems began about a week or so after upgrading from 8.0 to 8.2. The problem only seemed to manifestitself during the "busy hours" of the day, i.e., starting around 9AM until around 10:30 then again in the afternoon, starting around 1:30 PM until around 3 PM.
It was not affecting the entire network, but only certain floors of the (9-story, 550,000 s.f.) building. Our management was on the verge of demanding that we roll-back to the prior software load, even though by then we had been fighting the problem for a full and continuous 3 weeks.
Management, not understanding how VOIP phones work, swore it had to be a phone issue. Network professionals and Cisco TAC & our local CCIE/Voice swore it was a network problem.
We had 6 laptops set up in various wiring closets capturing packet traces. We could actually see packets drop when the problem occurred.
During those periods when the problem was manifesting itself, users were also complaining of long bursts of choppy audio and/or lots of distortion. Frequently if the caller would stay on the line the call quality would sometimes clear up.
On internal station-to-station calls, one onstrument would display "Fail" while the other instrument would display "UCM Down, Features disabled".
Overall it was literally a huge cluster-f*** of people pointing fingers and the higher-ups demanding immediate resolution.
To insulate our senior executives from the problem we temporarily connected a dedicated fiber between their wiring closet and the local segment where the CUCM was connected. That put their fire out, but didn't solve it for the rest of the building, although doing this did provide a lot of credence to the original diagnosis, that it was a network problem and not a phone system problem.
The issue was finally escalated to P1 and Cisco BACKBONE engineers were engaged. Both Cisco and the local VAR had people on site every day and late into the evening trying to identify the root cause. Finally TAC noticed that all of our ARP timers were set for 5 minutes. The recommended setting them back to default values (240 minutes) and suddenly "the problem" was gone.
As an epilog to this, we do not believe that the ARP timer setting was the actual root cause, but more likely the "trigger" mechanism. At the time our CORE consisted of a pair of 6500's running ancient IOS that had not been rebooted in years (literally) because "the powers that be" would never allow a maintenance window. Today we not have a pair of 7K's in the CORE, but ARP timers still at default 240 minute settings. For what it's worth, we have not had a recurrence of this problem in more than 2 years.
If your ARP timers are set for 5 minutes, I would recommend starting there.
HTH
Original MUG/NAMU Charter Member
