Network issue! And, just because you're running 6.4.0.14 doesn't mean you can't downgrade. You can't downgrade if your set's comcode/Avaya part # is the more recent release of 9611. So, if 700504845 or 700501429 isn't the # on the sticker on the bottom of the phone, you ought to be OK.
Live, if you trace the phone (and can reliably reproduce the problem) CM will print jitter and packet loss every 10 seconds if that call is using an Avaya DSP on a medpro or gateway. You can perhaps disable direct IP audio on the station form to force it to use a DSP to let you get those stats.
Further, depending on how QoS is setup and if you're in a WAN scenario, it can be beneficial to tie down audio between those sites to DSPs such that they're in charge of getting the audio between the 2 endpoints.
Example: You've got voice vlans for phones at each site, a core vlan for servers and gateways at each site, and QoS is setup to only prioritize traffic from either site if coming from the core VLANs. Disabling shuffling/direct IP between those network regions would make phones at either end of a WAN send audio to DSPs in their local core VLANs rather than directly to one another. QoS in the network might treat that differently.