VoIP over a regular internet VPN is possible, but it helps if you over-provision the network to help ensure better performance. (we have a minimum site bandwidth of T1-speeds, and only have 2 voice tie-lines to those sites).
In our experience, turning on bandwidth-saving options like silence suppression and higher compression on voice calls improved the situation, as inially we had issues like yours. Also, on our VPN routers between sites, we setup a QoS policy to prioritize (outbound at each site) the NBX voice traffic over other VPN traffic (which is already QoS tagged by the NBX as 'ef'). Note that we still don't yet have any QoS overall on our WAN environment, so VPN traffic (with the tagged NBX traffic) is still competing 1:1 with general internet access and browsing by users... Even with this setup, calls are 'acceptable' almost all the time now.
Our inter-site latency average is between 50ms and 90ms
In addition, 3COM's KB recommends:
Latency: Round trip delay should be <100 ms and not fluctuate to ensure toll voice quality.
Phones and remote devices such as Line Cards in a remote chassis, per port:
- 87 kilobits/sec sustained throughput per conversation, normal mode
- 55 kilobits/sec minimum requirement for low bandwidth and silence suppression options enabled
H.323
- 19.2 kilobytes/sec - 87 kilobits/sec, depending on compression codec used