Turn off VAD and change to G.729. VAD icreases processing time and latency.
Also, if your BCM is not near the edge of the network, you can have QoS issues if your LAN is not truly QoS enabled.
I know that this sounds counter-intuitive, but try setting your codec to G.711 for one user and test. It only uses ~175k per call, but adds no latency to processing where G.729 requires heavy processing and can add quite a bit of latency.
Finally, try changing the payload and jitter buffer sizes. I would recommend a smaller payload (especially in G.729)and no more than a medium jitter buffer.