I have a feeling that JeSTeROCK might agree with me here - I feel the best software level for VoIP is version 3. Our most complex site which works perfectly is on 3.0(44) simply because we've not had a chance to roll out 3.0(59).
My expriences have shown that 44 and 59 are the best builds.
2.1(44) and 3.0(44) are similar in terms of bug fixes and things like that, but version 3 has new features which version 2 doesn't. The other good thing about version 3 is that it allowed for the new 56xx series of IP phones, which work really well.
My remote offices and home users use Draytek 2600 routers for their ADSL VPN link, which allow me to give priority to voice traffic over data, which is great when remote workers are using web applications or downloads. Fair enough, I can't extend the QoS over the links, but it works well. You could also do the same with the switch on the back of the 5610 and 5620.
Your codec is also important. You have to play about until you find the one to suit. Usually, it's G729(a) but occassionally if your link allows it, you can run G711 for perfect voice without any problems.
Have a play with things like Silent Surpression, Out of Band DTMF, and H450 support to further improve things.
What IP Phones and which IP Office platform are you using?
You seem to know a fair bit about the networking side of things which is good, so perhaps we can look are your IP Office hardware and some of your settings in a little more detail.