NW6 works O.K. in a mixed environment, at least in out case. There is a problem mixing different versions of DS. This can be overcome (mostly) by applying the latest versions of the appropriate DS. We have been told that there can be problems if youi are running NDS 8 in NW 5.1. We are seeing a number of minor, but annoying, issues with DS, such as stuck obituaries, intermittantly degraded communication, and some sync issues. Regular use of DSRepair keeps this to a minimum.
Novell has some articles on their support page about this.
If you have the time, I would suggest upgrading the 4.11 boxes to either 5 or 6 quickly once you bring 6 into the tree.
You need to pay special attention to your replication. You can't have all 3 DS versions in the same ring. Start by working on the ROOT ring. What we did was bring a 6 box into a dummy tree, extend the remote schema to our production tree, then remove the Dummy ds from that server and install the server into the production tree. We then made it a R/W/ replica of root, and allowed DS to synchronize for a week. We then promoted it to master of ROOT. The other replicas of ROOT are all 5.1 boxes with DS 7.x
Don't do this without a lot of research and planning. It should not be undertaken lightly. Write out your replica rings so you know who has what replica, then make your plan from there. If you have the budget, think about bringing in Novell Consulting. This will be costly but might save a lot of heartache.