New VMware installation and Exchange errors

[unspecialeffect]

Hello ,

I have recently purchased a Dell R710 as a ESXi server. I am hosting 2 Terminal Servers and a Exchange 2003 server on the R710. 22 Users run Outlook 2003 from RDP sessions. The servers were on 3 seperate Dell 2600 servers before without issues.

Now outlook is responding noticably slower and running a search query in a users inbox with more than 200 emails will result a popup on the lower right of the screen with a triangle and exclamation point stating "outlook is requesting data from the server" and either outlook will freeze or respond extremely slowly.

Any ideas where to start looking for a solution?

 

[SimonDavies]

What is the spec of the R710, what's the spec of the Exchange server? What kind of storage are you using (shared or local)? How many network connections in?

We need to know the spec not just of the esxi box but also the guest vm.



Simon

The real world is not about exam scores, it's about ability.

 

[unspecialeffect]

Thanks for the response.

The R710 is

24GB of RAM
2 - E5520 Xeon Quadcore processors @ 2.26 w/ HT
5 - 450GB SAS 15K drives in Raid 5
4 Gb NICs available on board

The Exchange server is running Windows 2003 Server R2, Set to utilize 4 virtual processors, and 4 GB of ram, storage is local. Mailbox store is 24GB. Average 16 Network connections in (per cacti monitoring)

Each terminal server is running windows 2003 R2. 10 - 14 users on each box. 4 virtual processors , 4 GB of Ram. 100 GB virtual drive allocated to each. Retrieving mail with Outlook 2003 SP3 in each session.

I've noticed recently if I use outlook on a local PC (off the VM) it functions fine.

 

[baddos]

Probably a lot of disk contention. You really should have the information store on it's own raid volume.

 

[SimonDavies]

I agree with baddos, you really need to have dedicated storage for your Exchange environment.

Are your users also storing their Exchange data within the TS environment? If so then the disk usage will obviously increase and as you're sharing the same disks with the Exchange server as well that's where your "stuttering" will be coming from.

You will either need to offload the Outlook usage to local machines or add additional storage and move the Exchange server to that (usually you will have dedicated storage for high intensity servers such as Messaging and DB servers rather than sharing the disks across the board with your Print and File servers).


Simon

The real world is not about exam scores, it's about ability.

 

[baddos]

What version of office are you using on your terminal servers?

 

[unspecialeffect]

Office 2003 SP3

 

[baddos]

Make sure outlook is not using "Cached Mode". I know 2007 detects terminal server and disables this by default, but I'm not sure about 2003.

 

[unspecialeffect]

Yes, Cached mode is greyed out. So not even possible to select it.

 

[unspecialeffect]

I overlooked Simon's response up there, sorry. We are planning to do some sort of NAS or SAN type storage when we virtualize the other 6 servers in our enviroment , but just getting our feet wet right now. I guess we'll have to think about this sooner than later.

I'm looking at the performance counters in Vsphere under "disk". What should I be looking for that would suggest the disk is overused ? Or should I be running some performance counters on the Exchange VM by itself?

Thanks for your help with this guys.

 

[baddos]

On the host are you seeing lots of reads and writes? Running Exchange on the same array as a couple of other servers and on Raid5 is going to limit it's performance for sure.

When you get your new NAS or SAN make sure exchange has at least 1 raid10 array for itself.

 

[SimonDavies]

Also ensure that if you go down the NAS route that the device has iSCSI as a protocol it understands, not all devices do (I have two Terastation Pro 2's that don't).

If you're planning on building your own (which is not a bad thing to do) you could also have a look at either Openfiler or Freenas (

http://www.openfiler.com

or

http://www.freenas.org),

out of the two Openfiler is the more widely used on in the business arena.

As far as disk usage etc, I would have a look at something like Iometer to give you a better disk IO understanding (

http://www.iometer.org/)

Simon

The real world is not about exam scores, it's about ability.

 

[Hboogie]

The counters should be ran from inside the Exchange VM and the same perfmon counters applie ( disk queue length, etc)

Read this:

http://www.vmware.com/pdf/Virtualizing_Exchange2003.pdf

The problem almost certainly is related to the storage system. the OS/IS/logs are sitting on the same RAID5 array with a terminal server that is getting tons of R/W's to disk.

 

[cabraun]

I certainly agree that disk I/O is a likely candidate for a good part of the problem here. I however would propose that you have another problem as well.

Best practice would dictate that you assign a single virtual CPU to each VM and then add as needed, rather than to just out right assign 4 virtual CPUs to each VM. In the scenario the OP has posted, it is very likely that there is CPU contention.

Remember that in the virtual world that as many vCPUs are assigned to a VM, that many CPUs (cores) must be available for any command to be processed. Hence if a single thread is waiting to the processed on your exchange server, it must wait for 4 cores to be available to process that single thread and if you also have 2 terminal servers on the same host that also need 4 cores to process each single thread, than I bet your %Ready numbers are through the roof.

I would start by reducing your Virtual CPUs to 1 per VM (don't forget to update your HAL from multi to uniprocessor) and only increase that number if you can absolutely track a problem down to the need for more CPU.

 

[Hboogie]

Excellent point cabraun.

I do remember reading somewhere assigning anything more than 2 processors to exchange is futile, but don't have empirical data in front of me.

By changing the processor count, wouldn't this bluescreen the VM, like any other windows box ?

 

[cabraun]

It may result in a BSoD which is why you need to update your HAL from multi to uniprocessor. My experience has been that VMs tend to be more forgiving with regards to CPU additions and removals, than servers in the physical world, to allow you to make the HAL change after changing the number of vCPUs.

 

[unspecialeffect]

Cabraun , you hit it on the head. Changed the VM to a single processor, the errors are gone and exchange is running much better with the limited resources.

 

[Arisap]

It is really important to understand processor usage in a virtual environment. I have the "multiple vs single proc" argument all the time. Here's how this works. Your server has 8 processors. You have 3 VM's each with 4vCPU's. When a VM with 4vCPU's goes to execute a command (any command at all; even if the instruction set does not require SMP), the hypervisor has to locate 4 idle physical CPU's in order to run that instruction. In this scenario, one VM uses half the total processor count in the box; the second VM uses the other half; and the third must wait. As others have noted, always start with 1vCPU and go up from there. Windows 2003 SP2 and up all support changing the number of CPU's without causing BSoD (it just requires an extra reboot on Win2k3)

 

[said07]

I agree with Hboogie. After I virtualized my Front End box (It had dual processors, 4 gig of ram)it was so slow that I almost couldn't do anything on it.. I called Vmware performance team. we changed the processor count to one instead of two. That did it. Now it's working great.
I hope this helps.