Blackberry msgs slow or non existent

[stevenriz]

Hi, we have exchange 03 and BES 3.6.7.12 and all of a sudden our ATT blackberries started receiving messages really slowly or not at all. They also have trouble sending messages and lookups are now failing. Everything looks fine in the event viewers of both the Exchange and the BB server. This morning I restarted both the Exchange and the BB servers and by mid day everything seems back to slow again. I might suspect the Exchange server sluggish but I look at Task Manager and it is 99% idle although I see lots of disk activity.

we have 21 BB users and about 75 users overall on the Exchange server.

could this possibly be an ATT issue? I don't know what else to check...

 

[58sniper]

Well, that's one of that pitfalls of BES - it's overly harsh on Exchange storage (sometimes up to a 1200% increase over Windows Mobile/iPhone methods).

You haven't provided enough info. Look at the normal perfmon counters on both servers. Look at bandwidth utilization at the firewall.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[stevenriz]

thanks Pat, for the Exchange server, the default perfmon counters show the Avg Disk Queue Length to be pegged 75% of the time with the graph set at 100. On the BES server I see very low numbers on the same default graph. I'll start monitoring the external firewall port but I don't suspect that to be an issue...

 

[stevenriz]

Update:

I increased the graph on the Exchange Server perfmon to 15000 and every once in a whle Avg Disk Queue Length peaks.

I have the following being monitored and peak values with a 15000 graph...
- Avg Disk Queue Length - 15000+
- Disk Reads/sec - low
- Disk Writes/sec - 500
- Current Disk Queue Length - 2500
- Avg Disk sec/Read - 3000
- Avg Disk sec/Write - 5000

The CPUs are at 5% and it is a dual core dual CPU xeon Dell PowerEdge 1950... 8GB ram and if memory service, Exchange only uses 4GB... ?? right??

 

[stevenriz]

Update: 2

since this is sort of an "all of a sudden" scenerio I started poking around on the resident Virus Scanner for Exchange... here is what I found... I ran two reports, see below... Incoming messages has almost doubled in the past week. Could this possibly create the performance issues we are experiencing? We use Trend Micro Scan Mail for Exchange and it is running on the Exchange Server....

Report: 1
Date Range: 27-Jan-10 to 03-Feb-10
Total Messages Scanned: 65007
Total Attachments Scanned: 12501
Total Spam Detected: 17915

Report: 2
Date Range: 04-Feb-10 to 10-Feb-10
Total Messages Scanned: 112210
Total Attachments Scanned: 22556
Total Spam Detected: 18889

 

[58sniper]

Those storage permon counters are pointing to the problem. Your disk subsystem is taxed.

Are those the counters for the database volume, the log volume, or the OS volume?

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[stevenriz]

ya I see that as well. ugh.

We have everything, the OS, DB and logs are on the same volume. It has never been an issue before...

I wonder what is really going on. When Exchange is running, is it always hitting the disks? These reports I listed with the perfmon are from this morning with basically 15 users online right now. You would think it would be pretty idle.

I don't know why all of a sudden but we will address it one way or the other. We have an EqualLogic array waiting in the wings that I may move the DB over to next week but I don't want to move an IO problem from the local server to the iSCSI array...

 

[58sniper]

We have everything, the OS, DB and logs are on the same volume.

Well, that's certainly not recommended.

Configure recipient filtering and tarpitting in Exchange. What do the Exchange queues look like? Full of messages? Lots of connections in progress from outside?

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[stevenriz]

ok I will take a look. I attached a screen shot of process explorer. How do these numbers look?

 

[stevenriz]

I guess we can;t just attach a file here so nevermind. The process explorer shows IO Reads, Writes and Other as follows...

wmiprvse.exe 1,063,501 | 1,059,832 | 65,760
dsm_sa_datamgr32.exe 686,246 | 9,761 | 314,701,098
store.exe 9,249,517 | 5,916,184 | 8,437,911

those are the high ones...

I should add that it looks like recipient filtering is enabled. Also all of our email is filtered and cleansed "before" it reaches us at a third party company called Postini. I believe Google bought them but that's beside the point...

 

[58sniper]

Okay - look in the connections and make sure you only see occassional connections from Postini. If you're seeing connections from other outside sources, then you should lock the firewall to only allow SMTP from Postini servers.

Store.exe being high is normal.

Run Exchange Best Practices Analyzer and see what it says.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[stevenriz]

yes we only allow connections from postini. good suggetion. Also we have 175 queues. It doesn't seem much different then normal. most of them only have a message or two in them.

One thing is right now we are having a company meeting and there are no users on the system but me and the server is still getting hit with something. You'd think it would be idle in a normal condition, right?

I don't know much about the best practices analyzer so I will read up on that as well. Does it produce suggestions to a current condition??

thanks Pat and all for the attention here....

Steve

 

[58sniper]

It will look for config issues that might be causing problems.


Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[stevenriz]

thanks Pat...

we've pretty much verified we aren't getting hit with any external source because the network is basically idle too. do you concur? Plus I just checked the Postini config and we are configured to have only 7 connections at one time.

we have a consultant I've used in the past calling me in a few hours to help check it out. He is certified in many MS packages, Exchange, Win Server etc...

 

[stevenriz]

Since this server also serves up print jobs for a few printers, I am going to move those print queues to another server asap. After that we'll decide what to do. THe plan is to move to Exchange 07 and eventually VM it...

 

[58sniper]

You should think about going right to Exchange 2010. 2007 is already 3 years old, and not the current version.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[stevenriz]

I know but we already own it and cancelled our maintenance to save money. I will price it out or see what maintenance costs will be...

 

[stevenriz]

I have a 1.5TB volume created on our EqualLogic iSCSI array dedicated for Exchange. Do you suggest or go along with idea of moving the data store to the EL system? It is a PS5000 with 12TB (16 disks) usable... Any thoughts on that?

Currently the priv.db is about 120GB in size....

 

[stevenriz]

Just a thought. We believe exchange maintenance might be interfering with the backup etc which might cause a performance drop. Can this be possible and if so, can I check something to verify this? I see Exchange is configured for 12A to 4A for maintenance... In the event log I see a lot of "Online Defragmentation" events. The PUB always completes, the PRIV always says Interrupted and terminated and will resume at the point of interruption the next time it is started on this database... this happens a few times a day....

 

[58sniper]

Hold the backups one day, or alter the schedule and see if it changes. You can also do a log comparison to see when the Exchange backup actually started, and see if that's when the maintenance terminated.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/