Edb/stm drive full >:( 3

[cakestick]

Hello,

I've inherited an IT setup with an MS Exchange 2000 server, and came into a full hard drive where the EDB/STM files are stored. Apparently, there must have been some processes I should have been running because my priv edb is 31gb while the stm is 15gb. The hard drive has about 180k remaining and Exchange obviously will not start.

I have begun a backup of the system state and information store to see if this makes a difference, but I was assuming that the backup only removed the transaction logs that reside on another drive. This one has only the EDB/STM files for the mailbox stores and public folders, nothing more.. is there anything I can do aside from installing a new drive/moving the database? If that's the case, what will I have to do to ensure consistency?

Thanks in advance, any help is helpful!

 

[cakestick]

Update: Cancelled the backup since one was run on Friday afternoon, began offline defragmentation after removing unused mailboxes to free up ~1gb of space. Once the defrag is done, I am going to have users purge as much deleted items/junk mail as possible, and am instating a policy of permanently deleting items upon Outlook shutdown.

If anyone can provide any insight regardless, it's appreciated.

 

[cakestick]

Final update: Cancelled defrag, moved public folder store to another drive. Brought both stores back up, and Mailbox store began increasing in size at a decreasing rate. I will be monitoring the filesize increase, and running the offline defrag after hours.

Good to have this forum around anyways, just in case!

 

[lb63640]

You could also limit the storage per user mailbox.

 

[cakestick]

Once my users have deleted a large amount of their mail that is no longer necessary, and the defrag goes through, a storage limit policy is going into effect.

Thanks though!

 

[xmsre]

Whether you implement limits or not, THERE ARE LIMITS. If you choose not to implement mailbox limits, then you risk bumping into the ultimate limit; available hard drive space. That's what happend in your case.

Space isn't the only consideration when sizing storage for Exchange; you also have to consider performance. While available space on hard drives has increased 3,000,000%, in the past 20 years, performance has only improved 4,000% or so. Generally, you run into performance issues before you run out space. Disk performance issues translate into latency. Latency is user visible in the form of the dreaded "requesting Data" dialog box.

Try reading the MS paper "Optimizing Storage for Exchange Server 2003". The backgound on sizing and the methodology applies equally to Exchange 2000. You'll note that the paper says you cannot accurately size storage for exchange without limits.

In addition to size limits for mailboxes, you need to think about folder item counts as well. As the number of items in a folder increases, the size of the indexes for the 11 default views for the folder increases. Especially in Exchange 2000/2003, where there's only about 250K of database cache/user, the indexes get to a point where they can't effectively be cahced. That's when you see significant performance degradation; when the index has to rebuilt each time you view the folder. MS recommends no more than 5000 items per folder.

Aside from email server space and performance, there are business liability issues to assess as well. Is your industry regulated? Are there specific records retention requirements as relates to email? If there are, or even if there are not, does your business want to carry the added liability of retaining records that are not required and are available for discovery by anyone that wants to sue your company? When you work with managment to formulate an email retention policy, you should involve your legal department. Many times you'll find that they will champion the cause.

 

[cakestick]

xmsre,

Awesome post - I knew this setup was going to be a problem when my former manager was telling me that 60gb mirrored drives without mailbox size limits was going to be sufficient. Of course, he took off before the e-mail could hit the fan.

Your post is very valuable though for pointing me to the optimizing storage paper, I really appreciate it.

 

[xmsre]

Sadly, it's a common problem.

Just out of curiosity, mirrored what?

For Exchange, according to "Optimizing Storage for Exchange Server 2003", if you see average write latency abovc 20ms or spikes in latency above 50ms lasting more than a few seconds then you have a poorly performing disk subsystem. From a performance perspective then, you want your average latency below 20ms.

When you talk about drives and IOPS/spindle, the number is meaningless unless it is associated with a specific latency. The more IOPS you throw at a spindle, the higher the latency. For a 20ms response time,

A 10K RPM SCSI drive gets about 90 IOPS/spindle
A 15K RPM SCSI drive gets about 135 IOPS/spindle
A 7200 RPM SATA drive gets about 40 IOPS/spindle

When you combine spindles into RAID sets, the write capacity is not the same as the read capacity. This is because for a given RAID type writes require more operations. This is often expressed as a write penalty. It's important to know the read/write ratio when determining the expected performance of an array.

Where P is the performance in IOPS of a single spindle and N is the number of spindles in the array:

For RAID 1 or 10 or 0+1;

Write performance = P*N/2 (a write penalty of 2)
Read performance = P*N

For RAID 5;

Write performance = P*(N-1)/4 (a write penalty of 4)
Read performance = P*(N-1)


In a modern Exchange environment with Outlook clients in cached mode, we can expect to see read/write ratios on the order of 3:1 or 2:1. For Exchange 2007, it's closer to 1:1 after the host cache is populated. Applying a read/write ratio of 2:1 and assuming 10K SCSI drives, let's comapre a mirror (RAID 1) to RAID 5.

First the mirror:

write performance = 90*2/2 = 90 IOPS
read performance = 90*2 = 180 IOPS

applying the read write ratio, our composite performance is (180 + 180 + 90)/3 = 150 IOPS

Now, a RAID 5 set:

write performance = 90*(3-1)/4 = 45 IOPS
read performance = 90 * (3-1) = 180 IOPS

applying the read/write ratio our composite performance is (180 + 180 + 45)/3 =135 IOPS

In order to meet or exceed the performance of a mirror, you would need 4, or twice as many spindles, in RAID 5. This is probably why "Optimizing Storage for Exchange Server 2003" says "In general, Raid-5 does not provide the best trade-off between reliability/availability and performance."

As for layout, the ideal situation would be seperate physical disk (or arrays) for the OS, the Logs, and the databases. For many small organizations, a mirror for the OS, a mirror for the logs, and a mirror for the database. You didn't state what backup progam you are using, but a common issue with netbackup occurs when you combine a system state and and Exchange backup in the same job when the logs are on the same disk as the OS; it'll fail. You have to seperate the jobs. MS heavily stresses placing the logs on physically seperate spindles. If I had no other choice than to reduce spindle count, I'd rather go unmirrored on the OS and logs, than combine them. If you go this route you do lose some fault tolerance but maintain the sepration of the logs with a reduced spindle count (4). Whatever you do, don't create one big RAID 5 and carve logicals out of it; you'll regret it in the end.