I don't believe that limiting the memory available to the store will solve your problem, but it's worth a try. Something you should look at is circular logging. If it's enabled and you have a lot of e-mail traffic, this could cause a performance problem.
If circular logging is enabled (which it is by default), then exchange will only use 5 transaction log files (which is where messages reside until they are processed into the priv.edb or pub.edb. Since these files are never larger than 5MB, the system will become sluggish when there is a lot of mail to process. Another drawback to circular logging is that, in the event it is necessary to restore from backup, you can only restore to the time of the last backup - which means that if your backup was performed at 11:00 last night you will lose all messages that were received after that time.
Circular logging should be disabled, but with a caveat; ensure your using an exchange-aware backup program so that unneeded transaction log files (those that have been committed to the database) are deleted following the backup. Also, once circular logging has been disabled you should run the optimizer so that the log files are on a drive with plenty of space. When circular logging is disabled, the only limit to the number of log files is the available disk space. When space on the drive containing the log files gets below 10MB, the Information Store Service will stop - that's why you must have an exchange-aware backup program.
In case you're curious, another note: Even with circular logging disabled, there will always be at least five transaction log files. That's because Exchange wants to see the last five log files during startup, even though they will have been committed to the database during shutdown. Also, if it seems to take your server a long time to shut down, it's because the IS service will not stop until after all the transaction log files have been committed. I've seen this take 20 minutes on some servers.
Good luck,
Bob
p.s. Sorry if this seems like a disertation.