Go ahead... I challenge anyone out there to help Microsoft out to explain this intriguing long standing issue. Is it possible that a lot of issues of Black Screen/No Response and other "hung server" problems could all be wrapped around this driver (TERMDD.SYS)?
Excerpt from MS engineer response:
"From a technical perspective, investigating why CSRSS shows high I/O doesn’t provide any outcome worth the cost of the investigation. Determining the purpose of that I/O is another story entirely and likely impossible for a number of reasons, not the least of which is this is Server 2003. It is also a leap to label that amount of I/O as “unnecessary.” It is higher than the I/O observed after the application of the hotfix, but that doesn’t by itself indicate that this amount of I/O is bad, broken or unnecessary. It is the amount of I/O it was designed to have or that it requires in a pre-hotfix environment. Improvements in the TERMDD driver apparently optimizes its interaction with CSRSS (and perhaps other binaries) not as an intended consequence, but simply the result of an overall improvement in TERMDD’s design. The best I could probably do is verify what you have already documented and observed … higher I/O in CSRSS before this hotfix is applied than after."
Now... Go ahead... check a Windows 2003 server.
Prior to installing one of the noted hotfix(s) below. You'll find that the I/O numbers are RIDICULOUS. You won't find any of this documented anywhere in Microsoft land for Windows 2003 32bit.
Gee, some would say lighten up. At least there is This hotfix is one file... (TERMDD.SYS). But, the KB article doesn't state anything about the Csrss.exe process or other processes with elevated I/O. Not one mention of the drastic increasing level or “display” of I/O Other Bytes, when just making a simple RDP connection and doing "normal SA duties". But, the before and after effects on all the processes, across all the types of I/O is amazing!!
So, after opening a case with MS. And, a few email exchanges with screenshots. Really, not get anywhere. Other than a blow off, not worth any investigation, etc. Oh, and "... it is 2003". Surprised? Most IT people would say... Typical MS!! Upgrade to 2008.
We will get there eventually, someday. But, dare I say, the majority of Windows servers in the corporate world is 2003?
Another excerpt from the assigned MS engineer:
"Because the reduction in I/O is probably an unexpected or unplanned byproduct of this hotfix, it is entirely possible that we are unaware of this specific situation. We are probably unaware of this change because it ends up being irrelevant. Nobody I have spoken to has heard about this nor had to troubleshoot it. Unless you have a scenario that indicates that high I/O in CSRSS causes something to malfunction, it would likely never come to anyone’s attention. Even now, I haven’t heard you indicate something is broken or not functioning correctly, only that you see higher I/O before you apply the hotfix."
IS THIS THE REASON that there are nightly or weekly reboots of a Citrix environment is common? Or, maybe causing a heavier impact on an already "troubled" server, renders it inaccessible?
Other investigation lead me to find on a different forum pointing to: This hotfix is for 2003 64bit. WOW... the same driver!!??? TERMDD.SYS (this file in this hotfix, of course, is older (2007) a different version then the noted newer driver (2009) in the KB956438)
Neither KB article speaks of each other. Confusing? Since this KB935987 is also changing the "view" or the "displayed" values. Are these numbers relevant? Maybe only to ease the eyes of an alarmist that might be a user or an Admin?? But, what prey tell is really going on with this I/O activity and, the impact that appears to be "unnecessary"?
Correct me if I'm wrong. But, the hotfix sure does appear to have resolved a few things. Not just from a "display" point of view.
Both the KB articles aren't very forthcoming. Personally, I feel that this could be much bigger than just the "simple quick look at the running processes across the OS". It isn't just high I/O Other bytes per Second on running processes.
Go look for yourself on your 2003 servers... Enjoy!!! And, then, in my humble opinion, deploy the KB956438 at your earliest opportunity.
Excerpt from MS engineer response:
"From a technical perspective, investigating why CSRSS shows high I/O doesn’t provide any outcome worth the cost of the investigation. Determining the purpose of that I/O is another story entirely and likely impossible for a number of reasons, not the least of which is this is Server 2003. It is also a leap to label that amount of I/O as “unnecessary.” It is higher than the I/O observed after the application of the hotfix, but that doesn’t by itself indicate that this amount of I/O is bad, broken or unnecessary. It is the amount of I/O it was designed to have or that it requires in a pre-hotfix environment. Improvements in the TERMDD driver apparently optimizes its interaction with CSRSS (and perhaps other binaries) not as an intended consequence, but simply the result of an overall improvement in TERMDD’s design. The best I could probably do is verify what you have already documented and observed … higher I/O in CSRSS before this hotfix is applied than after."
Now... Go ahead... check a Windows 2003 server.
Prior to installing one of the noted hotfix(s) below. You'll find that the I/O numbers are RIDICULOUS. You won't find any of this documented anywhere in Microsoft land for Windows 2003 32bit.
Gee, some would say lighten up. At least there is This hotfix is one file... (TERMDD.SYS). But, the KB article doesn't state anything about the Csrss.exe process or other processes with elevated I/O. Not one mention of the drastic increasing level or “display” of I/O Other Bytes, when just making a simple RDP connection and doing "normal SA duties". But, the before and after effects on all the processes, across all the types of I/O is amazing!!
So, after opening a case with MS. And, a few email exchanges with screenshots. Really, not get anywhere. Other than a blow off, not worth any investigation, etc. Oh, and "... it is 2003". Surprised? Most IT people would say... Typical MS!! Upgrade to 2008.
We will get there eventually, someday. But, dare I say, the majority of Windows servers in the corporate world is 2003?
Another excerpt from the assigned MS engineer:
"Because the reduction in I/O is probably an unexpected or unplanned byproduct of this hotfix, it is entirely possible that we are unaware of this specific situation. We are probably unaware of this change because it ends up being irrelevant. Nobody I have spoken to has heard about this nor had to troubleshoot it. Unless you have a scenario that indicates that high I/O in CSRSS causes something to malfunction, it would likely never come to anyone’s attention. Even now, I haven’t heard you indicate something is broken or not functioning correctly, only that you see higher I/O before you apply the hotfix."
IS THIS THE REASON that there are nightly or weekly reboots of a Citrix environment is common? Or, maybe causing a heavier impact on an already "troubled" server, renders it inaccessible?
Other investigation lead me to find on a different forum pointing to: This hotfix is for 2003 64bit. WOW... the same driver!!??? TERMDD.SYS (this file in this hotfix, of course, is older (2007) a different version then the noted newer driver (2009) in the KB956438)
Neither KB article speaks of each other. Confusing? Since this KB935987 is also changing the "view" or the "displayed" values. Are these numbers relevant? Maybe only to ease the eyes of an alarmist that might be a user or an Admin?? But, what prey tell is really going on with this I/O activity and, the impact that appears to be "unnecessary"?
Correct me if I'm wrong. But, the hotfix sure does appear to have resolved a few things. Not just from a "display" point of view.
Both the KB articles aren't very forthcoming. Personally, I feel that this could be much bigger than just the "simple quick look at the running processes across the OS". It isn't just high I/O Other bytes per Second on running processes.
Go look for yourself on your 2003 servers... Enjoy!!! And, then, in my humble opinion, deploy the KB956438 at your earliest opportunity.
