Hardware Interrupts

[blister911]

We have 12 blade servers all configured almost identically and experiencing the same problem.

Server Configuration:
Windows Standard 2003
Quad processors
4GB Ram
Mirrored hard drives (some have 36GB drives and some have 72GB drives)

On occasion they will have high CPU utilization on one of the processors due to hardware interrupts. This information comes from running Process Explorer; hardware interrupts consuming almost 25% of total CPU with CPU3 running 85-95%. The only way I've found to resolve this issue is to reboot the server. Then it will run for a while (days to weeks, not consistent) before the problem re-occurs.

Is there a way to find out what is causing the interrupts to go crazy? What process(es)? What hardware?

Thanks.


----------------------------------
Light travels faster than sound. That's why some people appear bright until you hear them speak.

 

[dberg35]

This sounds more like a hardware issue not Windows but make sure you have all update for Windows and all server hardware. Have you looked at the event logs?

 

[blister911]

dberg35, thanks for the reply.

Windows updates were put on back in April and will be applied again in October. Those are our outage windows. But we were having these issues prior to the last set of updates.

Event log doesn't show anything that points to the root cause.

I agree that this is probably not an OS issue. However, I'm trying to determine what is causing the problem; poorly written software, bad drivers, failing hardware, etc. Seeing that I've got 12 servers all doing the same thing at different times and re-occurring at different intervals, I doubt it's failing hardware. More likely it's a driver or poorly written software.

So what I need to be able to do is track down what hardware is being accessed and by what process. All I have right now is Process Explorer telling me hardware interrupts are running amuck.


----------------------------------
Light travels faster than sound. That's why some people appear bright until you hear them speak.

 

[itsp1965]

Have you checked the vendor's site/knowledgebase for updates, issues for your particular blade model?

 

[blister911]

itsp1965,

The software we run on these servers is our ERP. The developers constantly have updates; some fixes and some enhancements. We have logged a case with them on this issue. However, like most software companies, they don't like to admit fault unless you can prove it to them or they can recreate it. (I'm not saying that's bad, I'm just saying that's how it is). Without knowing what is going on to cause the interrupts or what hardware is going whacko, I cannot do either.


----------------------------------
Light travels faster than sound. That's why some people appear bright until you hear them speak.

 

[itsp1965]

blister911 I was referring to the hardware vendor and not your application vendor. Sorry for the confusion.

 

[blister911]

Server bios/firmware is up-to-date. These are blade servers. I only have control of my blades, not the chassis.


----------------------------------
Light travels faster than sound. That's why some people appear bright until you hear them speak.

 

[GrimR]

This is stab in the dark, not a UPS or KVM causing the problem? Only other thing is network related which could be nic or switch.
Just strange to have all 12 experiencing the same problem, does not sound virus related.

MCITP:EA/SA, MCSE, MCSA, MCDBA, MCTS, MCP+I, MCP

 

[blister911]

First off, thanks to everyone trying to help me out.

GrimR,
I can't rule out anything for sure until I can figure out what hardware is involved. However, I highly doubt it's UPS or KVM related.
Here's why:
1. 6 of the blades are here at our operations center and 6 are at our DR site (for lack of a better term) about a quarter-mile away. They do not run on the same model UPS, they are on different power grids, and have different motor generators.
2. Not all of the servers at a site experience the problem at the same time, even though they are in the same chassis.
3. They are all running headless (no KVM).
4. None of our other servers are having this problem (some same-model blades, some stand-alone servers)

But I can only speculate until I can find some way to identify what hardware is having the issue.

Is there a way to find out what is causing the interrupts to go crazy? What process(es)? What hardware?

All I have right now is Process Explorer telling me hardware interrupts are running amuck.

Thanks.


Light travels faster than sound. That's why some people appear bright until you hear them speak.