Unusually high disk write load on one webserver

[TrojanWarBlade]

Hey guys,

I have a set of webservers that should be identical (mirrors of each other).
The web traffic is load balanced to each server but I am finding one server in particular seems to be heavily loaded.

On closer inspection, iostat seems to suggest that it is writing to the disk about 4 times as much as any other server.
Clearly something is different but I need to discover what the difference is.

Could anyone suggest ways of identifying which processes are writing to the disk at an unusually high rate?

Many thanks.


Trojan.

 

[North323]

how is it load balanced?

 

[TrojanWarBlade]

I have a separate machine taking the traffic and using round-robin load balancing.

I'm convinced that load balancing is not the problem as iostat shows similar or lower read counts on this server relative to the other servers but 4 times the write load (local disk).

The network load (in and out) on all webservers is similar.


Trojan.

 

[North323]

are they the same servers? are the hard drives different? did this just start or has it always been like this?

 

[TrojanWarBlade]

These are cloud servers so I don't actually know whether the disks are the same.
The load has been there since we started this server so although it could be specific to the hardware, I suspect we have a difference in the setup.
The fact that the read load is similar to the other servers but that the write load is 4 times higher or more than anything else is this thing that tells me we probably have a rogue process scribbling madly to disk.

I am looking for any ideas that might help me track that down.

Thanks



Trojan.

 

[North323]

may be a config issue. are all your NIC and port settings the same?

 

[TrojanWarBlade]

yep, all identical.

You do seem to be focussed on network when this appears to be a disk issue.

The iostat numbers I am talking about are for local raid disk (/dev/md0). The network numbers are not dissimilar.

All machines should be identical except for hostname.


Trojan.

 

[North323]

i apologize, i am not familiar with cloud computing. maybe a bad disk?

 

[Annihilannic]

Perhaps try iotop?

http://guichaz.free.fr/iotop/

(I can't vouch for that particular implementation personally, but I think there are other versions of it out there too...)

Annihilannic.

 

[stevexff]

Trojan

Asking the obvious questions, if the web site is down, do you still get the skewed load? I mean, is it caused by some other process entirely? What about logging? Do all the logs get written to one central server? Are the logging options different? Is your DB server on the same box?

You've been a bit quiet lately. Nice to see you back...

Steve

[small]"Every program can be reduced by one instruction, and every program has at least one bug. Therefore, any program can be reduced to one instruction which doesn't work." (Object:erlDesignPatterns)[/small]

 

[TrojanWarBlade]

Hi Annihilannic,

I did look at installing iotop and I might still do that but this is a live commercial server so I need to be careful with what I install.

Hey Steve, long time no speak.

I can't take the site down (there are over 2000) so I can't do that kind of testing. Would be nice if I could!

I have logging happening but it doesn't show anything obvious.

DB servers are not on webserver machines so that should not be a problem.

Log files are written to each server individually and *should* be identical (but clearly are not as the disk write load seems to be so crazy high on this particular server).



Trojan.

 

[TrojanWarBlade]

Guys,

I think we *might* have found the problem.
We are running reiserfs and it appears that these machines have the wrong mount options.
The /etc/fstab entry is correct but I suspect we mounted specifying a source *and* destination causing mount to ignore the options in /etc/fstab.

Anyway, when we're sure I'll drop another post here to confirm.

In the meantime, thanks to everyone for their ideas and suggestions.



Trojan.

 

[TrojanWarBlade]

Hi peeps,

Just as an update, we still seem have unusually high loads on this webserver but changing the mount options for reiserfs did seem to fix the excessive "writes" problem that I mentioned in the title of this thread so in that sense I think we can close this thread here.

Thanks for the feedback peeps.



Trojan.