How to determine "hot" file(s) on filesystem?

[linnorm]

I've got a V480 running Solaris 8 and Oracle9i. It has SAN attached storage for the database and uses the two internal disks for booting. The internal disks are mirrored via Veritas.

Via vxstat I've narrowed down the culprit to the rootvol volume on the internal disks, but I haven't been able to get down to specific file names that are busy. The rootvol volume holds all of the standard OS file systems except /export/home. (/, /opt, /etc, /var, /usr, etc.)

TIA,

 

[KenCunningham]

Have you looked into whether lsof might help, if run at regular intervals and written to a file for monitoring purposes perhaps?

 

[linnorm]

I've got lsof but was having trouble sifting the output down to a useful level. I noticed this morning that another system is having the same issue but with a different volume layout. For some reason /var was given its own volume and shows the same symptoms as the rootvol in the first box.

I used 'lsof +D /var' to see what's going on and none of the files it listed had timestamps within the last minute. I did find two special files '/devices/pseudo/tl@0:ticots->/var/tmp/.oracle/s<Oracle SID>' and '/devices/pseudo/tl@0:ticots->/var/tmp/.oracle/s#1477.1' which are obviously related to the Oracle installation, but I don't know how to get usage info for them.

 

[Annihilannic]

You can use the vxtrace command to identify which blocks are being hammered most often.

Unfortunately I can't figure out how to work back from the block number to the file that is using it. I expect you could do it with fsdb, but I've tried without much luck. I'd be curious to know how to do this!

Annihilannic.

 

[linnorm]

Annihilannic: I played around with vxtrace at your suggestion, but can't figure out how to correlate to a file name either. I did notice that something is doing large block writes to disk though, and the times associated with the END entries ramp up as the writes finish. I wonder if the two pseudo devices in /var/tmp are responsible. It looks like they're owned by the Oracle listener. I can't think of any reason why the listener would be doing writes like that, but I'm no expert.

 

[SamBones]

I don't have a complete answer, but the command "[tt]iostat -xzpn[/tt]" will give you IO statistics for each slice. The command "[tt]fuser[/tt]" can tell you which files are open and by which process (see the man page).