Inodes at 100% - intermitant problem

[linmatt]

linmatt (TechnicalUser) Nov 6, 2003
We have an interestin problem with a SCO Openserver 5.5 box. We have a partition that hold a lot of data in several million very small files,uses about 54% of the partition disk space and 19% of the inodes. We tried mounting a Windows2000 NAS box (Dell 725N) running NFS services on the SCO box to copy these files, but started to experience problems - firstly that the SCO box spontaneously rebooted with the NAS attached, but now every now and again, and it can be weeks apart, or could even be twice on the same day, we get the inode usage shoot up to 100%, some 13 Million inodes dissapear almost in an instant. We haven't seen any gradual build-up, and a reboot brings it back to 19%. There's nothing in any of the logs, and nothing that uses that partition that creates any temporary files, nothing that we can think of. We suspect some form of corruption on the FS, but are having trouble unmounting it and reformatting that partition. Scoadmin not being too helpful.

Thanks

Matt

 

[apeasecpc]

For your unmount problems, have you tried dropping to single user mode?

Is this the root file system?

If you haven't already, you should disconnect the Win2K NFS and see if that clears up the problem.

Are there any particularly large database files on the file system in question? I've seen some FS problems that resulted from attempting to copy large DB files while in use.

Is this FS on an array or a single hard drive? The problems could also be the results of a hard drive going bad.

 

[linmatt]

The problem is on /u1, it's a Compaq Proliant with RAID 5 across 6 disks, aprox 250GB in total, but the files are scanned images, all tiffs about 3k in size and quite a deep directory structure. The NAS box was taken off as soon as the problem appeared and hasn't been on since - have good old MS added something thier 'interpretation' of NFS that screws it up? We were able to copy 25GB in one hit without any problems, but on one occasion copying a 2k text file caused a reboot of the SCO box.

 

[apeasecpc]

Try restarting in single user mode, then run a fscheck on the filesystem. I am still thinking you may have a bad hard drive.

How long had this been a stable system before you mounted the NFS and ran into the problem?

 

[linmatt]

It's been running for 2 years without a hitch (it is Unix, after all) but the HDD were increased a year or so ago from 4 drives to 6 I believe, I can't remember the exact number. We're starting to think it's a drive, but it only started after adding the NAS. SCO/Caldera don't seem to know, our support company don't know, and I've never seen this before, even other 'nixes (HP-UX, Solaris and Linux).

Thanks for the responses - it's giving us more to go on than our support company have come up with.