HTFS: Out of inodes on HTFS 1

[piperent]

Maybe someone has seen this problem. System is 5.0.5 enterprise, 30 user, with 3 separate filesystems (/root, /usr2, /u). All application software is on the 2nd and 3rd filesystems with only the operating system on /root. The applications only make minimal use of the root drive, but something seems to be using all available inodes on that drive.

Example: at 3:00 pm there were 25,429 inodes available;

at 7:30 pm there were none availabe (HTFS: error message was displayed).

There was no unusual activity (printing, copying, etc.) going on, but something used all those inodes. What could the OS (?) be running that might cause this type of degradation. I've checked the crontab and limited all application oriented jobs to housecleaning chores ONLY, no backups, no summary prints, no file copies, etc.

I do have NFS implemented and am running 8 nfsd daemons. Maybe this has something to do with it. I have noticed that if the available inodes is at least functional (500 or so) when the system performs it's routine cron scripts, the lost inodes are recovered and made available. This kind of indicates that the OS does have a house-keeping routine to recover and restore un-used and available space. Darned if I know what routine it might be.

If someone has seen this or has any possible reason or solution, it would be greatly appreciated.

Thanks,
J Piper

 

[motoslide]

You need to find where the INODES are going, and this means counting files in different directories.
Once the system is totally out of INODES, it can get pretty cranky, and many of the little background tasks will quit working.
Try this:

# cd /usr
# find . -type f -print|wc
# cd /var
# find . -type f -print|wc
(do the same for other directories such as etc, dev, lib)

Once you find which directory tree is growing (in total number of files), you can go down that tree and narrow your search until you locate the culprit. That won't solve the problem, but it should point to a cause. Let us know what you find.

I've seen this before, and it's often either MMDF complaining or some script that needs attention. If it's MMDF, you'll have thousands of files (each using an INODE) in /usr/spool/mmdf/lock/home/*.

Best of luck.

 

[edfair]

Since it apparently gets cleaned you might want to look in the crontabs to see what is cleaned regularly.
In an earlier system I had a /tmp file generated whenever a specific program cranked up and coupled with some problem in the cleaning the system hit the wall regularly.

Ed Fair
Give the wrong symptoms, get the wrong solutions.

 

[piperent]

Thanks for the input. It's nice to know that I'm not the only person in the world to have seen this issue, although I am probably the only one who hasn't figured out how to avoid it. )

Is it possible that 'compress' could create a subjugate file in /tmp when files are compressed? I said I had cancelled all the 'application' based cron jobs and it just so happens that one of those jobs compresses a medium size database (225 mb). With that job cancelled, and approx. 530 inodes available, the evening process that cleans inodes (whatever that might be) completed just fine and freed 30,750 inodes.

The compress is performed on one of the secondary filesystems, but I'm just wondering if compress uses the /tmp directory as a storage facility to perform the compression. My thoughts are it has to use some file as a temporary holding area to complete what it does, and in the general mix of things, the most common place would be /tmp or at least somewhere on /root. Does this make any sense or am I grasping at straws?

J Piper

 

[piperent]

A final follow-up to this problem. Motoslide hit it right on the head. 'mmdf' has gone totally berserk and is filling the home/ directory with literally thousands upon thousands of 'msg' files. What a horrific waste of time. Nobody even uses the mail sub-system. I think I'll try to figure out how to disable it, or at least run a cleaning routine regularly.

Thanks for all you help.

J Piper

 

[edfair]

You need to fix the problem being reported. What you are proposing is "killing the messenger".

Ed Fair
Give the wrong symptoms, get the wrong solutions.

 

[cdlvj]

You may have a virus, as one of the very few that I have seen on SCO. Happened on my system, and just creates file after file.

 

[piperent]

Assuming I have a 'virus', how did you locate your's and what remedial action did it take to eliminate it. Nothing appears in my 'ps' display to indicate anything other than normal activity, and there is certainly nothing in the 'cron' file.

Also, if I temporarily suspend (stop) the 'deliver' routine, will that help resolve the current issue while I search for this thing? I'm not that familiar with MMDF since none of my sites have ever used it.

J Piper

 

[piperent]

A further note of interest. When I look at the messages that are spooled in lock/home/msg/, I find message contents like the following:


(Could not open message file.)
# more msg.az27699
Date: Mon, 27 Dec 2004 1:17:57 CST
From: fort. MMDF Mail System <mmdf@fort.>
Sender: mmdf@fort.
Subject: Waiting mail (msg.•b15245)
To: mmdf@fort.
Message-ID: <200412270117.az27699@fort.>

After 3 days (71 hours), your message has not yet been
fully delivered. Attempts to deliver the message will continue
for 5 more days. No further action is required by you.

Delivery attempts are still pending for the following address(es):

root

Problems usually are due to service interruptions at the receiving
machine. Less often, they are caused by the communication system.

Your message begins as follows:

(Could not open message file.)


To free up inodes, I had to delete the contents of the 'msg' directory. Is this the result of that cleaning, and does this tell me anything at all?

Thanks,
J Piper (an obvious MMDF neo-phyte)

 

[cdlvj]

Mine was sendmail, and by recylcing, was able to stop it. It was doing exactly the same, have installed spambouncer now. If you do not use mmdf, then disable it in init scripts /etc/rc? directory.

 

[edfair]

Guessing now, but mail is appended to /usr/spool/mail/root to be delivered. Owner root, group sys, 600 perms.
I would have expected that the file would be created if missing but have never checked.

Ed Fair
Give the wrong symptoms, get the wrong solutions.

 

[piperent]

The final word. First let me say thanks to everyone who contributed to my little query. Here's what I found and how I solved it. The culprit was a spurious 'lock' file sitting in the /usr/spool/mail directory. The strange thing is that it wasn't the normal 'lck' type of lock file, it was an actual 'lock' file on 'root' of all things. Well, this lock was apparently preventing any of the mail messages for 'root' from being disseminated properly. They would just sit on the queue waiting for a miracle to set them free. Once I deleted that lock, cleaned out the queue directories, and built a simple little scrubber script (I can do this since no one uses the mail sub-system) everything seems to be perking along just fine. Wow, what a learning curve.

Thanks to everyone.
J Piper