Do the number of inodes break restore and cp commands?

[crowep]

The restore and cp commands will not complete when the inode usage (bdf -i) is above 16% on a 4GIG drive. The operating system is HPUX-10.20. If you do a "top", these commands will be at the top of the list using all the available cpu time but they are not affecting the load average. One specific directory tree is responsable for the high inode counts.. I guess my questions are as follows;

Is this a problem with the operating system or a limitation with restore and cp (or any other command)? The kernel ninode parameter is set to 2048 on a TAC4 (770) HP machine.

Is there a fix? I've noticed that there are patches for fixing problems associated with the inode cache in that some flags are not set correctly. The descriptions don't indicate that this problem may also be mine.

thanks,

paul

 

[MikeLacey]

Hi Paul,

Irritating problem...

2048 doesn't seem an excessively high number of inodes to me.

How long do you leave the restore commands before you decide that they've hung?

The cp command is hanging as well? Wierd...

Could you post the exact commands you're using please?

Regards, Mike
michael.j.lacey@ntlworld.com
Email welcome if you're in a hurry or something -- but post in tek-tips as well please, and I will post my reply here as well.

 

[crowep]

The commands are part of backup and restore scipts written for off-site locations. For each instance a spare disk is mounted to /mnt. (these are not LVM, just straight file systems)

The "cp -pr /home2/stuff/data /mnt/backup_stuff" ran all night before I killed it the next morning. There is a directory structure under ../data that an application uses and doesn't clean up after itself that I have identified as the culprit for excesive idnode counts for this disk. Half of the 750,000 files (95% inide usage) I discovered at one site was some type of status file containing only 4 bytes. This I can manage by using a cron periodically to clean it up.

The cd /mnt; "/etc/restore rvfb "TAPE_HOST"":$TAPE_DEVICE" 126b" to restore from 8mm tape usually runs about an hour but hung when it started to retore the ../data tree. I killed it after 3 or so hours. There is no problem dumping this disk to tape.

There is a case on another drive, however, that has a large number of files that I cannot manage in the the same way. Some sites require a years worth of data and are starting to push the inode usage towards 35% an that drive.

I don't get any errors like "list to long" when you use wild cards like rm *. As I mentioned before, the restore or cp just goes to the top of the list (top) and uses all available cpu time. It gives it up when required and causes no problem with system performance.

thanks for the interest,

paul

 

[trunix]

This is just a long shot, but have you tried running fsck on the relevant filesystems. Maybe you have some corrupt files in there which are affecting cp.

 

[MikeLacey]

Ok - 3/4 million files, that's quite a lot. I assumed from your mention of ninode that the number was somewhat lower...

My understanding was that ninode had to be set so that all of the inodes on a system could be held in memory in the kernel table - but that's obviously not the case.

The kernel must be spending all of its time "managing" (badly) the inode table.

The obvious fix is to get that badly behaved application to clean up after itself - or to keep the number of status files down to a manageable level by deleting them every day.

Failing that: Have you tried setting ninode to be considerable higher? The kernel must be using it as a cache. I don't know what the upper limit is for ninode but I'll bet it's quite a lot lower than 3/4 million - go for the application and its status files if you can.

Go to

http://www.hp.com

and search for 'ninode' for more information. Hope this helps you. Mike
michael.j.lacey@ntlworld.com
Email welcome if you're in a hurry or something -- but post in tek-tips as well please, and I will post my reply here as well.

 

[crowep]

This is only reseach right now.. The developement system here gets rebuilt so frequently that that I don't have a system to experiment on. Only a couple of systems fielded are having the problem. We have enough problems trying to get them to go down long enough for backups much less taking down thier servers to play admin games. I'll be visiting one of those sites soon and was hoping to take some ideas with me.

 

[MikeLacey]

ok, sounds like you have your hands full... Mike
michael.j.lacey@ntlworld.com
Email welcome if you're in a hurry or something -- but post in tek-tips as well please, and I will post my reply here as well.