Unable to umount (all processes killed) 2

[aix4me2]

While trying to umount a file system keep getting message "...device busy" even after eliminating all the processes with open files.

fuser and lsof show nothing.

Ran slibclean. No change.

Used ipcrm to remove all share memory segments, messsage queues, semaphores. No change.

Killed all user processes and non essential root processes. No change.

Ran svmon -S. Found a number of ...lib.a files that were mapped to the file system. Tried modifying some and got the message files are open!? So what is keeping these files opened and how to clean up without rebooting? These are some 3rd party application files and are not owned by root. Nothing to do with AIX system files.

Alex

 

[bi]

I hate to ask this, but are you trying to umount a filesystem that is mounted within another filesystem?

For example, if you have a filesystem named /app, and you have another filesystem named /app/data, are you trying to umount /app?

 

[aix4me2]

No. This file system is mounted from /.

 

[sushveer]

Hi aix

Can you please post the exact command and out put your are getting.

Just paste it! It's strange!I would like to direct based on your next post

Thanks

sushveer
IBM certified specialist-p-series AIX5L System Administration
AIX/SOLARIS/WEBSPHERE-MQ/TIVOLI Administrator

 

[aix4me2]

Here goes...

Script command is started on Thu Aug 7 16:10:31 EDT 2003.
root@HTWMSTST1:/home/alexl # df /wmsapp
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/wmsapp_lv 15728640 10525728 34% 77430 4% /wmsapp
root@HTWMSTST1:/home/alexl # lsfs /wmsapp
Name Nodename Mount Pt VFS Size Options Auto
Accounting
/dev/wmsapp_lv -- /wmsapp jfs 15728640 -- yes
no
root@HTWMSTST1:/home/alexl # fuser -c /wmsapp
/wmsapp:
root@HTWMSTST1:/home/alexl # lsof /wmsapp
lsof: WARNING: compiled for AIX version 5.1.0.0; this is 5.2.0.0.
root@HTWMSTST1:/home/alexl # umount /wmsapp
umount: 0506-349 Cannot unmount /dev/wmsapp_lv: The requested resource is busy.
root@HTWMSTST1:/home/alexl # exit

Script command is complete on Thu Aug 7 16:11:22 EDT 2003.

 

[SPguru]

No /wmsapp/logs or such filesystems under /wmsapp?

 

[aix4me2]

Nothing is mounted under /wmsapp.

 

[bi]

Do you get any output when you use fuser -fxu /wsmapp?

 

[aix4me2]

No output.

 

[bi]

Is this filesystem exported to another system?

(I feel like we are playing 20 questions here!)

 

[aix4me2]

No. Believe me I checked all the obvious causes. I couldn't even get a satisfactory answer from AIX support. They told me to reboot!?

 

[sushveer]

Hi aix

The first point to be considered is : root user and owner of that dir(mount point) only can unmount.others can't.


Can you please try with the following:with root account use
#fuser -k -u /dev/wmsapp_lv
or
#fuser -k -u /wmsapp

Also ensure that ...You are not in the path of that directory(wmsapp)...confirm with this 'pwd' command
#pwd

If the problem is not resolved ...please try
#showmount -e
this will show the export list of your system...look for
/wmsapp ...whether it is exported or not...!

You can check the users using that /wmsapp file system
by using command:
#ps -ef|grep "/wmsapp"
If u find any process ..owner ...kill that proc id
by using kill -9 <proc-id> and then use umount /wmsapp

...Give me yur feedback asap....





sushveer
IBM certified specialist-p-series AIX5L System Administration
AIX/SOLARIS/WEBSPHERE-MQ/TIVOLI Administrator

 

[we2aresame]

Try this:
df
/dev/lv02 4063232 3935632 4% 17 1% /wmsapp
/wmsapp 4063232 3935632 4% 17 1% /example
do you find the file system /wmsapp like this, if it is,
umount /example
umount /wmsapp

 

[sushveer]

Hi we2aresame

This is good point but &quot;bi&quot; had already pointed out this in his first post...

So no chance...
There is something to lookinto...
so far we have considered all normal check points...
let us think something beyond that....!!??


sushveer
IBM certified specialist-p-series AIX5L System Administration
AIX/SOLARIS/WEBSPHERE-MQ/TIVOLI Administrator

 

[bi]

Have you talked to the vendor of the application that is using this filesystem? If the app I am guessing is using this filesystem, it very well could be the app is doing something flakey that is causing this problem.

Since you've checked all the obvious causes, am I right in assuming you have checked your syslog and errpt and found nothing of interest? (also assuming the level of logging you have for syslog is at a level that would report lots of errors.)

 

[aix4me2]

All of the above were tried again and again. (I've got 4 systems using this app, all exhibiting the same behaviour).

Right now I am using kdb to try to find out what is really going on. inode subcommand lists a number of inodes on the offending file system one of which did translate to a real file. My knowledge of AIX internals is very limited. Does anyone know how could I find out if anything is mapped to this inode?

 

[bi]

What did the vendor of the app say? You've got 4 systems using this app, they are all exhibiting the same behavior, and a group of SA's haven't been able to help you solve this. Has the vendor of the app ever suggested to you that you need to boot the system on a regular basis?

 

[aix4me2]

They've been looking at it for over a week and are clueless. They claim it doesn't happen elsewhere which is not surprising. I was never aware of the issue until I implemented HACMP on a couple of nodes.

 

[bi]

Has the app been certified for use with HACMP?

 

[aix4me2]

No but the problem is hapenning on non HA systems as well.