Unable to umount (all processes killed) 2

[aix4me2]

While trying to umount a file system keep getting message "...device busy" even after eliminating all the processes with open files.

fuser and lsof show nothing.

Ran slibclean. No change.

Used ipcrm to remove all share memory segments, messsage queues, semaphores. No change.

Killed all user processes and non essential root processes. No change.

Ran svmon -S. Found a number of ...lib.a files that were mapped to the file system. Tried modifying some and got the message files are open!? So what is keeping these files opened and how to clean up without rebooting? These are some 3rd party application files and are not owned by root. Nothing to do with AIX system files.

Alex

 

[SPguru]

Open file descriptor? Not sure exactly how an open file descriptor would effect an unmount, though.

Such as when people remove a file being actively written to and then ask why the filesystem reports 100% when there are no existing files.

Just tossing that one out.

 

[SPguru]

A kernel extension is loaded. fuser won't show references since they are not user processes. The genkex command will report all loaded kernel extensions.

 

[aix4me2]

genkex reports no user loaded kernel extensions. But open file descriptor is worth investigating. I am having problems, though, going through kdb without a map of where things are.

 

[sushveer]

Hi aix

Can you try with the follwing flags of umount

umount -k /wmsapp (to kill the process or file opened)
and umount -f /wmsapp (to forcibly unmount)

If they don't work...
As a last resort use
lazy mount...
#umount -l /wmsapp
and let me know asap.

sushveer
IBM certified specialist-p-series AIX5L System Administration
AIX/SOLARIS/WEBSPHERE-MQ/TIVOLI Administrator

 

[aix4me2]

-f is for remote mounts only. There are no -k and -l flags in aix.

 

[SPguru]

Maybe they're undocumented features or perhaps he has the source code and added his own options.

Even though I return this:

#umount -l /work
umount: Not a recognized flag: l
Usage: umount [-sf] {-a|-n Node|-t Type|all|allr|Device|File|Directory|Filesystem}

Maybe you have to run it from a trusted shell!

 

[sushveer]

Oh! I am sorry! Those flags will work on Linux.

I used once on linux ...long time back! for the same problem.
But in aix ...we don't have equivalent flags!!

sushveer
IBM certified specialist-p-series AIX5L System Administration
AIX/SOLARIS/WEBSPHERE-MQ/TIVOLI Administrator

 

[arvibm]

Hi aix4me2,

Something changed in the exports list or in the mounts or somewhere
else unexpectedly. rpc.mountd is confused. Usually happens when
something was mounted; exported; umounted; something else mounted
on that mount point...etc.,etc.
fix: stopsrc -s rpc.mountd; startsrc -s rpc.mountd
try this i hope it will resolve ur problem.

Regards

arvind

 

[Yegolev]

i think the syntax is:

lsof | grep /wmsapp

probably not helpful. you might want to compile your lsof again, noting your warning above.

IBM Certified -- AIX 4.3 Obfuscation

 

[rzs0502]

If you do a `ps -ef|grep lib.a`, do you get any info?

 

[aix4me2]

No.

In the meantime, looking through kernel tables I got a list of inodes on that file system with a non zero reference count. No wonder it can't umount. All these are .so and .a files, presumably some library modules.

The next thing to try to find out is: with the application running what processes are mapping these files.

 

[scottrisi]

I've seen this happen when the filesystem was exported. I've had to "exportfs -u xxxx" and also had to remove the filesystem entry from /etc/exports.
I've also had to kill the parent of a child process which may have used the filesystem.


hope this helps

 

[abutifilmnd]

Hey AIX4me2!

You must gone nuts while solving this challanging issue.

Just curious, what kind of application is it? IS it CA or BMC ?

Vendor must have found something in more than months time.

If still they are looking, here are few checks you can do.

1. Check any defunt processes are hanging out there.
2. Check all latest patches are applied from vendor.

x0x

 

[garry1624]

Did you try fuser -uck /filesystemname

 

[Yegolev]

one more guess: check for latest lsof version, i think 4.67, or get it from ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/

i know that when i upgrade from 4.3.3 to 5.1.0 i have to grab the latest copy from our repository, else it won't work properly.

IBM Certified -- AIX 4.3 Obfuscation

 

[unixtechie]

It's probably a running subsystem.

 

[scottrisi]

The last time this happened to me, I ended up copying and deleting files the with .so and .a suffixes. Somehow shared libraries remain open even though the application is stopped. Hope this helps.
I believe this happened with DB2 or BMC-patrol, it was a long time ago.

 

[unixtechie]

slibclean will remove unused modules from the kernel and library memory. You don't delete shared object (so) and archive (a) files to solve this.

And yes, both DB2 and BMC Patrol leave modules around after the apps are stopped. Hence, slibclean!

 

[tekken97]

Dear aix4me2;

Had you solved the problem?
We met the problem same as you now.
The filesystem cannot be umounted
Could you tell me how to find out and solve the problem?

Ken from Hong Kong

 

[baanman]

I faced the same problem as you a few moths a go , While I tried to takeover one of our servers to another from HACMP.
it could not unmout the filesystem.
and it was told us to reboot from our system support.

In our situation one thing was ,interesting
when I looked to logged in users wtih 'who' there was a username which actually was not logged in checked with ps -ef|grep username

if you have found any solution , I would like to hear.

PS: We are on aix5.2 and HAMP4.5
Baanman