root filesystem getting full and process trace 1

[arvibm]

Hi,

On one of my server the root filesystem getting full.which files i have to delete(i have deleted files smit.log and smit.script and files in var and tmp filesystem)but still it is 60% used.while on other servers root filesystem is not that much utlized.here is the output of df command.
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 1048576 426832 60% 1464 3% /
/dev/hd2 2097152 128464 94% 23275 58% /usr
/dev/hd9var 524288 151568 72% 678 4% /var
/dev/hd3 1572864 1340016 15% 463 1% /tmp
/dev/hd1 524288 489712 7% 512 1% /home
/proc - - - - - /proc
/dev/hd10opt 524288 509336 3% 316 1% /opt
/dev/crp_lv 31457280 4205344 87% 161299 5% /crpfm
/dev/patch_lv 40894464 9305904 78% 193420 4% /patches
/dev/loc_lv 41943040 13990376 67% 161279 4% /locfm
/dev/dm_lv 33554432 3872168 89% 164182 4% /dmfm
/dev/mxadmin_lv 20971520 5137208 76% 117503 5% /mxadmin
/dev/maxdemo_lv 10485760 8196816 22% 22250 2% /maxdemo
/dev/uat_lv 33554432 1178528 97% 166962 4% /uatfm
/dev/maxprod_lv 12582912 9860384 22% 35374 3% /maxprod
/dev/crpt_lv 33554432 4555440 87% 168343 5% /crptfm
/dev/maxuat_lv 10485760 7029032 33% 35495 3% /maxuat
/dev/uatcfm_lv 33554432 3083800 91% 168114 5% /uatcfm

how can i make my root filesystem clean with unwanted files.
sometimes my system get very slow.which command i can use to identify the process consuming maximum amount of memory and slowing down the system.
thanx to aix guru's in advance.


Regards

arvind

 

[rzs0502]

A / filesystem that is 60% full should not slow down your system.
Use sar and vmstat, iostat etc.
Alternatively, download top or nmon from

http://www.bullfreeware.com

Clearing files in /var and /tmp etc. will not matter as they are different filesystems.
Clearling logs, images or audit files from /

 

[dietz]

In addition to top and nmon as previously stated you might also find "topas" useful. It gives you a one page dynamic look at the system and is loaded with useful info. As for determining what to delete and not to, might I recommend a full mksysb or sysback prior to deleting to much. CYA...

I find "du -k |sort -rn" to be useful for determining which files are the largest and from there its just a little detective work to determine what and if the file is needed.

Remember backups are you friend...

 

[sushveer]

Hi
Also it is significant to clean up core files created in root.If u want find any...just delete it.To prevent any core files creation in root directory...just create a directory name :core....
eg:mkdir core

I wrote a script for cleanup/maintenance of all file systems
including root.
You can't use that bcoz...my environment is different from yours.
so You first check the growing files in root and delete it
by script if necessary...
Generally speaking in root apart from smit.log smit.script
you can delete core files if any...

You know ..what is the biggest chunk or junk in root..so act
accordingly and timely...


sushveer
IBM certified specialist-p-series AIX5L System Administration
AIX/SOLARIS/WEBSPHERE-MQ/TIVOLI Administrator

 

[petertp6]

I wonder some AP use / for store its log.
I suggest you use find command,
find / -newer +1 -exec ls -l {} \; >> /tmp/rootfiles
(I can not connect to a AIX now, PLS you use man to verify)
to find all files created/updated today, how their size change, used by which program.
You will find root cause after 3~5 days.

 

[bi]

check this thread, especially the starred item, for what to look for to clean up all of rootvg, not just /. You'll want to concentrate on /.

thread52-68802

 

[arvibm]

Dear Friends,

Thanx for all the tips.But when i use du -k |sort -rn command it will search the files in all the filesystems including oracle filesystems.giving a / with find command it will search for files in all the filesystems including oracle filesystems.
here is my ls -lrt output:-
# ls -lrt
total 384
drwx------ 2 root system 256 Mar 20 14:26 lost+found
lrwxrwxrwx 1 bin bin 8 Mar 20 14:26 lib -> /usr/lib
drwxr-x--- 2 root audit 256 Mar 20 14:26 audit
drwxr-xr-x 2 bin bin 256 Mar 20 14:26 mnt
lrwxrwxrwx 1 bin bin 8 Mar 20 14:26 bin -> /usr/bin
drwxrwxr-x 3 root system 256 Mar 20 14:27 .SPOT
drwxr-xr-x 2 root system 256 Mar 20 14:28 tftpboot
drwxr-xr-x 3 bin bin 256 Mar 20 14:31 sbin
drwxr-xr-x 6 root system 256 Mar 20 14:32 opt
drwxr-xr-x 2 root system 256 Mar 20 14:57 inst.log
drwxr-xr-x 22 root system 4096 Apr 27 11:20 var
-rwxr-xr-x 1 root system 3970 Apr 29 01:57 .dtprofile
drwxr-xr-x 2 root system 4096 Apr 29 02:01 TT_DB
drwxr-xr-x 2 root system 256 Apr 29 02:10 cdrom
dr-xr-xr-x 2 root system 256 Apr 29 03:57 .cd_x.qe7c
drwxr-xr-x 38 bin bin 4096 Apr 29 08:01 usr
-rw-r--r-- 1 root system 92 May 01 06:48 dsmerror.log
-rw-r--r-- 1 root system 26 May 06 13:01 .rhosts
drwxr-xr-x 2 root system 256 May 20 23:33 maxfm
drwxrwsrwx 4 sys sys 512 Jun 09 01:58 maxdemo
drwx------ 2 root system 256 Jun 09 08:42 nsmail
lrwxrwxrwx 1 bin bin 5 Jun 18 01:31 u -> /home
-rw-r--r-- 1 root system 44 Jun 18 07:42 .AATConfig
drwx------ 5 root system 4096 Jun 18 08:33 .netscape
drwxrwsrwx 10 crptappl dba 512 Jul 01 05:14 crptfm
drwxrwsrwx 10 crpappl dba 512 Jul 07 06:57 crpfm
lrwxrwxrwx 1 root system 21 Jul 07 13:18 unix -> /usr/lib/boot/u
nix_64
drwxr-xr-x 53 bin bin 4096 Jul 07 13:20 lpp
drwxr-sr-x 4 maxuat mxe 512 Jul 10 06:28 maxuat
-rw-r--r-- 1 root system 25287 Jul 21 04:36 core
drwxrwsr-x 11 locappl dba 512 Jul 23 13:24 locfm
drwxr-xr-x 28 bin bin 4096 Jul 25 04:43 home
drwxrwsrwx 4 maxprod mxe 512 Jul 27 11:10 maxprod
drwxrwsrwx 8 mxadmin mxe 512 Jul 28 08:57 mxadmin
-r-xr-xr-x 1 root system 906 Jul 29 08:34 bosinst.data
-rw-r--r-- 1 root system 14425 Jul 29 08:35 image.data
drwxrwsrwx 11 uatcappl dba 512 Jul 29 17:38 uatcfm
drwxr-xr-x 24 root system 8192 Aug 03 08:13 etc
drwxrwsrwx 11 dmappl dba 512 Aug 05 09:49 dmfm
drwxrwsr-x 7 uatappl dba 512 Aug 05 09:49 uatfm
-rw-r--r-- 1 root system 665 Aug 06 09:08 smit.script
-rw-r--r-- 1 root system 5678 Aug 06 09:08 smit.log
drwxrwxr-x 5 root system 8192 Aug 08 15:00 dev
-rw------- 1 root system 443 Aug 08 16:16 .Xauthority
-rw------- 1 root system 2190 Aug 08 16:16 .TTauthority
drwxr-xr-x 10 root system 4096 Aug 08 16:16 .dt
drwxrwsrwx 141 crptappl dba 2560 Aug 08 21:03 patches
drwxrwxrwt 31 bin bin 16384 Aug 09 09:03 tmp
dr-xr-xr-x 1 root system 0 Aug 09 09:03 proc
-rw------- 1 root system 3728 Aug 09 09:03 .sh_history

Pls let me know which files i can delete from rootvg.
sushveer here is my core file size:-
# ls -l core*
-rw-r--r-- 1 root system 25287 Jul 21 04:36 core
i don't think the file size is too big.which files i can delete otherthen core,smit.log,smit.script.which commnad i can use to search for a largest file only in rootvg.


Regards

arvind

 

[SLG]

If this box has a lot of users on it you may want to look at the /etc/security/failedlogin file. That file tends to get pretty big as users "fat finger" their passwords.

 

[arvibm]

Hi,

I have checked the size of file failed login and the file size is not too big.

# ls -l failedlogin
-rw-r--r-- 1 root system 64800 Aug 10 18:12 failedlogin


Regards

arvind

 

[bi]

arvibm,

That failedlogin file is 6% of the total size of /. The core file is about 2.5% of the size of /. Cut down on the failedlogin (or delete it), and delete the core file and you will get an additional 8.5% free.

When I am trying to find room, the first thing I look for is a core file and delete it (your core file is dated July 21, and if you haven't needed to look at it yet, you can probably safely delete it).

Then I do a du -sk | sort -n and concentrate on pruning directories (not filesystems) in /. An ls -l from / isn't going to help much in deciding which directories are large and should be looked at more.

 

[arvibm]

I have redirected output /dev/null to core and failedlogin file to make there byte size zero.but still my root filesystem is 60% utilized.i tried with commnad du -sk | sort -n but it dosen't show any output the cursor keeps on blinking and i have terminate the process by pressing control-C.

# ls -l core*
-rw-r--r-- 1 root system 0 Aug 12 11:12 core
# ls -l failedlogin*
-rw-r--r-- 1 root system 0 Aug 12 11:11 failedlogin

# df
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 1048576 426928 60% 1468 3% /
/dev/hd2 2097152 128416 94% 23275 58% /usr
/dev/hd9var 524288 59480 89% 711 10% /var
/dev/hd3 1572864 1518808 4% 103 1% /tmp
/dev/hd1 524288 207616 61% 546 3% /home
/proc - - - - - /proc
/dev/hd10opt 524288 509336 3% 316 1% /opt
/dev/crp_lv 31457280 4205344 87% 161299 5% /crpfm
/dev/patch_lv 40894464 9060400 78% 195535 4% /patches
/dev/loc_lv 41943040 13988856 67% 161281 4% /locfm
/dev/dm_lv 33554432 3871104 89% 164199 4% /dmfm
/dev/mxadmin_lv 20971520 4620888 78% 118278 5% /mxadmin
/dev/maxdemo_lv 10485760 8196816 22% 22250 2% /maxdemo
/dev/uat_lv 33554432 87392 100% 166415 4% /uatfm
/dev/maxprod_lv 12582912 9859392 22% 35378 3% /maxprod
/dev/crpt_lv 33554432 4555368 87% 168346 5% /crptfm
/dev/maxuat_lv 10485760 4778808 55% 35723 3% /maxuat
/dev/uatcfm_lv 33554432 2861024 92% 168809 5% /uatcfm
/dev/cd0 205856 0 100% 51464 100% /cdrom

when i use du -k command it will show the byte size of directories in all the filesystems.do u have any command to search for a largest file exclusively in root filesystem.

Regards

arvind

 

[arvibm]

Hi,

here is my directory output.Pls let me know which files can i delete.


# ls -lrt | grep ^d
drwxrwxrwx 3 root system 2048 Jan 09 2003 cdrom
drwx------ 2 root system 256 Mar 20 14:26 lost+found
drwxr-x--- 2 root audit 256 Mar 20 14:26 audit
drwxr-xr-x 2 bin bin 256 Mar 20 14:26 mnt
drwxrwxr-x 3 root system 256 Mar 20 14:27 .SPOT
drwxr-xr-x 2 root system 256 Mar 20 14:28 tftpboot
drwxr-xr-x 3 bin bin 256 Mar 20 14:31 sbin
drwxr-xr-x 6 root system 256 Mar 20 14:32 opt
drwxr-xr-x 2 root system 256 Mar 20 14:57 inst.log
drwxr-xr-x 22 root system 4096 Apr 27 11:20 var
drwxr-xr-x 2 root system 4096 Apr 29 02:01 TT_DB
dr-xr-xr-x 2 root system 256 Apr 29 03:57 .cd_x.qe7c
drwxr-xr-x 38 bin bin 4096 Apr 29 08:01 usr
drwxr-xr-x 2 root system 256 May 20 23:33 maxfm
drwxrwsrwx 4 sys sys 512 Jun 09 01:58 maxdemo
drwx------ 2 root system 256 Jun 09 08:42 nsmail
drwx------ 5 root system 4096 Jun 18 08:33 .netscape
drwxrwsrwx 10 crptappl dba 512 Jul 01 05:14 crptfm
drwxrwsrwx 10 crpappl dba 512 Jul 07 06:57 crpfm
drwxrwsrwx 10 crpappl dba 512 Jul 07 06:57 crpfm
drwxr-xr-x 53 bin bin 4096 Jul 07 13:20 lpp
drwxrwsr-x 11 locappl dba 512 Jul 23 13:24 locfm
drwxr-xr-x 28 bin bin 4096 Jul 25 04:43 home
drwxrwsrwx 4 maxprod mxe 512 Jul 27 11:10 maxprod
drwxrwsrwx 8 mxadmin mxe 512 Jul 28 08:57 mxadmin
drwxrwsrwx 11 uatcappl dba 512 Jul 29 17:38 uatcfm
drwxr-xr-x 24 root system 8192 Aug 03 08:13 etc
drwxrwsrwx 11 dmappl dba 512 Aug 05 09:49 dmfm
drwxrwsr-x 7 uatappl dba 512 Aug 05 09:49 uatfm
drwxr-sr-x 4 maxuat mxe 512 Aug 10 12:35 maxuat
drwxrwxr-x 5 root system 8192 Aug 11 15:00 dev
drwxr-xr-x 10 root system 4096 Aug 12 09:25 .dt
drwxrwsrwx 143 crptappl dba 2560 Aug 12 09:30 patches
drwxrwxrwt 11 bin bin 16384 Aug 12 11:54 tmp
dr-xr-xr-x 1 root system 0 Aug 12 11:54 proc


Regards

arvind

 

[bi]

We can't advise you on just a list of the directories -- we can't tell you to delete directories if we don't know what is in them -- or what their true size is.

A du -sk * | sort -n from root on your system will take a long time. You just have to be patient. You should probably redirect the output to a file.

Did you look at Thread52-68802 and follow those suggestions for /?

What's in .netscape and .dt?

 

[conniefu]

I usually do a "df -k" to identify which file system is getting full.

Then go to that directory. run the command
"du -k -x . |sort -rn|head" which will show you the largest sizes of files under that directory. But it will exclude anything which is mounted on that file system. This way you won't get confused by big files which are under different file systems (different devices).

cheers,

connie

 

[arvibm]

Hi,

Thread52-68802 IS USEFUL.specially the command to find files over 2mb is good.thanx to aixqueen.I have reduced the occupied space from 60% to 49%.Some users have deleted files from exceed and these unwanted files are in trash folders.i have deleted the files.but still it is 49%.

regards

arvind