Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations gkittelson on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
File systems at 100% 5
- Thread starter maxcrook
- Start date
-
2
- #2
If root (/) Is Full
Check the /etc/security/failedlogin file.
Use the following command to read the contents of the file.
who /etc/security/failedlogin
The condition of tty's respawning too rapidly will create failed login entries.
To clear the file after reading or saving the output, execute the following
command:
cp /dev/null /etc/security/failedlogin
Check the /dev directory.
If a device name is mis-typed, as in "rmto" instead of "rmt0",
a file will be created in /dev called "rmto". The command will normally
proceed until the entire root file system is filled before failing
(/dev is part of the / file system). Look for entries that are
not devices (that do not have a major or minor number).
Execute the following:
cd /dev
ls -l |pg
Whereas a file size on an ordinary file would normally be seen,
a device file will have two numbers separated by a comma.
Example
crw-rw-rw- 1 root system 12, 0 Oct 25 10:19 rmt0
If the output looks like the following, the file should be removed.
Example
crw-rw-rw- 1 root system 9375473 Oct 25 10:19 rmto
NOTE: The /dev directory has some valid file names. Look for a file
that has a large size (larger than 500 bytes).
NOTE: If system auditing is running, the /audit directory (default)
may rapidly fill up and require attention.
Check for very large files.
Large files can be located with the find command. For example,
to find all files in the root (/) directory
larger than 1 MB, type the following command:
find / -xdev -size +2048 -ls |sort -r +6
This will find all files greater than 1 MB and sort them in reverse
order with the largest files first.
NOTE: When checking the root directory, entries from the /dev directory
that have major and minor numbers instead of file sizes will be
interspersed with "real" files and can be ignored.
Other useful find command flags may be helpful, such as the -newer flag.
For versions of AIX prior to 4.3, use InfoExplorer to learn
more about such flags.
NOTE: Before removing any files, the user should check to see if the
file is currently in use by an active user
process. Execute the following command:
fuser <filename>
<filename> is the file name that is being checked by the active user
process. If a file is "open" at the time of removal it is only removed
from the directory listing. The blocks allocated to that file are
not freed until the process holding the file open is killed.
------------------------
If /var Is Full
In /var/tmp, check for old leftover files.
/var/adm/wtmp is a file that is used to log all "logins", "rlogins"
and "tn" sessions. If it is not monitored it will grow indefinitely
unless system accounting is running. System accounting will clear
it out nightly.
/var/adm/wtmp can either be cleared out or edited to remove old
and unwanted information.
To clear /var/adm/wtmp, execute the following:
cp /dev/null /var/adm/wtmp
To edit the file and remove unwanted entries, execute the following:
/usr/sbin/acct/fwtmp < /var/adm/wtmp >/tmp/out
Edit the /tmp/out file to remove unwanted entries then put the
edited version back in wtmp by executing the following command:
/usr/sbin/acct/fwtmp -ic < /tmp/out > /var/adm/wtmp
In the /var/adm/ras directory, clear the error log.
This directory contains the error log, errlog. It is never cleared
unless it is manually cleared. DO NOT cp /dev/null to it or it
will disable the error logging functions of the system.
A zero (0) length errlog file must be replaced from a backup tape.
The log can be cleared of all entries removed up to a
certain number of days using the errclear command.
To clear the error log, type the following command:
errclear 0
To clear the error log but leave the past week's error entries,
type the following command:
errclear 7
NOTE: The trcfile file in this directory may be large due to a
trace being run. The file can be removed by executing the following:
rm /var/adm/ras/trcfile
The /var/spool directory contains the queueing subsystem files.
Clear the queueing subsystem by executing the following commands:
1.stopsrc -s qdaemon
2.rm /var/spool/lpd/qdir/*
3.rm /var/spool/lpd/stat/*
4.rm /var/spool/qdaemon/*
5.startsrc -s qdaemon
The /var/adm/acct directory contains accounting records.
If accounting is running, this directory may
contain several large files. Information on how to manage these
files can be found in System Management Guide chapter 14 (SC23-2457-01).
The /var/preserve directory contains old terminated vi sessions.
Delete these.
While old vi sessions can be used to recover files that were abnormally
terminated, these files can be deleted. However, the user may want
to keep some of the newer ones in case users want to recover
files. To recover a file, execute the following:
-r filename or -r
This will list all available files that are recoverable.
Modify /var/adm/sulog.
This file tracks the number of attempted uses of su and
whether they are successful or not. This is a flat
file and can be viewed and modified with a favorite editor.
If it is removed it will be recreated by the next attempted su.
Modify /var/tmp/snmpd.log.
This is used by the snmpd daemon as a log. If the file is removed
it will be recreated by the snmpd daemon.
NOTE: The size of this file can be limited so that it does not grow
indefinitely by editing the /etc/snmpd.conf file under the section for size.
This is in bytes.
NOTE: The find command can also be used to discover large files in /var.
Execute the following:
find /var -xdev -size +2048 -ls| sort -r +6
NOTE: AIX provides a general system cleanup script called skulker
located in the /usr/sbin directory. Before attempting to run the skulker
command, look at the skulker entry in InfoExplorer(for versions of AIX
prior to 4.3). Read the script for details to determine what files
it will delete and what time frame it will allow files to exist
before deletion.
skulker may be run as a cron job using the following crontab entry:
0 3 * * * /usr/sbin/skulker
Consider limiting the errlog by the running these entries in cron:
0 11 * * * /usr/bin/errclear -d S,O 30
0 12 * * * /usr/bin/errclear -d H 90
-
3
- #4
Space problems are a common occurrence because applications are forever incresing in size. Often space problems can be caused by rouge processes or just careless users. Often the cause is obvious, but occasionally it can be a bit of a mystery.
After investigating numerous space problems over the years I have written the following scripts to ease the problem.
The First script is dfsort. All this script does is sort your filesystems into order of percentage space used. This immediately shows you where the problem is.
dfsort Script
#!/usr/bin/ksh
#
# Script: dfsort
# Aim: Show filesystems sorted by percentage full
#
awk="/usr/bin/awk"
basename="/usr/bin/basename"
df="/usr/bin/df"
egrep="/usr/bin/egrep"
grep="/usr/bin/grep"
sort="/usr/bin/sort"
tail="/usr/bin/tail"
uname="/usr/bin/uname"
#
# Functions
#
initialisation()
{
for parm in $PARMS
do
case "$parm" in
-k) DFOPT="-k" ;;
*) : ;;
esac
done
return
}
showAIXspace()
{
$df $DFOPT |
$tail +2 |
$grep -v ":" |
$grep "^/dev" |
$awk '{
space = substr($4,1,length($4)-1)
inode = substr($6,1,length($6)-1)
if ( space > inode )
{
print space, $0
}
else
{
print inode, $0
}
}' |
$sort -t" " +0 -1 -n |
$awk -v bs=$DFOPT 'BEGIN {
if ( bs == "" )
{
print "Filesystem 512-blocks Free %Used Iused %Iused
Mounted on"
}
else
{
print "Filesystems 1024-blocks Free %Used Iused %Iused
Mounted on"
}
}
{
printf "%-15s %9s %9s %7s %7s %7s %-s\n", $2, $3, $4, $5, $6, $7, $8
}'
return
}
showSpace()
{
OS=`$uname `
case "$OS" in
AIX) showAIXspace ;;
SunOS) showSUNspace ;;
*) : ;;
esac
return
}
showSUNspace()
{
$df -k |
$tail +2 |
$grep -v ":" |
$egrep -e "^/dev/|^/dev/|^swap" |
$awk '{
space = substr($5,1,length($5)-1)
print space, $0
}' |
sort -t " " +0 -1 -n |
$awk 'BEGIN {
print "Filesystem 1024-blocks Used Avail %Used
Mounted on"
}
{
printf "%-25s %9s %9s %9s %7s %-s\n", $2, $3, $4, $5, $6, $7 }'
return
}
#
# Start of main processing
#
SCRIPT=`$basename $0 `
PARMS="$*"
initialisation
showSpace
exit 0
Once the problem Filesystem has been identified, the fisrt method of attack is to determine the largest files within that filesystem. Use the script below finds. This script lists all the files within the filesystem in size order. I usually pipe the output through tail just to list the largest files, eg:
finds /usr | tail -10
Here is the finds script:
#!/usr/bin/ksh
#
# Script: finds
# Aim: List files in a filesystem by size
#
awk="/usr/bin/awk"
basename="/usr/bin/basename"
cut="/usr/bin/cut"
find="/usr/bin/find"
sort="/usr/bin/sort"
#
# Functions
#
checkParms()
{
FILELIST=""
for parm in $PARMS
do
if test "$parm" = "-r"
then
REVERSE="r"
else
check=`echo "$parm" | $cut -c1 `
if test "$check" != "-"
then
FILELIST="$FILELIST $parm"
fi
fi
done
return
}
listFiles()
{
$find $FILELIST -xdev -type f -ls |
$awk '{ print $7 ": " $0 }' |
$sort -t: +0 -1 -n$REVERSE |
$awk '{ printf "%7s %4s %10s %2s %-8s %-8s %10s %3s %2s %5s %s\n", $2,
$3, $4, $5, $6, $7, $8, $9, $10, $11, $12 }'
return
}
#
# Start of main processing
#
SCRIPT=`$basename $0 `
PARMS="$*"
checkParms
listFiles
exit 0
If this does not immediately highlight the cause of the problem, my next line of attack is to list the files in the filesystem sorted by modification time, in the hope that the files most recently changed are the cause of the problem. I use findt listed below.
Findt script
#!/usr/bin/ksh
#
# Script: finds
# Aim: List files in a filesystem by modification time
#
awk="/usr/bin/awk"
basename="/usr/bin/basename"
cut="/usr/bin/cut"
date="/usr/bin/date"
find="/usr/bin/find"
sed="/usr/bin/sed"
sort="/usr/bin/sort"
#
# Functions
#
checkParms()
{
FILELIST=""
for parm in $PARMS
do
if test "$parm" = "-r"
then
REVERSE="r"
else
check=`echo "$parm" | $cut -c1 `
if test "$check" != "-"
then
FILELIST="$FILELIST $parm"
fi
fi
done
return
}
showFiles()
{
(
$date +"%Y %m %d"
$find $FILELIST -xdev -type f -ls
) |
$awk '
NR == 1 {
thisyear = $1
lastyear = $1
thismonth = substr($2+100,2,2)
thisday = substr($3+100,2,2)
}
NR != 1 {
day = substr($9+100,2,2)
if ( $8 == "Jan" ) { month = "01" }
if ( $8 == "Feb" ) { month = "02" }
if ( $8 == "Mar" ) { month = "03" }
if ( $8 == "Apr" ) { month = "04" }
if ( $8 == "May" ) { month = "05" }
if ( $8 == "Jun" ) { month = "06" }
if ( $8 == "Jul" ) { month = "07" }
if ( $8 == "Aug" ) { month = "08" }
if ( $8 == "Sep" ) { month = "09" }
if ( $8 == "Oct" ) { month = "10" }
if ( $8 == "Nov" ) { month = "11" }
if ( $8 == "Dec" ) { month = "12" }
if ( length($10) == 4 )
{
year = $10
hour = "00:00"
}
else
{
if ( month <= thismonth ) { year = thisyear }
if ( month > thismonth ) { year = lastyear }
hour = $10
}
print year "/" month "/" day, hour, $0
}
' |
$sort $REVERSE |
$cut -c17-
return
}
#
# Start of main processing
#
SCRIPT=`$basename $0 `
PARMS="$*"
showFiles
exit 0
Once you have discovered the cause of your space problem, it may be best to copy the offending file elsewhere and then clear down the offending file, ie:
cat /dev/null > /filename
This has the advantage of freeing the disk space immediately, which removing a file will not do if the file is open to a running process. --
| Mike Nixon
| Unix Admin
| ----------------------------
After investigating numerous space problems over the years I have written the following scripts to ease the problem.
The First script is dfsort. All this script does is sort your filesystems into order of percentage space used. This immediately shows you where the problem is.
dfsort Script
#!/usr/bin/ksh
#
# Script: dfsort
# Aim: Show filesystems sorted by percentage full
#
awk="/usr/bin/awk"
basename="/usr/bin/basename"
df="/usr/bin/df"
egrep="/usr/bin/egrep"
grep="/usr/bin/grep"
sort="/usr/bin/sort"
tail="/usr/bin/tail"
uname="/usr/bin/uname"
#
# Functions
#
initialisation()
{
for parm in $PARMS
do
case "$parm" in
-k) DFOPT="-k" ;;
*) : ;;
esac
done
return
}
showAIXspace()
{
$df $DFOPT |
$tail +2 |
$grep -v ":" |
$grep "^/dev" |
$awk '{
space = substr($4,1,length($4)-1)
inode = substr($6,1,length($6)-1)
if ( space > inode )
{
print space, $0
}
else
{
print inode, $0
}
}' |
$sort -t" " +0 -1 -n |
$awk -v bs=$DFOPT 'BEGIN {
if ( bs == "" )
{
print "Filesystem 512-blocks Free %Used Iused %Iused
Mounted on"
}
else
{
print "Filesystems 1024-blocks Free %Used Iused %Iused
Mounted on"
}
}
{
printf "%-15s %9s %9s %7s %7s %7s %-s\n", $2, $3, $4, $5, $6, $7, $8
}'
return
}
showSpace()
{
OS=`$uname `
case "$OS" in
AIX) showAIXspace ;;
SunOS) showSUNspace ;;
*) : ;;
esac
return
}
showSUNspace()
{
$df -k |
$tail +2 |
$grep -v ":" |
$egrep -e "^/dev/|^/dev/|^swap" |
$awk '{
space = substr($5,1,length($5)-1)
print space, $0
}' |
sort -t " " +0 -1 -n |
$awk 'BEGIN {
print "Filesystem 1024-blocks Used Avail %Used
Mounted on"
}
{
printf "%-25s %9s %9s %9s %7s %-s\n", $2, $3, $4, $5, $6, $7 }'
return
}
#
# Start of main processing
#
SCRIPT=`$basename $0 `
PARMS="$*"
initialisation
showSpace
exit 0
Once the problem Filesystem has been identified, the fisrt method of attack is to determine the largest files within that filesystem. Use the script below finds. This script lists all the files within the filesystem in size order. I usually pipe the output through tail just to list the largest files, eg:
finds /usr | tail -10
Here is the finds script:
#!/usr/bin/ksh
#
# Script: finds
# Aim: List files in a filesystem by size
#
awk="/usr/bin/awk"
basename="/usr/bin/basename"
cut="/usr/bin/cut"
find="/usr/bin/find"
sort="/usr/bin/sort"
#
# Functions
#
checkParms()
{
FILELIST=""
for parm in $PARMS
do
if test "$parm" = "-r"
then
REVERSE="r"
else
check=`echo "$parm" | $cut -c1 `
if test "$check" != "-"
then
FILELIST="$FILELIST $parm"
fi
fi
done
return
}
listFiles()
{
$find $FILELIST -xdev -type f -ls |
$awk '{ print $7 ": " $0 }' |
$sort -t: +0 -1 -n$REVERSE |
$awk '{ printf "%7s %4s %10s %2s %-8s %-8s %10s %3s %2s %5s %s\n", $2,
$3, $4, $5, $6, $7, $8, $9, $10, $11, $12 }'
return
}
#
# Start of main processing
#
SCRIPT=`$basename $0 `
PARMS="$*"
checkParms
listFiles
exit 0
If this does not immediately highlight the cause of the problem, my next line of attack is to list the files in the filesystem sorted by modification time, in the hope that the files most recently changed are the cause of the problem. I use findt listed below.
Findt script
#!/usr/bin/ksh
#
# Script: finds
# Aim: List files in a filesystem by modification time
#
awk="/usr/bin/awk"
basename="/usr/bin/basename"
cut="/usr/bin/cut"
date="/usr/bin/date"
find="/usr/bin/find"
sed="/usr/bin/sed"
sort="/usr/bin/sort"
#
# Functions
#
checkParms()
{
FILELIST=""
for parm in $PARMS
do
if test "$parm" = "-r"
then
REVERSE="r"
else
check=`echo "$parm" | $cut -c1 `
if test "$check" != "-"
then
FILELIST="$FILELIST $parm"
fi
fi
done
return
}
showFiles()
{
(
$date +"%Y %m %d"
$find $FILELIST -xdev -type f -ls
) |
$awk '
NR == 1 {
thisyear = $1
lastyear = $1
thismonth = substr($2+100,2,2)
thisday = substr($3+100,2,2)
}
NR != 1 {
day = substr($9+100,2,2)
if ( $8 == "Jan" ) { month = "01" }
if ( $8 == "Feb" ) { month = "02" }
if ( $8 == "Mar" ) { month = "03" }
if ( $8 == "Apr" ) { month = "04" }
if ( $8 == "May" ) { month = "05" }
if ( $8 == "Jun" ) { month = "06" }
if ( $8 == "Jul" ) { month = "07" }
if ( $8 == "Aug" ) { month = "08" }
if ( $8 == "Sep" ) { month = "09" }
if ( $8 == "Oct" ) { month = "10" }
if ( $8 == "Nov" ) { month = "11" }
if ( $8 == "Dec" ) { month = "12" }
if ( length($10) == 4 )
{
year = $10
hour = "00:00"
}
else
{
if ( month <= thismonth ) { year = thisyear }
if ( month > thismonth ) { year = lastyear }
hour = $10
}
print year "/" month "/" day, hour, $0
}
' |
$sort $REVERSE |
$cut -c17-
return
}
#
# Start of main processing
#
SCRIPT=`$basename $0 `
PARMS="$*"
showFiles
exit 0
Once you have discovered the cause of your space problem, it may be best to copy the offending file elsewhere and then clear down the offending file, ie:
cat /dev/null > /filename
This has the advantage of freeing the disk space immediately, which removing a file will not do if the file is open to a running process. --
| Mike Nixon
| Unix Admin
| ----------------------------
- Thread starter
- #5
/home is easy to take care of... just send an email to the user(s) whose files you rm/compress/archive. Users are hardly ever cooperative since they absolutely need everything in their directories in order to keep the corporation from going belly-up, so sometimes you have to use your best judgement. I have found that those with ftp access sometimes like to store images of their C: drives on the AIX servers, but thankfully these sort of events are usually limited to the Win2k and Novell servers.
I could make a career out of filespace issues if I wanted to, so if I am unable to handle a complaint for whatever reason (I usually have to work on "real" issues) I ask them to please clean up their files, and that usually works when they can no longer write to their home. Explaining that DASD does not grow on trees is usually ineffective until there is some direct impact to the client.
I could make a career out of filespace issues if I wanted to, so if I am unable to handle a complaint for whatever reason (I usually have to work on "real" issues) I ask them to please clean up their files, and that usually works when they can no longer write to their home. Explaining that DASD does not grow on trees is usually ineffective until there is some direct impact to the client.
- Thread starter
- #7
aixmurderer
IS-IT--Management
And of course a simple "du" will list all your file sizes in a directory and its subdirectories as well. IBM Certified Specialist - MQSeries
IBM Certified Specialist - AIX 5 pSeries System Administration
IBM Certified Specialist - AIX 5 pSeries System Administration
- Status