file system inconsistency (problem with free space) 1

[Qwerty1979]

Hello,

df reports 50M free,

> df -k /usr

/dev/hd2 1007616 51576 95% 34606 14% /usr

then i try to copy 30M file

> cp anybigfile /usr/test

cp: tst1: There is not enough space in the file system.

df still reports free space...

> df -k /usr

/dev/hd2 1007616 47528 96% 34607 14% /usr

any ideas ?

the system is aix 4.3.3.0

Any help will be appreciated !

Regards: Filip

 

[PSD]

Filip,

It sounds like you have a corruption is /usr.

You could try fsck /usr - but you will of course not be able to unmount it.

The easiest way to fix this is to reboot.

Cheers

PSD
IBM Certified Specialist - AIX V4.3 Systems Support
IBM Certified Specialist - AIX V4 HACMP

 

[PSD]

Filip,

It sounds like you have a corruption in /usr.

You could try fsck /usr - but you will of course not be able to unmount it.

The easiest way to fix this is to reboot.

Cheers

PSD
IBM Certified Specialist - AIX V4.3 Systems Support
IBM Certified Specialist - AIX V4 HACMP

 

[Qwerty1979]

Hi there

I did reboot ... twice...

does aix fscks the filesystems at startup ?

Maybe i should boot from a CDrom and fsck /usr ?

Is that possible ? I'd do that with Linux, but i'm not sure how can one perform such things in AIX

Thanks for the help,
Filip

 

[aixqueen]

Note: By default, the /, /usr, /var, and /tmp file systems have the check attribute
set to False (check=false) in their /etc/filesystem stanzas.
===========================
Service mode is the best and then

1.Power down your system. If it has a key mode select switch (that is, if it is a
Microchannel-based system), turn the key to the Service position.

2.With bootable media of the same version and level as the system, boot
the system.

The bootable media can be any one of the following:
Bootable CD-ROM
NON_AUTOINSTALL mksysb
Bootable Install Tape

Follow the screen prompts to the following menu:

Welcome to Base Operating System
Installation and Maintenance

3.Choose Start Maintenance Mode for System Recovery (Option 3).
The next screen has the Maintenance Menu.
Choose Access a Root Volume Group (Option 1).

The next screen displays a warning that indicates you will not be able to
return to the Base OS menu without rebooting.
Choose 0 continue.

The next screen displays information about all volume groups on the system.
Select the root volume group by number.

Choose Access this volume group and start a shell before mounting the
filesystems (Option 2).

If you get other errors from the preceding option, do not continue with the
rest of the steps in this document.
Correct the problem causing the error.

4.Next, run the following commands to check and repair file systems.
(The -y option gives fsck permission to repair file systems when necessary.)

fsck -y /dev/hd2

5.Type exit. The file systems will automatically mount after you type exit.




JUST a thought...you are trying to copy the file to that directory as root right?
Because if you are a normal user...and trying to copy, the system reserves space
and you cannot write to 100%.

You can fsck, I would increase the space manually for the filesystem
and see if it then writes.

 

[ElgisRamon]

AIX by default checks critical file systems. Don't you believe me? look at your [tt]/etc/rc.boot[/tt], you can look for commands like
[tt]
fsck /dev/hd4
mount /dev/hd4 /mnt

fsck /dev/hd2
mount /dev/hd2 /usr
[/tt]
...
I hope it works...
Unix was made by and for smart people.

 

[Qwerty1979]

aixqueen,

i do try this being root...
i already tried increasing the file system and then i can write to the space i added, but still it says there are 40+Megs left and also says 'not enough space'...

there isn't anything but a network/power cables attached to the server so i doubt i can get into maintenance mode

could i just set the check=true in /etc/filesystems and reboot ? or add the necessary 'fsck' command in say /etc/rc file ? At what point is the /usr system mounted ? it is after a file system check according to /etc/rc ... only the fsck is run with the -f switch and it seems to be the one that skips filesystems that were unmounted clearly...

Thanks for the help,
Filip

 

[Qwerty1979]

i read the manual (i do it when everything fails

seems i cannot just set the check = true for the /usr
system and the only option left is booting from a CD (and i really doubt there is one around here) <sigh>

Filip

 

[aixqueen]

Note: By default, the /, /usr, /var, and /tmp file systems have the check
attribute set to False (check=false) in their
/etc/filesystem stanzas.

So let me think, you say you do not have original media to be able to get into
service mode. This is a NIM boot machine?
IS /usr nfs mounted?
It is best to get into service mode and run the fsck on /usr.

One other thing to look at?

lsfs -q and see if the lv is the same as the filesystem?
(lv size: 3276800, fs size: 3276800,)
sometimes people increase the lv but not the file system....and the two are not
equal in size which means you increased the logical volume but you cannot use
the space for the filesystem?

 

[Qwerty1979]

aixqueen,

lsfs -q says

Name Nodename Mount Pt VFS Size Options Auto Accounting
/dev/hd2 -- /usr jfs 2015232 -- yes no
Filesystem Helper: 0506-519 Device open failed

i already tried increasing the filesystem, so i beleive if someone just increased the logical volume, the chfs should have reclaimed that space.... and i beleive that if you just increase the logical volume, the added space won't show up in 'df'...

i might be able to get the CD, but the computer ain't got keyboard/monitor, and my boss' attitude towards this is that we just leave it as it is, instead of fiddling with the machine...

what is a NIM boot machine ?

no, the /usr is not NFS mounted...

i'm afraid if i leave it this way it might lead to something worse... for now it's just annoying not to know when the filesystem is running low on free space, but if it wents kaboom it'll proabably cost me a sleepless night or two

Thanks for the help,
Filip

 

[aixqueen]

Whoa...

you are not supposed to get an error with the lsfs -q command as you did...

http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/aixprob/msgdnref/7digitrecov.htm#BA89AD1338janb

YOu have a corrupt filesystem...it looks like.....and at some point you may not be
able to recover......Hopefully you will find your media....and also make sure they
are doing a mksysb regularly.......and you have their data backed up.....


0506-519
Device open failed
Possible Causes

You tried to mount or unmount a file system that may be damaged.
You ran the ff or fsck command on a file system after modifying the
/etc/filesystems file and specifying an incorrect device.

You attempted to run the fsck command on a cdrfs (CD-ROM)
file system.

Procedures for Recovery

Run the fsck FileSystem command on the file system to check whether there
are errors in the output.

1.If the fsck output displays errors, reboot the system from the boot media.
Select the Maintenance option, then select Access a Root Volume Group.

2.After identifying the correct volume group, select Access this Volume
Group and start a shell before mounting filesystems.

3.Run the fsck command on all the file systems, including the damaged one.
4.Enter exit after all the file systems are checked.
5.Shut down the system:
shutdown -Fr

Ensure that the device specified in the /etc/filesystems file is correct, or use
SMIT to remove and then recreate the file system.

Check the /etc/filesystem file for the FileSystem stanza. If the vfs field is
crdfs, do not run the fsck command.