File system getting Full but no such large files to fill it

[aixbuff]

Hi

I have a filesystem /usr/lpp/tracker which gets 100% full,
Now,the mounting of /usr/lpp/tracker and /usr/lpp/tracker/tmp is in correct order i.e /usr/lpp/tracker/tmp is mounted after /usr/lpp/tracker as specified in the /etc/filesystems,I have unmounted and checked the /usr/lpp/tracker/tmp and found no files filed in the /usr/lpp/tracker/tmp directory.
1.df -k|grep tracker

/dev/lv44 393216 58428 86% 181 1% /usr/lpp/tracker
/dev/lv09 3145728 3046628 4% 75 1% /usr/lpp/tracker/tmp

2.du -k output for under /usr/lpp/tracker filesystem
du -k
40 ./deinstl
4 ./tmp/lost+found
4 ./tmp/save
316 ./tmp
2096 ./bin
4 ./doc
952 ./log
4 ./methods
4 ./catalog
16 ./samples
32 ./etc
4 ./nls/msg/en_US
16 ./nls/msg/prime
24 ./nls/msg
272 ./nls/loc/iconvTable
276 ./nls/loc
304 ./nls
3788 .

3.du -k output for under /usr/lpp/tracker/tmp filesystem

4 ./lost+found
4 ./save
316 .

What is the cause,where are the hidden files?
AIX gurus any ideas.

Thanks in adv
AIXBUFF


AIXBUFF

 

[MoshiachNow]

HI,

1.You have to post the same checks WHILE the FS is 100%,now it's not as per your data.

2.Also ,when it's full,please run the following to check for any file larger then 1 MB:

find /usr/lpp/tracker/tmp -xdev -size +2048 -ls|sort -rn +6|head -10

3.Please post the output of:

lsvg -l <VG>

where <VG> is the VGname in question.
Thanks

&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[aixbuff]

Hi levw,

You are right the data does not say the FS is 100 % full as I have increased the filesystem size.

Here is what you asked for..
lslv lv44
LOGICAL VOLUME: lv44 VOLUME GROUP: rootvg
LV IDENTIFIER: 00508fcacb313e64.14 PERMISSION: read/write
VG STATE: active/complete LV STATE: opened/syncd
TYPE: jfs WRITE VERIFY: off
MAX LPs: 512 PP SIZE: 16 megabyte(s)
COPIES: 2 SCHED POLICY: parallel
LPs: 24 PPs: 48
STALE PPs: 0 BB POLICY: relocatable
INTER-POLICY: minimum RELOCATABLE: yes
INTRA-POLICY: middle UPPER BOUND: 32
MOUNT POINT: /usr/lpp/tracker LABEL: /usr/lpp/tracker
MIRROR WRITE CONSISTENCY: on
EACH LP COPY ON A SEPARATE PV ?: yes

> lslv lv09
LOGICAL VOLUME: lv09 VOLUME GROUP: data01vg
LV IDENTIFIER: 00508fca3d55f535.3 PERMISSION: read/write
VG STATE: active/complete LV STATE: opened/syncd
TYPE: jfs WRITE VERIFY: off
MAX LPs: 512 PP SIZE: 128 megabyte(s)
COPIES: 1 SCHED POLICY: parallel
LPs: 24 PPs: 24
STALE PPs: 0 BB POLICY: relocatable
INTER-POLICY: minimum RELOCATABLE: yes
INTRA-POLICY: middle UPPER BOUND: 32
MOUNT POINT: /usr/lpp/tracker/tmp LABEL: /usr/lpp/tracker/tmp
MIRROR WRITE CONSISTENCY: on
EACH LP COPY ON A SEPARATE PV ?: yes


lsvg rootvg
VOLUME GROUP: rootvg VG IDENTIFIER: 00508fcacb313e64
VG STATE: active PP SIZE: 16 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 1084 (17344 megabytes)
MAX LVs: 256 FREE PPs: 140 (2240 megabytes)
LVs: 16 USED PPs: 944 (15104 megabytes)
OPEN LVs: 15 QUORUM: 1
TOTAL PVs: 2 VG DESCRIPTORS: 3
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 2 AUTO ON: yes
MAX PPs per PV: 1016 MAX PVs: 32

lsvg data01vg
VOLUME GROUP: data01vg VG IDENTIFIER: 00508fca3d55f535
VG STATE: active PP SIZE: 128 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 54 (6912 megabytes)
MAX LVs: 256 FREE PPs: 29 (3712 megabytes)
LVs: 2 USED PPs: 25 (3200 megabytes)
OPEN LVs: 2 QUORUM: 2
TOTAL PVs: 1 VG DESCRIPTORS: 2
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 1 AUTO ON: yes
MAX PPs per PV: 1016 MAX PVs: 32

Now I cant wait for it again to get 100% full,your suggestion please

Thanks
AIXBUFF


AIXBUFF

 

[MoshiachNow]

HI,

Please post exactly the requested checks - the above posting is not very rellevant.
Thanks

&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[aixbuff]

Sorry about that
1.The query to the below command does not have any output
find /usr/lpp/tracker/tmp -xdev -size +2048 -ls|sort -rn +6|head -10

NO OUTPUT

Any further views

Thanks in ADV
AIXBUFF

AIXBUFF

 

[MoshiachNow]

Again,at the time when the system thinks the FS is 100% full:


1.df -k
2.mount
3.lsvg -l <yourVG>

How do you know it gets 100% full ?


&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[aixbuff]

levw,

This file system is for a application called tracker which require two filesystem /usr/lpp/tracker and /usr/lpp/tracker/tmp. This application behaves intermittently as soon as the usr/lpp/tracker get 95% full.

I increased the file system to take it to 85% of its usage and its behaviour is ok,this is what i have been doing,however when i do du -k i find the total size of files far too less than the size of the filesystem.

This two file system are on different VG the out put I have already provided.As I said I have increased the FS it is 86% and not 99% or 100% now hence the above log shows 86%.

Thanks in adv for any clue.

Aixbuff

AIXBUFF

 

[d3vzero]

Hi aixbuff,

I have seen this kind of behaviour where a process holds an open file descriptor for a file which has been removed (unlinked).

If you stop your application, so freeing its hold on that file descriptor, you may find that your df stats all of a sudden marry with your du stats.

Worth a try anyway ;-)

You could also use lsof -p PID, to see the Open Files that are held by any of your application processes.

Best of Luck.

 

[hfaix]

I agree with d3vzero.

I had this happen about a month ago and went through the same steps you did. Check to see if you have any hung processes or a process writing to this directory. Stop anything writing here and try your du/df commands again.