large file reading problem

[holdahl]

Has anyone using aix v5.2 had problems when reading large files from disk? I'm having problems on my AIX servers when reading files around 2GB from a backup disk.

Are there any known problems with AIX on this issue which could be solved by a fix?



-holdahl-

 

[khalidaaa]

What kind of problems do you notice?

 

[holdahl]

basically it makes the database server go 'down'.
system load like cpu and disk becomes very high.

to our users it is like the server is down.


-holdahl-

 

[TSch]

Could the system be swapping ?
How much memory do you have ?

 

[khalidaaa]

What's the vmstat -v after and before the reading?

vmstat 1 during the read?

Regards,
Khalid

 

[RMGBELGIUM]

hmz, the 2 Gb file issues make me think about a ulimit problem somehow ...

rgds,

R.

 

[holdahl]

The backup runs fine, but it use almost all system resources on the AIX machines. (we can see on output from monitoring system that there is a cpu and disk peak at the time when the backup runs)

The same problem on 3 machines which database server resides on.

-holdahl-

 

[RodKnowlton]

What do you mean by "reading files"?

What sort of files are they, what sort of process is reading them, and what sort of reading is it (random vs sequential)?

- Rod


IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+

A Simple Code for Posting on the Web

 

[RMGBELGIUM]

What you're saying is that your backup causes so much cpu and disk IO that the rest of the apps become nearly unavailable ( to high response time for databases )
What you could do , is using WLM ( workload manager ) to restrict your backup application to a limit of cpu and disk IO it can take.

 

[holdahl]

Is there any limitations on the file size on filesystems in AIX?
Still having the problem when reading large files either from disk to tape or disk from one machine to another.

Our database backup files are from a few gigabytes up to around 50 gigabytes.

Could filesystems be enabled for very large files, or is it just the large file enabled option?

-holdahl-

 

[Chapter11]

What's minpage% and maxpage% set to?

 

[khalidaaa]

If you could show me this:

What's the vmstat -v after and before the reading?

vmstat 1 during the read?

Then we can check the pbuf to see if you are running out of pbufs durning the backup! This could be a reason for high IO and CPU wait!

Regards,
Khalid

 

[holdahl]

Here is some output:
---------------------------------------------------------
vmstat -v at 23PM before backup:

1048576 memory pages
997185 lruable pages
138 free pages
1 memory pools
80195 pinned pages
80.1 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
47.2 numperm percentage
471424 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
80.0 maxclient percentage
93 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
514 paging space I/Os blocked with no psbuf
436734 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
--------------------------------------------------------

--------------------------------------------------------
vmstat -v at 05AM during backup:

1048576 memory pages
997185 lruable pages
147 free pages
1 memory pools
80236 pinned pages
80.1 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
79.9 numperm percentage
797608 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
80.0 maxclient percentage
0 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
514 paging space I/Os blocked with no psbuf
440923 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
----------------------------------------------------------

vmstat 1 during the backup:

System Configuration: lcpu=2 mem=4096MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 1 729554 126 0 7 6 575 330 0 809 9705 1669 8 22 65 4
4 0 729547 208 0 71 0 2976 35209 1 1457 22265 2530 14 51 9 25
6 2 729455 128 0 240 0 3007 5002 0 1560 15619 2908 5 40 23 33
2 0 729455 122 0 86 0 2863 4879 0 1375 15746 2606 11 34 33 22
1 0 729455 128 0 52 0 2940 3830 0 1389 18169 2432 10 44 32 15
2 0 729455 120 0 73 0 2825 3672 0 1458 16691 2552 15 39 27 19
1 1 729455 127 0 57 0 2825 3772 0 1380 15537 2577 5 34 40 20
2 0 729455 122 0 62 6 2792 4281 0 1349 19056 2452 9 50 25 15
1 1 729455 128 0 71 5 2845 3633 0 1380 16501 2489 8 42 32 19
2 0 729455 120 0 66 4 2819 4165 0 1375 18234 2494 13 45 26 16
1 1 729455 128 0 63 2 2830 3784 0 1342 17751 2494 6 44 34 17
1 0 729455 128 0 89 0 2953 3713 0 1448 13617 2631 9 28 33 29
3 0 729455 128 0 79 3 2839 3575 0 1369 16317 2500 11 38 34 18
0 1 729455 120 0 91 9 2942 4056 0 1401 16831 2545 11 38 30 20
0 2 729455 128 0 110 19 2894 4281 0 1451 16691 2489 10 45 17 27
1 2 729456 62 0 26 194 2796 3536 0 1574 8530 2037 18 17 21 44
1 1 729456 113 0 5 291 2889 4176 0 1673 5772 2253 47 6 15 32
1 1 729456 120 0 55 53 2766 6849 0 1360 17350 2184 12 47 22 19
2 0 729456 126 0 45 29 2982 4054 0 1399 15932 2475 9 36 37 19
1 0 729456 128 0 57 5 2777 6822 0 1332 17640 2461 9 44 32 15

 

[khalidaaa]

As you can see from the output of vmstat -v

440923 filesystem I/Os blocked with no fsbuf

You ran out of fsbuf during the backup so this might be a reason! I guess you are running under jfs filesystem. You need to increase the numfsbufs.

The other thing, which i don't think it is related to your case, but the memory is not tuned or it is on the default values. If what you are running is oracle database, then there are some recommendation to change the minperm and maxperm

Have a look at this nice link for tips:

http://www.tek-tips.com/faqs.cfm?fid=5685

Regards,
Khlaid