Limitation in number of rows read from a data file

[theodo116]

Dear all,
I am using gfortran and I am trying to read input from a large file (more than 2,100,000 lines...). I am using the following READ statement
DO i = 1, nnz
READ (111,*) a(i), irow(i), icol(i)
write(6,*) 'i=', i, 'a(i)=', a(i)
END DO
where 111 corresponds to the data file name. However, the loop exits after about 20,000 iterations. If I change the write statement within the loop to
write(6,*) 'i=', i
then the loop exits after 27,000 iterations. Any help would be much appreciated.

 

[FJacq]

Few questions :

How could the loop exit without EXIT statement?

Could you give us the real message showing that the loop exits in a strange manner ?

Are the arrays a, irow and ico dimensioned at the right size (at least nnz) ?

Could you print the values of nnz (before the loop) and the index i after the loop without printing anything within the loop ?

Could you add an iostat in the read statement ?

integer :: k
...
write(*,*) 'nnz=',nnz,' array sizes ',size(a),size(irow),size(icol)
do i = 1, nnz
  read(111,*,iostat=k) a(i), irow(i), icol(i)
  if(k /= 0) then
    write(*,*) 'problem at the line ',i
    stop
  endif
enddo
write(*,*) 'i= ',i

François Jacq

 

[theodo116]

Dear Francois,

many thanks for your reply and pardon me for not explaining the problem well. The loop I provided above simply terminates without completion and the code stops running.

The answers to your questions below:

Are the arrays a, irow and ico dimensioned at the right size (at least nnz)? YES

Could you print the values of nnz (before the loop) and the index i after the loop without printing anything within the loop?
Since the LOOP does not complete, then I cannot print the index i
after the loop. nnz has the right value before the loop

Could you add an iostat in the read statement ?

I did what you suggested and I got the following on the screen (I have changed the dimension of the arrays to 600001). No other messages whatsoever:

nnz= 600001 array sizes 600001 600001 600001

It is quite a strange problem! I am using this READ statement as part of a code that employs NAG routines with large matrices. So, it might be an issue related to the memory that is used by the routines... ??? I have contacted NAG and am waiting for their advice as well.

 

[xwb]

Have you looked at the data on the line where it stops? Any strange or spurious characters there or a change of format. There is a possibility that it may be fixed format rather than space separated and at the high numbers, there are no spaces in between so possibly two or three numbers become one and it is reading several lines at a time.

 

[theodo116]

There is no problem with the data file as far as I can see. What puzzles me is that the number of rows that are READ during the loop changes accordingly to the information that the print statement within the loop provides. The less information I print the more rows are read! If there was an issue with the format of the data, this wouldn't happen (the loop would stop at a fixed row location). I am suspecting that this might be a memory problem (maybe related to the NAG routines or the cooperation between gfortran and NAG).

 

[salgerman]

Here is my piece of advice: baby steps...baby steps.

Let's try to work our way towards the final goal.

Let's start with nothing, and progressively start adding stuff.

For starters, do not even declared any variables, except a character array 'line' long enough to read one line a time. Put a while statement that is always true and let's see if you can at least read all lines and reproduce them onto standard output...it may take a while, reduce the height of the terminal to a few lines, this should speed it up...if it takes too long, open another file and see if you can read lines from one file and write them to another.

Then, declare the loop variable 'i'...make sure is not too small, like INTEGER*2 because it will not be able to grow to 2,100,000! Convert the while into a loop using 'i' and see if it still works.

Declare a(), include the reading of it in the loop, but for now follow it with 'line'...see if you can still work.

Then, declare irow() and see if you can start reading that too in the loop...

and so on, you get the picture...

 

[xwb]

Try a mini program with just an open, your read loop and a close. See how far that gets.