I/O Line Limit?

[nperlong]

I am wondering if there is a limit to the number of characters you can have per line of an input file? My input file is arranged as follows. 4 lines of header. 40 lines of data. 1 header line. Thousands more lines of data. The regular READ(unit #,*) Header works to get rid of the first 4 header lines and simultaneously skip to the next line, however, when I reach the 2nd header line it ends up reading it twice. Following this apparent hiccup, the data values I see in my arrays for the first line of data after the 2nd header are hogwash and looks like it just grabbed random stuff from memory. These lines of data are ~1200 characters long. Is this an issue? Are there any other limitations to READ ?

Thanks for your help.. I've been slaving over this for a while now..

 

[gummibaer]

I am not aware of any limits. I have been reading up to 4kB of data in one read (unformatted though).

What kind of data are involved ?

You are doing formatted list directed input, so all depends on how the types of the variables you are going to read are declared.

If there are variables of type character involved, this reading can become tricky as every blank is interpreted as delimiter, unless declared otherwise in the open statement.

It would be a good idea to post the crucial parts of the code.

The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[nperlong]

The header lines are type character. If the spaces are delimiters with this type that explains why it only grabs the first word of this line, but I am not sure why its not skipping to the next line. The rest of my data is in x.xxxE+xx format.. declared as REAL. Here is some of my code.

CHARACTER*80 HEADER !CC If the header line is >80 characters will this break the code?


REAL TM1,TM2,NM1,NM2,TFM1,TFM2,TCM1,TCM2,NM,TFM,TCM,TS1,TS2, &
Ekap,Kappa,GAMMLN,FLUXMPAI2(40),FLUXMPAE1(40),EMPA_i(40),FLUXMPA(40),EMPA_e(40),&
FLUXMPAI1(40),FLUXMPAE2(40),EMPA(40),DATA(89)

OPEN(UNIT=16,FILE=NAME//'_MPAFlux.in',status='old')
657 DO I=1,4 ! 4 lines of header material
658 READ(16,*) HEADER
659 END DO
660 DO I=1,40
661 READ(16,*,IOSTAT=L) EMPA_i(I),EMPA_e(I)
662 END DO
663 READ(16,*) HEADER
664 print *, 'HEADER:',HEADER !CCCC Everyone working good up to this point (it got the correct numbers and header from the file)

READ(16,*,IOSTAT=L) HEADER
720 print *, HEADER !!!CCC Problem! Prints out same header again?!?!?
723 READ (16,*,IOSTAT=L) (DATA(I),I=1,89)
725 TM2=DATA(2)
726 DO I=1,89
727 print *, 'DATA(I)', DATA(I) !CCCCCC Prints out random numbers not in the file
728 END DO

Thanks for helping me out!

 

[salgerman]

I think, to start with, you should specify a format when reading header lines that have spaces. In other words, in your READ statement, you should add a spec format of "A80"; otherwise, Fortran is going to grab only the first word. Hopefully, your header lines are not wider than 80 columns.

Will you please post the first 50 lines of your input file?

 

[gummibaer]

CHARACTER*80 HEADER !CC If the header line is >80 characters will this break the code?

No. Your prog will read 80 characters and ignore the rest.

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[nperlong]

Thanks for the help guys... here is the beginning of my input file.

LANL MPA data for Hour of May 24, 2000
UT sequence for midnight-centered LT, +/- 6.0 hours
Adopted Energy Channels
Ion_eV Electron_eV
40924.000 40275.332
31344.000 30753.166
.
.
. (40 of these lines)
.
2.889 1.900
2.257 1.396
1.795 1.073
Sat UT,s LT,h N_p,/cc Tpe_p,eV Tpa_p,eV N_e,/cc Tpe_e,eV Tpa_e,eV En_p,eV En_e,eV
3 4.97232E+03 1.90160E+00 5.55700E-01 7.96500E+03 5.01200E+03 8.63600E-01.... (This goes 89 columns for about 2500 lines)

 

[salgerman]

I just happen to be busy to spare too much time on this at this time...so, here is how I would go about it, it always works for me when debugging:

"Baby steps...baby steps!"

Make a copy of your original file
Delete from the 46th line all the way to the end...just keep the first 45 line of the file.
For the lines after the second header, just keep the first 3 columns of data and delete the rest.
You could even reduce those 40 lines in between to just 5.
Read and write everything.
With this reduced input file, you can post the entire file and even an interactive session to see what the program is reading.
Get back to debugging.

In your reading of the 89 columns of data...do you really want to read 'Sat' into the same DATA() matrix? Sat seems to be the only INTEGER in there, the rest of the values are REAL. Maybe you can read Sat into a 1-column matrix of type INTEGER and even go ahead and specify a format?

Hope this helps. Keep us posted.

 

[gummibaer]

To break down into smaller steps is good advice. Then you can do it step by step.

What beats me is, why you would want to read this second header twice. An if you do, why you receive the same printout as in the first read. If you want to persue this, then I would cluear HEADER immediately before the second read (HEADER = ' ') and see what comes about. If you still receive the same printout I would assume that the second read is not executed. I'd guess - as I do right now anyway - that your program branches off before the read somewhere between line 664 and 720.

Then it might be a good idea, that you initilise DATA with 0.0, because not all compilers do that automatically. Put DATA = 0.0 before the first read, better still right at the beginning of your program. Then run your program and check the printout. If you see only 0.0, then once again, the read statement to input those 2500 lines is not executed and what you get in your printout before are the values that were in the PC-memory before you started your prog.

BTW, in your code between lines 720 and 728 there are some lines missing. What is happening there ?

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[nperlong]

Wow things are getting really weird now... I was trying to avoid posting so much code but I guess it could be useful. When I run this below I get some super strange things in my log file. The log file continues in that while loop until an underflow occurs because its not getting T from the file and it never exits the loop. I dont understand how it is/isnt reading this file.

Also I am not trying to read the header twice, I just noticed that it was for some reason when I was trying to debug.

Here I supplied the log file and lots more code. This is super nice of you guys to spend time helping me on this...

Opening the MPA file
LANL
UT
Adopted
Ion_eV
HEADER:Sat UT,s LT,h N_p,/cc Tpe_p,eV Tpa_p,eV N_e,/cc Tpe_e,eV Tpa_e,eV En_p,eV En_e,eV
CLEARED HEADER:
Success opening MPAFlux.in
Check the MPA data: 0.0000000 -1.0000000 0.0000000 0.0000000 -1.0000000 0.0000000 0.0000000
Update the MPA data variables: 0.0000000 -1.0000000 1 1 1 (these initial MPA data values are good, not read from file though)
I BREAK HERE
2nd header?


DATA(I) 0.0000000
DATA(I) 0.0000000
..... (bunch of zeroes)
DATA(I) 3.6960000E+06 (this is number is in the input file! but its from something like column 80...?!?!)
DATA(I) 0.0000000
...
DATA(I) 0.0000000
DATA(I) 0.0000000
DATA(I) 9.3210000E+06 (this is from the next column after 3.69e6)
DATA(I) 0.0000000
TM2 After If: 0.0000000 T: 0.0000000 TIME: 0.0000000
Im here
New MPA data: 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
I BREAK HERE
2nd header?

DATA(I) 0.0000000
..
DATA(I) 3.1550000E+06 (same column as last step in loop, 1 line down)
DATA(I) 0.0000000
...
etc..

CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

IF (IG7.EQ.1) THEN
631 TS2=TIME-1. ! Prepare SOPA input file
632 TS1=TS2
633 FS2(1:7)=0.
634 OPEN(UNIT=14,FILE=NAME//'_sopa.in',status='old')
635 DO I=1,3
636 READ(14,*) HEADER ! I have 2 files open.. this is ok yeah?
637 END DO
638 END IF
639 TM2=TIME-1. ! Prepare MPA input file
640 TM1=TM2
641 NM2=0.
642 TFM2=0.
643 TCM2=0.
644 NE2=0.
645 TEC2=0.
646 TEF2=0.
647 FLUXMPAI1(1:40)=0.
648 FLUXMPAE1(1:40)=0.
649 FLUXMPAI2(1:40)=0.
650 FLUXMPAE2(1:40)=0.
651 EMPA_e(1:40)=0.
652 EMPA_i(1:40)=0.
653 EMPA(1:40)=0.
654 DATA(1:89)=0. ! DATA IS NOW 0
655 print *, 'Opening the MPA file'
656 IF (I10.EQ.1) THEN
657 OPEN(UNIT=16,FILE=NAME//'_MPAFlux.in',status='old')
658 DO I=1,4 ! 4 lines of header material
659 READ(16,*) HEADER
660 print *, HEADER
661 END DO
662 DO I=1,40
663 READ(16,*,IOSTAT=L) EMPA_i(I),EMPA_e(I)
664 END DO
665 READ(16,30) HEADER
666 30 FORMAT(A150)
667 print *, 'HEADER:',HEADER !(Prints Entire Header perfectly)
668 HEADER = ' '
669 print *, 'CLEARED HEADER:',HEADER !(so far so goood....)
670 print *, 'Success opening MPAFlux.in'
671 ELSE ! (doesnt come in here)
672 OPEN(UNIT=16,FILE=NAME//'_mpa.in',status='old')
673 DO I=1,3 ! 3 lines of header material
674 READ(16,*) HEADER
675 print *, 'uh oh'
676 END DO
677 END IF
678 I2=0
679 IF (S.EQ.1) I2=3
680 END IF
681 END IF
682
683 DO S=1,NS
684
685 DO 5 L=1,LO
686 DO 5 K=1,KO
687 DO 5 J=1,JO
688 5 FGEOS(J,K,L,S)=0.
689
690 IF (SCALC(S).EQ.1) THEN
691
692 IF (TINJ.GT.TIME+2.*DT*NSTEP) THEN ! No injection, use IC for BC
693 DO L=1,LO
694 DO K=1,KO
695 DO J=1,JO
696 FGEOS(J,K,L,S)=F2(IO,J,K,L,S)
697 END DO
698 END DO
699 END DO
700 END IF
701
702 END IF ! SCALC check
703 END DO ! S loop
704
705
706 IF (I10.EQ.1) THEN ! LANL data injection
707 print *, 'Check the MPA data:',T,TM2,NM2,NE2,TM1,NM1,NE1 ! Works...
708 IF (TM2.LT.T) THEN ! MPA DATA
709 K=0
710 print *, 'Update the MPA data variables:',T,TM2,I10,,I11,IG7 ! This also works
711
712 DO WHILE (TM2.LE.T) ! Best if final TM2 > final T
713 K=K+1
714 TM1=TM2
715 NM1=NM2
716 TFM1=TFM2
717 TCM1=TCM2
718 NE1=NE2
719 TEC1=TEC2
720 TEF1=TEF2
721 FLUXMPAI1=FLUXMPAI2
722 FLUXMPAE1=FLUXMPAE2
723 print *, ' I BREAK HERE '
724 READ(16,*,IOSTAT=L) HEADER
725 print *, '2nd header?', HEADER
726
727 ! READ(16,*,IOSTAT=L) HEADER
728 print *, HEADER
729
730 ! READ(16,*,IOSTAT=L) HEADER
731 print *, HEADER
732
733
734 ! READ(16,*,IOSTAT=L) HEADER,HEADER
735 ! print *, HEADER
736 READ (16,*,IOSTAT=L) (DATA(I),I=1,89,40)
737 40 FORMAT (2X,I1,88(2X,ES11.9))
738
739 TM2=DATA(2)
740 DO I=1,89
741 print *, 'DATA(I)', DATA(I)
742 END DO
743 NM2=DATA(4)
744 TFM2=DATA(6)
745 TCM2=DATA(5)
746 NE2=DATA(7)
747 TEF2=DATA(9)
748 TEC2=DATA(8)
749 FLUXMPAI2=DATA(10:49)
750 FLUXMPAE2=DATA(50:89)
751 IF (L.LT.0) TM2=TIME+2*DT*(NSTEP+1)
752 print *, 'TM2 After If: ', TM2, 'T: ', T,'TIME: ', TIME
753 IF (T.EQ.TIME) THEN ! In case T2>T already
754 TM1=TIME
755 NM1=NM2
756 TFM1=TFM2
757 TCM1=TCM2
758 NE1=NE2
759 TEC1=TEC2
760 TEF1=TEF2
761 FLUXMPAI1=FLUXMPAI2
762 FLUXMPAE1=FLUXMPAE2
763 print *, 'Im here'
764 END IF
765 print *, 'New MPA data:',T,TM2,NM2,NE2,TM1,NM1,NE1
766 IF (K.EQ.10) STOP
767 END DO

 

[gummibaer]

DATA(I) 3.6960000E+06 (this is number is in the input file! but its from something like column 80...?!?!)

--> have I printed as well to know where you are.
print *, 'i, data (i)#, i, data(i)

Giving the IOSTAT = ... clause in your read statements suppresses eerormessages. So you should either skip them to receive them or check and printout L after each read to see what might have gone wrong.

READ (16,*,IOSTAT=L) (DATA(I),I=1,89,40)

This reads the first thre values only from each record and saves them to DATA (1), DATA(41), DATA(81), cause the 40 is the increment in I. But this was not part of your original code, was iut ? The format

40 FORMAT (2X,I1,88(2X,ES11.9))

is ignored as label 40 is not included in your read statement. And it would not fit to the output-list of the read as these are all reals whereas the format starts with an integer specifyer. I guess, what you wanted to write was

READ (16,40,IOSTAT=L) i, (DATA(I),I=1,89)
40 FORMAT (2X,I1,88(2X,ES11.9))

In that I am not so sure, if the ES format specifier does not create an error when it encounters en exponent not a multiple of 3. I would use E instead.

That's for now. Keep us posted on what is the result.

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[gummibaer]

Ooops

READ (16,40,IOSTAT=L) i, (DATA(I),I=1,89)
40 FORMAT (2X,I1,88(2X,ES11.9))

i is not so good a choice as it is used elsewhere. Give this integer a sensible name.

One more programming hint (don't know if this has any influence here though):
you have two different files that you open on line 657 and 672 with the same channel. I would assign different channels to be sure from which file I read with read (16, ....)

One request: If you post code, please enclose it with the tags code and /code, each enclosed in [ and ]. This woukd render your code more legibly in keeping the indentures and such.

But that's it for today. Promised.

Norbert

The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[nperlong]

Sorry I am new to the forums, i'll be sure to do that next time. Unfortunately, I am going on a trip for the rest of the week, so I wont be able to work on this project until Monday. I think I am going to be very close to solving this especially with your last couple posts. Thank you so much and I'll let you know how it goes next week.

 

[nperlong]

Hey guys, sorry for the delay I am finally returning to this project.

I changed the old read statement to the following as you suggested.

READ (16,40,IOSTAT=L) j, (DATA(I),I=1,89)
40 FORMAT (2X,I1,88(2X,E11.9))

Now the data array is simply all zeros. I don't understand what the "j," is doing imbetween the parentheses.

Thanks for the help

 

[gummibaer]

3 4.97232E+03 1.90160E+00 5.55700E-01 7.96500E+03 5.01200E+03 8.63600E-01.... (This goes 89 columns for about 2500 lines)

the first item in this line seems to be an integer, so it might be a good idea to save it to an integer variable. But this is a minor problem, just cosmetics.

If you receive all zeros, then your read statement in line 736 never gets executed or it reads the wrong file or create an error.

> If TM2 is greater than T the loop starting line 712 never gets executed.

> In line 657 and 672 you open two different files with the same unit number depending on the value of i10. So have this variable printed

> This iostat = L in the read blocks all errormessages that might result from this read. Skip the iostat = .. ( or printout L after the read) and see what you got.

Norbert




The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[nperlong]

Nice I didn't think to print out L. The first time I ran it it printed L to be 141. I looked this up at it means:


Invalid character in real input field
Integer,Parameter :: IOERR_BAD_INPUT_INTEGER = 141

I then removed the j, from

 READ (16,40,IOSTAT=L) j, (DATA(I),I=1,89)

and it then gives me 134 for L. Which means:

Character string edit descriptor used on input
Integer,Parameter :: IOERR_BAD_EDIT_FOR_REAL = 134

Can you tell what's going on? Furthermore, I checked I10, TM2, and T. Their values are all what they should be 1,-1,and 0 respectively.

-Nick

 

[gummibaer]

The meaning of the numerical is depending on the compiler, so you should check the documentation for any hints on what that means.

But the good news is, we now know that we have an error in the read statement which seems connected to the format.

Some hints from my side:

> doublecheck if there are really 89 columns. You might even try to read only 10 or 20 data to see if it is a problem in the number of elements or in the format itself.

> try list directed input

read (16, *, iostat = L) (data(I), I = 1, 10) ! reduced number of elements as above

and see what happens to the L.

Norbert




The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[nperlong]

I verified that there are indeed exactly 89 columns of data.

I then did what you said and what happened is fairly perplexing. The first entry in Data is 0.000. It then proceeds to fill the next 9 elements with the last 9 data points in the first row of my data. Following this, the rest are set to 0.

L is now reading 104. I couldnt find any documentation on compiler specific iostat codes, but the one I found for f90 (I believe I am using mpif90) says:

Exponent too large for w.d format
Integer,Parameter :: IOERR_INPUT_BUFFER_OVERFLOW = 104

My exponents are no greater than +08 though...

-Nick

 

[gummibaer]

Apparently you are not using the same compiler as me, cause in my docs there is no iostat value of 104 existing.

What I would do now is strip the problem down. That means copy and paste the first items of the first line of data to a new file and write a program just to read this line and print out the result.

The file to look like

3 4.97232E+03

and the prog

real a
integer i
open (16, file=test.dat, status = 'old')
rewind 16
read (16, *) i, a   ! this is lkist directed input, but you may well test format specifications here
print *, i, a
stop
end

and then play around with the format until you have proper results for just these two values. If you achieved this, add the next value and check on what the format has to be to read properly.

From the docs of my compiler I get it, that even the f-format should be able to read your exponential constants (did not try it myself though), so this might be worth a try. Just use something like f15.0 because the position of the colon found in the input record overrides the format defined position of the colon. (I do not know if this is standard FORTRAN, but my compiler works that way.)

As to the i:
If you do not read the first as integer, this may give confusing results.

So a proper format might be

read (16, 40) i, a
40 format (i5, f15.0)

(There is no need to exactly count the blanks and digits, as the format adapts to the actual record, as long as the fields are defined with more digits than present.)

Norbert

The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.