read from data file, special format

[sloebig]

Hi,

I need to read data from a file with a specific format. The program works fine up to the point where I want to read the specific data (open file, find row, etc.). Here is an example of how the data looks like:

...
1842:-3.45453139410687e-01,1.60844170520307e-01,-4.81282605408422e-02
1843:-3.35578544444257e-01,7.27210361860993e-02,-2.44364313569354e-02
1844:-3.33535918869363e-01,-1.41243420023581e-03,1.14044249262977e-03
1845:-3.39507325625196e-01,-6.45503538352583e-02,3.39581359009246e-02
...

So, each line starts with an integer (ranging from 0 to n) followed by a ":" and three real values in exponential form separated by commas. I don't really need the integer, only the real values are of interest to me.

What would be the correct READ statement for that?
Something like "READ(<filenumber>, ???)" variable1, variable2,...?

Many thanks in advance for your help!

 

[salgerman]

What have you tried?

 

[gummibaer]

I guess, the ':' will cause some trouble.

Is it allways in the same place ? Then you could just skip the charcters up to this place in the read format

read (10,'(5x, 3e25.0)')variable_1, varaiable_2, variable_3

If the ':' is not at the same place all the time, I would read the record into a string variable, search for the ':' and then read from there onward. This would look like this

implicit none
character*80 cBuffer    !big enough for a complete recird
real variable_1, variable_2, variable_3
integer iPosition
integer irslt

open (unit = 10, ......)  ! your file data
do
    read (10, *, iostat = irslt) cBuffer
    if (irslt .ne. 0) exit
    iPosition = index (cBuffer, ':') + 1
    read (cBuffer (iPosition :), *) variable_1, variable_2, variable_3
!
!   use the variables for evaluation
!
enddo

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[gummibaer]

Yes, German, you are right to ask first.
But it has been so quiet here the last couple of days, I just kind of exploded...

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[mikrom]

Instead of boring with specific file format I would first preprocess the file, i.e. transform to a format which could be read with fortran reliably.
There are several utilities suitable for this purpose like sed, awk, ..etc..

I found some older threads:

http://www.tek-tips.com/viewthread.cfm?qid=1599136

http://www.tek-tips.com/viewthread.cfm?qid=1585716

 

[mikrom]

As gummibär said that there is so quiet here, I post this little example to demonstrate what I said above:

sloebig.f95

[COLOR=#a020f0]program[/color] sloebig
  [COLOR=#2e8b57][b]integer[/b][/color] :: stat, num
[COLOR=#2e8b57][b]  real[/b][/color] :: var_01, var_02, var_03
  [COLOR=#0000ff]! command string - using sed to replace unwanted characters with space[/color]
  [COLOR=#2e8b57][b]character[/b][/color]([COLOR=#804040][b]*[/b][/color]), [COLOR=#2e8b57][b]parameter[/b][/color] :: cmd_string [COLOR=#804040][b]=[/b][/color] [COLOR=#804040][b]&[/b][/color]
     [COLOR=#ff00ff]'sed -e "s/[:,]/\ /g" sloebig0.txt > sloebig.txt'[/color]

  [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]"Preprocessing file"[/color]
  cmd_rc [COLOR=#804040][b]=[/b][/color] system (cmd_string)
  [COLOR=#0000ff]! if command not succesfull then stop the program[/color]
  [COLOR=#804040][b]if[/b][/color] ( cmd_rc [COLOR=#804040][b].ne.[/b][/color] [COLOR=#ff00ff]0[/color] ) [COLOR=#804040][b]stop[/b][/color] [COLOR=#ff00ff]'system: Error - command failed !!!'[/color]

  [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]"Reading file"[/color]

  [COLOR=#0000ff]! open input file[/color]
  [COLOR=#804040][b]open[/b][/color]([COLOR=#ff00ff]10[/color],[COLOR=#804040][b]file[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'sloebig.txt'[/color],[COLOR=#804040][b]status[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'old'[/color],[COLOR=#804040][b]iostat[/b][/color][COLOR=#804040][b]=[/b][/color]stat)
  [COLOR=#804040][b]if[/b][/color] (stat [COLOR=#804040][b].ne.[/b][/color] [COLOR=#ff00ff]0[/color]) [COLOR=#804040][b]then[/b][/color]
    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]'File cannot be opened !'[/color]
    [COLOR=#804040][b]go to[/b][/color] [COLOR=#ff00ff]99[/color]   
  [COLOR=#804040][b]end if[/b][/color]

  [COLOR=#804040][b]do[/b][/color]
    [COLOR=#804040][b]read[/b][/color]([COLOR=#ff00ff]10[/color],[COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]end[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#804040][b]99[/b][/color], [COLOR=#804040][b]iostat[/b][/color][COLOR=#804040][b]=[/b][/color]stat) num, var_01, var_02, var_03
    [COLOR=#804040][b]if[/b][/color] (stat [COLOR=#804040][b].ne.[/b][/color] [COLOR=#ff00ff]0[/color]) [COLOR=#804040][b]then[/b][/color]
      [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]'Error reading data !'[/color]
      [COLOR=#804040][b]go to[/b][/color] [COLOR=#ff00ff]99[/color]
    [COLOR=#804040][b]end if[/b][/color]
    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]'var_01='[/color], var_01, [COLOR=#ff00ff]', var_02='[/color], var_02, [COLOR=#ff00ff]', var_03='[/color], var_03   
  [COLOR=#804040][b]end do[/b][/color]

  [COLOR=#0000ff]! close file[/color]
[COLOR=#804040][b]  99 continue[/b][/color]
  [COLOR=#804040][b]close[/b][/color] ([COLOR=#ff00ff]10[/color])
[COLOR=#a020f0]end program[/color] sloebig

Now assuming I got this file
sloebig0.txt

1842:-3.45453139410687e-01,1.60844170520307e-01,-4.81282605408422e-02
1843:-3.35578544444257e-01,7.27210361860993e-02,-2.44364313569354e-02
1844:-3.33535918869363e-01,-1.41243420023581e-03,1.14044249262977e-03
1845:-3.39507325625196e-01,-6.45503538352583e-02,3.39581359009246e-02

the program above first calls sed which replaces ':' and ',' with spaces and creates the file
sloebig.txt

1842 -3.45453139410687e-01 1.60844170520307e-01 -4.81282605408422e-02
1843 -3.35578544444257e-01 7.27210361860993e-02 -2.44364313569354e-02
1844 -3.33535918869363e-01 -1.41243420023581e-03 1.14044249262977e-03
1845 -3.39507325625196e-01 -6.45503538352583e-02 3.39581359009246e-02

which could then be processed with Fortran without problems:

$ g95 sloebig.f95 -o sloebig

$ sloebig
 Preprocessing file
 Reading file
 var_01= -0.34545314 , var_02= 0.16084418 , var_03= -0.048128262
 var_01= -0.33557853 , var_02= 0.072721034 , var_03= -0.024436431
 var_01= -0.3335359 , var_02= -0.0014124342 , var_03= 0.0011404424
 var_01= -0.3395073 , var_02= -0.064550355 , var_03= 0.033958137

 

[salgerman]

Norbert: :-D ...thanks for the explanation, I was wondering why you got ahead of yourself.

mikron: With all due respect, but I wouldn't bother with a system call inside the Fortran program...what's the point? What if you are just putting together the fortran program, debugging, recompiling, and running again? There is no need to convert the original file, over and over and over....just once from the command line and that's it...and while you may know that, the inexperience OP may not.

Or, like Norbert suggest...a fortran program well written that know how to read exactly the original file without pre-processing.

 

[mikrom]

but I wouldn't bother with a system call inside the Fortran program...what's the point?

I placed sed in fortran, because we are here in Fortran forum
Normally I would preprocess the file outside the fortran program.

The point is, that the input file could be more complicated and IMHO fortran doesn't have very strong string processing facilities, so using an utility for this purpose seems to be simpler and more reliable.

 

[sloebig]

Thank you all for your help!

The ":" was really what troubled me the most since its position depends on the integer...
Your solutions seem all quite promising! I'll try your solutions on monday and give you a quick reply what worked for me best

See you and have a nice sunday!

 

[gummibaer]

... and what would be the advantage of the preprocessing approach ? It sure does not look more simple to code than a proper read statement if you include it into the fortran code.

And the seperate preprocessing before running the prog might be feasible if you only have very few users that know what they are doing, and operate on the same file most of the time. But if you design your program for external users, you would add just an additional complication, that the user has to observe to gain valid results. I would not want to have the unnecessary admin task to make sure that the preprocessed inputfile is up to date.

If your program is to be designed at least a little bit user firendly, then it should check if the current input file is up to date, e.g. by comparing the creation dates or save dates of the two files. And all of this compared to a read procedure, that consists of two additional statements ?

No harm intended, just my two cents.

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[mikrom]

Hi gummibaer,
I tried the example you posted with the data file provided by sleobig, but I got Fortran runtime error: End of file
on line

read (cBuffer (iPosition :), *) variable_1, variable_2, variable_3

I guess that the format used in read statement doesn't match the format of data in the file.

 

[mikrom]

... and what would be the advantage of the preprocessing approach ?

With the preprocessing approach the Fortran program gets data in easy-to-process format.

Consider, the input file would be a little bit more messy - something like this:
sloebig0.txt

1842 : -3.45453139410687e-01, 1.60844170520307e-01, -4.81282605408422e-02
1843 :     -3.35578544444257e-01,  7.27210361860993e-02,-2.44364313569354e-02
1844:-3.33535918869363e-01,-1.41243420023581e-03;1.14044249262977e-03
-3.39507325625196e-01;-6.45503538352583e-02; 3.39581359009246e-02

Now, to process this file, I only have to do little changes in the example posted above - i.e. in preprocessing part and in READ-statement:

...
character(*), parameter :: cmd_string = &
  'sed -e "s/^.*:\ *//g; s/[,;]\ */\ /g" sloebig0.txt > sloebig1.txt'
...
...
read(10,*,end=99, iostat=stat) var_01, var_02, var_03
...

Now, the preprocessing with sed creates the file
sloebig1.txt

-3.45453139410687e-01 1.60844170520307e-01 -4.81282605408422e-02
-3.35578544444257e-01 7.27210361860993e-02 -2.44364313569354e-02
-3.33535918869363e-01 -1.41243420023581e-03 1.14044249262977e-03
-3.39507325625196e-01 -6.45503538352583e-02 3.39581359009246e-02

which contains only the essential data, is properly formatted and therefore easy to process with Fortran.

IMHO it's not more effort, then code this in Fortran. However it's my personal opinion and you can disagree

 

[gummibaer]

Ooops,

list directed reading of character constant from file got me again.

implicit none

character*80 cBuffer    !big enough for a complete recird
real variable_1, variable_2, variable_3
integer iPosition
integer irslt
open (unit = 10, file = 'input.txt')  ! your file data
do    
!
!   read (10, *, iostat = irslt) cBuffer    
!
    read (10, '(a80)', iostat = irslt) cBuffer    ! field width to correspond with the number of characters in cBuffer
    if (irslt .ne. 0) exit    
    Position = index (cBuffer, ':') + 1    
    read (cBuffer (iPosition :), *) variable_1, variable_2, variable_3

! usage of variables

enddo

List directed input of character constants is tricky as ',' is interpreted as delimiter. The original read from file read only to the first ',' and therefore the second read encountered the end of the record. With the a80-format this code works.

As to the proprocessing:
You are right, to deal with such a messy file would not be that easy in fortran - luckily the files I work on are created by pc-output and usually are organised very well .

Still I would hate to code, test and debug that call to sed.

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[mikrom]

Hi gummibaer,
Thank yoou very much, now your example works for me. I mistakenly thought that the error was in the

read (cBuffer (iPosition :), *) variable_1, variable_2, variable_3

but it was in the

read (10, *, iostat = irslt) cBuffer

statement.

Althought I have some experience with Fortran, using formats with read steatements makes me difficulties

Still I would hate to code, test and debug that call to sed.

Yes you are right: I first tested the call on command line and then placed it in fortran system() call. As I wrote above I would normally call sed outside of the program. It's in the example above only for the fortran programmrs who are intersted to try it.

Maybe the regular expressions used in utilities like sed, awk, perl .. etc seems to be a little bit difficult, but they must be learned - it's only handwerk as other things in data processing and when you have to deal with them often you will understand them.

 

[gummibaer]

Well then, let's consider it a matter of mere personal likes or dislikes when you do programs that only you or your colleagues use. But if you have outside users - as I hope I will have when I am through with my project - which will be computer laymen in most cases, then I would not want to impose this on the users.

Norbert


The optimist believes we live in the best of all possible worlds - the pessimist fears this might be true.

 

[sloebig]

Thank you all again for your help!

I tried what seemed to me the most forward and easy solution, i.e. the one from gummibaer. Since I need to process many of these files and there is some other stuff in them in different formats that I need preprocessing them seemed a bit too complicated for me
Now everything works fine and I am a huge step closer to a working code

See you,

Stefan