several ascii files, open, read and average them

[Taiseer]

Hi,

I have the following problem, can any one help?

I have 744 simple ascii files. Each of them has the following format: One line header and a column of values. The column has 275648 values (rows).

The target is to create 248 simple ascii files (744/3). Each of the new files has the following format:
The same header as in the old files and the same number of rows (275648). But the first new file contains the average of the old files (file1, file2 and file3). The second new files will contain the average of the three old files (file4, file5, and file6).

How can i do this, below is my trial.

*****************************
program testInternalFiles
implicit none
integer i
integer ios

integer, parameter:: inFileUnit = 8
integer, parameter:: outFileUnit = 9

! Buffers to hold file names
character(80) inputFileName
character(80) outputFileName
character(180) skip

do i=1,248
jj=3*i
j=3*i-2
do k=j,jj
!Use internal files to convert integers to string values
! and embed it in the file names
write (inputFileName, '(a, I0, a)') 'NLDAS.VGRD.',k,'.sa'
write (outputFileName, '(a, I0, a)') 'NLDAS.VGRD.',i,'.dat'

open(inFileUnit, file=inputFileName,status="old", iostat=ios)
if (ios /= 0 ) then
write(*, '("Can''t open file ", a, " for reading.")') &
trim(inputFileName)
stop
endif

write(*, '("Opened file ", a, " for reading.")') &
trim(inputFilename)

!
! It will bail out rather than overwrite an existing file.
!
open(outFileUnit, file=outputFileName,status="new", iostat=ios)
if (ios /= 0) then
write(*, '("Can''t open file ", a, " for writing.")') &
trim(outputFileName)
stop
endif

write(*, '("Opened file ", a, " for writing.")') &
trim(outputFilename)
! Here's where you read stuff from the input file
! and write stuff to the output file.

read(inFileUnit,*) skip
read(inFileUnit,*) x
y=
write(outFileUnit,*) skip

! Close files before next pass through the loop
write(*, '("Closing files.")')
close(inFileUnit)
close(outFileUnit)
write(*,*)
enddo

end program testInternalFiles
*****************************************

thanks

 

[mikrom]

I tried it on a mini example with these 9 files:

NLDAS.VGRD.1.sa, NLDAS.VGRD.2.sa, NLDAS.VGRD.3.sa - all containing:

HEADER 1-2-3
123
123
123

NLDAS.VGRD.4.sa, NLDAS.VGRD.5.sa, NLDAS.VGRD.6.sa - all with:

HEADER 4-5-6
456
456
456

NLDAS.VGRD.7.sa, NLDAS.VGRD.8.sa, NLDAS.VGRD.9.sa - all with:

HEADER 7-8-9
789
789
789

The program is here
test_internal_files.f95

[COLOR=#a020f0]program[/color] testInternalFiles
[COLOR=#2e8b57][b]implicit[/b][/color] [COLOR=#2e8b57][b]none[/b][/color]

[COLOR=#2e8b57][b]integer[/b][/color], [COLOR=#2e8b57][b]parameter[/b][/color]:: nr_outfiles [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]3[/color]
[COLOR=#2e8b57][b]integer[/b][/color], [COLOR=#2e8b57][b]parameter[/b][/color]:: inFileUnit [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]8[/color]
[COLOR=#2e8b57][b]integer[/b][/color], [COLOR=#2e8b57][b]parameter[/b][/color]:: outFileUnit [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]9[/color]

[COLOR=#2e8b57][b]integer[/b][/color] :: i, j, k, first, last, idx, ios, nr_values, values([COLOR=#ff00ff]1000[/color])
[COLOR=#2e8b57][b]real[/b][/color]  :: sum_values, avg

[COLOR=#0000ff]! Buffers to hold file names[/color]
[COLOR=#2e8b57][b]character[/b][/color]([COLOR=#ff00ff]80[/color]) inputFileName
[COLOR=#2e8b57][b]character[/b][/color]([COLOR=#ff00ff]80[/color]) outputFileName
[COLOR=#2e8b57][b]character[/b][/color]([COLOR=#ff00ff]20[/color]) header

[COLOR=#804040][b]do[/b][/color] i[COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]1[/color], nr_outfiles
  first[COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]3[/color][COLOR=#804040][b]*[/b][/color]i[COLOR=#804040][b]-[/b][/color][COLOR=#ff00ff]2[/color]
  last[COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]3[/color][COLOR=#804040][b]*[/b][/color]i

  [COLOR=#0000ff]! initialize array index[/color]
  idx [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]1[/color]
  values [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]0[/color]

  [COLOR=#0000ff]! process 3 input files[/color]
  [COLOR=#804040][b]do[/b][/color] k[COLOR=#804040][b]=[/b][/color]first, last
    [COLOR=#0000ff]! compose file name[/color]
    [COLOR=#804040][b]write[/b][/color] (inputFileName, [COLOR=#ff00ff]'(a, I0, a)'[/color]) [COLOR=#ff00ff]'NLDAS.VGRD.'[/color],k,[COLOR=#ff00ff]'.sa'[/color]

    [COLOR=#804040][b]open[/b][/color](inFileUnit, [COLOR=#804040][b]file[/b][/color][COLOR=#804040][b]=[/b][/color]inputFileName,[COLOR=#804040][b]status[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]"old"[/color], [COLOR=#804040][b]iostat[/b][/color][COLOR=#804040][b]=[/b][/color]ios)
    [COLOR=#804040][b]if[/b][/color] (ios [COLOR=#804040][b]/=[/b][/color] [COLOR=#ff00ff]0[/color] ) [COLOR=#804040][b]then[/b][/color]
      [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color], [COLOR=#ff00ff]'("Can''t open file ", a, " for reading.")'[/color]) [COLOR=#804040][b]&[/b][/color]
      [COLOR=#008080]trim[/color](inputFileName)
      [COLOR=#804040][b]stop[/b][/color]
    [COLOR=#804040][b]endif[/b][/color]

    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color], [COLOR=#ff00ff]'("Opened file ", a, " for reading.")'[/color]) [COLOR=#804040][b]&[/b][/color]
    [COLOR=#008080]trim[/color](inputFilename)

    [COLOR=#0000ff]! read header[/color]
    [COLOR=#804040][b]read[/b][/color](inFileUnit, [COLOR=#ff00ff]'(A)'[/color], [COLOR=#804040][b]end[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#804040][b]99[/b][/color]) header

    [COLOR=#0000ff]! read rest of lines into the array[/color]
    [COLOR=#804040][b]do[/b][/color] [COLOR=#804040][b]while[/b][/color] ([COLOR=#ff00ff].true.[/color])
      [COLOR=#804040][b]read[/b][/color](inFileUnit,[COLOR=#ff00ff]'(I3)'[/color], [COLOR=#804040][b]end[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#804040][b]99[/b][/color]) values(idx)
      idx [COLOR=#804040][b]=[/b][/color] idx [COLOR=#804040][b]+[/b][/color] [COLOR=#ff00ff]1[/color]
    [COLOR=#804040][b]enddo[/b][/color]

[COLOR=#804040][b]    99 continue[/b][/color]

    [COLOR=#0000ff]! close input file[/color]
    [COLOR=#804040][b]close[/b][/color](inFileUnit)
    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]"...done"[/color]
  [COLOR=#804040][b]enddo[/b][/color]
  [COLOR=#0000ff]! Output file[/color]
  [COLOR=#804040][b]write[/b][/color] (outputFileName, [COLOR=#ff00ff]'(a, I0, a)'[/color]) [COLOR=#ff00ff]'NLDAS.VGRD.'[/color],i,[COLOR=#ff00ff]'.dat'[/color]
  [COLOR=#804040][b]open[/b][/color](outFileUnit, [COLOR=#804040][b]file[/b][/color][COLOR=#804040][b]=[/b][/color]outputFileName,[COLOR=#804040][b]status[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]"unknown"[/color], [COLOR=#804040][b]iostat[/b][/color][COLOR=#804040][b]=[/b][/color]ios)
  [COLOR=#804040][b]if[/b][/color] (ios [COLOR=#804040][b]/=[/b][/color] [COLOR=#ff00ff]0[/color]) [COLOR=#804040][b]then[/b][/color]
    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color], [COLOR=#ff00ff]'("Can''t open file ", a, " for writing.")'[/color]) [COLOR=#804040][b]&[/b][/color]
    [COLOR=#008080]trim[/color](outputFileName)
    [COLOR=#804040][b]stop[/b][/color]
  [COLOR=#804040][b]endif[/b][/color]

  [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color], [COLOR=#ff00ff]'("==> Opened file ", a, " for writing.")'[/color]) [COLOR=#804040][b]&[/b][/color]
  [COLOR=#008080]trim[/color](outputFilename)

  [COLOR=#0000ff]! write header[/color]
  [COLOR=#804040][b]write[/b][/color](outFileUnit,[COLOR=#804040][b]*[/b][/color]) header

  [COLOR=#0000ff]! number of array elements[/color]
  nr_values [COLOR=#804040][b]=[/b][/color] idx [COLOR=#804040][b]-[/b][/color] [COLOR=#ff00ff]1[/color]

  [COLOR=#0000ff]! compute average and write it to the file[/color]
  sum_values [COLOR=#804040][b]=[/b][/color] [COLOR=#008080]sum[/color](values, nr_values)
  avg [COLOR=#804040][b]=[/b][/color] sum_values [COLOR=#804040][b]/[/b][/color] nr_values
  [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]"...computed average ="[/color], avg
  [COLOR=#804040][b]write[/b][/color](outFileUnit,[COLOR=#804040][b]*[/b][/color]) avg

  [COLOR=#0000ff]! close output file[/color]
  [COLOR=#804040][b]close[/b][/color](outFileUnit)
[COLOR=#804040][b]enddo[/b][/color]
[COLOR=#a020f0]end program[/color] testInternalFiles

after compilation and running

$ gfortran test_internal_files.f95 -o test_internal_files

$ test_internal_files
Opened file NLDAS.VGRD.1.sa for reading.
 ...done
Opened file NLDAS.VGRD.2.sa for reading.
 ...done
Opened file NLDAS.VGRD.3.sa for reading.
 ...done
==> Opened file NLDAS.VGRD.1.dat for writing.
 ...computed average =   123.00000    
Opened file NLDAS.VGRD.4.sa for reading.
 ...done
Opened file NLDAS.VGRD.5.sa for reading.
 ...done
Opened file NLDAS.VGRD.6.sa for reading.
 ...done
==> Opened file NLDAS.VGRD.2.dat for writing.
 ...computed average =   456.00000    
Opened file NLDAS.VGRD.7.sa for reading.
 ...done
Opened file NLDAS.VGRD.8.sa for reading.
 ...done
Opened file NLDAS.VGRD.9.sa for reading.
 ...done
==> Opened file NLDAS.VGRD.3.dat for writing.
 ...computed average =   789.00000

it produces 3 files:

NLDAS.VGRD.1.dat

 HEADER 1-2-3        
   123.00000

NLDAS.VGRD.2.dat

 HEADER 4-5-6        
   456.00000

NLDAS.VGRD.3.dat

 HEADER 7-8-9        
   789.00000

 

[Taiseer]

Thank you very much,

but actually the output files (let us say the file with the index 3) whihc is the average of the files (7,8,9) should contain not one values but the same set of values as in the old three files. It means that there should be a vector.
Each value of this vector is the average of the corresponding values in the old three files.
For example, if the files (7,8,9) are the same as you suggested in your example, the the new file which is the average one should be:

HEADER 7-8-9
789.00000
789.00000
789.00000

Thanks

 

[mikrom]

When it should be so, then you have to modify the program so, that it computes the elemnt of average vector after the processing of every of the three files, i.e. in the inner loop - something like this:

  do k=first, last
    ...
    ! close input file
    close(inFileUnit)
    write(*,*) "...done"
    [COLOR=red]
    ! TODO: compute the value of the average vector here
    ...
    [/color]
  enddo

Then after the processig of all three files, you should do some modificaion in the outer loop - something like this:

  ...
  ! Output file
  write (outputFileName, '(a, I0, a)') 'NLDAS.VGRD.',i,'.dat'
  open(outFileUnit, file=outputFileName,status="unknown", iostat=ios)
  ...
  ! write header
  write(outFileUnit,*) header
  [COLOR=red]
  ! TODO: write all 3 elements of the average vector in the file
  ...
  [/color]
  ! close output file
  close(outFileUnit)
  ...

Try to do the modifications self as an exercise - in case of probems ask here.