Large file with variable length of both text and numbers!

[vallabhi]

Hello,

I would like to read from a file and write into another file. The input file is very large and has variable length and has both text and numbers. How do I copy? If I read 80 characters at a time, the resulting file is less in size to the original file. Since the file is huge unable to find the diff.

Please help!

Thanks,
Vallabhi

 

[mikrom]

How do I copy?

If you want o copy use the copy command from your operating system.

If I read 80 characters at a time, the resulting file is less in size to the original file.

Probably you try to read the file line by line and you assume that the line length is limited to 80 characters, but in reality the lines are larger.

 

[salgerman]

I don't remember the details, but it could very well be that the opposite of what mikrom describes also happens...when you read a line that is less than 80 characters wide and turn around and write your character variable, it could be that the length of such written line always matches the declared length of the variable.

To visualize the difference between two text files you can use applications like tkdiff in Linux and Winmerge in Windows; there are others.

 

[vallabhi]

I cannot do it using OS commands, I need to do it in Fortran only. Thats because I am using sticky bit and it works only executable and not in shell script. Please help!

 

[salgerman]

I do not understand. I don't think the sticky bit has anything to do with you being able to read the file or not; sticky bit is there to not allow you to delete file even if you have write permission to the directory.

Who owns the file and what are the permission on it?

If all you need to do is copy the file, I am certain that you can do it directly from the shell for as long as the permissions are right for both source and destination.

In any case, if you want some help with your fortran program, post it, first; show us what you have done, what the results are and what it is you don't like about them...what it is that you would like to see happen.

 

[vallabhi]

I need to write to a file of different owner, So I created a fortran file with that owner and set the sticky bit, so that anybody can write into that file. Please see below the code I have written:

The excerpts code is as below:

OPEN(UNIT=ISRC, FILE=ARG1, STATUS='OLD', ACTION='READ')
OPEN(UNIT=IDST, FILE=ARG2, STATUS='REPLACE', ACTION='WRITE')
DO [
read (UNIT=ISRC, '(80A)', IOSTAT=IERR) CHAR
# write(*, '(80A)') char
IF (IERR != 0) EXIT
write(UNIT=IDST, '(80A)') char
]
close (ISRC)
close (IDST)

After execution, the input and output file are like below:

cidcsmp01:/home/mb131% ls -lrt temp4_33766.dat H29285
-rw-r--r-- 1 mb131 d0000000 1750960 Dec 5 03:41 temp4_33766.dat -- Input
-rw-r--r-- 1 dataload d0000000 878318 Dec 10 01:17 H29285 -- Output

As you can see the bytes are not the same.

Is this a buffer issue?

Please advise.

 

[xwb]

How about

cp temp4_33766.dat H29285
chown newuser H29285
chmod 1644 H29285

Apart from that, is the input file binary or text?

 

[vallabhi]

Dear xwb,

This is not that simple, ofcourse I know what you have told, but I need any user to execute this, and for that Fortran program is the only way with sticky bit set. I need to do the copying in fortran only. This fortran program is part of a large program and any user should be able to execute this.

Thanks Anyway!

Regards,
Vallabhi

 

[mikrom]

I tried this

[COLOR=#a020f0]program[/color] copyfile
  [COLOR=#2e8b57][b]implicit[/b][/color] [COLOR=#2e8b57][b]none[/b][/color]
  [COLOR=#0000ff]! copies file record by record[/color]
  [COLOR=#2e8b57][b]integer[/b][/color] :: stat, rec_nr
  [COLOR=#2e8b57][b]character[/b][/color]([COLOR=#ff00ff]256[/color]) :: src_file, dst_file
  [COLOR=#2e8b57][b]character[/b][/color] :: my_rec

  [COLOR=#804040][b]if[/b][/color] (iargc() [COLOR=#804040][b].ne.[/b][/color] [COLOR=#ff00ff]2[/color]) [COLOR=#804040][b]then[/b][/color]
    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]'Usage: copyfile <src_file> <dst_file>'[/color]
    [COLOR=#804040][b]stop[/b][/color]
  [COLOR=#804040][b]end if[/b][/color] 

  [COLOR=#008080]call[/color] getarg([COLOR=#ff00ff]1[/color], src_file)
  [COLOR=#008080]call[/color] getarg([COLOR=#ff00ff]2[/color], dst_file)

  [COLOR=#804040][b]open[/b][/color]([COLOR=#804040][b]unit[/b][/color] [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]10[/color],[COLOR=#804040][b]file[/b][/color][COLOR=#804040][b]=[/b][/color]src_file,[COLOR=#804040][b]access[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'DIRECT'[/color],[COLOR=#804040][b]status[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'OLD'[/color],[COLOR=#804040][b]RECL[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]1[/color])
  [COLOR=#804040][b]open[/b][/color]([COLOR=#804040][b]unit[/b][/color] [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]20[/color],[COLOR=#804040][b]file[/b][/color][COLOR=#804040][b]=[/b][/color]dst_file,[COLOR=#804040][b]access[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'DIRECT'[/color],[COLOR=#804040][b]status[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'REPLACE'[/color],[COLOR=#804040][b]RECL[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]1[/color])

  rec_nr [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]1[/color]
  [COLOR=#804040][b]do[/b][/color]
    [COLOR=#0000ff]! read source[/color]
    [COLOR=#804040][b]read[/b][/color]([COLOR=#ff00ff]10[/color], [COLOR=#804040][b]rec[/b][/color][COLOR=#804040][b]=[/b][/color]rec_nr, [COLOR=#804040][b]iostat[/b][/color][COLOR=#804040][b]=[/b][/color]stat) my_rec
    [COLOR=#804040][b]if[/b][/color] (stat [COLOR=#804040][b].ne.[/b][/color] [COLOR=#ff00ff]0[/color]) [COLOR=#804040][b]then[/b][/color]
      [COLOR=#804040][b]go to[/b][/color] [COLOR=#ff00ff]99[/color]
    [COLOR=#804040][b]end if[/b][/color]
    [COLOR=#0000ff]! write destination[/color]
    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]unit[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]20[/color], [COLOR=#804040][b]rec[/b][/color][COLOR=#804040][b]=[/b][/color]rec_nr) my_rec
    [COLOR=#0000ff]![/color]
    rec_nr [COLOR=#804040][b]=[/b][/color] rec_nr [COLOR=#804040][b]+[/b][/color] [COLOR=#ff00ff]1[/color]
  [COLOR=#804040][b]end do[/b][/color]

  [COLOR=#0000ff]! close files[/color]
[COLOR=#804040][b]  99 continue[/b][/color]
  [COLOR=#804040][b]close[/b][/color] ([COLOR=#ff00ff]10[/color])
  [COLOR=#804040][b]close[/b][/color] ([COLOR=#ff00ff]20[/color])
[COLOR=#a020f0]end program[/color] copyfile

Output:

mikl1071@1_MIKL1071_01 ~/fortran
$ gfortran copyfile.f95 -o copyfile

mikl1071@1_MIKL1071_01 ~/fortran
$ copyfile ~/fortran/data/DATA-002.txt ~/fortran/data/copy_of_DATA.txt

mikl1071@1_MIKL1071_01 ~/fortran
$ ls -la  ~/fortran/data/*.txt
-rw-r--r--    1 mikl1071 Administ  1817498 May 10  2013 /home/mikl1071/fortran/data/DATA-002.txt
-rw-r--r--    1 mikl1071 Administ  1817498 Dec 10 10:18 /home/mikl1071/fortran/data/copy_of_DATA.txt

 

[vallabhi]

Thank you so much mikrom.... It is working! The power of internet is infinity!

 

[mikrom]

Hi vallabhi,
I'm curious about the syntax of the code you posted above

DO [
...
# write(*, '(80A)') char
IF (IERR != 0) EXIT
...
]

Is this any new dialect of fortran or a scripting language based on fortran which uses [..] for blocks and # for comments ?

 

[vallabhi]

This is ratfor code (.r files). Before compiling you have to do

$rafor < final.r > final.f

And then compile it.

 

[vallabhi]

Though my issue is resolved, I have one more question:

Please see below, I want to append 3 files together, How do I go about this? The last file is the huge one. First I have to replace then append two more files to the same file. Please advise.

OPEN(UNIT=ISRC, FILE=ARG1, STATUS='OLD', ACTION='READ')
OPEN(UNIT=IDST, FILE=ARG4, STATUS='REPLACE', ACTION='WRITE')
DO [
read (UNIT=ISRC, '(A)', IOSTAT=IERR) CHAR
# write(*, '(80A)') char
IF (IERR != 0) EXIT
write(UNIT=IDST, '(80A)') char
]
close (ISRC)
close (IDST)
ISRC = 3
IDST = 4
OPEN(UNIT=ISRC, FILE=ARG3, STATUS='OLD', ACTION='READ')
OPEN(UNIT=IDST, FILE=ARG4, ACCESS='APPEND',STATUS='OLD', ACTION='WRITE')
DO [
read (UNIT=ISRC, '(A)', IOSTAT=IERR) CHAR
# write(*, '(80A)') char
IF (IERR != 0) EXIT
write(UNIT=IDST, '(80A)') char
]
close (ISRC)
close (IDST)
ISRC = 7
IDST = 8
OPEN(UNIT=ISRC, FILE=ARG2, STATUS='OLD', ACTION='READ')
OPEN(UNIT=IDST, FILE=ARG4, ACCESS='APPEND',STATUS='OLD', ACTION='WRITE')
DO [
read (UNIT=ISRC, '(A)', IOSTAT=IERR) CHAR
# write(*, '(80A)') char
IF (IERR != 0) EXIT
write(UNIT=IDST, '(A)') char
]
close (ISRC)
close (IDST)

 

[mikrom]

I would do that similar as before

[COLOR=#a020f0]program[/color] appendfile
  [COLOR=#2e8b57][b]implicit[/b][/color] [COLOR=#2e8b57][b]none[/b][/color]
  [COLOR=#0000ff]! appends files record by record[/color]
  [COLOR=#2e8b57][b]integer[/b][/color] :: stat, inp_rec_nr, out_rec_nr, nr_files, k
  [COLOR=#2e8b57][b]character[/b][/color]([COLOR=#ff00ff]30[/color]) :: file_0, app_file
  [COLOR=#2e8b57][b]character[/b][/color] :: my_rec

  nr_files [COLOR=#804040][b]=[/b][/color] iargc()

  [COLOR=#804040][b]if[/b][/color] (nr_files [COLOR=#804040][b].le.[/b][/color] [COLOR=#ff00ff]2[/color]) [COLOR=#804040][b]then[/b][/color]
    [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]*[/b][/color],[COLOR=#804040][b]*[/b][/color]) [COLOR=#ff00ff]'Usage: appendfile <file_0> <app_file_1> .. <app_file_n>'[/color]
    [COLOR=#804040][b]stop[/b][/color]
  [COLOR=#804040][b]end if[/b][/color] 

  [COLOR=#0000ff]! open the first file[/color]
  [COLOR=#008080]call[/color] getarg([COLOR=#ff00ff]1[/color], file_0)
  [COLOR=#804040][b]open[/b][/color]([COLOR=#804040][b]unit[/b][/color] [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]10[/color], [COLOR=#804040][b]file[/b][/color][COLOR=#804040][b]=[/b][/color]file_0, [COLOR=#804040][b]access[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'DIRECT'[/color], [COLOR=#804040][b]status[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'OLD'[/color],[COLOR=#804040][b]RECL[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]1[/color])

  [COLOR=#0000ff]! count number of records in the first file[/color]
  out_rec_nr [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]1[/color] [COLOR=#0000ff]! number of records in original (output) file[/color]
  [COLOR=#804040][b]do[/b][/color]
    [COLOR=#0000ff]! read source[/color]
    [COLOR=#804040][b]read[/b][/color]([COLOR=#ff00ff]10[/color], [COLOR=#804040][b]rec[/b][/color][COLOR=#804040][b]=[/b][/color]out_rec_nr, [COLOR=#804040][b]iostat[/b][/color][COLOR=#804040][b]=[/b][/color]stat) my_rec
    [COLOR=#804040][b]if[/b][/color] (stat [COLOR=#804040][b].ne.[/b][/color] [COLOR=#ff00ff]0[/color]) [COLOR=#804040][b]then[/b][/color]
      [COLOR=#804040][b]exit[/b][/color]
    [COLOR=#804040][b]end if[/b][/color]
    [COLOR=#0000ff]! count records[/color]
    out_rec_nr [COLOR=#804040][b]=[/b][/color] out_rec_nr [COLOR=#804040][b]+[/b][/color] [COLOR=#ff00ff]1[/color]
  [COLOR=#804040][b]end do[/b][/color]

  [COLOR=#0000ff]! append all other files to the first file[/color]
  [COLOR=#804040][b]do[/b][/color] k[COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]2[/color], nr_files
    [COLOR=#0000ff]! get file for reading[/color]
    [COLOR=#008080]call[/color] getarg(k, app_file)
    [COLOR=#804040][b]open[/b][/color]([COLOR=#804040][b]unit[/b][/color] [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]20[/color], [COLOR=#804040][b]file[/b][/color][COLOR=#804040][b]=[/b][/color]app_file,[COLOR=#804040][b]access[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'DIRECT'[/color], [COLOR=#804040][b]status[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]'OLD'[/color], [COLOR=#804040][b]RECL[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]1[/color])

    inp_rec_nr [COLOR=#804040][b]=[/b][/color] [COLOR=#ff00ff]1[/color] [COLOR=#0000ff]! number of records in each input file[/color]
    [COLOR=#804040][b]do[/b][/color]
      [COLOR=#0000ff]! read source[/color]
      [COLOR=#804040][b]read[/b][/color]([COLOR=#ff00ff]20[/color], [COLOR=#804040][b]rec[/b][/color][COLOR=#804040][b]=[/b][/color]inp_rec_nr, [COLOR=#804040][b]iostat[/b][/color][COLOR=#804040][b]=[/b][/color]stat) my_rec
      [COLOR=#804040][b]if[/b][/color] (stat [COLOR=#804040][b].ne.[/b][/color] [COLOR=#ff00ff]0[/color]) [COLOR=#804040][b]then[/b][/color]
        [COLOR=#804040][b]exit[/b][/color]
      [COLOR=#804040][b]end if[/b][/color]
      [COLOR=#0000ff]! write destination[/color]
      [COLOR=#804040][b]write[/b][/color]([COLOR=#804040][b]unit[/b][/color][COLOR=#804040][b]=[/b][/color][COLOR=#ff00ff]10[/color], [COLOR=#804040][b]rec[/b][/color][COLOR=#804040][b]=[/b][/color]out_rec_nr) my_rec    
      [COLOR=#0000ff]![/color]
      inp_rec_nr [COLOR=#804040][b]=[/b][/color] inp_rec_nr [COLOR=#804040][b]+[/b][/color] [COLOR=#ff00ff]1[/color]  
      out_rec_nr [COLOR=#804040][b]=[/b][/color] out_rec_nr [COLOR=#804040][b]+[/b][/color] [COLOR=#ff00ff]1[/color]
    [COLOR=#804040][b]end do[/b][/color]
    [COLOR=#0000ff]! close input file[/color]
    [COLOR=#804040][b]close[/b][/color] ([COLOR=#ff00ff]20[/color])
  [COLOR=#804040][b]end do[/b][/color]

  [COLOR=#0000ff]! close first (output) file[/color]
  [COLOR=#804040][b]close[/b][/color] ([COLOR=#ff00ff]10[/color])
[COLOR=#a020f0]end program[/color] appendfile

Usage:

mikl1071@1_MIKL1071_01 ~/fortran
$ gfortran appendfile.f95 -o appendfile

mikl1071@1_MIKL1071_01 ~/fortran
$ appendfile file_0.txt file_1.txt file_2.txt

The example abowe appends file_1.txt and file_2.txt to file_0.txt

 

[mikrom]

Thanks for information. I never heard about RATFOR before. It's interesting.

 

[xwb]

Ratfor is one of those preprocessor tools for people that came out in the early 80s. I've only heard of it because it was on many of the early PDP-11 Unix machines.

 

[mikrom]

Hi vallabhi,

I have typo in the program appendfile above.
Please correct

if (nr_files [COLOR=red].le.[/color] 2) then
  write(*,*) 'Usage: appendfile <file_0> <app_file_1> .. <app_file_n>'
  stop
end if

to

if (nr_files [COLOR=blue].lt.[/color] 2) then
  write(*,*) 'Usage: appendfile <file_0> <app_file_1> .. <app_file_n>'
  stop
end if

 

[vallabhi]

Thanks a lot mikrom..... I have used the first tip and finished the development. Will get back to you in case of more questions! Thanks Again!!