Adding test to the first line in a large text file.

[Bminer]

Howdy,

I am a new user of TCL macros with a software package used in house.

The software creates a text file. The first line of the file is supposed to contain both the path and name of a partner file. Due to a bug is the software, the path is not being added to this first line of the test file.

I am not expecting a rapid bug fix for this issue so I need a workaround in the meantime.

I know that the a new file can be created and the 2nd to final line of the text file can be parsed into a new file, but these text files can be very large so avoinding a parsing regimen would be preferred.

Here is an example first line

"l0_grid.str,8148161600;algorithm=standard;fields=x,y"

it should read

"data_files/strings/l0_grid.str,8148161600;algorithm=standard;fields=x,y"

No other lines in the file need to be changed.

Thanks in advance for any suggestions.

Brian

 

[Bminer]

Title should read "Adding TEXT to the first line of a large text file"

Brian

 

[thacoda]

Hi,

Is there a particular reason why you want to use tcl/tk for solving this problem? You say you want a workaround only. In that case, I would propose to use "sed", to manipulate (large) files. See Link. There is a version available of this ancient utility for most OS, if I correctly remember.

assume your filename=test.dat

sed s/l0_grid.str/data_files\/strings\/l0_grid.str/ test.dat > new_test.dat

thacoda

 

[thacoda]

btw,
we can execute 'sed' from within a tcl-script with the 'exec' tcl-command...

 

[Bminer]

thacoda,

Thanks for the reply. Two points on using sed. First, the programs I am using are on a Windows 7 machine. It could be loaded it onto this machine and then follow your suggestion. I am researching this option (little prior experience with Unix/ Sed before).

The downside of this option is that the macros that will likely be used by a coworker as well. I do not want to have to be involved in maintaining his machine for him (hes more a chisel and stone tablet guy).

Any other tricks inside TCL are appreciated.

Brian

 

[feherke]

Hi

I know that the a new file can be created and the 2nd to final line of the text file can be parsed into a new file, but these text files can be very large so avoinding a parsing regimen would be preferred.

You can edit in the middle of an existing file only by overwriting the current content. As you need insert new content, it is not possible without the creation of a new file.

If you always want to insert at the very beginning of the existing file, even [tt]sed[/tt] is an overkill. You can just write the missing part in a new file, then use the system's copy functionality to concatenate it with the old file to create a new file. On Windows that would be the [tt]copy[/tt] command. Not a speed champion, but probably faster than coding the same in Tcl.

[b]set[/b] fil [teal][[/teal][b]open[/b] [green][i]"prefix.txt"[/i][/green] w[teal]][/teal]
[b]puts[/b] [teal]-[/teal]nonewline [navy]$fil[/navy] [green][i]"data_files/strings/"[/i][/green]
[b]close[/b] [navy]$fil[/navy]

[b]exec[/b] [teal][[/teal] copy prefix[teal].[/teal]txt[teal]+[/teal]original[teal].[/teal]txt temporary[teal].[/teal]txt [teal]][/teal]

[b]file[/b] rename [teal]-[/teal]force temporary[teal].[/teal]txt original[teal].[/teal]txt
[b]file[/b] delete prefix[teal].[/teal]txt

[small][maroon]Warning[/maroon] The above code was tested only partially[/small]

Really ugly, but I think is the fastest option in case of large files.

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[Bminer]

Thanks for the reply feherke,

After pondering over this, for now, I can stand parsing the old file to a new file and "Concat"ing the new text in.

Here the first two lines from the initial file

L1_grid.str,6251187200;algorithm=standard;fields=x,y
1, 672673.000, 1278320.000, 465.414,


Here is my code so far

set fi [open data_files\\geological_database\\stratalines\\dtms\\l1_grid.dtm r]
set fo  [open data_files\\geological_database\\stratalines\\dtms\\l1_grid_temp.dtm a]

gets $fi line

puts $fo [concat "data_files\\geological_database\\stratalines\\dtms\\" $line]

while {[gets $fi line] >= 0} {puts $fo $line}

close $fi
close $fo

and here is the output of the first two lines of the resulting file

data_files\geological_database\stratalines\dtms\ L1_grid.str,6251187200;algorithm=standard;fields=x,y
1, 672673.000, 1278320.000, 465.414,

All looks great except for one item. There is whitespace at the CONCAT point in the first line of the output file. This will cause an error is subsequent file loads as the path\filename is now wrong. How do I eliminate this?

Thanks for the info so far.

Brian

 

[feherke]

Hi

Why the [tt]concat[/tt] ?

[b]set[/b] fi [teal][[/teal][b]open[/b] data_files[teal]\\[/teal]geological_database[teal]\\[/teal]stratalines[teal]\\[/teal]dtms[teal]\\[/teal]l1_grid[teal].[/teal]dtm r[teal]][/teal]
[b]set[/b] fo  [teal][[/teal][b]open[/b] data_files[teal]\\[/teal]geological_database[teal]\\[/teal]stratalines[teal]\\[/teal]dtms[teal]\\[/teal]l1_grid_temp[teal].[/teal]dtm a[teal]][/teal]

[b]puts[/b] [teal]-[/teal]nonewline [navy]$fo[/navy] [green][i]"data_files[/i][/green][lime][i]\\[/i][/lime][green][i]geological_database[/i][/green][lime][i]\\[/i][/lime][green][i]stratalines[/i][/green][lime][i]\\[/i][/lime][green][i]dtms[/i][/green][lime][i]\\[/i][/lime][green][i]"[/i][/green]

[b]while[/b] [teal]{[/teal][teal][[/teal][b]gets[/b] [navy]$fi[/navy] line[teal]][/teal] [teal]>=[/teal] [purple]0[/purple][teal]}[/teal] [teal]{[/teal][b]puts[/b] [navy]$fo[/navy] [navy]$line[/navy][teal]}[/teal]

[b]close[/b] [navy]$fi[/navy]
[b]close[/b] [navy]$fo[/navy]

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[Bminer]

I am trying to add a path to the .str file referenced in the header line of the text file. I use concat all the time in excel so that experience led me down that path.

Do I need APPEND instead? I am trying it now but cannot seem to get it to work (yet).

Brian

 

[Bminer]

Figured it out!

Here is my code

# code to add missing path in .dtm file

set fi [open data_files\\geological_database\\stratalines\\dtms\\l1_grid.dtm r]
set fo  [open data_files\\geological_database\\stratalines\\dtms\\l1_grid_temp.dtm a]

append line "data_files\\geological_database\\stratalines\\dtms\\" [gets $fi ] 

puts $fo $line

while {[gets $fi line] >= 0} {puts $fo $line}

close $fi
close $fo

file rename -force "data_files\\geological_database\\stratalines\\dtms\\l1_grid_temp.dtm" "data_files\\geological_database\\stratalines\\dtms
\l1_grid.dtm"

THanks for the space to work this out.

I am going to look at some of the other commands you both suggested as I progress down the TCL path.

Brian

 

[feherke]

Hi

See & try my code posted on 29 Aug 12 10:26. No [tt]concat[/tt], no [tt]append[/tt].

Note that [tt]concat[/tt] is intended to be used on lists, not strings. That is why you got a whitespace separator. [tt]append[/tt] is a correct choice for this task, but no need for it. Just put the values into the output file and omit the terminating newline when putting the prefix piece.

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[Bminer]

feherke,

I am trying your approach. I generated a permenant prefix file using the put noneline command. This file will never change. The other files will routinely be regenerated as new information is added thus requiring the path name being added back into the regenerated file.

However I am getting errors and not sure what I am doing wrong (it might be the path)?

exec [ copy /A "data_files/geological_database/stratalines/dtms/prefix.txt"+"data_files/geological_database/stratalines/dtms/ovb_grid.dtm" "data_files/geological_database/stratalines/dtms/ovb_grid_temp.dtm"]

I tried it with and without quotes and with and without the "/a" parameter am still getting errors.

Warning: Tcl generated error at line 13:
extra characters after close-quote
while executing
"exec [ copy /A "data_files/geological_database/stratalines/dtms/prefix.txt"
(file "test.tcl" line 13)
invoked from within
"default_source "test.tcl""

Any thoughts?

Brian

 

[feherke]

Hi

Oops, those brackets should no be there.

Regarding the quoting, just quotes around the parameters are enough in Linux, no idea about Windows :

exec copy /A "data_files/geological_database/stratalines/dtms/prefix.txt+data_files/geological_database/stratalines/dtms/ovb_grid.dtm" "data_files/geological_database/stratalines/dtms/ovb_grid_temp.dtm"

Sorry, I have no Tcl on Windows to test it properly.

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[Bminer]

Removed the [] and got

Warning: Tcl generated error at line 13:
couldn't execute "copy": no such file or directory
while executing
"exec copy data_files/geological_database/stratalines/dtms/prefix.txt+data_files/geological_database/stratalines/dtms/ovb_grid.dtm data_files/geologica..."
(file "test.tcl" line 13)
invoked from within
"default_source "test.tcl""

For now, I just cut and pasted snippets of my earlier code to get paste the current stopping point. I certainly think your solution is much more elegant than my chunks of code.

Of course if the parent software would correct the original bug (of not saving the path with the file name) then this all would not be necessary. In spite of that, I have learned much in the past 2 days so not all is lost.

Thanks again.

Brian

 

[Bminer]

reading

http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M24

there are notes about windows systems

I might not have a console attached to the application which apparently prevents exec from functioning. TCL is functioning within a 3rd party program.

Brian

 

[feherke]

Hi

couldn't execute "copy": no such file or directory

Oops again. The old issue - [tt]copy[/tt] is internal command so can not be called directly :

exec [highlight]cmd.exe /c[/highlight] copy /A "data_files/geological_database/stratalines/dtms/prefix.txt+data_files/geological_database/stratalines/dtms/ovb_grid.dtm" "data_files/geological_database/stratalines/dtms/ovb_grid_temp.dtm"

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[Bminer]

I tried code and while it attempted to run, the error message states

Warning: Tcl generated error at line 13:
The system cannot find the path specified.
child process exited abnormally
while executing
"exec cmd.exe /c copy /A "data_files/geological_database/stratalines/dtms/prefix.txt+data_files/geological_database/stratalines/dtms/ovb_grid.dtm data_..."
(file "test.tcl" line 13)
invoked from within
"default_source "test.tcl""

Perhaps the program's path and the path expected by cmd.exe are different?

Brian

 

[feherke]

Hi

Possible. Given Windows' habit to spread executables all over the harddisk, I have no idea what path should be used there.

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[thacoda]

Hi,

I am successful in the following case (created 2 dummy files with the same filenames you used as well as put these in a folder called 'rotz' in C-drive):

exec cmd.exe /c copy /A c:\\rotz\\prefix.txt+c:\\rotz\\ovb_grid.dtm c:\\rotz\\ovb_grid_temp.dtm

so, basically, 3 differences:
1. used drive name
2. removed quotes
3. used \\ instead of / as path seperators (you are only uising this workaround for Windows, right?)

I am sure there is another way doing it, but hey, if you are looking for a quick solution, these are my 2 cents.

thacoda

 

[Bminer]

thacoda,

I suspect that your solution would have solved my issue. I ran the routine with and without quotes. Adding the quotes gave me the final error of "the system cannot find the specified path". Without the quotes I got a "does not recognize this command" or somesuch error. The last error message posted above appears to indicate that the command was valid but the path was not. Adding the full path should work.

Ultimately, the need to make the file correction vanished. when I returned to the work after a weekend (and a fresh hard boot of my machine), the original routines worked without having to add the text into the files I was trying to modify. The master program uses java and the support desk indicated that sometimes after several errors, weird things happen at which he recommended rebooting the machine (and providing the opportunity to take a short walk).

Thanks to both you and feherke for your time in trying to sort this out. It will ultimately help me out since there are several other tasks that I need to tackle that can be done using tcl and not relying on the long drawn process the master programs require but instead directly edit the text files using some tcl functions. Years ago, I did many similar things in lisp and now see how many of these tasks can again be handled outside the primary software.

/bows to honor

Brian