TCL to open and combine multiple text files extracting only integer values at specified lines

[RadicalGiant]

Hi I am new to tcl programming. I am in process of creating a automation tcl and i would like to request for ur help.

Scenario:

I have multiple text files which contains integer data of some variables called nodes and ele.The sample file looks like this.

#C
##Id Files
Ele 15235,16845,555845,21452,451254
Nodes 15478,26558,78549,12547,12546,13556
Thickness 2.5
Nodal thickness 0​

I have hundreds of those text files.

One similarity in all files is that in line 3 and 4 , we have the element ids and node ids

Now, the thing i would like to automate is that , i want to build a scripts in tcl, after running which, it reads all the node ids and elements ids in lines 3 and 4; only integer values (in all of those 100 txt files) and stores them in a separate txt file, each one for node and for element.

1. Final node txt file should have only node ids from all files

2. Final elem txt file should have only elem ids from all files

It is possible through array? how can we best code it in tcl. pls help. its urgent.
I lack direction. can u help me and guide me , how to create that tcl?

thank you in advance

 

[mikrom]

Hi RadicalGiant,

OK, you need a little help but first show us an effort what have you done so far.

You need to divide your complex task into simple tasks -i.e.:
1. create a procedure which reads one file line by line - for that you can use the tcl commands open, close and gets.
2. create a procedure which processes one line and if possible it extracts the desired numbers - for this you can use several string and list operations
3. create a procedure which delivers list of all files in a given directory and process this list - use the command glob.

Have you tried to solve something of these tasks ?

 

[mikrom]

Hi RadicalGiant,

As for the task no.2 (see my previous post): I would create 2 global lists (i.e. arrays): one for elements and other for nodes.
The procedure which processes lines should read this data from every file and append them to the global lists - e.g.:

[COLOR=#a52a2a][b]proc[/b][/color] process_line {line} {
[COLOR=#0000ff]  # input: line[/color]
[COLOR=#0000ff]  # output: modified global: lists elem_list, node_list[/color]
  [COLOR=#a52a2a][b]global[/b][/color] elem_list
  [COLOR=#a52a2a][b]global[/b][/color] node_list
[COLOR=#0000ff]  # try to match Elements[/color]
    ...
  [COLOR=#a52a2a][b]if[/b][/color] {[COLOR=#008b8b]$result[/color]} {
[COLOR=#0000ff]    # return a list of the numbers by the regex[/color]
    [COLOR=#a52a2a][b]set[/b][/color] temp_list [get_list_of_numbers [COLOR=#008b8b]$line[/color]]
    ...
    [COLOR=#a52a2a][b]foreach[/b][/color] ele [COLOR=#008b8b]$temp_list[/color] {
       ...
[COLOR=#0000ff]        # add the element to the global list[/color]
        [COLOR=#a52a2a][b]lappend[/b][/color] elem_list [COLOR=#008b8b]$ele[/color]
       ...
    }
  }
  ...
[COLOR=#0000ff]  # try to match Nodes[/color]
  ...
}

[COLOR=#a52a2a][b]proc[/b][/color] get_list_of_numbers {line} {
  ...
  [COLOR=#a52a2a][b]return[/b][/color] [COLOR=#008b8b]$ret_list[/color]
}

 

[mikrom]

Hi RadicalGiant,

Other tasks (from my previous post) are not difficult:
1. This procedure opens a file with a name from the argument for reading, then
reads it line by line and for every line calls other procedure (see my previous post. And at end it closes the file.

[COLOR=#a52a2a][b]proc[/b][/color] process_file {[COLOR=#a52a2a][b]filename[/b][/color]} {
[COLOR=#0000ff]  # input: filename to process[/color]
  [COLOR=#a52a2a][b]set[/b][/color] input_file [[COLOR=#a52a2a][b]open[/b][/color] [COLOR=#008b8b]$filename[/color] [COLOR=#ff00ff]"r"[/color]]

  [COLOR=#a52a2a][b]while[/b][/color] { [[COLOR=#a52a2a][b]gets[/b][/color] [COLOR=#008b8b]$input_file[/color] line] != -[COLOR=#ff00ff]1[/color] } {
[COLOR=#0000ff]    # process line[/color]
    process_line [COLOR=#008b8b]$line[/color]
  }
  [COLOR=#a52a2a][b]close[/b][/color] [COLOR=#008b8b]$input_file[/color]
}

3. If your data files would be in same directory named e.g. *.txt, then you could create a list of them using the glob command - like this:

[COLOR=#a52a2a][b]proc[/b][/color] get_list_of_files {} {
[COLOR=#0000ff]  # return sorted list of files matching a given pattern[/color]
  [COLOR=#a52a2a][b]set[/b][/color] files_to_process [[COLOR=#a52a2a][b]glob[/b][/color] ./*.txt]
  [COLOR=#a52a2a][b]return[/b][/color] [[COLOR=#a52a2a][b]lsort[/b][/color] [COLOR=#008b8b]$files_to_process[/color]]
}

Let me know if you have other questions ...

 

[mikrom]

I posted some answers and I see that the OP is not responding ...
However, in the meantime as I tried to help OP how to start with this task, I created fully functional script which works according to the initial specification. Maybe, it wouldn't be good idea simply to throw it away, because it could help other beginning Tcl programmers how to start.
So, I rather post it here:
First I created some data files: radicalgiant_01.txt, radicalgiant_02.txt, radicalgiant_03.txt
which contain the data like given above - e.g.:
radicalgiant_01.txt

#C
##Id Files
Ele 15235,16845,555845,21452,451254
Nodes 15478,26558,78549,12547,12546,13556
Thickness 2.5
Nodal thickness 0

... etc

Then I created the program - some procedures are discussed above:
radicalgiant.tcl

[COLOR=#0000ff]#!/usr/bin/tclsh[/color]

[COLOR=#0000ff]# ***** functions *****[/color]
[COLOR=#a52a2a][b]proc[/b][/color] process_file {[COLOR=#a52a2a][b]filename[/b][/color]} {
[COLOR=#0000ff]  # input: filename to process[/color]
  [COLOR=#a52a2a][b]set[/b][/color] input_file [[COLOR=#a52a2a][b]open[/b][/color] [COLOR=#008b8b]$filename[/color] [COLOR=#ff00ff]"r"[/color]]

  [COLOR=#a52a2a][b]while[/b][/color] { [[COLOR=#a52a2a][b]gets[/b][/color] [COLOR=#008b8b]$input_file[/color] line] != -[COLOR=#ff00ff]1[/color] } {
[COLOR=#0000ff]    # process line[/color]
    process_line [COLOR=#008b8b]$line[/color]
  }
  [COLOR=#a52a2a][b]close[/b][/color] [COLOR=#008b8b]$input_file[/color]
}

[COLOR=#a52a2a][b]proc[/b][/color] process_line {line} {
[COLOR=#0000ff]  # input: line[/color]
[COLOR=#0000ff]  # output: modified global: lists elem_list, node_list[/color]
  [COLOR=#a52a2a][b]global[/b][/color] elem_list
  [COLOR=#a52a2a][b]global[/b][/color] node_list
[COLOR=#0000ff]  # try to match Elements[/color]
  [COLOR=#a52a2a][b]set[/b][/color] result [[COLOR=#a52a2a][b]regexp[/b][/color] {^\s*[Ee]le\s*} [COLOR=#008b8b]$line[/color] match]
  [COLOR=#a52a2a][b]if[/b][/color] {[COLOR=#008b8b]$result[/color]} {
[COLOR=#0000ff]    # return a list of the numbers by the regex[/color]
    [COLOR=#a52a2a][b]set[/b][/color] temp_list [get_list_of_numbers [COLOR=#008b8b]$line[/color]]
    [COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"  ==> This was matched: %s %s"[/color] [COLOR=#008b8b]$match[/color] [COLOR=#008b8b]$temp_list[/color]]
    [COLOR=#a52a2a][b]foreach[/b][/color] ele [COLOR=#008b8b]$temp_list[/color] {
      [COLOR=#a52a2a][b]if[/b][/color] {[[COLOR=#a52a2a][b]lsearch[/b][/color] -exact [COLOR=#008b8b]$elem_list[/color] [COLOR=#008b8b]$ele[/color]]==-[COLOR=#ff00ff]1[/color]}  {
[COLOR=#0000ff]        # add the element to the global list if it isn't already present[/color]
        [COLOR=#a52a2a][b]lappend[/b][/color] elem_list [COLOR=#008b8b]$ele[/color]
      } [COLOR=#a52a2a][b]else[/b][/color] {
        [COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"  Warning: element %d already exists in elem_list!"[/color] [COLOR=#008b8b]$ele[/color]]
      }
    }
  }
[COLOR=#0000ff]  # try to match Nodes[/color]
  [COLOR=#a52a2a][b]set[/b][/color] result [[COLOR=#a52a2a][b]regexp[/b][/color] {^\s*[Nn]odes*\s*} [COLOR=#008b8b]$line[/color] match]
  [COLOR=#a52a2a][b]if[/b][/color] {[COLOR=#008b8b]$result[/color]} {
[COLOR=#0000ff]    # return a list of the numbers by the regex[/color]
    [COLOR=#a52a2a][b]set[/b][/color] temp_list [get_list_of_numbers [COLOR=#008b8b]$line[/color]]
    [COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"  ==> This was matched: %s %s"[/color] [COLOR=#008b8b]$match[/color] [COLOR=#008b8b]$temp_list[/color]]
    [COLOR=#a52a2a][b]foreach[/b][/color] node [COLOR=#008b8b]$temp_list[/color] {
      [COLOR=#a52a2a][b]if[/b][/color] {[[COLOR=#a52a2a][b]lsearch[/b][/color] -exact [COLOR=#008b8b]$node_list[/color] [COLOR=#008b8b]$node[/color]]==-[COLOR=#ff00ff]1[/color]}  {
[COLOR=#0000ff]        # add the element to the global list if it isn't already present[/color]
        [COLOR=#a52a2a][b]lappend[/b][/color] node_list [COLOR=#008b8b]$node[/color]
      } [COLOR=#a52a2a][b]else[/b][/color] {
        [COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"  Warning: node %d already exists in node_list!"[/color] [COLOR=#008b8b]$node[/color]]
      }
    }
  }
}

[COLOR=#a52a2a][b]proc[/b][/color] get_list_of_numbers {line} {
[COLOR=#0000ff]  # return a list of the numbers by the regex[/color]
  [COLOR=#a52a2a][b]set[/b][/color] ret_list [[COLOR=#a52a2a][b]regexp[/b][/color] -all -inline {\d+} [COLOR=#008b8b]$line[/color]]
  [COLOR=#a52a2a][b]return[/b][/color] [COLOR=#008b8b]$ret_list[/color]
}

[COLOR=#a52a2a][b]proc[/b][/color] get_list_of_files {} {
[COLOR=#0000ff]  # return sorted list of files matching a given pattern[/color]
  [COLOR=#a52a2a][b]set[/b][/color] files_to_process [[COLOR=#a52a2a][b]glob[/b][/color] ./radicalgiant_??.txt]
  [COLOR=#a52a2a][b]return[/b][/color] [[COLOR=#a52a2a][b]lsort[/b][/color] [COLOR=#008b8b]$files_to_process[/color]]
}

[COLOR=#a52a2a][b]proc[/b][/color] write_data_fo_file {[COLOR=#a52a2a][b]filename[/b][/color] data} {
  [COLOR=#a52a2a][b]set[/b][/color] output_file [[COLOR=#a52a2a][b]open[/b][/color] [COLOR=#008b8b]$filename[/color] [COLOR=#ff00ff]"w"[/color]]
  [COLOR=#a52a2a][b]puts[/b][/color] [COLOR=#008b8b]$output_file[/color] [COLOR=#008b8b]$data[/color]
  [COLOR=#a52a2a][b]close[/b][/color] [COLOR=#008b8b]$output_file[/color]
} 


[COLOR=#0000ff]# ***** main *****[/color]
[COLOR=#0000ff]# initialize lists[/color]
[COLOR=#a52a2a][b]set[/b][/color] elem_list {}
[COLOR=#a52a2a][b]set[/b][/color] node_list {}

[COLOR=#0000ff]# get list of files to process [/color]
[COLOR=#a52a2a][b]set[/b][/color] file_list [get_list_of_files]
[COLOR=#0000ff]#puts [format "files found : %s " $file_list][/color]

[COLOR=#0000ff]# process all files[/color]
[COLOR=#a52a2a][b]foreach[/b][/color] fname [COLOR=#008b8b]$file_list[/color] {
[COLOR=#0000ff]  # process file[/color]
  [COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"* processing file: %s"[/color] [COLOR=#008b8b]$fname[/color]] 
  process_file [COLOR=#008b8b]$fname[/color]
  [COLOR=#a52a2a][b]puts[/b][/color] [COLOR=#ff00ff]"current global lists:"[/color]
  [COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"Ele : %s"[/color] [COLOR=#008b8b]$elem_list[/color]]
  [COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"Node: %s"[/color] [COLOR=#008b8b]$node_list[/color]]
  [COLOR=#a52a2a][b]puts[/b][/color] [COLOR=#ff00ff]""[/color]
}

[COLOR=#a52a2a][b]puts[/b][/color] [COLOR=#ff00ff]"Resulting global lists:"[/color]
[COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"Ele : %s"[/color] [COLOR=#008b8b]$elem_list[/color]]
[COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"Node: %s"[/color] [COLOR=#008b8b]$node_list[/color]]
[COLOR=#a52a2a][b]puts[/b][/color] [COLOR=#ff00ff]""[/color]

[COLOR=#0000ff]# write global lists to files[/color]
[COLOR=#a52a2a][b]set[/b][/color] fname [COLOR=#ff00ff]"radicalgiant_elem.txt"[/color]
[COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"Writing result to file: %s"[/color] [COLOR=#008b8b]$fname[/color]] 
write_data_fo_file [COLOR=#008b8b]$fname[/color] [COLOR=#008b8b]$elem_list[/color]
[COLOR=#a52a2a][b]set[/b][/color] fname [COLOR=#ff00ff]"radicalgiant_node.txt"[/color]
[COLOR=#a52a2a][b]puts[/b][/color] [[COLOR=#a52a2a][b]format[/b][/color] [COLOR=#ff00ff]"Writing result to file: %s"[/color] [COLOR=#008b8b]$fname[/color]] 
write_data_fo_file [COLOR=#008b8b]$fname[/color] [COLOR=#008b8b]$node_list[/color]
[COLOR=#a52a2a][b]puts[/b][/color] [COLOR=#ff00ff]"done."[/color]

Then running the script above causes this output and produces 2 files:

mikrom@mikrom-Lenovo-S500 ~/Work/Tcl_Tk $ tclsh radicalgiant.tcl
* processing file: ./radicalgiant_01.txt
  ==> This was matched: Ele  15235 16845 555845 21452 451254
  ==> This was matched: Nodes  15478 26558 78549 12547 12546 13556
current global lists:
Ele : 15235 16845 555845 21452 451254
Node: 15478 26558 78549 12547 12546 13556

* processing file: ./radicalgiant_02.txt
  ==> This was matched: Ele  15235 16855 665845 22452 461254
  Warning: element 15235 already exists in elem_list!
  ==> This was matched: Nodes  15478 26558 79549 13548 13546 14556
  Warning: node 15478 already exists in node_list!
  Warning: node 26558 already exists in node_list!
current global lists:
Ele : 15235 16845 555845 21452 451254 16855 665845 22452 461254
Node: 15478 26558 78549 12547 12546 13556 79549 13548 13546 14556

* processing file: ./radicalgiant_03.txt
  ==> This was matched: Ele  65235 56845 455845 31452 251254 111111
  ==> This was matched: Nodes  11478 11558 11549 11547 11546 11556 66666
current global lists:
Ele : 15235 16845 555845 21452 451254 16855 665845 22452 461254 65235 56845 455845 31452 251254 111111
Node: 15478 26558 78549 12547 12546 13556 79549 13548 13546 14556 11478 11558 11549 11547 11546 11556 66666

Resulting global lists:
Ele : 15235 16845 555845 21452 451254 16855 665845 22452 461254 65235 56845 455845 31452 251254 111111
Node: 15478 26558 78549 12547 12546 13556 79549 13548 13546 14556 11478 11558 11549 11547 11546 11556 66666

Writing result to file: radicalgiant_elem.txt
Writing result to file: radicalgiant_node.txt
done.

The result file radicalgiant_elem.txt contains the numerical data for elements

15235 16845 555845 21452 451254 16855 665845 22452 461254 65235 56845 455845 31452 251254 111111

and the file radicalgiant_node.txt contains similar data but for nodes.