vbs script not processing 55MB file but 20MB works well

[Yuggy16]

Hello All,

I have a windows vbs script which take a txt file as input and do some validation for the content of the file.
As output, the vbs script create another 2 txt files based on the input file.

Yesterday, i got a 55MB file to process but the vbs script could not process it nor give an error message.
I broke the 55MB file into chunks files and process them, all works fine with all the chunks file.

Does vbs has a problem for processing large text file? and how can this be solved?
Many thanks

 

[guitarzan]

Does vbs has a problem for processing large text file?

Depends on the method you use Since you didn't offer up any code or give a hint as to which method you are using, I will give you a link to an interesting article on the subject. It's quite old but still accurate I believe:

http://www.4guysfromrolla.com/webtech/010401-1.shtml

 

[Yuggy16]

Hello,

Thank you very much for this link, it gives me a better idea of how to handle text files.

In my case, after doing some validation for each line in the text file, i am adding the valide lines in a dictionary via the function below:

Private Sub AddLigneFicSCD(in_Line)
Call Dic_LinesFicSCD.Add(Dic_LinesFicSCD.Count, in_Ligne)
End Sub

Dic_LinesFicSCD has been initialised as:

Set Dic_LignesFicSCD = CreateObject("Scripting.Dictionary")

Then to create the ouput file I have created the function below:

Private Sub CreateFicSCD()
Dim Dic_key
Dim Fs
Dim Fd

Set Fs = CreateObject("Scripting.FileSystemObject")
Set Fd = Fs.OpenTextFile(Str_PathFicSCD & 'Output.txt', 2, True)

If Dic_LinesFicSCD.Count > 0 Then
For Each Dic_key In Dic_LinesFicSCD
Fd.Writeline(Dic_LinesFicSCD(Dic_key))
Next
End If

Fd.Close

End Sub

Is it because I am working with buffer before sending whole valide lines to output?
Many Thanks

 

[Geates]

Depends on the method you use Since you didn't offer up any code or give a hint as to which method you are using, I will give you a link to an interesting article on the subject. It's quite old but still accurate I believe:

http://www.4guysfromrolla.com/webtech/010401-1.shtml

Interesting. The method they refer to as "extremely fast, but it’s a bit of a cheat" is my method of choice. However, I can see .read being more efficient.

@Yuggy16, did you find a solution? I would recommend, as this article eludes to, reading and processing chuncks of the file.

-Geates

http://raqxtr.sstar.com/caspdoc/html/chapter_5__developer_s_reference.htm

 

[Yuggy16]

Hello Greates,

Yes the simplest solution is instead of storing the lines in buffer dictionary, write each lines on the fly directly to the files. This works well but slow down a bit the process. No matter for me, the most important is that I made this thing works.