Captureing large amounts of data

[wilsonza]

I am trying to download a large amount of data for bank accounts. I login, then start entering account numbers that I read from a text file. I enter in all of the account #'s, (About 10,000 of them), enter a date, then start a capture and transmit a go for download.

For some reason I keep getting disconnected after right around 13,500 records recieved. Is there some sort of limit that I am not aware of? Here is my code:


proc main
string account
delfile "c:\financial\data\account_data.txt"
set capture file "account_data.txt"
set capture path "c:\financial\data\"
connect telnet "Datasource"
waitfor "begin logging on ****"
transmit "^M"
waitfor "Username: "
transmit "xxxxx^M"
waitfor "Password: "
transmit "xxxx^M"
waitfor "#ENTER USERCODE/PASSWORD PLEASE -"
transmit "xxxx xxxx^M"
waitfor "(MM/DD/YY) OR HIST FOR HISTORY"
transmit "hist^M"
if fopen 0 "c:\financial\data\accounts.txt" READ TEXT
while not feof 0
fgets 0 ticker
waitfor "MM/DD/YY OR BYE"
transmit "("
transmit account
transmit &quot01/01/97 01/01/98^M"
endwhile
else ; Display error message
errormsg "Could not open accounts!"
endif
fclose 0
waitfor "MM/DD/YY OR BYE"
transmit "bye^M"
waitfor "C TO CONTINUE"
capture on
transmit "c^M"
waitfor "&& 0 0.01^M^J"
capture off
transmit "end^M"
disconnect
endproc

 

[hankm3]

Wilson,

I perform something close to what you are doing. I have a Capture file that is 991K in size. I can open it and extract what I need from it. Not sure but I don't think there's a File Size Limit, not on Win95 and up.

Hank

 

[knob]

I'm not aware of any limits on a capture file's size. One test you can run is commenting out all of the capture-specific commands in your script, then run your modified script. If your script continues to have problems, then you have eliminated the capture file as a possibility. Is this a fairly consistent problem, or does it just occur now and then?

 

[wilsonza]

The problem seems to have been the last waitfor command. The download is about 350 MB, so the default time seems to be about 20-25 seconds to wait. I think my script did the wait, timed out, then disconnected. I added a FOREVER to the end of the last wait statement, that seems to have fixed it.

Now I am having trouble with the end of download symbols. Based on success or failure, I get one of two codes (&&& .001 or ERR) My script will recognize the &&& .001, but I cannot get it to recognize the ERR as a close code.

I've tried waitfor "&&& .001 || ERR" but that doesn't seem to work. The problem is, I need the wait forever to stop my connection from being closed, but if I do that, I get stuck in a connection due to the waitfor forever holding open the connection and not recognizing the ERR. Any ideas?

 

[knob]

The waitfor command can only look for one string, so trying to have it do an or operation will not work. You might want to look into using two when target commands instead. One when target command would call a procedure if &&& .001 is received and perform the necessary operations to finish the script's execution, while another when target command would call a second procedure if the ERR result code was detected. If it is possible for your script to receive either the &&& .001 or ERR string before the transfer is complete, then you will not want to have the when target commands in your script until right before you anticipate receiving &&& .001 or ERR. The discussion of the when command in the ASPECT help file has an example of when target that probably explains the command better than I am.

 

[wilsonza]

I tried that, it seems to disconnect me before I am able to download any data. Is there any way to tell Procomm "Wait until I am done recieving this data"? I just need to get all of the data down, and be able to end the script.

Also, is there a way to close the procomm window? If I wanted to say, schedule a procomm script to run as part of a batch file, the rest of the batch file will not begin execution until the procomm instance is destroyed. Since procomm seems to only disconnect you and/or stop your script, Im wondering how I could use a batch file...

 

[knob]

Is the disconnection happening due to the disconnect command you have in your script? If so, what you could do is put these lines at the end of your script in lieu of the disconnect command:

while 1
yield
endwhile

The two procedures that are called by your when commands can then include a disconnect command at the end of each procedure. This will keep the disconnection from happening before your script is done processing all of the data.

As for closing the Procomm window, you can use the pwexit command to perform this task.