Yield and CPU usage 1

[madesjardins]

Hi there,

Currently writing a script that, at some point, needs to read through a file which will have more than 50000 lines and sometimes 200K or more. I'm doing so with a while not feof, and then some fgets'.

If I don't put a yield in the while loop, pw5.exe takes up 100% cpu usage. If I do put yield in the loop, CPU usage stays pretty low, but the script slows down to a crawl, making reading those 50k-200k files tedious and unpractical.

Is there any other to go about handling big files in an efficient way?

Thanks.

 

[knob]

I can't think of a happy medium between the two results you have seen. Have you checked to see if the problem is due to the fgets command and not other commands inside the loop by commenting out everything except for the yield and fgets commands?

http://www.aspectscripting.com

 

[madesjardins]

Just tried that. Commenting out the fgets doesn't change anything. I still hit 100% as soon as I get in the while feof loop.

I'm thinking I might have to write a dll to handle the file processing. I've never done that before but I'll look it up on your site and see what I can come up with.

 

[knob]

Can you post the contents of the while loop?

http://www.aspectscripting.com

 

[madesjardins]

Here you go. It'll a little all over the place as it still needs some polishing.

while not feof 443

fgets 443 LineBuffer
substr ReportDate LineBuffer 18 5
substr ReportName LineBuffer 27 6
ReportFileName = LineBuffer

if strfind LineBuffer ".z"

pwtitlebar Linebuffer

for i = 0 upto ReportNumberToGet-1
if stricmp ReportsToGet[0] ReportName
atol ReportsToGet[1] lReportDays
atoi ReportDate iReportDate
LongReportDate = JulToDate(iReportDate)
if ($LTIME/86400)-(LongReportDate/86400) <= lReportDays
strfmt julianfile "C:%d F:%d D:%d C:%s F:%s" ($LTIME/86400) (LongReportDate/86400) lReportDays ReportsToGet[0] ReportName
pwtitlebar julianfile

ltimeints LongReportDate YEAR MONTH DAY HOUR MINUTE SECOND

strfmt DownloadFile "r:\%s\%s" Region ReportName
mkdir DownloadFile

strfmt DownloadFile "r:\%s\%s\%04d-%02d" Region ReportName YEAR MONTH
mkdir DownloadFile

if strcmp ReportCurrent "Y"
strfmt CurrentPath "r:\%s\%s\Current" Region ReportName
mkdir CurrentPath
endif

strfmt ReportDate "%04d-%02d-%02d" YEAR MONTH DAY
strfmt DownloadFile "r:\%s\%s\%04d-%02d\%s %s %s.txt" Region ReportName YEAR MONTH StoreNumber ReportName ReportDate
strfmt DownloadFile "r:\%s\%s\Current\%s %s %s.txt" Region ReportName StoreNumber ReportName ReportDate

if not isfile DownloadFile

strfmt ReportFetchString "cp %s /home/ReportMining%s.gz;sz -b /home/ReportMining%s.gz^J" ReportFileName StoreNumber StoreNumber
transmit ReportFetchString
waitfor "**"
getfile ZMODEM

while $XFERSTATUS
yield
endwhile

transmit "rm /home/ReportMining*.gz^J"
waitfor "root@"

strfmt ReportFetchString "gzip -d -f ..\Download\ReportMining%s.gz" StoreNumber
run ReportFetchString HIDDEN GzipTaskID
while taskexists GzipTaskId
yield
endwhile

strfmt ReportFetchString "..\Download\ReportMining%s" StoreNumber
if strcmp ReportCurrent "Y"
strfmt LatestReportFile "%s\%s %s*.txt" CurrentPath StoreNumber ReportName
if findfirst LatestReportFile
delfile $FILESPEC
while findnext
delfile $FILESPEC
endwhile
endif
delfile CurrentFile
copyfile ReportFetchString CurrentFile
endif

rename ReportFetchString DownloadFile
loopfor
endif
endif
endif
endfor
endif

; yield

endwhile

 

[madesjardins]

You know what? As I pasted it I realized what you asked for in the first place, which was to comment everything BUT the fgets and the yield commands. I did that and found that the for loop is execute way too many times when it doesn't have to.

I added an extra check to prevent the script from entering the for loop unless it's needed (it's only needed on few of the lines in the huges files) and it's much more reasonable now.

Thanks a lot.