How to read the tail 1Mbyte text out of a 700Mbyte log file 1

[strongm]

You may find the following a little faster in retrieving the last ByteCount bytes from a file as a string:

[blue]Private Function tail(strFile As String, Optional ByVal ByteCount As Long = 1024& * 1024) As String
    Dim hFile As Long
    
    hFile = FreeFile
    Open strFile For Input Access Read Lock Read As hFile
    If ByteCount > LOF(hFile) Then ByteCount = LOF(hFile)
    Seek hFile, LOF(hFile) - ByteCount + 1
    tail = Input(ByteCount, hFile)
    Close hFile
End Function
[/blue]

 

[coolago]

After sevel week's running, the size of the tomcat log file will get very large. E.g. yesterday evening the size of the log file gets 700M bytes large. But when tomcat goes un-normally, it's the tail 1M bytes of the log file that can do we help. Just according the tail 1M bytes of this large file, then we can know what's the problem with the tomcat service.

But we meet a problem, we cann't open the 700Mbytes file in the running server computer, it's a dangerous and time-wasting operation. If we open it directly with Win2000's Notepad problem, the performance of the server will be affected!

Can we get the tail 1M bytes out of this 700Mbytes file in another method?

Now I have tried two ways to do this in VisualBasic programing:

Firstly using the FSO object. I.e use SkipLine() to move the file point to some tail of the log file, then use ReadLine() function to get the tail lines. But it'll take too many time to move the file point the the some tail of the file. The SkipLine() wastes too many times.

Secondly through the file split method. The code pasted below. This method still wasted some time, even let the EMS memory overflowed some time if the log file get larger than 800Mbytes.

Can any one help please, i.e. how to write a VisualBasic program to read the tail 1M bytes(or the last 10000 lines) from the 700Mbytes big file?

Thanks in advance for your reading, and your so thankful advice!
Thank you very much!

Vincent Wu

Vincent
---------------------
If u think u can, then u can!

 

[coolago]

The main function that my second method uses:

Public Function SplitFile(SplitFileName As String, BeginningNumber As Long, ReturnErrorDes As String, Split As Long, oName As String) As Boolean
'Use this function to do file spliting. The function return a boolean value, true for successful, false for fail.
'The function has 4 parameters, i.e. the file name of the split file, the begin number, the file size of the generated file after split,
'and the generated file name after split
'After split, it will just generate the last 2 files in the disk, but ignoring all the other files, becasue only the last 2 files are useful to us.

Dim SaveName As String 'the generated file name after split

Dim fnum As Integer, fnum1 As Integer 'file channel variant


SplitFile = True 'if all does ok, then return true

On Error GoTo CleanUp

Dim CurrentFile As SectionedFile, m_lngNumFil As Long, m_LngLoop As Long, FilesLen As Long


FilesLen = FileLen(SplitFileName) 'get the file size of the generated file after slit

If FilesLen <= Split + 1 Then

SplitFile = False

ReturnErrorDes = "the new file size is too large, plese set it again!"

Exit Function

End If '


fnum = FreeFile 'return the first idle channel to read

Open SplitFileName For Binary As fnum 'open the source file by binary method

If CInt(FilesLen / Split) >= FilesLen / Split Or CInt(FilesLen / Split) = FilesLen / Split Then

m_lngNumFil = CInt(FilesLen / Split)

Else

m_lngNumFil = CInt(FilesLen / Split) + 1

End If 'calculate the number of the files that will generated after split

ReDim CurrentFile.Files(1) 'assign a Ems memory associated array

For m_LngLoop = 1 To m_lngNumFil

'this loop do the split work, and generate the files after split

If m_LngLoop < m_lngNumFil Then 'if not the last loop

ReDim CurrentFile.Files(1).Bytes(1 To Split)

're-assign the EMS memory array as the size of the split size

Get #fnum, , CurrentFile.Files(1).Bytes

'read binary data as the split size into the EMS memory array

Else 'if it the last loop


ReDim CurrentFile.Files(1).Bytes(1 To FilesLen - ((m_lngNumFil - 1) * Split))


're-assign the EMS memory that the same size as the lefted

Get #fnum, , CurrentFile.Files(1).Bytes

Close #fnum 'close the file reading channel

End If



'minus 2 for only generate the last 2 file, ignoring all the other files
If m_LngLoop > m_lngNumFil - 2 Then

SaveName = oName & "." & Format(BeginningNumber - 1 + m_LngLoop, "00#") 'calculate the generated file name, the extend name is 00x

fnum1 = FreeFile 'get the second idle channel to do writing

Open SaveName For Binary As fnum1

Put #fnum1, 1, CurrentFile.Files(1)

DoEvents

Close #fnum1 'writing the generated file by binary method

End If

Next

Exit Function

CleanUp: 'if it has exceptions

ReturnErrorDes = Err.Description

SplitFile = False 'return false

End Function

Vincent
---------------------
If u think u can, then u can!

 

[coolago]

Thanks to Strongm's hint. It works! Now display the tail 1Mbytes of a 800Mbytes file only takes 3 seconds. It's wonderful!
As a summary, first use the seek() function to move the file point to the some tail of the big file, then use the get function to read the text into the EMS memory. At the same time when reading, write it into the new generated file.
It done!
Thanks again!

Vincent
---------------------
If u think u can, then u can!

 

[DrJavaJoe]

Vincent, click that link that says:
Thank strongm
for this valuable post!



Two strings walk into a bar. The first string says to the bartender: 'Bartender, I'll have a beer. u.5n$x5t?*&4ru!2[sACC~ErJ'. The second string says: 'Pardon my friend, he isn't NULL terminated'.