Advice for reading BIG files please.

[ringd]

Hello!

I'm a JAVA newbee and am working on an exercise to read in a file, check the record text for a certain character string then write the record to an output file when the record has a certain character string.

Problem is, the input file is massive. The only class I have seen used in examples is 'BufferedReader' and it's 'readLine' method. The input file is way too big to buffer.

The 'FileReader' class has no associated 'readLine' method (at least I can't find one documented!).

Anyone have any ideas? Performance for this is not too much of an issue (if I have to read 1 record at a time). Can BufferredReader be used to read 'chunks' of the file at a time, rather than the whole?

Cheers,
Dave.

mailto:dave.ring@barclays.co.uk

 

[HavaTheJut]

I have never had any problems reading large text files with BufferedReader (have tried up to 3megs of plain text).
If you are having problems you can try specifying the buffer size

BufferedReader(myReader, bufSize)

Also, the read method in FileReader can be used as follows:

FileReader fIn = new ...
int numToRead = ...
char[] charBuffer = new char[numToRead];
int numRead;
numRead = fIn.read(charBuffer, 0, numToRead);

This tries to read numToRead characters into the array and returns the actual number read.

 

[cwinans]

I think you are confused on how the BufferedReader works... the buffer deals with the current line that is being read. Your file may be large but I'm sure that each line is not an insane amount of characters. This class should work for you without a hitch. Have you tried implementing this class into your project or did you assume that BufferedReader would not work? I hope this helped! ;-)
- Casey Winans

 

[meadandale]

A BufferedReader simply buffers some of the I/O to make performance faster and more efficient. The way this works is, the reader uses a buffer to 'buffer' data that it has read from the I/O source, in this case, the file. Rather than going out and reading a character from the I/O everytime you want one, the reader reads in a whole buffers worth and then serves you up the data locally out of the buffer until the buffer is empty. When the buffer is empty, it goes back out to the file and reads in a whole buffers worth of data again. The buffers are usually pretty small, say 4-8k. The data in the buffer is not only a line at a time. The buffer may contain one (large) input line or many. In the later case, several readLine() calls may be fulfilled from the buffer. In the former case, just one.