Working With Text Files 1

[SSJ]

hi, I'm in the middle of a problem working with text files in java.

How can I delete a certain line on a text file on Java?

I have a file like

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
111111111111111111111111111111
sadsdssssssssssssssssssss
sssssssssssssssssssssss

If I'd like to delete line 2 (the one with the 1111111111) how should I do it?
I don't need to know what's written there assume I only need to delete the line number 2(or any other number) of my text file?
It's easy to append text to the end of a text file, but deleting it seems to be really complicated. Any tips?

thank you

 

[alebu]

Hm... Well, lets look at this situation closely...
You have text file and you KNOW number of line which you
want to delete. It means that file is allready readed and is in memory in accesible form ( like array of String or something else ). So all you need is WRITE those String's to original file ( overwrite it ). Am i right ? But if you want to CUT some data from file ( it is solution of your problem to ) then you need CHANGE your queswtion on something like "How to cut data from file when I know offset and length of "CUT BLOCK". ;-). Last will require to read data after CUT STRING to some buffer, then set file pointer on position where CUT STRING begins and flush buffer here. And don't forget to change size of file! I don't see any REALLY easy solutions, sorry, try to formulate your Question better and I will answer again. ;-)

 

[jfryer]

If you are just wanting to remove a specified line (eg line 2), so your original file -
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
111111111111111111111111111111
sadsdssssssssssssssssssss
sssssssssssssssssssssss

now looks like this -
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
sadsdssssssssssssssssssss
sssssssssssssssssssssss


Then I would read each line in the file to an Arraylist (each line of the file is an array element stored as a string) ignoring the line numbers you wont to delete (ie reading them but not storing them tothe arraylist, and then overwrite the file writing each line in the arraylist back.

There is a problem in the amount of memory you will need for the arraylist if the text file is very large. If this is the case then a better suggestion may be to read each line from the original file and write it straight to a new file, skipping the line(s) you wish to ignore. You can then delete the old file and rename the new file to the old files name.

Not sure which solution is better in terms of performance, the first solution reads data from the disk, to a memory location and then back to the disk, the second reads data from the disk straight to another part of the disk. Neither are brilliant solutions but they will solve your problem.

----------------------------------------
There are no onions, only magic
----------------------------------------

 

[SSJ]

ok, let me explain a little better what I need to do.

I need to read a text file, and each line of this text file has a SQL Insert or Update statement that I'll have to run to a DB.
So I need to read the file line by line, run that SQL statement and then delete that line from the file.
The problem here is the time it will take to process if I read the file line by line into a ArrayList delete the line and the re-write all the info again into another file. If I have 100 lines I'll have to rewrite the file 100 times.
I'm tryed to use a MappedByteBuffer, but I'm having lots of trouble with it. I was able to get the file into a MappedByteBuffer, but I don't know how to rewrite the file so the line will disappear...

Anyone has a good sugestion for this then?

 

[alebu]

Something is still unclear to me... Way do you need to delete line from file after each request to DB? Isn't it better to read all lines to Array then one by one make requests to DB and in the end save only those lines that should be saved. Because deleting lines from file each time will requere to overwrite that file each time! Offcouse, in most cases this file will be keeped in memory by OS but anyway it is very "not optimized" solution. Try to change your algorhitm by organizing some sort of buffer for your text file... But if there still are some reasons for deleting line each time then by my opinion main steps should be like those:
----
1) Read file into Array ( because we always will know what changes were made in file )
2) read string from that Array and make whenever you wanna do with it ( Including call to DB ). Simply after working with line delete it from Array ( its easier then from file )

3) Save all Array line by line in original file, not appending but overwriting it ( by default as I remember )

4) return to (2)
----
Rereading file after each writing have sence only in case then this file can be changed from outside ( by other program or thread... )
Deleting line from Array is much much faster then deleting it from file...
But my first sugestion still is the same ( exchange 3 with 4 ) :-D

If you have questions on details then write...

 

[SSJ]

alebu: That's the easier way, but what would happen if there is a power cut b4 u actually write the new lines into the file. All your updates would be lost and the file would contain lines that were already executed! That's the reason I need to update each time I execute a line.

And when u say:
3) Save all Array line by line in original file, not appending but overwriting it ( by default as I remember )

This also leads to a problem I think. If your original file has say 10 lines and after the updates you execute 5 of that lines if u overwrite the original file, the last 5 lines of it will remain there!

I realise writing the file all over again is very time consuming, so I'll probably have to think in another workaround like writing the executed lines into a new file and then compare it with the original one. Suggestions are appreciated...

 

[sedj]

SSJ,

I have a couple of questions/comments :

Why do you need to update a database with transactions from a text file ? Is this a transaction log from a live machine that you are updating a shadow db with the day's transactions ? If this is so, then there must be better ways to do it ...

If you are not updating a shadow db ,then is it really necessary to carry out your goal using the suggested methods ? It is all very well reading a file into memory - but it seems a bit messy - plus if the file was huge, the system resources may strain considerably.
If you do the line by line method, then you are accessing the physical disk loads of times which is also resource intensive ...

I suggest reconsidering what your end goals are, and trying to find another path to them...

 

[SSJ]

ok, things are like this. This system will be used in a manufacturing line. There will be around 30 workstations receiving production data from the line and they will write that in their local DB and will send the SQL updates to a server through sockets so the data can be rewritten into a server DB ( it's +/- a shadow DB). Things is I need to store the messages physically to avoid losing data if the power goes down. So I receive the messages insert them into a array list and at the same time into a file. I work with the array list all the time, but I need to maintain in disk a copy of the current array list state otherwise in case of power cuts I wouldn't know where to start from.
I'm quiting the idea of deleting the lines from the file and maybe writing the lines I exectued to a new file and at the beginning compare the 2 files to see if there are any non-exectued statements and if not delete the files. I don't know if this is the best method so if u have any better idea please let me know

 

[sedj]

These thirty workstations - if they can connect to a server, then would it not be better to have just one database on the server, and then using a ConnectionPooling type jdbc application which updates the server's database on the fly from each workstation ?

Plus, buy a UPS device (big battery basically), which will cut in to power your servers if mains power ever dies.

 

[SSJ]

About the workstations connecting directly to the server I'm not using it since they must be able to work isolated, even if there is no network for example so they need to have a local database. And besides the software that's running on the local machines is done in VB not in Java.
About the UPS, we already have one, but we need to assure data isn't lost, like if someone shuts the server down or if we need to make a reboot or in case of really long power downs.

 

[sedj]

Criky !
I guess then, all things considered ... I would read in the updates file line by line, executing sql as you go, and then writing out each executed line to a new file, which then, as you said, could be compared by line count to the original file ...

Good luck !

 

[SSJ]

Thanks for the replies sedj.
I've already implemented that and I'm testing it at this point, it's quick the only consideration here is the file growth that can be too big. I'll have to think on a method to delete the file from time to time maybe during the weekends. But so far it seems to have no time consuming issues

 

[sedj]

Or ... if you wanted to be a bit flash, you could zip up the file as you wrote it - check out the java.util.zip package - if you can use Streams, you can use the .zip packagae !

 

[SSJ]

That could be a really interesting solution since the files can eventually grow to unexpected sizes. I've used the java.util.zip package once, but I think that I can't write to a file inside the zip archive on the fly can I?
Or would I have to write the line in to a file zip the file into the archive and delete the txt file and so on?

 

[sedj]

Try compiling and running the below - its not the best bit of code out, and could do with some refining, but it basically does what I think you are after - ie - it opens up zipoutputstreams etc, and lets you write out compressed lines on the fly, and appears to unzip those lines quite happily (though with a stack of whitespace, but I'm sure you could get around that with some String.trim() somewhere)
once the streams are all closed up. See what you think anyway !

Ben

import java.io.*;
import java.util.zip.*;

public class ZipFileTest {

FileOutputStream fos;
GZIPOutputStream gzos;
Checksum check;
static CheckedOutputStream cos;

public void openStreams() {
try {
fos = new FileOutputStream(new File("file_compress_out.txt&quot);
gzos = new GZIPOutputStream(fos);
check = new Adler32();
cos = new CheckedOutputStream(gzos, check);
}
catch (IOException e) {
System.out.println("IO Error .... " +e.toString());
}
}

public void addZIPLine(String s) {
StringBufferInputStream sbis = new StringBufferInputStream(s);
try {

byte[] buffer = new byte[4096];
int bytes_read = 0;

while ((bytes_read = sbis.read(buffer)) != -1) {
cos.write(buffer);
}
}
catch (IOException e) {
System.out.println("IO Error .... " +e.toString());
}
}

public void closeStreams() {
try {
cos.close();
gzos.close();
fos.close();

}
catch (IOException e) {
System.out.println("IO Error .... " +e.toString());
}
}


public void GZIP_UnCompressFile() {
try {
File out_file = new File("file_uncompress_out.txt"
File in_file = new File("file_compress_out.txt"

FileInputStream fis = new FileInputStream(in_file);
GZIPInputStream gzis = new GZIPInputStream(fis);
Checksum check_in = new Adler32();
CheckedInputStream cis = new CheckedInputStream(gzis, check_in);
FileOutputStream fos = new FileOutputStream(out_file);

byte[] buffer_in = new byte[4096];
int bytes_read_in = 0;

while ((bytes_read_in = cis.read(buffer_in)) != -1) {
fos.write(buffer_in);
}

fos.close();
long sum = check_in.getValue();
}
catch (IOException e) {
System.out.println("IO Error .... " +e.toString());
}
}

public static void main(String args[]) {
ZipFileTest app = new ZipFileTest();
app.openStreams();
app.addZIPLine("hello"
app.addZIPLine("there"
app.closeStreams();
app.GZIP_UnCompressFile();
}
}

 

[SSJ]

ok now this is getting somewhere
The only problem with that code of yours is the whitespaces after compression and that's because of the buffer size(4096), if I make a trim b4 writing into the file or make the buffer with the exact size of my messages I won't have the whitespace problem. So in theory this might work. I'll have to give it a better try later on. I will start testing things as they are right now (ie with no compressed files) and at the same time working on implementing this way to see if it's worth it or not.
Thanks a lot for the code it can be really useful!

 

[Stevoie]

I had this problem aswell awhile back and got round it by overwriting the first 2 chars in a line with XX if the line was executed successfully. This means after a power cut the program checks for XX at the start of a line and just skips it.

This doesn't change the file size but it is a fast, easy solution. Hope it helps

StevoIE