Memory Management

[ralphtrent]

Hi
I have an app that I wrote that takes a text file, reads it line for line. If the line meets certain criteria, then I store the line in an ArrayList. Then I take the array list and sort it so I can analyze each line. That part is all done and complete. I just got a text file that 1.5 gig, yes, 1.5 GIG. When running the app against the file, the memory usage is about 900 meg and then my system freezes. Is there a better way to do this? The reason I through it in an array list is because of its sorting feature and I can go through each item, one by one.

Thanks
Ralph

 

[SqueakinSweep]

Ralph

Thats a damn big file....I would have thought it wouldnt take much work to swap the arraylist to a datatable.



Sweep
...if it works dont f*** with it
...if its f****ed blame someone else
...if its your fault that its f***ed, say and admit nothing.

 

[stsuing]

You may think about using SQL Server (or some other storage medium)to hold and sort the information. I imaging 900 meg of strings bogging down a system

 

[Dimandja]

Use Access or SQL to sort.

 

[ralphtrent]

well the 1.5 gig is a rarity, the app handled a 250 meg file very nicely. I was thinking about using a datatable, but would I have the same problem with memory being that they both are stored in memory?

Thanks,
Ralph

 

[chrissie1]

Of course you would. Where do you think he puts them? You should try and split it up some way or use a database server. For example split it up in ten pieces, sort each them and then it depends on the data you have.

Christiaan Baes
Belgium

If you want to get an answer read this FAQ faq796-2540
There's no such thing as a winnable war - Sting

 

[ThatRickGuy]

do you need to store the entire line? Can you filter the listing down from the 1.5gig? You mention that you only want certain lines, are you sure you are only storing those you need?

-Rick

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.

 

[ralphtrent]

yes I am sure I am only getting the lines that I need. I tried the Datatable theory, the program ran slightly fast, but use more memory. I think I may have to try the database theory.

 

[chiph]

Are you changing the line, or just looking for it?

Because if you're just looking for it, you can read the file as a series of bytes, and when you get a match against the first byte in your "match" array, you set a flag and start doing more matching. Only if all the bytes line up do you say that you found a match.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first