Reading and writing a large text file

[wawalter]

Hello,

I am trying to read in a large text file (2-6 gig), parse what I need and write it out to a smaller text file which is about 20% of the size of the original. The problem is in this read/write and amount of memory the program uses just keeps growing and growing until with a 4 gig or larger input file the program will crash. The error I usually get is something about not having MDAC 2.6 (I have 2.81), so that doesn't make sense to me.

Here is the basic code I am using. I hoping someone has a better way of doing this that will not cause the memory growth. Thanks for your help.

            using (StreamWriter swOut = fiOut.CreateText())
            {
                if (fi.Exists)
                {
                    using (StreamReader sr = fi.OpenText())
                    {
                        string input = null;
                        string output = null;
                        while ((input = sr.ReadLine()) != null)
                        {

			    //
			    // parse the input and create the output line here
			    //
                            swOut.WriteLine(output);


                            count++;
                        }
                    }
                }
	    }

 

[jmeckley]

will the exact code above crash the system? if so then you need to find another way to parse the file. if not, then your problem is something else, not reading the large text files.

my understanding is MDAC deals with database connectivity, not file access. Maybe this is a symptom of actual problem.

2-6 GB text files are huge. PERL was designed to parse files. this may be a better choice for manipulating 6Gb text files.

Jason Meckley
Programmer
Specialty Bakers, Inc.