Reading / Writing Files, What's the most efficient way

[Webrookie]

I put together an app for my company which splits large text files (3gb+) into two files so that some of our other software may work with the files. This app was done in VB.

I was recently introduced to C# (last week) and was expecting it to be faster / more efficient. But when I finished the C# app and ran it, it took approximately the same amount of time.

So I'm wondering if the method of opening, reading, and writing data I chose is inefficient and if there is a preferred, better way of doing so. It seems like I've seen several different ways of accomplishing this from searching the internet

			string InputPath = this.lblInput.Text;
			string OutputPath = this.lblOutput.Text;
			string InputFileName = Path.GetFileNameWithoutExtension(InputPath);
			string OutputFileName1 = OutputPath+"\\"+InputFileName+"-split1.dan";
			string OutputFileName2 = OutputPath+"\\"+InputFileName+"-split2.dan";

FileStream InputFile;
FileStream OutputFile;

InputFile = new FileStream (InputPath,FileMode.OpenOrCreate,FileAccess.Read);
StreamReader InputStream = new StreamReader(InputFile);			
OutputFile = new FileStream(OutputFileName1,FileMode.CreateNew,FileAccess.Write);
StreamWriter OutputStream = new StreamWriter(OutputFile);
currentLine = InputStream.ReadLine();			
while (currentLine != null)
{
...
//if at position to split file
{
OutputFile.close()
OutputFile = new FileStream(OutputFileName2,FileMode.CreateNew,FileAccess.Write);
OutputStream = new StreamWriter(OutputFile);
}

OutputStream.WriteLine(currentLine);			
currentLine = InputStream.ReadLine();
Application.DoEvents();

}

OutputStream.Close();
InputStream.Close();

That's the basic gist of it. Is there anything I can do to speed it up? Thanks

 

[chiph]

Try running some perfmon counters when your programs are running.

I suspect you'll see that the programs aren't CPU bound, but I/O bound as the poor disk drive struggles to keep up.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[Webrookie]

Thanks Chip, that's what I had suspected, just making sure I was on right track.

 

[chiph]

I suggest you get some hard numbers, just in case the boss asks why it didn't improve.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[stevexff]

VB.NET and C# both compile down to IL, which gets run by the CLR. All of the .NET framework classes you are using in your code are written in C#. When you take those out of the equation, there is probably very little code left that's actually a direct result of the things you have coded yourself, in either C# or VB.NET.

Any minor differences between the IL produced by the two compilers probably won't have any impact on performance.

I don't know if you can get any improvement by using one of the filestream constructors that allow you to specify a buffer size?

 

[cappmgr]

//if at position to split file
Does this parse every line looking for something. This could be a big factor.
Marty

 

[cappmgr]

stevexff,
I'd like to respectfully disagree a little about the compiling down to IL code. It's sometimes hard to tell when you are using C# code and not VB code.
I would think that
VB
If str = "" Then
and
C#
if (str == "")
would perform the same. They do not.
if (str == "") is much faster. I wondered why.
disassembling the code

VB str = "" results in
int32 [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.StringType::StrCmp(string, string, bool)

C# (str == "")
bool [mscorlib]System.String:p_Equality(string, string)

I would not use this to check for an empty string it is just to prove a point.
Marty

 

[chiph]

Marty's post proves my point. Unless you have some performance metrics, both before and after, you can't:

1) Figure out where the slow-down is happening.
2) Prove that you improved things after changing code.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first