C# Multithreading question, looking for some direction

[robertbcoding]

Using Directives

namespace myProject
{
class myClass
{
Global Variables

static void Main(string[] args)
{
// get a list of files
string[] files = Directory.GetFiles(myDirectory);

Thread[] workers = new Thread[files.Length];

for(int i = 0; i < files.Length; i++)
{
Work w = new Work(new object[] { files });

ThreadStart threadDelegate = new ThreadStart(w.dowork);

Thread worker = new Thread(threadDelegate);

workers = worker;

worker.Start();
}

foreach (Thread workerThread in workers)
workerThread.Join();


Console.WriteLine("press enter to close...");
Console.ReadLine();
}
}



public class Work
{
private string myFile;

public Work(params object[] list)
{
this.myFile = (string)list.GetValue(0);
}


public void dowork()
{
Thread.CurrentThread.Priority = ThreadPriority.Normal;
Thread.CurrentThread.IsBackground = true;

string myLine = null;

StreamReader sr = new StreamReader(myFile);

while ((myLine = sr.ReadLine()) != null)
{

using (Process proc = new Process())
{
// I set my arguments from the line read from the file
// set my output to write to a unique file
// each thread has it's own file, each line in each file executes a process
// and each process writes output to a unique file

proc.Start();
proc.WaitForExit();
proc.Close();
}

}

sr.Close();
}
}
}


The code above is no where near complete, I took any extra code I felt was uneccessary for this problem.

Basically this code gets a list of files from a directory and creates a thread for each one of those files. (3 files means 3 threads get created) The file name is passed to the Work class and the thread is started. In the dowork() method above the file is opened and for each line in the file a process is executed. After the process is executed a new file in a seperate location is created and the output from the process is written there. (3 files, 100 lines in each file means 300 files are created with output written to each one. This of course is done by 3 seperate threads, one for each initial file.) I hope that's clear enough.

The completed code works without error creating the correct number of threads, creating the correct number of output files containing the correct output from the process executed in the dowork() method.

Here is my problem:
Lets say that there are 3 files each containing 100 lines. For testing purposes I specifically created these 3 files to be completed in a specific amount of time. I know for a fact that file 1 will take exactly 1 minute to complete, file 2 will take 2 minutes and file 3 will take 3 minutes to complete.

When I run my code with these 3 files all 3 threads finish at the exact same amount of time of 3 minutes. This is because the 3 threads created will each grab the first line from their file and execute the process in dowork(). The first thread should finish the process using the first line and move on to the second line before threads 2 or 3 becuase that's the way the files were engineered. However, this is not the case. Even though thread 1 finishes line 1 execution before the other 2 threads it will not move on to the second line of the file until the slower of the 3 threads finishes it's first line execution. I know this from watching the output files being created. Why is this so? I want the threads to run independantly. The first thread should be executing the processes and moving on to the next line 3x faster then the slowest thread, and creating output files. These threads seem to be synced based on reading the lines and I don't know why. The Thread.Join() statement does not have any effect on this issue.

 

[JurkMonkey]

Threads are not as independent in .Net as you would expect. They also don't get work time distributed evenly - which is actually by design.

An alernative is to create a separate exe to do the parse and start each job in it's own process. That may allow windows to dedicate a little more to each one. The other option is to use only 1 thread and let each job do one line in turn. Once a job is finished the next job can take it's place.


File 1 File 2 File 3
--- --- ---
--- --- ---
--- --- ---
--- --- File 4
--- --- ---
--- --- ---
--- File 5 ---
--- --- ---
--- --- ---


Someone else may have a better suggestion?

http://www.ubuntu.com/

 

[chiph]

I know for a fact that file 1 will take exactly 1 minute to complete, file 2 will take 2 minutes and file 3 will take 3 minutes to complete.

Actually, you don't know this.

If the machine is under a high load, it can take longer to create those files than you anticipate.

Thread.CurrentThread.Priority = ThreadPriority.Normal;

You shouldn't have to do this, as the launching thread will already be running at Normal priority.

Also, you're using your threads to launch a separate process -- that's overkill. You might want to just wait for your Process instance to raise it's Exited event. You know how many processes you launched, and when Exited is called that many times, you know you've at least tried to read all the files.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first