WebRequest and threads cause problems

[juhaka]

Hi again!

One problem solved, a new one occurs...

I have an application, which reads and writes to a website periodly every 5 seconds. I'm using (Http)WebRequest object. The application makes about 1200 successful read and write -process, but after that an error occurs. There were not enough free threads in the ThreadPool object to complete the operation. The error occurs especially in read process.


The first, I have openConnection() -method, where I create
a new instance of HttpWebRequest (named _webReq) for every
read and write process.

public XmlDocument readData()
{
XmlDocument data = null;
if(this._webReq != null)
{
WebResponse webRes = null;
StreamReader sr = null;
Stream stream = null;
try
{
//Reading xml over HTTP
this._webReq.Method = "GET";
webRes = this._webReq.GetResponse();
stream = webRes.GetResponseStream();
if(stream != null)
{
sr = new StreamReader(stream);
string xmlStr = sr.ReadToEnd();
//Let's create a new Xml -document
data = new XmlDocument();
data.InnerXml = xmlStr;
}
else
{
LogWriter.error("Getting a responsestream failed!", null);
}
}
catch(WebException ex);
{
data = null;
LogWriter.error("WebExcepion was catched in readData method.", ex.Message);
}
finally
{
if(sr != null)
{
sr.Close();
}
if(webRes != null)
{
if(stream != null)
{
stream.Close();
}
webRes.Close();
}
}
}
return data;
}


And I have own method for closing the connection


public bool closeConnection()
{
if(this._webReq != null)
{
try
{
this._webReq = null;
}
catch(Exception ex)
{
return false;
}
}
//else...the object doesn't exist, there is no connection to close
return true;
}



When the error occurs, the stack trace looks like 'at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state) at System.Net.HttpWebRequest.GetResponse()'

So, where is the problem? Should I use ThreadPool or something....

br.
JuhaKa

 

[alexboly]

I think that the web server uses thread pooling for requests. That means that it has already a number of threads created and it only passes them to the applications that make requests. The number of threads available is a parameter of the web server.
Now, in your case, because you are doing a request on a periodical basis and at a very short time period(5 sec), it may happen that the previous requests occupy all the threads of the web server and you cannot take resources anymore.
Now, I don't know exactly what kind of operation you are doing. To solve the problem you have to use fine tuning. Either add new threads to the web server, or increase the time between two requests to the server, or try to optimize your application so that it is faster. Without knowing the nature of the processing, it is rather difficult to say. And I have to say that I find it odd that you would need something like this...
I hope this helps,
Alex

 

[juhaka]

Thanks, Alex.

If I understood right your message, I have no errors in my code and this is further a recourse problem, isn't it. Or is there still a bug somewhere, like WebRequest -object? (I have installed ServicePack 2)

I have watch the process with netstat and Task manager. Task Manager shows there are about 17 threads running in 'normal' state and just before the error orrurs, the number of threads raises to 35. Same time in netstat, there is a lot of connections, which have TIME_WAIT status. If I have understood right, TIME_WAIT status means, that the connection is waiting for to be closed by protocol in 240 secs. Is there any way to destroy these connections before, because I haven't time to wait so long?

desparete 8=) Juha Ka