Web Scraping - Input Ignored

[MadJock]

Hi,

I am trying to web scrape some data from a particluar site. From IE, I do the following:
browse to site
enter date in text box
press submit

In my c# code (desktop app - not ASP.Net) I do the following:
request to site
get viewstate
add POST variables (including viewstate) to header
Send request

My issue is that doing the above in code makes the site disregard the value I have set for the textbox in the second request.

I've used Fiddler to sniff and the only difference I can see is the 'Proxy-Connection' header. This is 'Keep-Alive' when sniffing IE and does not exist when sniffing from c#. I found a couple of people with the same problem from Googling but no answers!

I've done the following:

myHttpRequest.KeepAlive = true;

but this does not appear to have made a difference.

Does anyone have any suggestions? Code or HTTP trace available if needed.

Thanks in advance,

Graeme

"Just beacuse you're paranoid, don't mean they're not after you

 

[KDavie]

What site are you scraping? It would be easier to give a suggestion if we knew how the site behaves.. For example, are values being passed in a query-string or are you forced into a POST?