httpmodule header / footer speed issue

[Trebor100]

Afternoon everyone,I'm working on a httpmodule for adding a custom header and footer to all my sites. The concept is great but the performance is slow

I'm after adding the header and footer from a specific location (ie not hard coded). What i'm doing is scraping the html from the location and inserting via regex on the filter string.

Does anyone have any ideas as to how i can get this to work faster?

I'm wondering if this can be cached in the httpmodule but don't have a clue how I can do this in a httpmodule. Another handy way would be to get the #include line somehow added on the initial request.

Any thoughts or ideas would be grand. I'm working in C#

Regards,Rob

 

[jmeckley]

as per my C# forum post:
you should not need to use an http module to manage headers/footers. with webforms you can use master pages to manage this. with an MVC framework you can use partial views/view components/layouts/content sections/etc.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[Trebor100]

Hi Jason - with regards to the use of master pages:

I currently have 200+ sites that I would like to get the branding out on all at the same time. Whilist I fully agree that master pages are great for this purpose I am looking for a solution that does not require modifying every single site (hence httpmodule and machine.config).

Regards,
Rob

 

[jmeckley]

that is an entirely different requirement altogether. can you quantify

The concept is great but the performance is slow

and are you sure the module is where the bottleneck is? Ants Profilers or dotTrace can assist with locating the bottleneck(s). if this is the bottle neck how are you injecting the text? we may be able to better assist if we can view the code you are currently using.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[Trebor100]

Hi Jason,

I've been doing some time testing on the solution and it appears the scrapping of the html from the header / footer is the big slow down. The code for the scrape is as follows:

string url = "

http://google.com";

string strResult = "";

WebResponse objResponse;
WebRequest objRequest = System.Net.HttpWebRequest.Create(url);

objResponse = objRequest.GetResponse();

using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
strResult = sr.ReadToEnd();
// Close and clean up the StreamReader
sr.Close();
}


the injection of the html is being done by httpfilters / regex and runs at a good rate. My thoughts on this are:

- can i insert an #include .... on the request rather than inserting full html on the response?
- can i cache the request so that i dont need to scrape it?

Thanks again and have a good weekend all.

Rob

 

[jmeckley]

1. not sure that an include would help. aren't includes a server side action, not a client action. if they are a server action, the html still needs to be rendered before sending to the client. also, how will the includes be processed this late in the pipeline? if includes are a client side action, a second and third request will need to be issued for this content.
2. you can cache it using the asp.net cache.

questions:
1. if you want to add a header/footer to the response, why are you creating a new request? I'm assuming you want to add the header/footer to your website(s), not add your header/foots to 3rd party website.

where are you pulling the header and footer html from? file, database, memory, webservice? there may be room for improvement here as well.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[Trebor100]

Hi Jason,

Sorry for the lateness is my response – it has been like a mad house!

With regards to point 2 (asp.net cache). How would you achieve this? I’ve added

<add name="OutputCache" type="System.Web.Caching.OutputCacheModule"/>


To the web.config of my test application however wont this just be at an application level? I’m after caching the entire requests made from the httpmodule. If you could point me in the right direction on achieving this that would be great.

With regards to your question:

“1. if you want to add a header/footer to the response, why are you creating a new request? I'm assuming you want to add the header/footer to your website(s), not add your header/foots to 3rd party website. where are you pulling the header and footer html from? file, database, memory, webservice? there may be room for improvement here as well.”

I’m creating a new request to scrape the html from the header / footer html files (note these header and footer files are managed via a cms solution). If I can scrape this html without a new request please advise as I’ll modify accordingly.

I’m adding the header and footer to all website. This includes internally developed solutions and third party solutions.

I’m pulling the html from a url and as such treating it as a webpage. This element cannot be changed as this links in to a CMS solution.

Once again thank you for any help and assistance.

Rob

 

[jmeckley]

oh, your using the declarative syntax. I don't use that stuff. too messy and difficult to test. I would put the header/footer in the cache programmatically with HttpContext.Cache. a google search will produce plenty of results on the topic.

as for performance; the remote request to get the header/footer is most likely where the cost is. a profiler like dotTrace or Ants Profiler will determine the biggest bottlenecks.

if you are making this remote request each time it's going to kill performance. I would cache the header/footer locally to improve throughput to. This way you only make the remote call if the value doesn't exist.

depending on whether the header/footer are context sensitive will determine the complexity of the logic.

one other thought...
if the header/footer are static content (or changes very rarely) then why not hard code the header/footer into variables and just inject into the request? this gets you a quick fix now and buys you time to properly flesh out a solution of styling your sites.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[ca8msm]

I'm confused. If you are implementing a httpModule, then why do you need to do any scraping? You already have access to the Context object, so just inject your HTML wherever relevant. Here's the MSDN example that writes out a very simple header/footer message to every processed request:

http://msdn.microsoft.com/en-us/library/aa719858(VS.71,lightweight).aspx

Mark,

Darlington Web Design
Experts, Information, Ideas & Knowledge
ASP.NET Tips & Tricks

 

[Trebor100]

Jason your a legend - I'll start the google-ing of HttpContext.Cache and see how I get on. I'll probably come back with a hundred questions on its use in this context over the next few days. I take your point on hard coding the header / footer. I'm thinking of adding the cache and adding a timeout on the scrape and if the timeout exceeds X then using the hard-coded header / footer. Any thoughts on this approach?

Mark - The scraping is used because the content of the header and footer is handled by a content management system so I need to grab the header / footer content. Using the method as described in the MSDN example would also put a fixed header / footer at the start / end of the html output where as my solution is putting it in the correct place (ie inside the <body> </body> tags of the html thus complying to standards.

 

[jmeckley]

I'm thinking of adding the cache and adding a timeout on the scrape and if the timeout exceeds X then using the hard-coded header / footer. Any thoughts on this approach?

1. check cache for header/footer
2. if exists, use it, otherwise, preform scrape.
2.b. put scaped content in cache
3. inject header/footer into response stream


Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[ca8msm]

Mark - The scraping is used because the content of the header and footer is handled by a content management system so I need to grab the header / footer content.

I thought your scraping was getting the content of the original page not the header/footer that it has to append?

Using the method as described in the MSDN example would also put a fixed header / footer at the start / end of the html output where as my solution is putting it in the correct place (ie inside the <body> </body> tags of the html thus complying to standards.

That's just a basic example to show you how easy it is - you can inject the HTML wherever you want.


Mark,

Darlington Web Design
Experts, Information, Ideas & Knowledge
ASP.NET Tips & Tricks

 

[Trebor100]

Hi Mark,

The scraping is to get the content of the header and footer to inject into the page.

Cheers,
Rob

 

[ca8msm]

Sorry, I had misunderstood your post. In that case, yes I would just cache the header and footer and then use the response object to insert them wherever relevant.

Mark,

Darlington Web Design
Experts, Information, Ideas & Knowledge
ASP.NET Tips & Tricks

 

[Trebor100]

Thanks for all the help / pointers everyone. Cache worked great and performance is now at a good level.

 

[Trebor100]

Back on this one again - the use of the httphandler has stopped the required field validators from working on the .net solution. Has anyone run into this before or know the cause of this?

 

[Trebor100]

Just found this: "HTTPModule that modifies the response output stream seems to break WebResources.axd"

all is not well in Rob-land. any thoughts / solutions to this?

 

[Trebor100]

problem now sorted - it needed a catch for WebResource.axd in the begin request... Most odd

 

[ca8msm]

That makes sense, the WebResource.axd will be handled by the ASP.NET service so attempting to inject your HTML into it would most likely break it.

Mark,

Darlington Web Design
Experts, Information, Ideas & Knowledge
ASP.NET Tips & Tricks