site search engine efficiency

[GIGN]

I am investigating a custom website search engine, which appears to be very straight forward, the basic idea I have is :

- Visit and "read" each page searching for keywords.
- Record results as xml.
- Process xml and display ordered results.

The principle is working, but I wonder about efficiency, and whether or not this is a heavy load for the server to cope with.

Is caching possible in this situation? I could actually save the xml results each time - but don't know if that is worth it.

I have tried to find search engine guides, but have not been successful, any ideas or experiences would be valued highly.

<bb/>

 

[harmmeijer]

If this search engine is only to be used on your own (intranet) site. And this site is run with IIS.

You can use indexing service, this service will index all word documents, textfiles, excel files, html and some more for you.
You can later on query this index.

 

[GIGN]

It is for an internet site so, what is your opinion on the server load for a system as outlined above?

As I see it, this is the only method available - I can't see any way to cache results or anything - is this the way ordinary search engines work?

I guess I could try one of those external ones, but I have reservations about blending it into existing design environment, and I don't want ads.

<bb/>