Search engine tips req'd

[LucieLastic]

hi All

For my new project I have to search thousands of .txt files (containing) news stories for keywords.

I was just wondering if anyone has any tips they could give me. We can manipulate the text files (eg add headers etc) but the main story should stay the same, any manipulation will also be automated because of the volume coming in.

I was thinking of using a similar idea like Meta tags, that is, filtering out the keywords and putting them at the top of each file. There will be approx 5000 files a day. Also having an index file containing the headers of each story and a link to the relevant file.

We're thinking of creating directories for each year, then month, then day and storing files like that.

Any other ideas?

many thanks
lou

 

[LucieLastic]

hello

Just found some info on the Boyer-Moore algorithm, just have to code it now...