Apache and search engines

[everythingisinuse]

Hi,
I have 1.3.20 running with many virtualhosts, none of them get indexed by Google, Altavista etc. I have tried every kind of META-tags.

Is Apache default install somehow denieing access or is there some module you have to add when compiling ?

In 1 virtualhost directory is file called robots.txt with these lines:
User-agent: *
Disallow: /

BUT it's not in webserver root, so it should not block search engine bots?

Any ideas would be appreciated!

 

[Wullie]

Hi,

Can you post the URLs?

I am currently building the database for a seach engine and I can try spidering the pages and tell you the results.

Hope this helps Wullie

http://www.survivorhelp.co.uk

 

[everythingisinuse]

Hi wullie,
try these, they have some META tags:

http://www.cmautocaravan.com

http://www.takatukka.com

/Kimmo

 

[Wullie]

Hi,

Both sites are returning results.

The caravan site is using frames which always confuse search engines. Spiders don`t follow frame links so only the actual content of that page will be indexed.

If possible, you should make an entry page to submit to search engines that doesn`t use frames.

The second site returns fine.. Can`t see any problems there.

You should add a META tag with the following to your pages:

<meta NAME=&quot;robots&quot; CONTENT=&quot;INDEX,FOLLOW&quot;>

How long ago did you submit the sites? It normally takes a few months to appear..

Hope this helps Wullie

http://www.survivorhelp.co.uk

 

[Wullie]

Hi again,

Check out

http://spider-food.net/frames.html

Hope this helps Wullie

http://www.survivorhelp.co.uk