Index page dismissed in robots.txt - reason why Google ignores info? 2

[makemusic]

Hi

If you do a search for followtheboat (the name of my new site) in google it appears but with no information. Is this because I have only just submitted the page?

Also, I made my robots.txt AVOID my index page and instead go to the alternative home page which sits in a sub directory of the root. This is better optimised for search engines. Is this a silly thing to do?

Thanks!

 

[ChrisHirst]

Also, I made my robots.txt AVOID my index page and instead go to the alternative home page which sits in a sub directory of the root. This is better optimised for search engines. Is this a silly thing to do?

Yes extremly. How in the name of whatever do you expect to get your site crawled if you exclude the robots from the only way into your site they can find!
Robots.txt is an EXCLUSION protocol it doesn't tell a bot to go elsewhere for content.

and the photos entry in robots.txt is incorrect it should be /photos/ to disallow all files aand folders.

Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.

http://www.cram-system.com

Nightclub counting systems

So long, and thanks for all the fish.

 

[makemusic]

So they can ONLY enter the site via the index.asp page? And I'm ok with it being an asp page not htm?

Thanks for the comments. I'm learning.....

 

[Foamcow]

Providing the index.asp page generates HTML.
That is what the crawler is looking at, what is sent to the user/crawler.

Put some robot info into your index page that says to follow links within it but not to index it's content.
Although it doesn't matter if it DOES index the content really does it?
Remember SE's index PAGES and not SITES.

Foamcow Heavy Industries - Web design and ranting
Toccoa Games - Day of Defeat gaming community
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
"I'm making time

 

[ChrisHirst]

doesn't matter what the extension is. Just use the default document for the server. As you are running on IIS6 that would be default.asp. (or index.asp, index.htm, default.htm default.aspx)

No they can eventually arrive at any page on the site but as most external links point to the domain name that's where they will start from.


Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.

http://www.cram-system.com

Nightclub counting systems

So long, and thanks for all the fish.

 

[makemusic]

Cheers guys. Any other comments on my structure / coding is always appreciated