Googlebot came and requested my old/deleted files??? HELP!!! 3

[JoJoH]

Hi all,

I've read my log files today, it showed that Googlebot has come and crawled my site on 7-3-03... But the log files showed that Googlebot is crawling/requesting OLD files that no longer exist/deleted in my site!!! The case is I already have a site up like a few months ago, recently(a month ago) I scarp/delete my entire site and uploaded a brand new site with different file names, content... Now on 7/3/03, which is a few days ago, Googlebot come and decided to pick up my no longer exist, deleted old files??? Please advice!

Thanks in advance.


JoJoH

http://www.jadeboutique.com

 

[ulteriormotif]

Step 1: Don't panic.

Step 2: If your site is on an Apache server and you are have access to the .htaccess file, edit it so that the 'gone' pages return a 301-permanently moved response rather than a 404. If not, go to Step 3

Step 3: Make sure your new pages are all linked with spiderable links, and sit back and wait.
.................

Basically Google and any other SEs who already know about your pages will return to check for changes. Of course they will get a 404 response because the page can't be found. A 404, however, doesn't tell them anything except that the page couldn't be found AT THAT TIME, so you can expect to see them come back a few more times to see if the page has turned up... they may do this for a few months.

If you have access to an .htaccess file, you can instruct the server to return a different response code to the spiders when they request those pages. A 301-permanently moved response tells the spider not to bother checking back later because the page really is gone.

Whether you serve up a 404 or a 301 doesn't in the end make a lot of difference, as the spider will continue to spider the rest of your site normally, including your new pages.

 

[ulteriormotif]

Distraction is my middle name.

http://www.seoconsultants.com/w3c/status-codes.asp

http://www.webmasterworld.com/forum23/1494.htm

These two pages will give you a bit of a starting point if you want to learn more about access codes and editing the .htaccess file.

 

[ChrisHirst]

You possibly have some backlinks to the old pages that Googlebot is following and\or it is looking for the existing links in the database. By removing all the old pages you have every chance of google dropping your site totally from the index. I should get some of your old page names back in with some content on and text links to your new pages, use one as another sitemap, and\or have a custom 404 page put on that has links to the site map. don't forget that anyone who had your old pages bookmarked will be getting the same 404 errors and may not bother to try again. When the new pages have been indexed you should get a 301 (permanent redirect) put on for these old pages, then finally delete them (may take about 6 months) when the requests stop.

BTW naming folders and pages with keywords has zero effect for SEO but can be a real pain for maintaining the site not forgetting that some spiders may have a length limit to the filepath.


Chris

Indifference will be the downfall of mankind, but who cares?

 

[JoJoH]

Thanks Distraction(ulteriormotif), thanks Chris

Distraction(ulteriormotif) Good thoughts, Unfortunately I don't have access to .htaccess files... You know what you said about "spiderable links"? Well almost all of my links are dynamically generated, they are considered "spiderable links" in Google terms now right? If yes than all I can do right now is to wait till they realize there is a new site? Or should I submit my new site to Google? Here is my scenario:
I have never submitted my site to Google before because my site has not been optimized until recently. I have never even want it to crawl my site until last week, but about 3 months ago Googlebot picked up my totally unoptimized site through other site that links to me. So now what's in Google's index was the unoptimized site(and of course it ranks looooow). Now I have a brand new, optimized site, do you think I should submit that site to Google? Or should I wait for it to realize that the old site no longer exist and there is a new site for it to crawl? If I should submit my new site, should I ask Google to remove my old site before I submit my new site?


Chris Good thoughts, thanks You know what you said about putting back up my old pages and have a link in them to my new pages? That is a good idea but the thing is my links are dynamically generated, so I might have to create like a page for each diffrerent link(I have to create one for this page "../mypage.asp?Category=shoes" and one for this page "../mypage.asp?Category=handbags" and so on, it could get quite tedious...) any ideas?

Please advice.

Thanks in advance!

JoJoH

http://www.jadeboutique.com

 

[ChrisHirst]

You only need something like your sitemap and your homepage to be linked on maybe a couple of pages that get requested most, this will then allow the spider to follow the rest. You've got Google there don't let it get away!

Don't resubmit to Google, if it's already visiting, a submission can have a negative effect and so long as it can find links it will eventually crawl the whole site. For the first few months it can help to keep adding or updating the text, Googlebot likes more and new in context content (it then appears as an important site) this is how message boards (even with the spam) and forums get high rankings.
There is never a need to submit to Google it prefers sites it finds all on its own.

Another thing to do is have the sites that link to you update their links. If possible get them to use some relevant text in the link (it helps the rankings).

And dynamic links are followed, it seems to be if there are more than 2 ampersands (&) in the address some spiders will give up.


Chris.

Indifference will be the downfall of mankind, but who cares?

 

[JoJoH]

Thanks Chris! I don't understand, when I was reading my log files, it showed Google visited on 7-3-03 and requested some of my old files, but it did not request my home page... If it did request my home page on 7-3-03 then it would have no problem getting to my new pages and indexing them! How come it didn't request my home page? Is what I should do now is to put up my old page and link it to my new pages and let Google crawl and find them next time?

Please advice.

Thanks in advance.


JoJoH

http://www.jadeboutique.com

 

[ChrisHirst]

You don't often find home pages in the Google database as these are usually set up as landing pages or splash pages (yeuch) with little or no content, so because they don't rate as important it never indexes them again or list them in the SERPs. If you get some useful content on there it will get listed.
All you will need is the pages that were requested, add a bit of keyword content, not enough to make them important though, a link to the home page (use the FQDN rather than the filename) a link to the sitemap and some text along the lines of;
'The Jadeboutique website has been redeveloped and for (add keyword) click here' make all that a link to a category page and then put a note on to say please update your bookmarks/favourites when you visit the home page.
The site visitors (human) will have a way into the site and the spiders will have the links to follow, after a few weeks remove the keywords from the content and links and you should find requests for these old pages will start to drop off, then you can delete them, the 404s will mark them as dead and they will be removed at the next dance. by then your new pages will be indexed and be appearing in the SERPs.

Chris.


Indifference will be the downfall of mankind, but who cares?

 

[JoJoH]

Thanks Chris! Last question, what is FQDN?

JoJoH

http://www.jadeboutique.com

 

[JoJoH]

Ah, looked it up FQDN=fully qualified domain name

Thanks Chris for all the wonderful help! Thank you ulteriormotif too! Stars for both of you!



JoJoH

http://www.jadeboutique.com

 

[mThomas]

If the rest of your site has lots of good content and there are links to and from your other pages, then I wouldn't loose sleep.

Google comes to my site every day. I have taken pages down. I've even forgoten to take out links to those pages. I've still got the same basic ranking for my keywords.

My advise, relax..do the best you can and you'll be just fine.

mike

http://www.udraw2.com

 

[JoJoH]

Thank you Mike, indeed I am losing sleep! Wow Google comes to your site EVERYDAY!? Lucky you! How did you get it to do that? Please share it with me!

Thanks in advance!

JoJoH

http://www.jadeboutique.com

 

[ChrisHirst]

There's no trick to getting Googlebot to visit everyday. If your content changes or updates on a regular basis it will arrive more often. So long as it visits and it can find some of the pages it will keep crawling the rest.
One site you may want to submit to is

http://www.sleepyshopper.com

I submitted a new site to a few directories and minor SEs when this popped up from one of them (scrubtheweb I think) but the site was listed within 30mins and got visitors from sleepyshopper within 2hrs of being listed (and the online purchase wasn't live, damn)

Best of luck, enjoy your site and don't lose sleep fretting about ranking.

Chris.

Indifference will be the downfall of mankind, but who cares?

 

[JoJoH]

Thank you Chris for the wonderful advices! Without you telling me I would never have known there is a "sleepyshopper.com" and its benfits! Thank you once again! Better go get some sleep....

JoJoH

http://www.jadeboutique.com