Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations Mike Lewis on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
The opposite of SEO!
- Thread starter jmcg
- Start date
You can do a search on this forum for 'robots.txt'. A quick search found which shows how to ask spiders not to index certain areas of your site.
___________________________________________________________
If you want the best response to a question, please check out FAQ222-2244 first.
'If we're supposed to work in Hex, why have we only got A fingers?'
Drive a Steam Roller
Steam Engine Prints
___________________________________________________________
If you want the best response to a question, please check out FAQ222-2244 first.
'If we're supposed to work in Hex, why have we only got A fingers?'
Drive a Steam Roller
Steam Engine Prints
- Thread starter
- #3
Surely the admin section of a site should be behind a login which is impassable by a search engine?
If you are saying that the pages were indexed prior to being locked away then you can request they be removed from Google's index by using Google's webmaster tools.
I would take a 3 prong approach.
1. Request the removal via webmaster tools
2. Create a robots.txt
3. Add some kind of redirect via htaccess (assuming you are using Apache) that sent a suitable HTTP response code to search engine spiders.
--
Tek-Tips Forums is Member Supported. Click Here to donate
<honk>*:O)</honk>
Tyres: Mine's a pint of the black stuff.
Mike: You can't drink a pint of Bovril.
If you are saying that the pages were indexed prior to being locked away then you can request they be removed from Google's index by using Google's webmaster tools.
I would take a 3 prong approach.
1. Request the removal via webmaster tools
2. Create a robots.txt
3. Add some kind of redirect via htaccess (assuming you are using Apache) that sent a suitable HTTP response code to search engine spiders.
--
Tek-Tips Forums is Member Supported. Click Here to donate
<honk>*:O)</honk>
Tyres: Mine's a pint of the black stuff.
Mike: You can't drink a pint of Bovril.
- Thread starter
- #5
There is a protected login via an application but it is indexing the admin page itself.
They cannot progress from there and see anything behind it.
Using htaccess or protecting the folder was out because of the volume of users with their own username password that directs them to particular content.
Thanks for the advice anyway.
They cannot progress from there and see anything behind it.
Using htaccess or protecting the folder was out because of the volume of users with their own username password that directs them to particular content.
Thanks for the advice anyway.
Just confirming that you're right, I've been in the same situation myself and I find it helpful when several people affirm responses, it adds credibility y'know?
In fact you should go further and also add a meta to the page itself:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
But the quick approach is a robots.txt entry
User-agent: *
Disallow: /path/to/page
In my experience though, robots.txt does unintentionally get bypassed by robots if you still have links pointing to it from other pages.
In fact you should go further and also add a meta to the page itself:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
But the quick approach is a robots.txt entry
User-agent: *
Disallow: /path/to/page
In my experience though, robots.txt does unintentionally get bypassed by robots if you still have links pointing to it from other pages.
ChrisHirst
IS-IT--Management
No they don't.
The robots.txt protocol is to stop the compliant bots indexing the page and accessing the content.
It does NOT prevent them from listing the URL only, should the page be linked to.
Chris.
Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems
So long, and thanks for all the fish.
Well, yes sort of - and anyway this was exactly JMCG's original problem. "admin areas of a site that are ranking higher than the main pages". Ie being listed.
I've personally found myself that even if you do robots.txt AND meta voodoo, pages can still get both listed and spidered if there are links pointing there, especially from other sites. It shouldn't, but it does. Maybe it's simply that once they're in, they didn't get revisited. Or a timing issue.
Ie imagine this, say the page is in one of the Google indexes and suddenly us humans slap a robots.txt "don't go there" instruction on it and "noindex". Well the robot just does what we tell it to. It knows about the page, tries to go there, but because it's a good little robot obeying robots.txt it doesn't go there because of the exclusion. So it doesn't get a chance to see it's got noindex on there now. Lots of variants on this - after all computers just do what we tell them to.
I've personally found myself that even if you do robots.txt AND meta voodoo, pages can still get both listed and spidered if there are links pointing there, especially from other sites. It shouldn't, but it does. Maybe it's simply that once they're in, they didn't get revisited. Or a timing issue.
Ie imagine this, say the page is in one of the Google indexes and suddenly us humans slap a robots.txt "don't go there" instruction on it and "noindex". Well the robot just does what we tell it to. It knows about the page, tries to go there, but because it's a good little robot obeying robots.txt it doesn't go there because of the exclusion. So it doesn't get a chance to see it's got noindex on there now. Lots of variants on this - after all computers just do what we tell them to.
- Status
Similar threads
- Locked
- Question
- Locked
- Question