Paid Content Available to Search engines but hidden from users

[noyes99]

hi,

I am a newbiew in this area. I was wondering if there is a way we can modify robots.txt such that the content is available to the search engines and not available to the real users. I was wondering if people know what are the common techniques that people use in instances such as these.

Thanks

 

[ChrisHirst]

Dangerous game to play, Technically it's cloaking because you are showing SEs one thing and users another and once discovered could well get your site banned.

robots.txt can only block robots as well.

The safest way is to show snippets of the paid content on crawlable pages

Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[noyes99]

Thanks for responding Chris and bringing this up. Ill just a give a brief description of what I want.

We have content, that is behind registration barrier. Someone who is registered on the site can freely read that content. Some of the content is paid content. i.e. you have to pay a monthly or an annual fee to read that content. A registered or a paying user is always able to see that content, via website, via search engines, whenever he/she is logged in. We just want the search engines to index that content too, so that users can get to the content. We will have snippets of paid content on the pay block page, but we want the search engines to index everything, not just parts of the content.

Does it really fall into that category? I wouldn't think so, because what we want indexed is available, just behind registration/ pay barriers.

What is the best way to achieve this? Thanks

 

[ChrisHirst]

It's a thin line and has sparked a few debates.

You will need to detect the user-agent or IP of known SE crawlers and bypass logon/cookie checks, allowing them to access content that other UAs will be blocked from without having explicit consent of the SEs (unlike trusted feed). This is content cloaking regardless of intent.
You also have the issue of your paid content being available from the SEs cache. Although there are ways around this of course.

Now depending on the industry and market place it may never come under the spotlight and be given a look over, but on the other hand of course ...

Personally I prefer showing "teaser" content and a "register/login to read more" link to the full article.


Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[Foamcow]

I wrote a post about "synopsis content" and forgot to actually post it. lol
Here's a truncated version from memory...

I would echo Chris' sentiments here.

Imagine if a user came to your site via some means other than an SE. What is there to make them think the info you have stashed behind that registration barrier is going to be of interest?

Some sample content, or a synopsis of what they might find would go a long way to improve your site's useability and overall "niceness".

The old adage stands here, build your site for PEOPLE and not Search Engines.

Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r

 

[coldfused]

Also if you take those "snippets" and place them on "holder pages" so to speak that are written well enough (weighted keyword density, etc. etc...) it can only help your content get ranked more efficiently.

And as the guys said above it helps with the overall usability and adds worth to the content thats there making them more likely to purchase the extended content.

----------------------------------------
Florida Web Design
Orlando Web Hosting
Florida Coldfusion Hosting

 

[noyes99]

Thanks everyone for your input. As a programmer I always try to keep the useability issues in mind. I hate popup's but have to program them because the managers/marketing guys want me to. Similarly, I'd rather index only the snippets that i am displaying, but it is really not up to me to decide. I can make suggestions, but ultimately I have to implement what is asked.

What I'd be doing is checking for user agents, and if it is a spider (from a list of known spiders), we would bypass the cookie verification process.

 

[Foamcow]

That's kind of like cloaking then.
Showing one thing to "people" and another to crawlers is a bad thing to be doing.

The solution is actually really simple.
Just present sample, synopsis, material to the world at large.


And I understand that you have to program popups because the marketing guys want them. But do the marketing guys understand WHY they are a bad idea in most circumstances?
Marketing is about communicating your brand or product. If anything you do in the process is going to get in the way, then you are failing in your marketing.

If you present a synopsis of the info on your site for public view then you are effectively marketing your "product" are you not?



Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r