do spiders see ASP generated text pulled from a database

[slickwillyslim]

i have a big product line, everything is being pulled from a database and written to a products page by category. my question is, do the search engine spiders see this content? if they can't see the product descriptions then the page content wouldn't match up with my meta tags. i've seen some sites with their products pages in HTML. are they doing this for ranking? i know it's ok for my cart to be in asp, but does it hurt my product pages (and overall site ranking)? due to the fact that i have so many products it is much easier to make product changes in the database. i guess my real question should be, how does a spider crawl a page? does it open the page like a browser does, and does the vbScript, ADO, and SQL stuff run to pull the database contents for the spider to read. any suggestions would be appreciated.

 

[ChrisHirst]

yes the spider behaves much the same as a browser apart from the fact that they don't trigger client-side scripts or events and can't submit forms.
Provided there is a link to the dynamic page the spider will crawl it. There are however some caveats to this,
most spiders will not crawl 2nd generation dynamic pages, so if your catalogue system generates category page links with querystrings and that page then has products with querystrings the product pages may not get crawled.

Use of session IDs in the URL may stop the pages being crawled.
Requiring cookies to enter the catalogue will stop the crawler dead.
More than 3 parameters in the URL may stop the pages being crawled.

Depending on your host server you can use mod_rewrite (*nix) or an ISAPI filter to rewrite the urls to appear static, this does however add overhead to the server so may not be suitable.


Chris.

Indifference will be the downfall of mankind, but who cares?

http://www.candsdesign.co.uk

A website that proves the cobblers kids adage.

http://www.cram-system.com

Nightclub counting systems

 

[slickwillyslim]

thanks for that information Chris. about what you said:

"most spiders will not crawl 2nd generation dynamic pages, so if your catalogue system generates category page links with querystrings and that page then has products with querystrings the product pages may not get crawled."

are you referring to question marks in the URL? my homepage is setup with drop-down boxes and "go" form buttons. each drop-down box represents a major category with subcategories for options. if you click an option and click "go". the option is sent as form data to an ASP page called category.asp. this page parses the form data and compares the option to category field values in the DB. so it filters all the products based on the option picked and displays all products in that category. i only have one product page, where all products get pulled to after the category filtering. if spiders can't submit forms, then i guess they'll never make it to my product page. looks like i'll need to create a product page for each category with a link to it, and back. i guess if the spider can link to each individual product page then once it reaches the page, it can see the generated product list. does this sound like the proper way to go? it kinda sucks because i'm dealing with over 4,000 products :-( anyway, thanks again for the help!

 

[ChrisHirst]

yep querystrings are the "/pagename.asp?cat=something" so 2nd gen dynamic would be links on pagename.asp looking like "/anotherpage.asp?prod=12345"

quickest way round the form problem is to create categorised site maps with plain text links and optimised anchor text for the target product page and link these from your home page. It gives the spiders and visitors an alternative route around the site.

problem is so few catalogue\cart solutions are SEO friendly out of the box, so I'm writing my own!



Chris.

Indifference will be the downfall of mankind, but who cares?

http://www.candsdesign.co.uk

A website that proves the cobblers kids adage.

http://www.cram-system.com

Nightclub counting systems