Here's (I think) a better solution which avoids most of these problems. Implement this, and pay me a commission on revenue (just kidding), or give me a job and I'll implement it for you (kidding a bit less ;) ).
----
1) Update the sitemap spec to support the following true/false flags for each page
* Free View (Is it cost free to view the full page)
* Free Preview (Is it cost free to view a preview of the page)
* Registration View (Is registration required to view the page)
* Registration Preview (Is registration required to view a preview of the page)
* Cache Preview (Should a preview of the page be cached)
* Cache page (Should the full page be cached)
(Depending on the model you require, you could base this on pages or articles - which would require some other modifications to the sitemap spec. This could easily be done without breaking for existing sitemaps.)
For each sitemap, indicate if SSL is supported for the URLs spidered (or alternate URLs, etc)
2) Equip googlebot with a client certificate to identify itself.
When googlebot spiders a site, the site can decide if it wants to let Google index the full article or not. If they are a big enough site to support SSL, they can validate the client is Google with higher certainty.
3) In the search listings, show flags depending on whether the information is free to view/preview and whether registration is required. Allow users to filter search results on these criteria, and setup default preferences.
Penalise sites if they are reported (and confirmed) as lying about the cost and registration flags.
Result: You can index the hidden web in a way allowed by the content authors, but still enable the end users to be in control of the type of sites their search will return.
Webmasters could build their registration and authentication system based on the rules in their sitemap file, so everything would be automatically up-to-date and in agreement between themselves and the search engines.
Just pop the cheque in the post ;)
الاشتراك في:
تعليقات الرسالة (Atom)
ليست هناك تعليقات:
إرسال تعليق