A Closer Look at Microsoft's MSNBot
A Closer Look at Microsoft's MSNBot: "Towards the end of February of this year a new search engine spider began showing up in server logs everywhere. 'MSNBot' had made its first appearance, and quickly became the topic of discussion. Just as some webmasters became excited to see those first few traces of MSNBot, others became a bit curious as they found evidence of the deepest crawl cycles in months by any engine...In a surprising maneuver though, a MSN Search representative (under the alias of "msndude") informed webmasters that as of June 11th, MSNBot was able to review what your preferences were for crawling time. Using the massively popular Webmaster World forum system, the following discussion had begun:
Webmasterworld forum Microsoft Introduces new Robots.txt Cmds
posted by Jen @ 7:59 AM
<< Home