On Thu, 30 Sep 2004, Frederick Noronha (FN) wrote:
---------- Forwarded message ----------
http://lxer.com/module/newswire/view/23328/index.html
I could not track the source of old content showing up at the top of
google searches on my name. According to the number of hits I
generate, I have a significant mind share on Google. Finally, we
tracked down the problem and found that MSNbot was hammering and
multiplying links old and irrelevant content, pushing that above
recent articles about Linux in Government, case studies, etc.
I also found other open source advocates on the web trying to find
out why they couldn't stop MSNbot using a standard robots.txt
directive in the root directory of their web sites. Some of the
complaints related to increased fees for bandwidth.
In this article, I discuss the issue. I will also post the fix we
discovered by accident. BTW, a standard redirect will not work.
I'm not very pleased to discover Microsoft is manipulating the
content on the Internet.
Tom Adelsten