Thursday, June 16, 2011

Web Crawlers Detection in ASP.Net

Here is a way of detecting search engine (Google, live…) crawlers,spiders etc..


System.Web.HttpBrowserCapabilities clientBrowserCaps = Request.Browser;
if(((System.Web.Configuration.HttpCapabilitiesBase)clientBrowserCaps).Crawler)
{
Response.Write ( "Browser is a search engine.");
}
else
{
Response.Write ("Browser is not a search engine.");
}

//Here is another approach..

if(Request.ServerVariables["HTTP_USER_AGENT"].Contains("Googlebot"))
{
//log Google bot visit
}
else if(Request.ServerVariables["HTTP_USER_AGENT"].Contains("msnbot"))
{
//log MSN bot visit
}
else if(Request.ServerVariables["HTTP_USER_AGENT"].Contains("Yahoo"))
{
//log yahoo bot visit
}
else
{
//similarly you can check for other search engines
}

No comments:

Post a Comment