Web Crawlers Detection in ASP.Net

Here is a way of detecting search engine (Google, live…) crawlers,spiders etc..

System.Web.HttpBrowserCapabilities clientBrowserCaps = Request.Browser;

if(((System.Web.Configuration.HttpCapabilitiesBase)clientBrowserCaps).Crawler)

{

Response.Write ( "Browser is a search engine.");

}

else

{

Response.Write ("Browser is not a search engine.");

}

//Here is another approach..

if(Request.ServerVariables["HTTP_USER_AGENT"].Contains("Googlebot"))

{

//log Google bot visit

}

else if(Request.ServerVariables["HTTP_USER_AGENT"].Contains("msnbot"))

{

//log MSN bot visit

}

else if(Request.ServerVariables["HTTP_USER_AGENT"].Contains("Yahoo"))

{

//log yahoo bot visit

}

else

{

//similarly you can check for other search engines

}