Block search engines from crawling the web server

Here's a neat little trick to block search engines from indexing your entire web server contents.

Say you have a web server with many websites that serves as your testing and development sites for your clients.  It's happened to the best of us that Google has gone and indexed one of your sites, either beacuse you've forgot to add password protection or you've accidently dropped in the LIVE robots.txt file.

As described here Google respects the: X-Robots-tag:

http://googleblog.blogspot.hk/2007/07/robots-exclusion-protocol-now-with-even.html

You can add the X-Robots-tag tag at your IIS web server level so that it sends a noindex response for every page that is requested from your web server:

Now inspect that the response headers includes the noindex command