Dynamic Robots Exclusion File in ASP.NET

Engineering

June 16, 2013

Serving environment-specific robots.txt files via IIS URL rewriting to keep non-production environments out of search indexes.

Prevent search engines from indexing the non-production versions of your web application.

Many websites maintain multiple environments: development, testing, staging, and production. Organizations typically use username and password authentication to prevent search engine indexing of non-production versions while also protecting them from unauthorized access.

A challenge arises when stakeholders need to test social media sharing functionality on staging environments. Temporarily disabling security measures can expose links publicly, allowing search engines to index the website. This oversight can result in diluted search rankings, duplicate content problems, and customers making actual purchases on staging systems.

Prevalence of Unfinished Websites on Google Search Results

Unfinished websites are web pages that are in development, test, uat, or staging phases and are not ready for public viewing. They may contain incomplete information, outdated data, technical issues, broken links, errors, or malware. Such exposure negatively affects reputation, user experience, and search rankings.

Google search queries revealing unfinished websites:

https://www.google.com/search?q=site:dev.*
https://www.google.com/search?q=site:test.*
https://www.google.com/search?q=site:uat.*
https://www.google.com/search?q=site:staging.*

How to Prevent Search Engines from Indexing Non-Production Websites

Common prevention methods include:

Using a meta noindex tag on pages to hide from search engines, instructing crawlers not to index or display the page
Using an X-Robots-Tag HTTP header with noindex or none values for non-HTML resources like PDFs, videos, and images
Using a robots.txt file to block crawlers from accessing certain site sections (less reliable due to potential search engine non-compliance)

Dynamic Robots.txt

A robots.txt file instructs search engine crawlers regarding what they can or cannot access. The objective is allowing production crawling while blocking non-production access.

Create two plain (ANSI encoded) text files — ANSI encoding is recommended because some crawlers lack UTF-8 support or become confused by byte order marks.

robots.test.txt content:

#DO NOT INDEX ANYTHING ON THIS WEBSITE
User-agent: *
Disallow: /

robots.live.txt content:

#INDEX EVERYTHING YOU CAN FIND ON THIS WEBSITE
User-agent: *
Disallow:

Important distinction: the "Disallow: " (without the forward slash) allows access to all directories. Incorrect syntax risks complete search engine de-indexing.

Copy files to the website folder. Verify the IIS URL Rewriting module installation.

Add these IIS rewriting rules to the web.config system.webServer section:

<rewrite>
  <rules>
     <rule name="Rewrite LIVE robots.txt" enabled="true" stopProcessing="true">
       <match url="robots.txt" />
         <action type="Rewrite" url="/robots.live.txt" />
         <conditions>
            <add input="{HTTP_HOST}" pattern="^(www.)?tekcent.com" />
         </conditions>
    </rule>
    <rule name="Rewrite TEST robots.txt" enabled="true" stopProcessing="true">
       <match url="robots.txt" />
         <action type="Rewrite" url="/robots.test.txt" />
         <conditions>
           <add input="{HTTP_HOST}" pattern="^(www.)?tekcent.com" negate="true" />
         </conditions>
    </rule>
    </rules>
</rewrite>

Verify the Results

After deploying changes, test different website versions. An online robots.txt validation tool can verify file validity.