How to programmatically update robots.txt for multiple sites in Optimizely 11? - Stack Overflow

I am working with Optimizely 11 and need to update the content of robots.txt for multiple sites programmatically. However, I haven't been able to find any documentation on how to achieve this.

Could someone provide guidance or a sample code snippet to help with this implementation?

I am working with Optimizely 11 and need to update the content of robots.txt for multiple sites programmatically. However, I haven't been able to find any documentation on how to achieve this.

Could someone provide guidance or a sample code snippet to help with this implementation?

https://support.optimizely.com/hc/en-us/articles/4413191642637-Edit-the-Robots-txt-file

• optimizely

Share Improve this question Follow edited Feb 3 at 12:22 Mark Rotteveel 110k229229 gold badges156156 silver badges224224 bronze badges asked Feb 3 at 3:39 user3928241user3928241 16311 gold badge33 silver badges1010 bronze badges 1

• 1 robots.txt is a simple text file. You can't find anything on this site or the internet about how to programmatically update a text file in your language of choice? – Ken White Commented Feb 3 at 5:42

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

There's no need to host robots.txt as a physical file in .NET. Instead add an API that will "fake" the robots file, then you can modify the contents as well, very practical when having multiple environments.

Here's how we do it in .NET 8 and CMS 12, but this should be portable to your needs as well

[Route("robots.txt")]
[ApiController]
public class RobotsController : ControllerBase
{
    private readonly IWebHostEnvironment _hostEnvironment;

    public RobotsController(IWebHostEnvironment hostEnvironment)
    {
        _hostEnvironment = hostEnvironment;
    }

    [HttpGet]
    public ContentResult GetRobotsFile()
    {
        var content = "User-agent: *\n";

        if (_hostEnvironment.EnvironmentName.StartsWith("Prod", StringComparison.OrdinalIgnoreCase))
        {
            // add content to your preference
            content += "Crawl-delay: 10\n";
            content += "sitemap: https://www.yourdomain.org/sitemap.xml";
        }
        else
        {
            // allow no crawling in public test environments
            content += "Disallow: /";
        }
        return Content(content, "text/plain");
    }
}