How to achieve environment specific robots.txt

vikram.bishnoi
vikram.bishnoi Spryker Solution Partner Posts: 15 🧑🏻‍🚀 - Cadet

Hi Team,

Can you please how can we create environment specific robots.txt?

The requirment is to avoid crawling of lower environments and allow only PROD environment.

Please suggest.

Thanks

Vikram

Answers

  • fsmeier
    fsmeier Senior Software Engineer & Developer Enablement Advocate Sprykee Posts: 1,081 ⚖️ - Guardians (admin)

    Heyhey,

    how are you creating the current robots.txt? And why is your robots.txt from a hopefully protected staging/testing environment publicly available to the crawlers?

    All the best,

    Florian

  • vikram.bishnoi
    vikram.bishnoi Spryker Solution Partner Posts: 15 🧑🏻‍🚀 - Cadet

    Hi Florian,

    We have created two versions of robots.txt. In UAT/Staging robots.txt we have added:

    User-agent: *
    Disallow: /

    This way lower environments are not crawlable.

    For PROD, we have added different version on robots.txt in our PROD/Master branch.

    Thanks