How to achieve environment specific robots.txt

vikram.bishnoi

Hi Team,

Can you please how can we create environment specific robots.txt?

The requirment is to avoid crawling of lower environments and allow only PROD environment.

Please suggest.

Thanks

Vikram

fsmeier

Heyhey,

how are you creating the current robots.txt? And why is your robots.txt from a hopefully protected staging/testing environment publicly available to the crawlers?

All the best,

Florian

vikram.bishnoi

Hi Florian,

We have created two versions of robots.txt. In UAT/Staging robots.txt we have added:

User-agent: *
Disallow: /

This way lower environments are not crawlable.

For PROD, we have added different version on robots.txt in our PROD/Master branch.

Thanks