This course is free — log in to save your progress and unlock the full library.
Log in free →One rogue robots.txt on a subdomain tanked all our rankings
1:00Rankings tanked: 7→11, 2→49. The culprit: a status-page subdomain shipped its own broken robots.txt that let Google crawl everything — junk pages exhausted our crawl budget. One config file, all of it.
Transcript
This one small mistake has caused all of our ranking to tank from going 7 to 11, 2 to 49. This is crazy and it's really affecting us. And here's what happened. So, months back, we connected a website which is called BetterTime to show the status of our APIs. We created a page for status. We connected a subdomain. So, it was available on status.magicslides.app. Now, what happened was this website somehow created a robot.txt file which was not a valid file. It was allowing every single path to be indexed. On the other hand, the robot.txt file which we have on magicslides is very documented. We make sure to block all the next year's chunks, pages or all the stuff which we don't want it to include. Well, when this robot.txt got in the hands of Google, it just scraped all the long pages, exhausted our crawl budget and now it has way more pages than we actually have and it has affected all of our ranking.