Just a headsup incase anyone uses these rules. Google is dropping support for Robots.txt Noindex rules but still supports other ways of blocking pages from indexing:
I didn’t even know NoIndex in robots.txt was a thing. I thought NoIndex had to go in a meta tag.
So, from my perspective, nothing has changed.
No I’ve never used it. At first i thought they meant robots.txt was no longer supported but its just some rules that they wont support.
I used to block Google’s bot tracking my jump script in robots.txt. But now Google ignored all the rules in robots.txt. I have seen google’s bot start to crawl everything, including my jump script, even though the result does not make any sense.
How do you know it’s Google? Are you going off user agent, or a hostname lookup on the IP?