@Nfoonf
Why would it be? Robots.txt is not even a standard, it's a convention. It takes on true/false switch in the crawling algorithm and it will ignore it.
@phooky @0x5021