@phooky @0x5021 has anyone ever tried to exploit webcrawlers via malicious content in Robots.txt? I mean, it most probably is not illegal, right?
@Nfoonf Why would it be? Robots.txt is not even a standard, it's a convention. It takes on true/false switch in the crawling algorithm and it will ignore it.@phooky @0x5021
Gregory's Server
@Nfoonf
Why would it be? Robots.txt is not even a standard, it's a convention. It takes on true/false switch in the crawling algorithm and it will ignore it.
@phooky @0x5021