Moving all my web content into "robots.txt" files on the presumption that they'll be ignored by crawlers
Moving all my web content into "robots.txt" files on the presumption that they'll be ignored by crawlers 5 comments
@phooky oh so that's why web developers invented the Single Page App, so it fits into that one file! |
@phooky @0x5021 has anyone ever tried to exploit webcrawlers via malicious content in Robots.txt? I mean, it most probably is not illegal, right?