Email or username:

Password:

Forgot your password?
phooky

Moving all my web content into "robots.txt" files on the presumption that they'll be ignored by crawlers

5 comments
Nfoonf

@phooky @0x5021 has anyone ever tried to exploit webcrawlers via malicious content in Robots.txt? I mean, it most probably is not illegal, right?

Julianoë

@Nfoonf
Why would it be? Robots.txt is not even a standard, it's a convention. It takes on true/false switch in the crawling algorithm and it will ignore it.
@phooky @0x5021

phooky

or just the similar format gobots.txt, which is just robots.txt in disguise

lexi :neocat_floof_happy:

@phooky oh so that's why web developers invented the Single Page App, so it fits into that one file!

Go Up