Moving all my web content into "robots.txt" files on...

All posts phooky's posts Post Back to profile

Moving all my web content into "robots.txt" files on the presumption that they'll be ignored by crawlers

Like 12 May at 12:12 | Open on octodon.social

5 comments

Nfoonf

@phooky @0x5021 has anyone ever tried to exploit webcrawlers via malicious content in Robots.txt? I mean, it most probably is not illegal, right?

12 May at 13:06 | Open on chaos.social

Julianoë

@Nfoonf
Why would it be? Robots.txt is not even a standard, it's a convention. It takes on true/false switch in the crawling algorithm and it will ignore it.
@phooky @0x5021

12 May at 21:58 | Open on mastodon.xyz

phooky

or just the similar format gobots.txt, which is just robots.txt in disguise

12 May at 13:47 | Open on octodon.social

herbert

@phooky / cc @mnot

12 May at 14:36 | Open on w3c.social

lexi :neocat_floof_happy:

@phooky oh so that's why web developers invented the Single Page App, so it fits into that one file!

12 May at 18:44 | Open on catcatnya.com

Go Up