@darius what is the mechanism that prevents crawlers...

Darius's posts Post Back to profile

Top-level

Michael Sokolov

@darius what is the mechanism that prevents crawlers from indexing posts here? Is it advisory like robots.txt?

Like 4 Dec 2022 at 11:28 | Wall-to-wall | Open on fosstodon.org

3 comments

Darius Kazemi

@msokolov yes

4 Dec 2022 at 16:06 | Open on friend.camp

Darius Kazemi

@msokolov it is "noindex" https://developers.google.com/search/docs/crawling-indexing/block-indexing

4 Dec 2022 at 16:08 | Open on friend.camp

Michael Sokolov

@darius thanks, interesting in a geeky way... TIL Using robots.txt to block the crawler can prevent it from reading this noindex tag paradoxically exposing it in search results ... Due to links? Somehow...

4 Dec 2022 at 18:01 | Open on fosstodon.org

Go Up