@tedivm Do you happen to know if a blanket disallow like:
User-agent: *
Disallow: /
Will be honored by GPTBot? I wouldn't put it past them to ignore the root disallow and require a specific `User-agent: GPTBot`, but it's not called out in the doc either way.
There's some inconclusive evidence in my Nginx access logs that they crawled my robots.txt, then proceeded to crawl other URLs. But I only see 6 hits total, and they all happened within 10 seconds. 🤷♂️
@null it should be- I'd be interested if anyone has examples of it ignoring those