@jon but does this spider respect my robots.txt?