@paulcuth Hi! I noticed your crawler gets a 406 Not Acceptable status from my site – probably because it sends requests with Accept: text/html, and my site is XHTML. Most HTML parsers can handle XHTML, so this is probably just a matter of changing the header.