@amin

re: searxng
there is documentation for it: https://docs.searxng.org/dev/plugins.html#example-plugin

although I think it would just be enough to provide a good API I think and well focus on publishing it, I think a plugin is written out rather fast.

re: scope for my intranet it would be enough to access the public parts so basically what you plan anyway. I just need something that can crawl the company internal wiki and a couple of other public doc.