Email or username:

Password:

Forgot your password?
Aires's posts Post Back to profile
Top-level
James 🌈💜

@aires Ooh. I can weigh in on this one!

I started writing my own search engine and crawler. The index is uselessly small right now, but it prioritises written content and small websites.

rustysearch.org/

8 June at 13:01 | Wall-to-wall | Open on mastodon.social
3 comments
skua

@shaknais @aires

Tried #RustSearch out.

It returned apposite results.

Brought back memories of a #SmallWWW

What are the current limiting factors on it crawling much further?

9 June at 0:53 | Open on mastodon.social
James 🌈💜

@skua @aires
The results are weighed for accuracy, and of the top N of those, they are reweighed by content, and presented in that order. It isn't the most accurate thing, not yet, but the web as a whole is weighted for recency, not content, at the moment.

Crawling is just a time factor, really. I'm not spending much on the project, so I only have a single multi-threaded crawler

I haven't implemented keywords like "not", because I'm experimenting with a few ways of filtering. They should help

@skua @aires
The results are weighed for accuracy, and of the top N of those, they are reweighed by content, and presented in that order. It isn't the most accurate thing, not yet, but the web as a whole is weighted for recency, not content, at the moment.

Crawling is just a time factor, really. I'm not spending much on the project, so I only have a single multi-threaded crawler

9 June at 1:49 | Open on mastodon.social
James 🌈💜

@skua @aires
Oh, on "tried it out", I don't yet have a page on data collection. I should get to it soon.

But as for your data - I have nginx logs, and I do plan to keep and analyse them. Mostly for caching.

However, I use no 3rd party analytics, and sell data to no one, and you'll never find that stuff passed onto someone like Google.

9 June at 2:32 | Open on mastodon.social
Go Up