Email or username:

Password:

Forgot your password?
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ's posts Post Back to profile
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

I'm proud to announce the launch of the public beta for Clew, my independent web search engine! ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰

Try Clew: https://clew.se
Add to your browser: https://clew.se/add/

Clew maintains an independent index and is aiming to be a copyleft (APGLv3), self-hostable, privacy-respecting, customizable search engine which prioritizes independent creators/bloggers/writers and penalizes sites with ads and trackers.

Many features are yet to come. If you'd like to support Clew's development and server costs, you can donate to its Liberapay team: https://liberapay.com/Clew/

Boosts are very welcome, as is feedback of any kind!

21 April at 4:09 | Open on alpha.polymaths.social
84 comments
D. Moonfire
Quinn9282โœŒ๏ธ๐Ÿ‘‹

@amin Cool project! Always nice to see another independent search engine come onto the scene!

21 April at 4:32 | Open on mas.to
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@Quinn9282

Thanks so much!

I've been working on this off and on since Thanksgiving break last year; I took extra long because when I started I kinda had a database phobia and had to relearn pretty much everything I now know about SQL and databases from scratch. XD

In the meantime it seems that a bunch of similar projects have been popping up, and I've tried to add them as options for Clew's bangs because we gotta stick together: https://clew.se/bangs/

Probably gonna add quick links to try a search again on some of these independent sites and indexes, too.

@Quinn9282

Thanks so much!

I've been working on this off and on since Thanksgiving break last year; I took extra long because when I started I kinda had a database phobia and had to relearn pretty much everything I now know about SQL and databases from scratch. XD

In the meantime it seems that a bunch of similar projects have been popping up, and I've tried to add them as options for Clew's bangs because we gotta stick together: https://clew.se/bangs/

21 April at 4:37 | Open on alpha.polymaths.social
Carlos Francisco ๐Ÿฆฃ

@amin it looks really interesting. I'll try it. ๐Ÿ‘Œ๐Ÿป

21 April at 5:19 | Open on lile.cl
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@carloshr

Let me know what you think!

And if the index is thin in a few spots, send me a few good RSS feeds and sites and I'll add them to the index. ;)

21 April at 5:20 | Open on alpha.polymaths.social
Dane Henson

@amin wow, this is very cool. Canโ€™t wait to make heavy use of it and get lost in the independent web!

21 April at 5:51 | Open on hachyderm.io
Alex :anarcho_punk: :demiboy_flag:
Alex :anarcho_punk: :demiboy_flag:
:debian: ๐šœ๐šŽ๐š•๐šŽ๐šŠ :t_blink:

@amin

How can we add sites to Clew to index?

21 April at 7:11 | Open on social.linux.pizza
Joacim Melin
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@joacim @selea

Since Iโ€™ve only just gone public, currently the process is โ€œsend me a linkโ€. ;)

For best results the site should have a RSS/Atom/JSON feed since thatโ€™s how new posts/pages are primarily discovered.

In my plans for this Beta phase is to develop a better site submission process.

21 April at 7:28 | Open on alpha.polymaths.social
:debian: ๐šœ๐šŽ๐š•๐šŽ๐šŠ :t_blink:
:debian: ๐šœ๐šŽ๐š•๐šŽ๐šŠ :t_blink:
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@selea @joacim

Not currently, no. I'm undecided on whether they give me much more value than crawling an RSS/Atom/JSON feedโ€ฆ I'm also wary about using any information that's not user-facing; probably not a big concern with a sitemap, though.

That said it would be really easy to implement into the crawler with the way I've got things coded currently (I'd treat it like another web feed, with some custom parsing code to extract the links). So it probably will be implemented at some point.

21 April at 9:12 | Open on alpha.polymaths.social
Captain Janegay ๐Ÿซ–

@amin @selea @joacim I'd encourage using sitemaps, as they're another indicator of quality - the WCAG accessibility guidelines require sitemaps, so you're more likely to find them on sites run by people who give a shit!

21 April at 12:59 | Open on mastodon.coffee
Joacim Melin
Shamar

@amin

Here my blog (mostly political and tech stuffs) encrypted.tesio.it/atom.xml

Here the blog of an Italian organization that fought #BigTech in Italy with a certain success before the new fake #PrivacyShield was adopted: monitora-pa.it/atom.xml

Both are manually crafted indipendent website (including the feeds), with no tracking or ads.

@joacim @selea

21 April at 9:27 | Open on qoto.org
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@Shamar @joacim @selea

Perfect, I've added both feeds! Since I'm currently running the crawler locally for easier development, it'll start showing up in results when I next upload an update to the index.

21 April at 9:48 | Open on alpha.polymaths.social
Shamar
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@Shamar @joacim @selea

Since so many people have sent me links to their sites or sites they're interested in I'll probably be doing that later "today". (It's 5am here and I still haven't been able to fall asleep after repeated tries. Guess I should give it one more go.)

21 April at 9:58 | Open on alpha.polymaths.social
pancake
McSinyx

Impressive, very nice, @amin, and Iโ€™m looking forward to hosting the copyleft crawler!

btw thanks to the currently smol index, iโ€™m now the primary source for ham and thแป‹t kho tร u :floofSmug:

21 April at 7:46 | Open on larkspur.one
13 barn owls in a trench coat

@amin

Hey, nice work, that produces really interesting results. Not necessary the results I'd expect, but interesting ones, which is certainly useful for me in some cases.

Now pinned to my bookmarks bar!

Here's the RSS feed for one of my sites, to contribute to the data pool a little.

hauntedgames.net/feed/

21 April at 7:55 | Open on eldritch.cafe
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@HauntedOwlbear

Awesome, imported your feed; it'll show up next time I upload an update to the index! (I currently run the crawler on my laptop for easier development.)

I'm glad you find the results interesting! That's definitely what Clew is tuned for right now: showing you new ideas. Like, when trying to decide how to license this project I searched up "permissive licenses" and got a bunch of blog posts debating whether or not copyleft should be used. :)

As for bookmarks barโ€”depending on your browser you can probably add Clew directly to it as a search engine option, usually with the ability to add a keyword to trigger using it instead of your default search engine: https://clew.se/add/

@HauntedOwlbear

Awesome, imported your feed; it'll show up next time I upload an update to the index! (I currently run the crawler on my laptop for easier development.)

I'm glad you find the results interesting! That's definitely what Clew is tuned for right now: showing you new ideas. Like, when trying to decide how to license this project I searched up "permissive licenses" and got a bunch of blog posts debating whether or not copyleft should be used. :)

21 April at 9:16 | Open on alpha.polymaths.social
:mima_rule: Mima-sama

@amin@alpha.polymaths.social Nice, it even indexed my website! โ€‹:mimahappy:โ€‹
21 April at 7:57 | Open on makai.chaotic.ninja
PurpleJillybeans :PrideDisk:

@amin Nice! I certainly hope that your index grows and this ends up becoming the next great tool in the answer-finding toolbox. I know I was one of many who were asking "what do we use now?" when our old standby #DuckDuckGo jumped on the AI bandwagon. Maybe this will be the answer? ๐Ÿค”

21 April at 8:13 | Open on kind.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@PurpleJillybeans

I hope it'll be useful, for sure! If nothing else, it's been a great way for me to find new ideas on topics I'm curious about, which is really where it shines at the moment. Like in deciding how to license this project, I searched up ("clewed up"?) "permissive licenses" and got a bunch of blog posts debating whether or not to use copyleft.

But yeah, I have no intention of ever including AI and am actually doing some things by design to intentionally make the search index non-useful as training data for Machine Learning.

@PurpleJillybeans

I hope it'll be useful, for sure! If nothing else, it's been a great way for me to find new ideas on topics I'm curious about, which is really where it shines at the moment. Like in deciding how to license this project, I searched up ("clewed up"?) "permissive licenses" and got a bunch of blog posts debating whether or not to use copyleft.

21 April at 9:19 | Open on alpha.polymaths.social
Nafeon the Beaw

@amin I have a suggestion and a question. I use searxng as a meta search engine. I could imagine it would be neat to add clew as plugin to see how much better it is to the other (now AI tainted) engines.

the second one is: could I scope it to index only the intranet of a company, when self hosting?

21 April at 8:40 | Open on bears.town
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@NafiTheBear

re: searxng

Hmm, that would definitely be cool! Do you know if there's some kind of API I could implement that would make that easier? I don't really know much about searx so any guidance in helping make that a reality is appreciated.

re: scope

Not currently, but the ability to do that definitely fits into my roadmap for the project. One specific idea I've had is to be able to scope it to crawl sites from a webring, so you can search the webring, like Lieu, and scoping it to an intranet would not be much harder.

I assume, though, that would need some kind of authentication to crawl the intranet? That might be the tricky part depending on how the intranet is implemented. (Unless it's just a "be on the local network" thing in which case it's probably pretty easy.)

@NafiTheBear

re: searxng

Hmm, that would definitely be cool! Do you know if there's some kind of API I could implement that would make that easier? I don't really know much about searx so any guidance in helping make that a reality is appreciated.

re: scope

Not currently, but the ability to do that definitely fits into my roadmap for the project. One specific idea I've had is to be able to scope it to crawl sites from a webring, so you can search the webring, like Lieu, and scoping it to an intranet...

21 April at 9:23 | Open on alpha.polymaths.social
Nafeon the Beaw

@amin

re: searxng
there is documentation for it: docs.searxng.org/dev/plugins.h

although I think it would just be enough to provide a good API I think and well focus on publishing it, I think a plugin is written out rather fast.

re: scope for my intranet it would be enough to access the public parts so basically what you plan anyway. I just need something that can crawl the company internal wiki and a couple of other public doc.

21 April at 15:33 | Open on bears.town
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@NafiTheBear

Sounds perfect, then! I'll keep that on my radar. You're also welcome to submit feature requests here to make sure it stays on my radar: https://codeberg.org/Clew/Clew/issues

21 April at 21:18 | Open on alpha.polymaths.social
Eric the Cerise

@amin

As something of a search engine connoisseur, I look forward to putting it through it's paces.

But before I start bitching about specifics, let me congratulate and thank you for even trying this.

Building a functional SE these days seems Herculean. I'd sooner try to create a new programming language, or new OS from scratch (TempleOS still around?).

Edit: Hey! Throw some relevant hashtags into an announcement like this. Don't you know how Internet search works? ๐Ÿ˜‰

21 April at 8:46 | Open on kolektiva.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@ErictheCerise

Awww you're gonna make me cry. ๐Ÿ˜ญ

Thank you so much for the kind words! Means a lot.

Speaking of the "herculean" bit, I feel you might get a kick out of reading blog posts about my initial attempt earlier this year, which was to code the web crawler in pure bash scripts. Oh yeah, and I was also going through something of a database phobia. XD

https://benjaminhollon.com/musings/starting-work-on-a-search-engine/
https://benjaminhollon.com/musings/web-crawler-sort-order/
https://benjaminhollon.com/musings/this-shouldve-been-a-database/
https://benjaminhollon.com/musings/using-grep-to-increase-index-quality/
aaaand where I give in and decide to go with python: https://benjaminhollon.com/musings/announcing-clew-my-search-engine/

;)

(Yeah, I tend to forget hashtags existโ€ฆ whoops! I will make a mental note for future releases like thisโ€ฆ)

@ErictheCerise

Awww you're gonna make me cry. ๐Ÿ˜ญ

Thank you so much for the kind words! Means a lot.

Speaking of the "herculean" bit, I feel you might get a kick out of reading blog posts about my initial attempt earlier this year, which was to code the web crawler in pure bash scripts. Oh yeah, and I was also going through something of a database phobia. XD

21 April at 9:27 | Open on alpha.polymaths.social
Eric the Cerise

@amin

Are you familiar with Stract.com ? Another mostly-one-person search engine, still a bit Beta but already quite nice.

21 April at 12:31 | Open on kolektiva.social
ULTROS_PROFESSIONAL

@amin do you plan to integrate ai? i would love to use a search engine that actively avoids this trend.

21 April at 8:55 | Open on cuties.cloud
Elias Mรฅrtenson

@amin how big is the index? I tried searching for my programming language and the index doesn't know anything about it.

kapdemo.dhsdevelopments.com/

21 April at 8:58 | Open on functional.cafe
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@loke

Very tiny at the moment, I'm afraid. Still working on developing the crawler and it's not quite ready to go at full speed yet. :)

On results pages should be a count of the current number of pages in the index (32,000 pages at this time).

I've added that site to my local index, it'll show up on the live site when I next upload an update. ๐Ÿ‘๏ธ

21 April at 9:31 | Open on alpha.polymaths.social
Andrรกs Nรฉmeth
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@sunarch

Thank you!!!! :BlobCatHeart:

I'm so excited. I should be asleep but after lying in bed for a couple hours couldn't fall asleep. :)

21 April at 9:31 | Open on alpha.polymaths.social
OrbitalMartian :GoToSocial:

@amin On add page, under Safari, youโ€™ve put โ€œlaodedโ€ not โ€œloadedโ€ :).

21 April at 9:09 | Open on alpha.polymaths.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@orbitalmartian

Oh, scubrats. Fixed, good catch!

~/projects/clew] git commit -a
[main 6583dcb] Fix another tpyo
 1 file changed, 1 insertion(+), 1 deletion(-)
21 April at 9:33 | Open on alpha.polymaths.social
Shamar

@amin

Very nice work!

I see the engine is a single #python script that connects to a #PostgreSQL db.

Both are cool technologies but require quite a bit of technical expertise to be self-hosted.

Over the years I've seen that projects based on less cool technologies (php, cgi-bin, sqlite...) enable both #selfhosting and #SAFEhosting, that is using cheap, local, non #BigTech hosting providers.

It's something I realized reading free software based on the #permacomputing values.

Not really a suggestion or a feature request (maybe a note to myself, for a fork when I'll have more free time), but something I think you might consider.

Another one: it would be cool to enable a sort of federation among the instances, either by simply proxying the trusted instances (and excluding the duplicated urls) on user's search, or by enabling trusted #fediverse users to add websites to be crawled.

"Trust" here is a key concept: federation should be optional and disabled by default.

Anyway: good luck and good work!

And thanks for using a network #copyleft!
(I prefer the #HackingLicense over #AGPLv3 in the age of #GitHubCopilot/CopyALot, but at least AGPL protects the work you donated to the world from direct privatization...)

@selea

@amin

Very nice work!

I see the engine is a single #python script that connects to a #PostgreSQL db.

Both are cool technologies but require quite a bit of technical expertise to be self-hosted.

Over the years I've seen that projects based on less cool technologies (php, cgi-bin, sqlite...) enable both #selfhosting and #SAFEhosting, that is using cheap, local, non #BigTech hosting providers.

21 April at 9:15 | Open on qoto.org
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@Shamar @selea

Thanks for the thoughts, they're good ones!

Postgres I mostly picked because it's the database I've heard the highest praise of from everyone I've talked to who works daily with databases. Myself, I had something of a database phobia (๐Ÿ˜ฌ) before starting work on this project so I was entirely working off other people's recommendations in that matter.

As for Python, it was mostly for access to a really great natural language processing library, Polyglot, which works well cross-language. Most of the heavy work is done by a SQL query, so it likely wouldn't be hard to reimplement in another backend language if someone so desired.

And for hosting apart from big tech, I'm currently running the live version on Uberspace which I highly recommend and which I expect will continue being able to host something like clew for the forseeable future (storage space perhaps being the main tricky bit).

@Shamar @selea

Thanks for the thoughts, they're good ones!

Postgres I mostly picked because it's the database I've heard the highest praise of from everyone I've talked to who works daily with databases. Myself, I had something of a database phobia (๐Ÿ˜ฌ) before starting work on this project so I was entirely working off other people's recommendations in that matter.

21 April at 9:39 | Open on alpha.polymaths.social
๐‘๐ž๐›๐ž๐œ๐œ๐š ๐…๐ข๐ง๐ง ๐Ÿณ๏ธโ€โšง๏ธ ๐Ÿ‡บ๐Ÿ‡ฆ

@amin is it mostly aimed for English speaking audiences? I tried a couple of searches in Spanish and got results that got me some thoughts that, if you are interested in, I could share with you.
Cheers.

21 April at 10:30 | Open on topspicy.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@rebeccafinn

I do want it to work well in non-English languages, but of course that does depend on whether the underlying index has pages in those languages ranked.

I'm very interested in hearing any feedback you have on the experience of searching in Spanish.

Also, to help the index get better seeded, if you have any great sites and blogs in spanish, send the links my way and I'll import them!

21 April at 11:40 | Open on alpha.polymaths.social
iooioio

@amin So cool! Haven't seen anyone else mention this but I think the (visual) design is great.

21 April at 12:38 | Open on fosstodon.org
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@iooioio

Yayyyy I'm glad you like it! :D

Much of the visual design is a reusable classless CSS framework I made: https://readable-css.freedomtowrite.org/

Anything else is a layer on top of that.

And of course @joel gets credit for making that beautiful logo. :BlobCatHappie:

21 April at 12:40 | Open on alpha.polymaths.social
cyocum

@amin Interesting, I searched for "Celtic Studies" and it came up with a bunch of "JMK Engineering Inc. " stuff.

I would suggest indexing the following:

* codecs.vanhamel.nl/Home
* dias.ie/celt/
* bill.celt.dias.ie/
* celt.ucc.ie/

21 April at 12:57 | Open on mastodon.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@26aafa19

Done, those'll show up next time I upload an index update. Thanks for the recommendations! :D

(As an explanation for your odd results; since it couldn't find anything for "Celtic", it defaulted to just "studies". That's on my list of things to improve.)

21 April at 13:02 | Open on alpha.polymaths.social
cyocum

@amin Thank you! My area is honestly rather niche to be sure ๐Ÿ˜‚

21 April at 16:02 | Open on mastodon.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@26aafa19

Well, it takes a lot of niches to make a nache. (No, that's not a saying or a real word, but it sounds like it should be so I'm gonna say it anyway. ๐Ÿ˜‰)

21 April at 21:19 | Open on alpha.polymaths.social
Joshua Wagner :debian: :gnome:

@amin ohhh! I'll have to try this out.

21 April at 12:58 | Open on fosstodon.org
MarvinFreeman

@amin The "bangs" thing is cool. Makes using the engine much more usable for a first search if one can use the bang system when clew's results are less than helpful.

21 April at 13:11 | Open on mastodon.online
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@MarvinFreeman

It definitely is! I admit to having completely stolen the idea from DuckDuckGo though. ;)

https://duckduckgo.com/bangs/

(Will probably be changing the character at the beginning to ! like DDG since it's been discovered that / conflicts with some preset shortcuts in people's browsers.)

21 April at 13:12 | Open on alpha.polymaths.social
lily ๐Ÿณ๏ธโ€โšง๏ธ

@amin@alpha.polymaths.social woah this is cool, i was thinking of doing this but this looks way better than mine

21 April at 13:33 | Open on possum.city
lily ๐Ÿณ๏ธโ€โšง๏ธ

@amin@alpha.polymaths.social also is there any way to add your own site to the index? does it require any special requirements (like a sitemap.xml) or does it just crawl through all links?

21 April at 13:35 | Open on possum.city
hyolo

@amin@alpha.polymaths.social Very interesting. How does this compare to Stract?
Also, have you heard of YaCy?

21 April at 13:35 | Open on den.raccoon.quest
Lucas ๐Ÿ’พ๐Ÿฆ‰โ™พ๏ธ

@amin Cool project! ๐Ÿ‘ And nice to see that its code is hosted on @Codeberg
Can't wait to see the crawler code too.
Boosted and followed ๐Ÿ˜Ž

21 April at 13:44 | Open on mastodontech.de
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@lhinderberger @Codeberg

Yup! Been using Codeberg ever since Github launched Copilot. :P

Crawler code is a bit of a mess right now but steadily improving. ;)

21 April at 14:15 | Open on alpha.polymaths.social
LionelB

@amin

My impression is that the selection of test sites is curated?

For an objective evaluation, a random subset would be more realistic, even if it returns a lot of junk.

21 April at 13:52 | Open on expressional.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@lionelb

Well, the initial index was seeded with RSS feeds I follow, which are definitely curated, and I'm still trying to get the discovery code to where I'm satisfied with it, so your impression is unfortunately fairly accurate. :P

That said if there's an area you see underrepresented send me some links to good sites/blogs and I'll index them. (Planning to add a better process for this, but with the site only just now public it wasn't a priority.)

21 April at 14:19 | Open on alpha.polymaths.social
LionelB

@amin

I tried just one test, which is always the most difficult.

The history of hotels, architecture of hotels, regulation of hotels, hotels slept in by famous people etc. etc. are impossible to probe using a search engine because the first 100,000 pages will try to sell a hotel room.

21 April at 14:27 | Open on expressional.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@lionelb

That, hopefully, is something that this will be better at than most engines once the index is broader, since it tries to focus on noncommercial, independent sites.

21 April at 14:28 | Open on alpha.polymaths.social
David Hall

@amin โ€Norrkรถpingโ€ seems to give almost the same results as โ€œpingโ€.

21 April at 14:22 | Open on social.tchncs.de
Packbats, creative

@amin How does the algorithm handle multiple words? We tried wurlitzer synth design as a test, and the first search result that wasn't on page one of the synth search or the design search was all the way down at number thirteen. That seems low to us - combining search terms to find something at the intersection is a pretty standard approach.

21 April at 15:25 | Open on indiepocalypse.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@Packbat

Currently it does combine tokens into keywords; two maximum at the moment but I will probably expand on that. The current thing I'm dealing with is that it's not great at the fallback if it doesn't find that specific combination, which I'm guessing is the situation here.

To demonstrate, there are no results for "wurlitzer": https://clew.se/search?q=wurlitzer

And the "suggestions" path which is filled in using keywords from the database has nothing for "synth design", meaning that specific combination doesn't appear in the index at this time: https://clew.se/suggestions?q=synth+design

It's a pretty small index, only 30k words. Enough for the testing stage I'm at but it'll definitely need to be scaled up significantly before this is fully ready to go.

@Packbat

Currently it does combine tokens into keywords; two maximum at the moment but I will probably expand on that. The current thing I'm dealing with is that it's not great at the fallback if it doesn't find that specific combination, which I'm guessing is the situation here.

To demonstrate, there are no results for "wurlitzer": https://clew.se/search?q=wurlitzer

21 April at 21:17 | Open on alpha.polymaths.social
Packbats, creative

@amin *nods*

That would pretty well explain it - the oddity of the single result that popped up from the depths to the first page made us wonder if something else was happening. Thanks for the response!

21 April at 21:21 | Open on indiepocalypse.social
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@Packbat

No problem!

Any feedback on this process is also of course very welcome. :)

21 April at 21:31 | Open on alpha.polymaths.social
Packbats, creative

@amin our leading concern reading the methodology is that white reactionaries with money to burn on disseminating their propaganda will be overrepresented in search results, which isn't great - there are a lot of alt-right news sites with no ads

no idea how to address that, though

21 April at 21:40 | Open on indiepocalypse.social
Codrus :archlinux: ๐Ÿ‡บ๐Ÿ‡ฒ

@amin @amin Adding it to my KDE desktop search plugin now! Thanks for making it OpenSearch-compatible!

I look forward to many future rabbit-holes and time-sinks...

21 April at 21:02 | Open on linuxrocks.online
Amin Hollon ๐Ÿ‡บ๐Ÿ‡ธ๐Ÿ‡ฒ๐Ÿ‡พ๐Ÿ‡ฎ๐Ÿ‡ณ๐Ÿ‡ฆ๐Ÿ‡ซ

@codrusofathens

Hope you enjoy! :D

I also added OpenSearch search suggestion support though that's a little hit-or-miss since at least in Firefox many queries take longer than the ridiculously-low 500ms timeout. :)

21 April at 21:03 | Open on alpha.polymaths.social
Sire

@amin congratulations ๐Ÿฅณ๐Ÿฅณ๐Ÿฅณ๐Ÿฅณ๐Ÿฅณ

21 April at 21:12 | Open on alpha.polymaths.social
Synth Morxemplum

@amin Good shit. We need more competition in the web search space so we can dissolve the effectiveness of SEO. Canโ€™t wait to see more!

22 April at 20:08 | Open on mastodon.social
Go Up