Email or username:

Password:

Forgot your password?
Gregory's posts Post Back to profile
Gregory

Whoever you are whose crawler this is, fix your thing already. You're supposed to use nodeinfo instead of hardcoded API paths with typos. And maybe Mastodon's /api/v1/instance, but that's debatable.

#activitypub @activitypub
27 Jul 2021 at 19:45 | Open on mastodon.social
17 comments
Michael Vogel
@grishka Please have a look at the user agent string. This should reveal where the requests come from. Most likely this is an older Friendica version, then the user-agent string tells you not only the version of the system but also the exact site name.

In newer versions we now use a different order of requests to reduce the amount of needed queries per server. BTW: All the queried endpoints are valid in some versions of Friendica (which at one point hat the name Friendika) and GNU Social (which had the name Statusnet before).

The current server detection looks like this: github.com/friendica/friendica…

In current versions we search for "/.well-known/nodeinfo" first. And we only query those endpoints above only when this query hadn't been successful (and also when the "host-meta" path is present). This seems to work fine with Smithereen (I just did a quick test against a known server).
@grishka Please have a look at the user agent string. This should reveal where the requests come from. Most likely this is an older Friendica version, then the user-agent string tells you not only the version of the system but also the exact site name.
28 Jul 2021 at 3:33 | Open on pirati.ca
Gregory

@heluecht added logging, now gotta wait for these requests to have my curiosity satisfied :D

28 Jul 2021 at 13:19 | Open on mastodon.social
Gregory

@heluecht you're right, these are coming from Friendica, and apparently two instances running some really old version
28 Jul 2021 at 14:44 | Open on mastodon.social
Michael Vogel
@grishka Oh yeah ... 3.6. is really old. These versions don't even speak AP at all. I guess we introduced AP in 2019 or late 2018.
28 Jul 2021 at 15:00 | Open on pirati.ca
Gregory

@heluecht and by the way while we're at it — am I right that many of additional profile fields, as well as things like photos, aren't exposed in AP yet? Are there any plans about this? It'd be great if we could make something compatible where our features intersect.

28 Jul 2021 at 15:07 | Open on mastodon.social
Gregory

@heluecht btw I'm getting this when trying to fetch your profile, but it does somehow work fine from my server
28 Jul 2021 at 15:07 | Open on mastodon.social
Michael Vogel
@grishka What is "Paw"? I'm using a bot detection on that server.
28 Jul 2021 at 15:25 | Open on pirati.ca
Gregory

@heluecht it's a native REST client for macOS, a very nice one. They were giving away licenses for retweets a while ago.
paw.cloud/

28 Jul 2021 at 16:16 | Open on mastodon.social
Michael Vogel
@grishka Okay, then I'm putting it on the whitelist
28 Jul 2021 at 16:20 | Open on pirati.ca
Michael Vogel
@grishka We should have published everything. If something isn't working, please tell.
28 Jul 2021 at 15:20 | Open on pirati.ca
Gregory

@heluecht everything that's in this tab pirati.ca/profile/heluecht/pro in your profile isn't returned anywhere in the AP actor.

28 Jul 2021 at 16:33 | Open on mastodon.social
Michael Vogel
@grishka This is what the system is returning: 
  [@context] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams
            [1] => https://w3id.org/security/v1
            [2] => Array
                (
                    [vcard] => http://www.w3.org/2006/vcard/ns#
                    [dfrn] => http://purl.org/macgirvin/dfrn/1.0/
                    [diaspora] => https://diasporafoundation.org/ns/
                    [litepub] => http://litepub.social/ns#
                    [toot] => http://joinmastodon.org/ns#
                    [manuallyApprovesFollowers] => as:manuallyApprovesFollowers
                    [sensitive] => as:sensitive
                    [Hashtag] => as:Hashtag
                    [directMessage] => litepub:directMessage
                    [discoverable] => toot:discoverable
                )

        )

    [id] => https://pirati.ca/profile/heluecht
    [diaspora:guid] => 0704b08f5c777566
    [type] => Person
    [following] => https://pirati.ca/following/heluecht
    [followers] => https://pirati.ca/followers/heluecht
    [inbox] => https://pirati.ca/inbox/heluecht
    [outbox] => https://pirati.ca/outbox/heluecht
    [preferredUsername] => heluecht
    [name] => Michael Vogel
    [vcard:hasAddress] => Array
        (
            [@type] => vcard:Home
            [vcard:country-name] => Germany
            [vcard:region] => Hamburg
            [vcard:locality] => Hamburg
        )

    [summary] => Pirat aus Hamburg Mitte. Fahrrad-Freak, Social-Media-Junky, Open-Source-Programmierer, Fan von islieb, Franzbrötchen und guter Schokolade.<hr>Wunschliste: alles unter <a href="https://www.rausch.de/schokolade/" target="_blank" rel="noopener noreferrer">https://www.rausch.de/schokolade/</a><br><br>Wer Amazon mag: <a href="https://www.amazon.de/hz/wishlist/ls/3VWK0ZL3MN3ZT" target="_blank" rel="noopener noreferrer">https://www.amazon.de/hz/wishlist/ls/3VWK0ZL3MN3ZT</a><br><br>Liberapay: <a href="https://liberapay.com/heluecht/donate" target="_blank" rel="noopener noreferrer">https://liberapay.com/heluecht/donate</a><br>BTC: 1AtJ9JVysdhWjSs5qQvp7Xt9xFdjMKSSA7<br>BCH: qpjg2gwgr35fgz3dxy6lcpw3lt4szrfgev90uk3tfv
    [url] => https://pirati.ca/profile/heluecht
    [manuallyApprovesFollowers] => 1
    [discoverable] => 1
    [publicKey] => Array
        (
            [id] => https://pirati.ca/profile/heluecht#main-key
            [owner] => https://pirati.ca/profile/heluecht
            [publicKeyPem] => -----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxuBagH7XdLM6GWJ261As
Gj4GKhDnn+OLqxpaEMCe/6UGC9YBvk2Fhda9d4P1PY1vg4iy1DS+pyf55P9G6TgJ
dAN1FpkUnsxEwi9d2rOnDhsacJgwWwS6snDl3uMi1MO7AQhdoNgmb3nqAAvSDhsX
Gr9HwKXf8HnydPDLL4k9Y5AFKhrrHSdVqiLk4oxgGJRymjPCW9rgQTKsVY5U7iuk
mpPLhhTKeqhPvQ5vwQC03iKbfgZ4YQifbK7qubnnwLulaSEwJjlRVEPXnLWp4RrI
vCZnyUoshpVB8empRQdgJKQ2249Ll9Xdkqh62+1EAh8A/m4j4ZbKYgngW/850mOd
scrnJrnLqTuFz3cdVhC30u6hj1dQ5cYINrYfKOyFJjI91Xwan9fPpoG4AJCjeuXZ
cRhwd/GmhowXj4pSWWI0oMtBdw4uLYGFyhT3Hezg1u9fElBMvauPS/pHAj8NY19Z
X/+Xk3icQLrHN5KBvM2E4Ha/ddBt66M+NLyi+qWrBoMc14wDCNs8cbkimyA7jVNa
hZld6kIrojxfKHPA8KRKeTkWrgC5WliXNQ7HOO3G4QI9l2pfMtWfaF6bOQsirE1Y
E8kGIdipWOakWOki65zhgxbdUA7TGMmW4dNDFnbbR6sPnUHE3v3J39dUTZn7Pe56
GD111cTk8E5pzH4XVO71Vw0CAwEAAQ==
-----END PUBLIC KEY-----

        )

    [endpoints] => Array
        (
            [sharedInbox] => https://pirati.ca/inbox
        )

    [icon] => Array
        (
            [type] => Image
            [url] => https://pirati.ca/photo/contact/1?ts=1613722415
            [mediaType] => image/png
        )

    [generator] => Array
        (
            [type] => Service
            [name] => Friendica 'Siberian Iris' 2021.09-dev-1430
            [url] => https://pirati.ca
        )
)


The name, the avatar, the description and the location is exported. The homepage, the XMPP address and the variable profile fields aren't. Mainly because I haven't found corresponding AP fields.
@grishka This is what the system is returning: 
 [@context] => Array ( [0] => https://www.w3.org/ns/activitystreams [1] => https://w3id.org/security/v1 [2] => Array ( [vcard] => http://www.w3.org/2006/vcard/ns# [dfrn] => http://purl.org/macgirvin/dfrn/1.0/ [diaspora] => https://diasporafoundation.org/ns/ [litepub] => http://litepub.social/ns# [toot] => http://joinmastodon.org/ns# [manuallyApprovesFollowers] => as:manuallyApprovesFollowers [sensitive]...
28 Jul 2021 at 17:05 | Open on pirati.ca
Gregory

@heluecht yeah that corresponds with that I receive.

You could use schema.org for those fields, it has schemas to cover everything under the sun. Like I did for first/last/middle names for example, because AP specifies just "name" as a single entity.

28 Jul 2021 at 17:28 | Open on mastodon.social
Michael Vogel
@grishka You wrote "everything" - which confused me. The variable profile texts can't be exported though, since they are freeformat.
28 Jul 2021 at 17:52 | Open on pirati.ca
Gregory replied to Michael

@heluecht Mastodon does a thing where it returns an array of PropertyValue's in attachment field, like this. If I understood correctly what you meant by variable profile texts.
28 Jul 2021 at 18:13 | Open on mastodon.social
Michael Vogel
@grishka Or do you mean that it isn't included in the activities? We only provide the URL there, since this is just a waste of space to provide this info with each and every activity.
28 Jul 2021 at 17:10 | Open on pirati.ca
Go Up