Email or username:

Password:

Forgot your password?
Ilya's posts Post Back to profile
Top-level
Ilya Zverev

This one came up yesterday, while preparing for @organicmaps GSoC student work.

When developing an editor, you need to work with OSM Sandbox, a dev copy of API with no data. Sometimes it's frustrating because complex data is where an editor breaks. Usually I downloaded an OSM extract in JOSM, fixed ids, and uploaded it to the Sandbox, but on the 100th time I gave up.

github.com/Zverik/osm_to_sandb

This script helps you transfer a bbox worth of data in two commands, one of which is pip install.

3 May at 6:36 | Open on en.osm.town
40 comments
Organic Maps replied to Ilya

@zverik Does the script also change IDs? What does "fix ids" mean? For readers' info, Organic Maps Editor doesn't store IDs and doesn't rely on them.

5 May at 17:25 | Open on fosstodon.org
InsertUser replied to Organic

@organicmaps
I'd guess it flips them to be negative and resets any version numbers so that the dev server recognises them as new and assigns IDs to them rather than rejecting them as unknown.
@zverik

5 May at 18:32 | Open on en.osm.town
Ilya Zverev replied to Organic

@organicmaps To upload objects to the sandbox, you need to give them negative ids and keep the references. This is not related to OM, just a low-level API stuff.

6 May at 10:48 | Open on en.osm.town
Ilya Zverev replied to Ilya

Okay I skipped three days, so here are three tweets on a big thing I did at Maps.Me. It's on the deep end of the iceberg.

In short, I was tasked with bringing subway networks to the app. Which was based on OSM data, so naturally I thought how to fix the data model: every city had its own tagging schema.

To monitor the process, I wrote the validator, which now powers @organicmaps :

cdn.organicmaps.app/subway/ind

Alexey later improved it, a lot. Now there are over 300 cities, all uniformly mapped!

Okay I skipped three days, so here are three tweets on a big thing I did at Maps.Me. It's on the deep end of the iceberg.

In short, I was tasked with bringing subway networks to the app. Which was based on OSM data, so naturally I thought how to fix the data model: every city had its own tagging schema.

"Subway Validation Results. 305 of 308 networks validated without errors. To make a network validate successfully please follow the (link) metro mapping instructions. Commit your changes to the OSM and then check back to the updated validation results after the next validation cycle, please. See (link) the validation instance description for the schedule and capabilities". There is also a map link, and a long list of countries, mostly in green, each having 1 to 15 cities with subways.
6 May at 19:23 | Open on en.osm.town
Ilya Zverev replied to Ilya

It was made possible with planning, iterating, and preparing a detailed tagging schema that incorporated most tagging practices:

wiki.openstreetmap.org/wiki/Me

It started as a proposal, which failed the voting. Then I made another proposal, but didn't have the nerve to go through voting the 2nd time.

But also I noticed how most objections come from just a few people in EU, so I just started tidying up networks in many cities, and when we passed 80 (with help of other OSMers), it became the standard.

It was made possible with planning, iterating, and preparing a detailed tagging schema that incorporated most tagging practices:

wiki.openstreetmap.org/wiki/Me

It started as a proposal, which failed the voting. Then I made another proposal, but didn't have the nerve to go through voting the 2nd time.

Hand-drawn schema of a complex subway station. There are two platforms with two intersecting lines. Each platform has overground exits, and some escalators.
6 May at 19:37 | Open on en.osm.town
Ilya Zverev replied to Ilya

At that point I was ready to tackle the general public transport schema. After all, PTv2 is impossible to route over and a pain to use.

So, with experience in processing PT data for routing and validation, I generalized it for all kinds of transport, and presented at @sotm in Milan:

wiki.openstreetmap.org/wiki/Pr

Technically it's flawless. It has full backward compatibility, easy to map, tagging is pretty and conscise.

But at that point I got burnt out at my job and couldn't make the final step.

At that point I was ready to tackle the general public transport schema. After all, PTv2 is impossible to route over and a pain to use.

So, with experience in processing PT data for routing and validation, I generalized it for all kinds of transport, and presented at @sotm in Milan:

wiki.openstreetmap.org/wiki/Pr

"Stop Areas and Groups": three pictures denoting correct and incorrect ways of grouping stop elements into stop_area relations.
6 May at 19:45 | Open on en.osm.town
Ilya Zverev replied to Ilya

(should mention that there are TONS of nuances that don't fit into tweets.)

Two main issues with the proposal that don't make it a perfect one:

- It's too wordy, nobody would read it, despite the core idea is very simple (stop nodes).

- It challenges the current tram stop mapping (and nothing else).

The new PT schema should've been perfect in 2018 to be approved, and now it should be double-perfect, otherwise it won't pass any discussion. Everybody in OSM has their own opinion on PT mapping.

6 May at 19:55 | Open on en.osm.town
Ilya Zverev replied to Ilya

Keeping an up-to-date rendering database means relying on worldwide replication streams from osm.org or per-country streams from @geofabrik . But what if I want my own extract area?

In 2014 I dug into osm2pgsql's "slim" tables, and wrote a script to trim the data outside a given polygon:

github.com/Zverik/regional

I use it for my "veloroad" tile server, but not sure if it works: the database grows pretty fast. Maybe it's just because people are mapping more than ten years ago.

8 May at 7:23 | Open on en.osm.town
Ilya Zverev replied to Ilya

This one is new. Couple days ago I released @everydoor 5.0, and for it I made a new thing.

GeoScribble takes inspiration from OpenStreetBugs, which was where we put our OSM notes before 2013. Just a simple server with no auth and a few fast endpoints.

wiki.openstreetmap.org/wiki/Ge

The idea is, you draw lines and leave short notes on the map. Just like with walking papers.

I see it already being used in multiple countries. The layer can be added to @josmeditor and Rapid, or browsed online.

This one is new. Couple days ago I released @everydoor 5.0, and for it I made a new thing.

GeoScribble takes inspiration from OpenStreetBugs, which was where we put our OSM notes before 2013. Just a simple server with no auth and a few fast endpoints.

wiki.openstreetmap.org/wiki/Ge

The idea is, you draw lines and leave short notes on the map. Just like with walking papers.

Map of an italian castle grounds. There are few red solid and dashed lines marking missing footways.
8 May at 7:49 | Open on en.osm.town
TheStroyer replied to Ilya

@zverik @everydoor @josmeditor wow, that looks like it can improve the editing workflow a lot

8 May at 12:11 | Open on mastodon.social
Ilya Zverev replied to TheStroyer

@TheStroyer @everydoor @josmeditor

Indeed! Just last Sunday I was walking with my friends in a forest, and surveyed a few streams and boardwalks this way. They would be lost if not for the drawing feature.

Editing ways properly (e.g. in @vespucci_editor ) needs too much involvement, compared to just a single finger movement in ED.

8 May at 14:27 | Open on en.osm.town
TheStroyer replied to Ilya
vespucci replied to Ilya

@zverik @TheStroyer @everydoor @josmeditor why draw? just record a GPX track ...

PS: in a forest you probably want to use a DTM anyway for tracing.

PPS: not to mention that you are massively overstating the effort to simply directly add things.

8 May at 15:23 | Open on en.osm.town
Ilya Zverev replied to Ilya

Since we're mentioning @josmeditor , here's one thing I made a long time ago:

josm.ru/

Yeah, that's a proper landing page (in Russian), with a few buttons, shortcuts to most important pages like a forum thread or a tag list, and a tutorial video.

I even browsed the changelog and added interesting changes to the feed, but stopped in 2020. I still use the page to download updates though.

"Start with JOSM! Download stable version 19067. Installers for Windows, Mac OS X, WebStart, jar file. Development version 19075." Also a few wiki buttons below, and a youtube video.
9 May at 13:01 | Open on en.osm.town
Ilya Zverev replied to Ilya

I'm a bit out of breath, but def not out of projects :) Gonna keep doing this until I get an answer from NLNet.

In 2010 Microsoft granted to OSM permission to trace Bing imagery. It was the only detailed imagery layer we had in Russia, so that enabled making a validator that found illegal imagery usage (google / yandex):

vakhtyor.osmz.ru/

Basically it watched where people uploaded buildings and checked for available Bing imagery.

In 2014 we got Mapbox layer, and I turned this off.

"OpenStreetMap Watchdog. This table lists mapped buildings in places where there's no detailed Bing imagery. <...> Many traced buildings might mean an import, drawing from memory, or — what we're trying to catch — tracing illegal imagery or proprietary maps."
11 May at 18:34 | Open on en.osm.town
Ilya Zverev replied to Ilya

There are different kinds of imports. Safest are those done by hand, but also the dirtiest, in my opinion: it's easy to miss or to get lazy and just press upload.

For towns and villages, the best source we had was GNS, an american database of toponyms. I converted it to OSM format and wrote a short manual on copying it onto the map:

gns.osmz.ru/

Interesting that while box=yes parameter doesn't work on osm.org nowadays, parameters minlat, maxlat etc still do.

GNS data converted to OSM format. Open in JOSM as a new layer, copy tags with Ctrl+C, then Ctrl+Shift+V onto a node placed on IRS imagery. Don't forget to fix the place type. <..> Made on 9.12.2015 with gns2osmr.pl script. Everything is in public domain.
11 May at 18:44 | Open on en.osm.town
SK53 replied to Ilya

@zverik Still loads of redundant GNS tags on place nodes in Ireland (probably imported by Dermot - mackerski). Until the townlands project got going they were still the major source of places in rural Ireland. Still a lot of dedup & other cleanup needed.

I also did a bit with them for Pakistan, but never could trust the data in a country where the population had increased so much : sk53-osm.blogspot.com/2011/03/ (probably should correct GNIS->GNS

12 May at 9:05 | Open on en.osm.town
Ilya Zverev replied to SK53

@SK53 yeah, what felt a good idea in 2011, not that much now :)

12 May at 9:55 | Open on en.osm.town
Ilya Zverev replied to Ilya

So, imports. When I was working at Maps.Me, many companies asked if we can put them on the map. So I made another thing.

OSM Conflator takes a data source and carefully merges it with OSM data, with a way to update it later.

wiki.openstreetmap.org/wiki/OS

While it can take a plain JSON with tags, it is most powerful with profiles: Python scripts that control the behaviour.

It can export a preview GeoJSON and has other ways of adhering to the guidelines. Alas, few people had enough skill to use it.

So, imports. When I was working at Maps.Me, many companies asked if we can put them on the map. So I made another thing.

OSM Conflator takes a data source and carefully merges it with OSM data, with a way to update it later.

wiki.openstreetmap.org/wiki/OS

While it can take a plain JSON with tags, it is most powerful with profiles: Python scripts that control the behaviour.

12 May at 18:34 | Open on en.osm.town
Ilya Zverev replied to Ilya

Having a GeoJSON to preview an import is good, but how to suggest amendments?

That's why I made CF Audit: an interactive website to go through changes one by one, cross-reference them, and fix anything wrong.

Those fixes are kept for consequent imports with OSM Conflator, so less work next time.

audit.osmz.ru/

I know RocketData iterated on it for their operations, and there's a new IFS fork by @sfkeller team: cfauditgui.geoh.infs.ch/

CF Audit screenshot. There are two maps to the right: OSM and Satellite, with markers for a location of a fuel station being imported. The text on the left says, "Update tags on a point. You can move the marker to a better location. Imagery may be misaligned.". Some tags are available for modification (e.g. brand and opening_hours), and there are three buttons: "Good", "Don't Change", and "Skip".
13 May at 6:31 | Open on en.osm.town
Ilya Zverev replied to Ilya

So I got the conflator, and I got the audit tool, and the entire pipeline as shown on the picture. Did I use it?

Oh did I even. 5k Walmart supermarkets, tens of thousands Shell fuel stations, ~10k addresses in Moscow, various restaurant and shop chains... It all worked perfectly.

Of course it was just a few imports. I envisioned an automated system for high volume of data, still controllable by OSMers, but nobody asked for that, and I didn't have time.

Still, mentioned it at SotM EU 2023.

A chart. External Dataset → Profile → OSM Conflator → Changet GeoJSON → cf_audit browse → cf_audit validate → Audit JSON → OSM Conflator (+Overpass API) → JOSM XML → JOSM → OSM.
14 May at 17:00 | Open on en.osm.town
Ilya Zverev replied to Ilya

OSM Cards is a postcrossing thing for OpenStreetMap members. You get a person's postal address, write them a few encouraging words on a postcard and add a 5-letter code. When received, they type those letters in a form, and the connection is set up.

I announced this website at SotM 2020 and got a few beautiful cards. I lived in Minsk back then and printed like 50 postcards with the city map to send out — and now idk what to do with them :)

Send me one if you like: osmcards.org/profile/bqh2on123

OSM Cards is a postcrossing thing for OpenStreetMap members. You get a person's postal address, write them a few encouraging words on a postcard and add a 5-letter code. When received, they type those letters in a form, and the connection is set up.

I announced this website at SotM 2020 and got a few beautiful cards. I lived in Minsk back then and printed like 50 postcards with the city map to send out — and now idk what to do with them :)

Amateur radio card with id "uz9osm".
15 May at 12:28 | Open on en.osm.town
Ilya Zverev replied to Ilya

You know OpenWhateverMap.

openwhatevermap.xyz/

Grant made the original version, then we lost the domain, and then I updated it with modern tiles and some interactivity: clicking on a tile reveals its name and URL.

Map of Lodz, Poland (where SotM EU 2024 is held). Every rectangular tile is in different style.
16 May at 19:11 | Open on en.osm.town
David replied to Ilya

Very cool, I'm just missing #openseamap ⛵

17 May at 7:57 | Open on hachyderm.io
Ilya Zverev replied to Ilya

For the @sotm in Firenze I went all-out with the remote participation part. One thing I did was a Telegram bot, where people from anywhere could introduce themselves.

The idea was to put a face to hundreds of names on online channels. Because once you have a face and a voice attaches to words, it's impossible not to feel a proper conversation going when you type.

Not many people contributed, but still, was fun. The bot was disabled after the conference ended of course.

Screenshot of a telegram bot, "SotM People". I sent "/random", and it replied with a video of Gregory Marler introducing himself. The video is 17 seconds long.
18 May at 9:08 | Open on en.osm.town
Ilya Zverev replied to Ilya

I feel like I've invented another job for myself, posting those tweets regularly. Let's do a few small things now.

Like, I love doing efficient focused tooling. GeoJSON Point Editor, while not entirely related to OSM, is exactly that. Faced with making a GeoJSON for ~100 point amenities, geojson.io was frustrating to use.

So I made this. Double click for a new point, type some words for properties (in a already focused field!), filter by keys, download the result.

zverik.github.io/point_ed/

I feel like I've invented another job for myself, posting those tweets regularly. Let's do a few small things now.

Like, I love doing efficient focused tooling. GeoJSON Point Editor, while not entirely related to OSM, is exactly that. Faced with making a GeoJSON for ~100 point amenities, geojson.io was frustrating to use.

Screenshot of GeoJSON Point Editor. The left panel has those words: "Double-click on the map or click here (button) to add a point. Download the result (link) or download filtered (link). Filter: (field)." To the right is a map with dozens of small markers. Near one is a label: "Kotelsky".
21 May at 15:28 | Open on en.osm.town
Ilya Zverev replied to Ilya

Another one I made for my last job, and it's for annotating GeoJSON geometries: "Geometry Review".

You load a set of geometries to compare into it, and it shows a bunch of buttons for each set: "OK", "Bad" etc, your choice.

No extra clicking or studying required, you glance at the map, press a 1/2/3 key, see the next item. And export the result in the end.

Used this tool to annotate hundreds of routes and review predictions.

github.com/Zverik/geometry_rev

Geo is easy until you see it on the map.

Another one I made for my last job, and it's for annotating GeoJSON geometries: "Geometry Review".

You load a set of geometries to compare into it, and it shows a bunch of buttons for each set: "OK", "Bad" etc, your choice.

No extra clicking or studying required, you glance at the map, press a 1/2/3 key, see the next item. And export the result in the end.

Screenshot of Geometry Review. There is a big map with blue and black lines. On the top there are buttons: "Previous", "Next", "Good 9", "Broken 4", "Bad 1". And title: "Order (long string): raw 1398 m, snapped 974 m (70%).". On the right sidebar is a list of route ids grouped into "Bad", "Broken", and "Good" sections, and two buttons: "Copy to clipboard" and "Download".
Another screenshot. There is a map with three markers: blue "2" and black "1" and "2". For black "1", a popup is shown with "car_id", "distance_m", and "duration_sec". On the top there are multiple buttons: "previous", "next", "good", and "bad".
21 May at 15:40 | Open on en.osm.town
Ilya Zverev replied to Ilya

For a very long time the leading repository on my github was "leaflet-grayscale" with a hundred stars.

I love @leaflet and made too many plugins for it. This one was quite simple — but I guess useful enough.

It just makes the map grayscale.

There are too few layers fit for a visualization background. Sometimes removing colors (and labels if possible) is good enough.

github.com/Zverik/leaflet-gray

Grayscale map from around Chernivtsi, Ukraine
21 May at 15:49 | Open on en.osm.town
watmildon replied to Ilya

@zverik @leaflet Yes! TIGERMap uses greyscale carto. Thankfully maplibre has a color saturation setting!

21 May at 18:32 | Open on en.osm.town
Robin Wilson replied to Ilya

@zverik @leaflet leaflet-grayscale is great and I use it a lot, thanks for making it!

21 May at 22:36 | Open on mastodon.me.uk
Ilya Zverev replied to Ilya

Back to OSM, here's one you'd ban if you could.

Sisyphus make a person's mapping work as pointless as rolling a stone up a mountain. It plain reverts every changeset a person uploads to OpenStreetMap.

We used it in Maps.Me for reverting edits of our QA team.

Not ideal, which Frederik noted, but it works, the map is clean, no hassle with moving data to the sandbox.

Uses simple_revert as a library, of course. That library is underrated.

github.com/Zverik/sisyphus

23 May at 16:38 | Open on en.osm.town
Ilya Zverev replied to Ilya

People usually need three things from maps: tiles, geocoding, and routing. Three things too mundane to work on.

For Maps.Me Changes Monitoring, I filtered edits by countries. For that I wrote a simple reverse geocoder: QueryAt.

It returns a country, a region, and a populated place for a given location. 3 GB database for the entire world, no configuration or query parameters.

Won't recommend using it now though: 8 years later, you could do better.

github.com/zverik/queryat

23 May at 16:46 | Open on en.osm.town
Ilya Zverev replied to Ilya

The recent discussion around "class:bicycle" tag reminded me of my "surface:grade" proposal — basically the same thing, but done 13 years earlier, with 25k usages as of now and no clear documentation.

wiki.openstreetmap.org/wiki/Pr

Basically it's a measure of surface quality relative to roads with that surface quality. Not all roads ever, like smoothness=*.

You see a perfect gravel road? surface:quality=3. You're on a technically asphalt road and moving 1 km/h wishing they left it unpaved? Zero.

The recent discussion around "class:bicycle" tag reminded me of my "surface:grade" proposal — basically the same thing, but done 13 years earlier, with 25k usages as of now and no clear documentation.

wiki.openstreetmap.org/wiki/Pr

Basically it's a measure of surface quality relative to roads with that surface quality. Not all roads ever, like smoothness=*.

26 May at 17:16 | Open on en.osm.town
Ilya Zverev replied to Ilya

Having tag proposals on this list feels like cheating — despite those take weeks to materialize, mostly because of research and writing and discussing in our Russian-speaking community.

For example, my turns lanes proposal took two weeks researching eight other proposals, charting examples, writing the proposal and FAQ.

wiki.openstreetmap.org/wiki/Pr

It failed because it was hard to understand, unlike the semantically weird but simple turn:lanes schema.

Mappers need simple, not consistent.

Having tag proposals on this list feels like cheating — despite those take weeks to materialize, mostly because of research and writing and discussing in our Russian-speaking community.

For example, my turns lanes proposal took two weeks researching eight other proposals, charting examples, writing the proposal and FAQ.

26 May at 18:06 | Open on en.osm.town
Ilya Zverev replied to Ilya

When I set out to write a reverse geocoder, I had a very narrow scope in mind: basically the east coast of the US. But the longer you study something, the more edge cases you find. Got a few talks out of it:

youtube.com/watch?v=sjE6xy0BBg

Juno Reverse Geocoder is covered in tests no other geocoder would pass, and is built on SQL. I bragged that it was faster than Nominatim, but recently I've tested it and it's like 8 times slower. Still, worked fine for our company.

github.com/gojuno/jrg

When I set out to write a reverse geocoder, I had a very narrow scope in mind: basically the east coast of the US. But the longer you study something, the more edge cases you find. Got a few talks out of it:

youtube.com/watch?v=sjE6xy0BBg

Juno Reverse Geocoder is covered in tests no other geocoder would pass, and is built on SQL. I bragged that it was faster than Nominatim, but recently I've tested it and it's like 8 times slower. Still, worked fine for our company.

28 May at 12:53 | Open on en.osm.town
SK53 replied to Ilya

@zverik sounds a bit like user GeneralDreedle, although that account automatically everted edits by fantasy mappers

23 May at 20:08 | Open on en.osm.town
Rihards Olups replied to Ilya
Go Up