There’s more than one way to draw a map - Basil Dubyk, Alexander Gaevsky

Speaker 1: Hello everyone. Uh our talk is uh there's uh more than one way to draw a map. We're uh Metaclass Agency. Uh I'm Aleksandr Gayevsky, this is my colleague Basil Dubik. Hi We're developing custom solutions for e-commerce, fintech , and other domains using Python 3, Django, and also Flask and FastAPI. In our spar uh spar time we um uh contribute to the open source um projects like uh Django Oscore and uh Django. This year we're giving a talk from my living room, not for the conference uh call, and this is kind of uh a weird experience Third party uh special services are key parts of the

Speaker 1: worst platforms in different domains. And traveling at delivery uh peer-to-peer marketplaces, transportation. And Google recommended itself as a liable and solid provider, making itself practically irreplaceable part of those services. As time goes by and special integration becomes more and more tight. We were busy shipping uh fishes uh for uh the client project uh when in the early fall of 2018 our client approached us since the bill um for the Google Mass API. um dramatically grew. Until July 16 , 2018

Speaker 1: , 1000 additional requests after initial quote of 25 A thousand of requests cost uh 50 cents and it grew to $7 afterwards. Uh not that we weren't satisfied by the Google Maps, uh, but then But the estimated costs exceed the client expectations and our budget, so we started to seek for the solutions. and prove this prove the existence of full-fledged uh flexible and powerful uh alternative of uh um Google Maps which Django spatial project could use And also the current price spike not only made us to start looking for a placement but also to

Speaker 1: re-evaluate the whole approach of external API usage and free API limits more carefully. So but uh before we go and uh review alternatives, uh let's see what um special uh um and pattern service um consist of So first of all it's tile server which renders uh map tiles and uh um serves them Also it's JS or other front-end library which allows to um read the whole map uh out of the tiles and uh uh display in the browser. And um uh geocoding API uh which allows you uh

Speaker 1: to uh uh get the place uh get the location uh coordinates and other um um geodata and um also reverse geocordinate which allows uh vice versa uh get their um uh location name uh from the um uh coordinates Basically, nowadays there are open source components which allows you to build all of the parts of the special service uh for the tiles server you could use Tigola uh open map tiles or you could use uh OS to PGL Component by OpenStreetMaps,

Speaker 1: modile for the Apache or custom solution by for the Flask. to render the uh to to serve the tiles and mapnik uh to uh render the tiles um of the server There are also plenty JS map libraries, decent ones, like Leaflet JS, Open Layers. uh and pubbox gel I would mention those ones. Also there are geocoding um API servers like Pilias and Nominating Let's um see what alternatives we have.

Speaker 1: Um uh OpenStruet maps, mapbox tom, here maps, location IQ, graphhopper, uh , So first of all, the criteria for the service was reasonable pricing and a sufficient free ETF. And map widget, customizable markers and informational model which could display some um additional data when you click on the marker And we had plenty of markers on the map already, so we needed to group them into the cluster which we already had in the Google Maps.

Speaker 1: Uh as for the geocoding we needed both forward and uh reverse geocoding. Um Last but not least we uh use uh location autocomplete and it would be nice to have uh a similar one uh on this new service as well and uh a native python sdk would be cool as well uh also Also, our client platform focuses on the clients in Germany and Austria. So we wanted that. uh the special data for those locations uh uh is um

Speaker 1: maximum uh accurate and also uh we needed uh all uh labels and everything to be localized in German. And uh we already had uh 10,000 of markers and maps, so rendering uh clusters uh also needed um a good performance. Otherwise uh customers would be um Frustrated and open map follows uh too slow. So OpenStreetMaps is a good uh is a decent service uh which uses uh open data And it's also open source and it has pre free pricing. Already it has the Python SDK, GSmap CSDK compatible with it, and the RESTful

Speaker 1: API and on the on uh dataset uh which is uh uh crowdsourced by uh by the users and it has localization The only con is that the data is open and free, but server is not uh services usage is not free. And then their services run entirely on donated um resources and they have limited capacities. So basically uh uh you as soon as you uh reach the quote, which is n they don't have a hard quote, but as soon as you um utilize their services um uh too hard

Speaker 1: um uh they would uh block requests. And they suggest you to uh deploy your own uh tiles server using uh for instance uh OpenMap tiles. So um uh next alternative is TomTom. TomTom is Dutch GPS navigator manufacturer, uh which is widely used in the UA and uh European Union and uh North America and um uh navigation the mass uh software producer. It has a um uh three tier uh for um uh two thousand five hundred requests per day uh which is seventy-five thousand requests per months and limitation of five requests per second for non-tile-based APIs

Speaker 1: and 1000 for tile bell tile-based APIs And flexible um pricing uh you can pay as you grow more quests um um more costs Although there are some qubits here, one requests equals one transactions, but not always accept um traffic API tiles and so on and so on. They have a native SDK for Android, iOS, and web and uh it supports uh Marcus and Marcos uh clusterization um um

Speaker 1: Uh RESTful API has forward and reverse geocodian search and the dedicated endpoint for uh after complete and it has uh localization And uses own uh dataset um Uh Mapbox uh allows to use uh map uh fifty uh thousand uh times for free and um it allows to do um uh one hundred thousand uh requests to the temporary geocoding api and uh five hundred thousand Up to 500,000 requests to permanent GeoCode and API.

Speaker 1: It has Python SDK, but it's not currently actively developed and for now It has mobile SDK for IS and R8. It has augmented reality SDK And it has a RESTful API with both forward, reverse geocoding and autocomplete. And it has localization as well. Um HereMap the HereMaps uh it's a service uh uh run by uh Here Technologies, uh which is a Netherland-based international technology company previously It was Nokia Maps or OVMaps. It's now owned by a group of

Speaker 1: companies which are mainly automotive. Such as Audi , BMW, and Aimler. Her maps were used by Facebook and by Microsoft Maps. So the pricing is um two hundred thousand um a transaction per month. Um it has um five thousand uh mobile sdk mostly active users uh and um it it doesn't have a mobile uh native uh Native Python SDK, it has uh uh JavaScript SDK for building uh maps

Speaker 1: um and extensive restful API for allocation discovery and geocoding back uh uh both uh forward and reverse I would suggest uh location browsing uh lookup by d uh reverse uh geocoding and uh localization Graf Hopper. Graf Hopper is a German service for road planning and optimization. uh and um his it's my foc focus on this functionality. Um it has um free tier of uh 500 crits per day, but only for non-commercial use.

Speaker 1: It has limited radio account per minute and cannot use flexible mode Doesn't have uh native uh Python SDK, doesn't have uh maps. sdk, and uh doesn't have um dedicated um up to complete endpoint The dataset is provided by multiple sources and depends on the source you will be built a different different amount of credits. Uh location IQ. Uh location IQ and has a um uh free tier of uh ten thousand requests per day

Speaker 1: or uh two requests per second or sixty requests per minute, uh which leads to uh three uh hundred thousand requests per month uh and it has limited commercial use so you need to add the link uh uh to the location qcom site in the application or on your on your site So um it has um and it has bunch of um native uh software development kits uh for uh python, C sharp, Dart, Ruby, Objective-C, Perl. Kotlin , PHP, Swift, Scala, Rust, Java, Haskell, R Go, C, and Clujo.

Speaker 1: It doesn't have own um uh maps uh JavaScript SDK and mobile SK as well. Although it has um Uh forward and revest geocoding and location search and dedicate it endpour after complete. It has localization, but since uh it depends on the um external um uh data providers um uh like open street map street map um If the uh if the uh data provider uh translated uh uh particular uh G data into the local language. It will be translated on the local location IQ

Speaker 1: as well. But they don't guarantee that all results will be translated A D A D is a Dutch um service. It has a free free tri of ten thousand requests per month. Uh it doesn't have uh any SDKs, but it has um uh powerful and dedicated uh map tile server. and RESTful API for forward and report your coding but uh does it have um dedicated point for uh after complete Uh joke. Uh joke maps um uh is a French uh service

Speaker 1: uh uh which does not have a um uh which has uh free pricing only for uh commercial uh only for non-commercial use uh it's uh 50 uh thousand uh map use um uh ten thousand um uh static maps you and uh ten thousand place uh search requests Doesn't have any SDKs natively. Respole API has a forward, reverse geocoding allocation search and dedicated endpoint for after compute. and localization the dataset um uh they are using is from uh open suite maps open addresses and your

Speaker 1: names So there is a precaution that uh the credits and transactions and requests it's not all the same when you choose the service you need to carefully I read the agreement and the terms of use because some of the requests to the APIs would cost a few credits And consequently they will be uh built uh not as one of course but as multiple Finally , we um selected HereMaps as the replacement of the Google Maps Uh first of all because it has a uh a built-in uh marker marker clustering

Speaker 1: uh which would um allow us uh quickly to switch uh to the um here maps uh and to save our clusters It has uh generous free tier um and it has uh dedicated autocomplete API endpoint, which we use in uh two places in our platform. uh dedicated AP endpoints allow you to um fetch uh results um Only to fish results and only receive the full location names and location IDs, just the minimum amount of information and to

Speaker 1: um uh later on um uh retrieve the uh details of this particular uh location. So also based on the Counterpoint research company, Here Technologies was named as the world's number one location platform uh and it has uh leading position uh uh during the last uh three years. Uh here maps um Has uh eighty thousand sources of data and provides maps in uh one hundred ninety countries in the world. So um

Speaker 1: uh finally uh auto's combination of the um criteria uh made us uh uh choose the here mass. So um Um yeah and uh yeah we could uh now uh proceed with the um immigration and my colleague uh would uh tell you about it

Speaker 2: As you may know, with Google adding address autocompletion is quite easy since you just need to include their JavaScript library into your HTML and provide some callback. for uh when the selected address is changed. This is general everything that you need to do to make it work. I mean that you are fine with the default styling But this is not the case with here maps. They do not provide ready-to-use JavaScript library for these. And so since we already use view framework in many places in the project, we decided to implement after completion with view, means like making requests, showing drop-down, etc. For this, as you can see on the slide, we use simple input to get

Speaker 2: the place required by the user for the search and initial value for it will be rendered by the Django and not by view. In the request, uh in general in the query string, in the request for the here maps API, it's possible to set many different settings that required by a project and for example like result results language and the countries that will be used for the search and the max result and similar thing And also, this I think is very important, you can indicate what HTML tag will be used to highlight the uh to highlight the results with uh places that match uh

Speaker 2: text typed by the user Here you can see the example of suggestions from the maps API. From here, in general, we use just label to show in the drop-down and location ID that we use to. Get coordinates of the place. When when the user adds their equipment to the platform, they can use address autocompletion to select like concrete address but when we use autocompletion for searching equipment in desirable areas we limit the results To the ones where only the CT matches with the text entered by the user, and you can see this filtering on the slide.

Speaker 2: Now we are talking about how we Draw and use markers. As you know, Google Maps as well as Here Maps allow you to use custom images for the markers So the the simplest uh what we can do for the start, we just use the same image for the icon that we used previously with Google Maps, we use the same for the uh here maps. But we found that performance is not quite good because we use like fancy PNG with transparency and other. So we decided to use SVG instead of. P

Speaker 2: G and additionally uh and additionally here now we start to uh set the marker size and other other things. The time we That time we decided to use SVG instead of PNG to improve the performance of marker rendering and with SVG icons for markers, we started to indicate needed marker size and anchor point Now the size of markers depends on the user browser's pixel ratio, and this is very useful to show markers in correct size on high DPI and routine displays. To show additional info

Speaker 2: when a user is clicking on the marker, it's possible to use info bubbles. For it, you just need to provide the the coordinates and the HTML content. And here's an example of an info bubble is very simple, but It showed that you can display all data related to the clicked point on the map. In our case, we shows a photo of the equipment, user's profile picture, and Score, price, etc. It's possible to style content in the info bubbles by updating CC's classes added by Info bubble automatically and by styling custom CCS classes inside the info bubble

Speaker 2: And additionally, to you you can customize your map during initialization by setting needing tiles. And we think that this is very customizable, so probably can be used in any range of requirements. However, after the first part was finished, a trickier one popped up, marker's clusterization. This metal allows to group markers of one location, hide single markers and show a group of markers instead for the given scale. When the users close down, group breaks down into the other groups or to the separate markers. Google Maps provide a ready-to-use JavaScript library in the official repo

Speaker 2: and a couple of examples how you can use it and integrate. Our successor here maps luckily has built-in markers clusterization under the hood and three different algorithms. or strategies powering it. It's first grid, grid, and dynamic grid for different for different dataset sizes. And we can compare them The fast grid is an efficient way to cluster large sets of data points, and we use it for the pages where we show search results. not related to some like concrete place or when we show all the markers on the map. Grid strategy is slower but has greater precision.

Speaker 2: We use it for maps on the user's profile pages. And dynamic grid clustering uses the same algorithm of clustering as the grid, but clusters on the viewport basis. It was suited for uh for data sets that are subject to frequent updates and we don't use it in our project since we do not make Updates to the map since we put all the needed markers to the maps once. We made our own comparison for the different strategies and we can see that Fast grid strategy a lot for a lot of markers, clusters appear on the initialized map much faster than with grid

Speaker 2: strategy. Last but not least, we have multiple different equipment items tied to the same address. So that previously we plugged it. In an additional JavaScript library which renders markers within the same location as a literal web of markers. With Spiderfire we used markers in the open cluster as single markers on click we show an info window. Since here does not Provide the same functionality as Spiderfire. This time we used a different strategy. We just decided to display only one marker and show available gear in the informational

Speaker 2: pop-up. Which appears when the user clicks on a marker. Now in the info window for the cluster, the user will be able to review all the gears by using just arrows. We found this solution even better. Than what we have with Spider Fire, since uh previously users can accidentally skip some markers in the open Spider-Fire, if there are many of them, they are all the same in general, and now this is not possible And additionally, we can say that it's possible to use custom cluster icon. In general, cluster icons can be the same as the marker icons, but since cluster groups a couple of

Speaker 2: Markers, it's useful to show number of markers in the group. And we have achieved this by putting just text part into our SVG template.

Speaker 1: Now that we have finished with the uh front end migration, uh we can look at the uh back end On the back end we were using native Python SDK for Google Maps and now we obviously needed SDK replacement for here maps. Uh it doesn't have uh Python decay, but it's not um necessary after all. Uh we can quickly uh wrap around the RESTful API Since we only need three endpoints for geocoding, reverse geocoding and location lookup. Also, there is a nice looking third-party package called Herepy uh which is also suitable for uh here integration although we uh use only three endpoints

Speaker 1: um which doesn't seem enough for bringing uh bringing in another uh dependence on board As stated before, auto-suggest endpoint basically returns only location IDs and has minimum information about the location. Obviously uh due to the um performance reasons so we can later on through location details separately naturally uh there are popular locations big cities like Berlin uh Colonia uh Munich uh which customers often look for equipment around and in order to uh utilize um a pick what uh more wisely we could um uh cache um allocation data for uh those um uh

Speaker 1: locations uh and uh you can see on the slide um This cache implementation in the lookup method. Now we could review some advanced features like spatial search uh for uh the um uh Django project. Uh we have s have a similar one uh on our platform as well So um there were choice uh two choices to implement um uh full full text uh at the same time full text and spatial uh search uh in the Django project. You could use uh Postperscale as the database engine or uh uh

Speaker 1: Elasticsearch. So um For the PostgreSQL , for the post SGRSQLE, you would need PostGIS and obviously Joe Geojle And um uh on the slide you see that we outlined the simple model and implemented uh Yan. uh simple uh endpoint for the junk rest uh framework uh which uh is doing um queries in search um uh filter and uh also special uh search using uh divisin uh

Speaker 1: um filter The implementation for the Elastic requires also a Django has tagged, and on the slide you see that we outline the um index class for the Django Hastack and the implementation for the Django Rest framework it looks uh quite similar except the um queer set we use a separate method uh d within uh not not filter and the parameters Are um quite similar. Um uh all in all, um we don't stick to the

Speaker 1: your maps as the ideal solution uh since there are plenty of uh decent um uh products available and uh we can't avoid um uh mapbox who build special JavaScript library supercluster which shows quite uh decent performance with millions of points in the dataset Also, um the Leaflet has a marker cluster um uh module uh which also allows to build a uh uh cluster for um then uh markers. Thank you very much for uh listening.

Speaker 1: Uh you can ask your questions or uh uh contact us uh in the twitter or uh via the email.