[mkgmap-dev] mkgmap-r3906 (optimize-index)

Fri Apr 21 14:49:47 BST 2017

Hi Arndt,

seems that -Xmx5300 was just a little too less, on my machine it worked fine  with -Xmx6000m and also crashed with -Xmx5300m.
I've downloaded your map and used MapReverseConverter
from http://www.javawa.nl/cne2009_multilang.html
to convert the .gmap format back to *.img files so that I could use mkgmap to re-calculate the index.
java -Xmx6000m -jar d:\mkgmap\dist\mkgmap_r3910.jar -c mkgmap_options_mdrmdx.args  --block-size=65536 ..\Speiche_Europa_170419\88*.img
(r3910 is exactly like r3908 for this purpose)

I am working on some improvements and corrections for the prefix/suffix support now, maybe this will require less memory, maybe a bit more.

Gerd
________________________________________
Von: Arndt Röhrig <arndt at speichenkarte.de>
Gesendet: Freitag, 21. April 2017 10:40:48
An: Development list for mkgmap; Gerd Petermann
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi Gerd,
the map tiles from Speiche_Europa 190419 was build with mkgmap-r3906. The index with mkgmap-r3908. (I start it before 3908 was available)
Arndt

> Gerd Petermann <GPetermann_muenchen at hotmail.com> hat am 21. April 2017 um 10:32 geschrieben:
>
>
> Hi Arndt,
>
> strange, the index itself is ~ 1GB, that's very close to the size for the default style in r3908.
> Did you use r3908 or an older version?
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt at speichenkarte.de>
> Gesendet: Freitag, 21. April 2017 10:28:25
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Hi Gerd,
>
> yes, i use 170419.
>
> The OSM-Data is downloaded by "Geofabrik" (europe). To be added 5GB SRTM data. osm and srtm data merged with the splitter.
>
> Arndt
>
>
> Gerd Petermann <GPetermann_muenchen at hotmail.com> hat am 21. April 2017 um 07:34 geschrieben:
>
> Hi Arndt,
>
> thanks for the flowers ;-)
> I am still surprised that your style requires so much more memory compared to the default style.
> Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt at speichenkarte.de>
> Gesendet: Mittwoch, 19. April 2017 17:04:43
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Thank you for that great work!
>
> Now my old nostalgia PC is able to create the index for Speiche_Europa!
>
> My PC has only 6GB RAM.
>
> -Xmx5300M -> fails with "java heap space error"
>
> -Xmx10500M -> fails with "overflowed directory with max block".
>
> * Xmx10500M and Option --block-size=65536 -> That works!
>
> Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.
>
> The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.
>
> Special thanks to Gerd, who show me many things to improve my map building procedere!
>
> Best regards
>
> Arndt
>
> speichenkarte.de
>
> .
>
> Gerd Petermann <GPetermann_muenchen at hotmail.com> hat am 18. April 2017 um 16:41 geschrieben:
>
> Hi all,
>
> as a follow up:
> In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
> r3906 failed with OutOfMemoryError even with -Xmx5000 .
> So I think memory is no longer a problem unless you want to create an index for planet ;-)
> No other changes, means r3908 can be used to create index for *.img files created with r3906.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen at hotmail.com>
> Gesendet: Montag, 17. April 2017 20:05:15
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Hi all,
>
> with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).
>
> I think address / road search works very well now, at least with west european languages.
> I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.
>
> IMPORTANT:
> If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.
>
> The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
> The created index is a bit smaller although it now also contains roads with an empty string as first label.
>
> If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
> keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
>
> If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
> I think it is a big step forward, but there may still be special cases with other languages.
>
> I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index
>
> Gerd
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev