[mkgmap-dev] mkgmap-r3906 (optimize-index)

Fri Apr 21 06:34:37 BST 2017

Hi Arndt,

thanks for the flowers ;-)
I am still surprised that your style requires so much more memory compared to the default style.
Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?

Gerd
________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt at speichenkarte.de>
Gesendet: Mittwoch, 19. April 2017 17:04:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Thank you for that great work!

Now my old nostalgia PC is able to create the index for Speiche_Europa!

My PC has only 6GB RAM.

-Xmx5300M -> fails with "java heap space error"

-Xmx10500M -> fails with "overflowed directory with max block".

- Xmx10500M and Option --block-size=65536 -> That works!

Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.

The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.

Special thanks to Gerd, who show me many things to improve my map building procedere!

Best regards

Arndt

speichenkarte.de

.

Gerd Petermann <GPetermann_muenchen at hotmail.com> hat am 18. April 2017 um 16:41 geschrieben:

Hi all,

as a follow up:
In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
r3906 failed with OutOfMemoryError even with -Xmx5000 .
So I think memory is no longer a problem unless you want to create an index for planet ;-)
No other changes, means r3908 can be used to create index for *.img files created with r3906.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen at hotmail.com>
Gesendet: Montag, 17. April 2017 20:05:15
An: mkgmap-dev at lists.mkgmap.org.uk
Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi all,

with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).

I think address / road search works very well now, at least with west european languages.
I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.

IMPORTANT:
If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.

The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
The created index is a bit smaller although it now also contains roads with an empty string as first label.

If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.

If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
I think it is a big step forward, but there may still be special cases with other languages.

I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index

Gerd

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev