[mkgmap-dev] MDR building out-of-memory

Tue May 11 08:04:49 BST 2021

Hi Ticker,

thanks, good findings!

the patch doesn't use trimToSize() on cities. Did you change your mind?
The part about the scope is interesting. At first glance I thought this should make no difference but it probably helps GC to detect that this array cannot be garbage collected.

The SortKeys stuff is really eating memory and it would good to find a better solution.  One approach is to use the cache as in attached patch but that only helps when memory is really the problem, it slows down processing for other situations.
Maybe better would be to first collect all combinations of region+country, sort them, and use the position in that list to sort other objects?

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
Gesendet: Montag, 10. Mai 2021 23:05
An: mkgmap development
Betreff: [mkgmap-dev] MDR building out-of-memory

Hi Gerd

Since downloading loading britain-and-ireland-latest.osm.pbf I had been
unable to build a gmapsupp because of running out of heap (my hardware
is 32 bit, -Xmx1540M is largest value allowed)

My problem is mainly because I have 1731146 cities (along with 1046096
streets)

Looking at Mdr5 processing, I've changed it in 3 ways to improve memory
usage and garbage collection.

1/ use trimToSize() after all the cities are loaded from the individual
tile .img. I presume that the growth factor gradually increases as it
runs out of allocated array space. I had to change the declaration from
List<Mdr5Record> to ArrayList<Mdr5Record> to allow this, but I can't
see any problem in this.

2/ Move the main part of preWriteImpl into its own method so the first
sortKeys ArrayList and Sort can be freed before calcMdr20/1/2() each
create another massive SortKeys and Sort.

3/ Move the scope of mdr20s to a class variable. This is referenced by
all the Mdr5Records and the scope of where it was declared before
seemed to to cause the garbage collector major problems - it churned
for 5 mins using all the processors before running out of memory. After
moving it, the whole of mdr is built in a couple of mins with cpu usage
mostly < 125%.

Ticker