[mkgmap-dev] MDR building out-of-memory

Tue May 11 09:10:49 BST 2021

Hi Ticker,

sorry, here's the patch.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
Gesendet: Dienstag, 11. Mai 2021 10:07
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MDR building out-of-memory

Hi Gerd

allCities.trimToSize() is at the top of preWriteImpl().
cities is empty at this point. I should trim it at the end of
genCitiesAndMdr20s or delay allocation so it can be made the same size
as allCities.

Is there any reason not to share "sort" between genCitiesAndMdr20s,
calcMdr20/1/2SortPos() - I'll try it.

Your patch wasn't attached.

I had given up trying to understand the different combinations of sort
keys for mrd20/1/2 and, until it runs out of memory again, I'd rather
not touch it.

I'll experiment a bit more and send another patch in a while.

Just seen your change to show the stacktrace when OOM - good.

Ticker

On Tue, 2021-05-11 at 07:04 +0000, Gerd Petermann wrote:
> Hi Ticker,
>
> thanks, good findings!
>
> the patch doesn't use trimToSize() on cities. Did you change your
> mind?
> The part about the scope is interesting. At first glance I thought
> this should make no difference but it probably helps GC to detect
> that this array cannot be garbage collected.
>
> The SortKeys stuff is really eating memory and it would good to find
> a better solution.  One approach is to use the cache as in attached
> patch but that only helps when memory is really the problem, it slows
> down processing for other situations.
> Maybe better would be to first collect all combinations of
> region+country, sort them, and use the position in that list to sort
> other objects?
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> Gesendet: Montag, 10. Mai 2021 23:05
> An: mkgmap development
> Betreff: [mkgmap-dev] MDR building out-of-memory
>
> Hi Gerd
>
> Since downloading loading britain-and-ireland-latest.osm.pbf I had
> been
> unable to build a gmapsupp because of running out of heap (my
> hardware
> is 32 bit, -Xmx1540M is largest value allowed)
>
> My problem is mainly because I have 1731146 cities (along with
> 1046096
> streets)
>
> Looking at Mdr5 processing, I've changed it in 3 ways to improve
> memory
> usage and garbage collection.
>
> 1/ use trimToSize() after all the cities are loaded from the
> individual
> tile .img. I presume that the growth factor gradually increases as it
> runs out of allocated array space. I had to change the declaration
> from
> List<Mdr5Record> to ArrayList<Mdr5Record> to allow this, but I can't
> see any problem in this.
>
> 2/ Move the main part of preWriteImpl into its own method so the
> first
> sortKeys ArrayList and Sort can be freed before calcMdr20/1/2() each
> create another massive SortKeys and Sort.
>
> 3/ Move the scope of mdr20s to a class variable. This is referenced
> by
> all the Mdr5Records and the scope of where it was declared before
> seemed to to cause the garbage collector major problems - it churned
> for 5 mins using all the processors before running out of memory.
> After
> moving it, the whole of mdr is built in a couple of mins with cpu
> usage
> mostly < 125%.
>
> Ticker
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mdr5Mem-with-cache.patch
Type: application/octet-stream
Size: 3421 bytes
Desc: mdr5Mem-with-cache.patch
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210511/937c68f1/attachment.obj>