[mkgmap-dev] MDR building out-of-memory

Tue May 11 20:15:16 BST 2021

Hi Ticker,

I think the cache is not meant to improve run time, it is used to deduplicate and thus reduce memory. Maybe it would be better to use a smaller chunk size and no cache.
No idea why I didn't use
merged = new ArrayList<>(len);

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
Gesendet: Dienstag, 11. Mai 2021 19:19
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MDR building out-of-memory

Hi Gerd

I've looked at this, and if Mdr5 space becomes a problem again, I'll
consider converting to it.

A couple of comments:

My map had 2909735 poi so the sort chunk size was ~727433. The cache
sizes after each chunk were 563095, 603595, 597239 & 605718. Is the
cache worth-while for this low hit-rate? Just running the Gmapsupp
combiner on existing tiles (without --route, so no streets), I got a
run time of 1 min 44 secs with the cache and 1 min 30 without! However
most of this time is copying the tiles into gmapsupp.img so not an
accurate statistic.

You could pre-allocate List<> "merged" with the correct size.

Ticker

On Tue, 2021-05-11 at 14:40 +0000, Gerd Petermann wrote:
> Hi Ticker,
>
> I've committed the patch as is. I've not seen big changes in
> performance, but I've used a different (already existing) set of
> files which was created with my own style. For me,
> Mdr11.preWriteImpl() is the most problematic part reg. OOM errors.
>
> Maybe look at the code which uses LargeListSorter.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> Gesendet: Dienstag, 11. Mai 2021 13:27
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] MDR building out-of-memory
>
> Hi Gerd
>
> Here is updated version of patch.
>
> Changes from last:
>
> Uses your cache code for region and country (in 2 places). For
> British
> Isles, there are 190 regions and 7 countries, so I don't think the
> extra memory will be a problem and there should be some performance
> benefit.
>
> Delays allocating cities until it can use sortKeys.size() for initial
> allocation. For above map this is 0.07% too big, so I don't think
> trimToSize() is worthwhile.
>
> Shares the Sort object between the 4 methods.
>
> Ticker
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev