[mkgmap-dev] [Patch v1] reduce peak memory usage for global index

Thu Mar 30 09:33:23 BST 2017

Hi all,

attached is a patch which seems to help without changing the output of mkgmap.
Compared to r3873 I see ~ 33% less memory usage when used with --x-split-name-index and ~50% less without it.
Runtimes were similar. Actual numbers depend on the style and the used options.

A binary based on 3873 is here:
http://files.mkgmap.org.uk/download/341/mkgmap.jar

It consists of different changes, each of them maybe used independently:
1) always release memory for mdr19, not only if it was filled for the device.
2) check if sort key has zero length, if yes, don't allocate new buffer
3) don't use MultiSortKey for mdr7 if --x-split-name-index is not used
4) count occurences of the generated key strings, cache sort keys for those keys which occur more than 3 times.
5) create smaller copy of byte array if number of allocated bytes was too high

@Steve: Please review, I think 1-4 are okay, 5) may cause problems if my understanding of Sort.fillkey() is wrong.
I was also surprised to see that we require such long sort keys, even with --latin1.
I see many 0 bytes in the created keys, maybe that can be optimized further?

Gerd

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mdr-sort-v1.patch
Type: application/octet-stream
Size: 7645 bytes
Desc: mdr-sort-v1.patch
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20170330/8faaba86/attachment-0001.obj>