[mkgmap-dev] Memory limits for mkgmap and splitter

Tue Aug 4 11:38:58 BST 2009

Chris Miller wrote:
> Note that I've started working on overhauling the splitter to try to reduce 
> the memory requirements as much as possible without compromising too much 
> on performance. I'm also hoping to overcome the limitations on the number 
> of tiles (255) and number of areas for each relation (currently 4). My plan 
> is to rework some of the algorithms being used, tune the memory requirements 
> wherever possible (though Steve's already done a good job here), and switch 
> algorithms and page to disk (using eg custom B-Trees, R-Trees, not a database) 
> when heap space starts to run out.
> 
What currently happens when not enough memory is available is ofcourse 
that the heap is getting swapped in and out to disk by the os' memory 
manager. This is so slow that it's not workable. Do you think that doing 
this intelligently in Splitter will be much faster? I.e. would it 
actually be of any real use to switch to temporary disk storage for 
multiple gigabytes of data?

> So far I've cut the memory requirements for the areas.list file generation 
> in half, and hopefully this week I'll check in a further change that slightly 
> speeds up the XML parsing and reduces memory requirements a fraction more. 
> Further gains beyond that will unfortunately take me a while since it is 
> a reasonable amount of work and I don't have a lot of free time to work on 
> this. Of course I'll be sure to post to this list when I make any significant 
> improvements beyond the above.
> 
I assume the actual splitting uses the most memory of the two stages?

> As an aside: If you split a file that results in more than 255 tiles, my 
> understanding (from looking at the splitter code) is that it will fail 'silently', 
> ie the output will be corrupt (nodes/ways/relations end up in the wrong area 
> files) without it being obvious that this has occurred. Be careful!
> 
Ah, this might explain the 'POI only' tiles I have without good reason 
in the NE USA and Canada. Is there a possibility for a quick fix for 
this behavior, as this would be most welcome...? :-)