[mkgmap-dev] Memory Consumption on Index Creation

Tue Sep 6 00:17:04 BST 2011

On 30.08.2011 17:23, Steve Ratcliffe wrote:
> On 29/08/11 08:31, Felix Hartmann wrote:
>> 1. I noticed that (at least for my style-file) 7500MB of RAM available
>> does not suffice to compile a map of Europe including index! However if
>> I recreate the index in another go, then it is no problem.
>>
>> How comes? The index creation seems to be completly independent of the
>> maptiles creation. Is mkgmap leaving stuff inside the memory, that is in
>> reality not needed anymore?
>
> Yes, the index is completely independent of the map tile creation 
> phase and all the memory used for a single map tile is released after 
> it is finished.
could it be that the index starts to be worked one, when the first of 
the 4 worker threads (to compile the maps) is finished, with 3 still 
working and hence overshooting the memory limit?? I really cannot see 
why it would have failed me 4 times in a row on a single call, but 
single index creation allways worked...

I have the problem that on my local machine, it ain't powerful enough to 
compile all of Europe, so I cannot test really what happens (like 
decreasing the Xmx value to values that crash the index creation alone, 
are so low, that I cannot compile the maps anyhow).
On the server I use to render the map, however I only have a limited 
account, and can not see actual memory consumption.
>
> Profiling confirmed in my tests that all the memory was indeed released.
>
> Also if the memory was not released you would see gradually increasing 
> memory usage the more tiles that were compiled at once.  Do you see an 
> evidence of that happening?
>
> So I can't reproduce this at the moment.
>
> I do have some ideas for some reduction in the memory use by the index 
> generation.
>
Well that would be great. It is currently using much more memory than 
any other tool in the chain and growing faster than any other memory 
consumption (for running mkgmap with 4 threads, about 2.5GB is enough, 
for splitting Europe 3GB is enough, however for index creation below 
4.5GB it is a no go).
Best would even be if on single index creation, one could specify an mdr 
file for existing maps, and just add stuff for maps that you want to add 
to the compilation (use case, user has a map of Germany in several .img 
with mdr/mdx and wants to add contourlines into the mapset, it would be 
great if the index of the Germany map could be given, and the 
contourline maps just added to the mdr without needing to regenerate the 
full mdr -- don't know the format well enough, but would hope that this 
is possible..).

Sorry for taking so long to respond, I wanted to test it out locally, 
and didn't have a working build environment anymore, hence I took some time.

Regarding the -XX:+UseCompressedOops option I'll add it to my 
commandline....

Felix