[mkgmap-dev] Memory Consumption on Index Creation

Tue Sep 6 09:53:19 BST 2011

On 06/09/11 00:17, Felix Hartmann wrote:
> could it be that the index starts to be worked one, when the first of
> the 4 worker threads (to compile the maps) is finished, with 3 still
> working and hence overshooting the memory limit?? I really cannot see
> why it would have failed me 4 times in a row on a single call, but
> single index creation allways worked...

Maybe - if something else is failing while tiles are being generated.

If you apply the attached patch and run, it will print out how much
memory is available, what threads are running and version information
just before it starts the combining step.

On my machine it looks like this:

	About to start combine steps
	Thread[Signal Dispatcher,9,system] RUNNABLE
	Thread[Reference Handler,10,system] WAITING
	 at java.lang.Object.wait(Native Method)
	 at java.lang.Object.wait(Object.java:485)
	 at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	Thread[main,5,main] RUNNABLE
	 at java.lang.Thread.dumpThreads(Native Method)
	 at java.lang.Thread.getAllStackTraces(Thread.java:1530)
	 at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:423)
	 at 
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
	 at uk.me.parabola.mkgmap.main.Main.main(Main.java:132)
	Thread[Finalizer,8,system] WAITING
	 at java.lang.Object.wait(Native Method)
	 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	 at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
	Memory 2M 105M 881M
	Java Version 1.6.0_22 Sun Microsystems Inc., on Linux 
2.6.40.3-0.fc15.x86_64

>> I do have some ideas for some reduction in the memory use by the index
>> generation.
>>
> Well that would be great. It is currently using much more memory than

I looked into this a bit more and found that the simple ideas wouldn't
save much memory unfortunately.

> Best would even be if on single index creation, one could specify an mdr
> file for existing maps, and just add stuff for maps that you want to add
> to the compilation (use case, user has a map of Germany in several .img
> with mdr/mdx and wants to add contourlines into the mapset, it would be
> great if the index of the Germany map could be given, and the
> contourline maps just added to the mdr without needing to regenerate the
> full mdr -- don't know the format well enough, but would hope that this
> is possible..).

It would be possible, but the existing file would have to be read so
that the new information could be merged in the correct sorted order.
It would be a fair amount of work and would use a similar amount of
memory as doing the whole lot.

..Steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: index_memory_use1.patch
Type: text/x-patch
Size: 1724 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20110906/50124330/attachment.bin 