[mkgmap-dev] Memory Consumption on Index Creation

Wed Sep 7 11:15:54 BST 2011

Hi steve,

Okay strange, tonight it went through without problems (255M reserved 
when starting). Maybe something else was wrong then...

However could you change mkgmap to delete broken mdr files if it crashes 
on low memory??
That would be a real troublesaver!!!!

I don't know how to find out using a batch/bash if the mdr file is okay 
or not otherwise!
(I can't simply look at the size, as for the Europe file when it crashed 
it was always around 3MB, while simple maps a working mdr is much smaller).

Felix

    /About to start combine steps/
    /Thread[Reference Handler,10,system] WAITING/
    / at java.lang.Object.wait(Native Method)/
    / at java.lang.Object.wait(Object.java:485)/
    / at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)/
    /Thread[Finalizer,8,system] WAITING/
    / at java.lang.Object.wait(Native Method)/
    / at java.lang.ref.ReferenceQueue.remove(Unknown Source)/
    / at java.lang.ref.ReferenceQueue.remove(Unknown Source)/
    / at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)/
    /Thread[Attach Listener,5,system] RUNNABLE/
    /Thread[main,5,main] RUNNABLE/
    / at java.lang.Thread.dumpThreads(Native Method)/
    / at java.lang.Thread.getAllStackTraces(Unknown Source)/
    / at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:423)/
    / at
    uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)/
    / at uk.me.parabola.mkgmap.main.Main.main(Main.java:132)/
    /Thread[Signal Dispatcher,9,system] RUNNABLE/
    /Memory 255M 7569M 7569M/
    /Java Version 1.6.0_21 Sun Microsystems Inc., on Windows Server 2008
    R2 6.1/

On 06.09.2011 10:53, Steve Ratcliffe wrote:
> On 06/09/11 00:17, Felix Hartmann wrote:
>> could it be that the index starts to be worked one, when the first of
>> the 4 worker threads (to compile the maps) is finished, with 3 still
>> working and hence overshooting the memory limit?? I really cannot see
>> why it would have failed me 4 times in a row on a single call, but
>> single index creation allways worked...
>
> Maybe - if something else is failing while tiles are being generated.
>
> If you apply the attached patch and run, it will print out how much
> memory is available, what threads are running and version information
> just before it starts the combining step.
>
> On my machine it looks like this:
>
>     About to start combine steps
>     Thread[Signal Dispatcher,9,system] RUNNABLE
>     Thread[Reference Handler,10,system] WAITING
>      at java.lang.Object.wait(Native Method)
>      at java.lang.Object.wait(Object.java:485)
>      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>     Thread[main,5,main] RUNNABLE
>      at java.lang.Thread.dumpThreads(Native Method)
>      at java.lang.Thread.getAllStackTraces(Thread.java:1530)
>      at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:423)
>      at 
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
>      at uk.me.parabola.mkgmap.main.Main.main(Main.java:132)
>     Thread[Finalizer,8,system] WAITING
>      at java.lang.Object.wait(Native Method)
>      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>     Memory 2M 105M 881M
>     Java Version 1.6.0_22 Sun Microsystems Inc., on Linux 
> 2.6.40.3-0.fc15.x86_64
>
>>> I do have some ideas for some reduction in the memory use by the index
>>> generation.
>>>
>> Well that would be great. It is currently using much more memory than
>
> I looked into this a bit more and found that the simple ideas wouldn't
> save much memory unfortunately.
>
>> Best would even be if on single index creation, one could specify an mdr
>> file for existing maps, and just add stuff for maps that you want to add
>> to the compilation (use case, user has a map of Germany in several .img
>> with mdr/mdx and wants to add contourlines into the mapset, it would be
>> great if the index of the Germany map could be given, and the
>> contourline maps just added to the mdr without needing to regenerate the
>> full mdr -- don't know the format well enough, but would hope that this
>> is possible..).
>
> It would be possible, but the existing file would have to be read so
> that the new information could be merged in the correct sorted order.
> It would be a fair amount of work and would use a similar amount of
> memory as doing the whole lot.
>
> ..Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20110907/710912de/attachment.html 