logo separator

[mkgmap-dev] Patch to reduce memory usage by interning strings.

From WanMil wmgcnfg at web.de on Fri Apr 2 11:15:07 BST 2010

Am 02.04.2010 06:16, schrieb Scott A Crosby:
> On Thu, 01 Apr 2010 12:10:22 +0100, Steve Ratcliffe<steve at parabola.me.uk>  writes:
>
>> On 31/03/10 14:56, Scott A Crosby wrote:
>>> I noticed that mkgmap does not intern any strings. In particular, this
>>
>> This is true.  There is a pending patch that deals with excessive
>> memory use in a slightly different way which is on the style-speed
>> branch, with a pre-built one available at the bottom of the mkgmap
>> download page.
>
> I pulled it out of SVN at 1569.
>
>> I drops all tags that are not used by the current style and as an
>> extra feature it ensures that there is only copy of all the strings
>> on the key side.
>>
>> Could you try it out please. It may be worth also interning the values
>> but when I was looking at it there was much less benefit for the maps
>> I was looking at (but it might vary with the input).
>> I'd be happy to apply a patch that interned the values too if there
>> was a decent memory saving without a significant performance drop.
>
> The style-speed branch builds the tile in question within 1gb of ram
> using the default style. I suspect that that style isn't using the
> problematic tags. A different style that did use the tags would likely
> still blow up. This seems fragile.
>
> I benchmarked the problematic tile on the style-speed branch with and
> without interning all keys and values in the Tag() constructor using
> ordinary String.intern(). The 18 character change implementing
> interning seems to increase performance by about a second, from 66 to
> 65 seconds.
>
> I'd say to go for it.
>
> Scott

Scott,

thanks for posting hard performance values! That's good news because it 
makes possible to use a simple intern() solution.

Before I read your post I compared some german tiles. The single 
threaded SVN r1625 took 328s with the default style. Using the intern() 
patch it took 335s. That seems to be an unsignificant change.
I am not able to post multithreaded comparisons because my geriatric CPU 
has one core only...

I also used the VisualGC plugin of JVisualVM to view the GC. The 
permanent space (where all the interned Strings are stored) did not 
exceed 11mb. So interning all Strings will probably not exceed the 
permanent space. However we might add a note to the README file that in 
some special cases it is necessary to increase the permanent size with 
-XX:MaxPermSize=128m (or more) to avoid OutOfMemoryExceptions.

WanMil




More information about the mkgmap-dev mailing list