[mkgmap-dev] Patch to reduce memory usage by interning strings.

Fri Apr 2 05:16:04 BST 2010

On Thu, 01 Apr 2010 12:10:22 +0100, Steve Ratcliffe <steve at parabola.me.uk> writes:

> On 31/03/10 14:56, Scott A Crosby wrote:
>> I noticed that mkgmap does not intern any strings. In particular, this
>
> This is true.  There is a pending patch that deals with excessive
> memory use in a slightly different way which is on the style-speed
> branch, with a pre-built one available at the bottom of the mkgmap
> download page.

I pulled it out of SVN at 1569.

> I drops all tags that are not used by the current style and as an
> extra feature it ensures that there is only copy of all the strings
> on the key side.
>
> Could you try it out please. It may be worth also interning the values
> but when I was looking at it there was much less benefit for the maps
> I was looking at (but it might vary with the input).
> I'd be happy to apply a patch that interned the values too if there
> was a decent memory saving without a significant performance drop.

The style-speed branch builds the tile in question within 1gb of ram
using the default style. I suspect that that style isn't using the
problematic tags. A different style that did use the tags would likely
still blow up. This seems fragile.

I benchmarked the problematic tile on the style-speed branch with and
without interning all keys and values in the Tag() constructor using
ordinary String.intern(). The 18 character change implementing
interning seems to increase performance by about a second, from 66 to
65 seconds.

I'd say to go for it.

Scott