[mkgmap-dev] [PATCHv4]splitter memory usage

Sat Nov 12 13:46:21 GMT 2011

Hello WanMil,

the attached patch reverts the changes that calculated the MaxNodeId and nodeCount values. 
The default is now to assume a highest node id of 2^31 and resize when needed. 
This still needs much fewer memory compared to r181 because it allocates exactly two of these large arrays, while r181 typically allocated 8.
So, on my machine splitting germany.osm.pbf took 938 secs with r181 (GC very busy because of -Xmx1600m ) and only 638 secs with patchv4 (optimize-mem=false).
With optimize-mem=true, the program is typically ~ 10% slower.

I've also increased the allowed number of areas per pass, because this is now limited only by available memory and by the number of elements in 
the dictionary. If the latter reaches 32767 the program will crash, but I doubt that this will happen with real OSM data.

Ciao,
Gerd

> Date: Sat, 12 Nov 2011 12:41:11 +0100
> From: wmgcnfg at web.de
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] [PATCH]splitter memory usage
> 
> Am 11.11.2011 14:58, schrieb Gerd Petermann:
> >  > Date: Wed, 9 Nov 2011 19:24:37 +0100
> >  > From: wmgcnfg at web.de
> >  > To: mkgmap-dev at lists.mkgmap.org.uk
> >  > Subject: Re: [mkgmap-dev] [PATCH]splitter memory usage
> >  >
> >  > Thanks Gerd!
> >  >
> >  > I didn't yet review the patch but I have commited (r184) it so it's
> >  > available at http://www.mkgmap.org.uk/splitter/
> >  >
> >  > I am looking forward that your patches will improve splitter for several
> >  > use cases.
> >  >
> >  > WanMil
> >
> > Hello WanMil,
> >
> > please apply the attached patch on the memory_optimization branch. It
> > contains:
> > a) code simplification
> > b) correction regarding the possibilty that a way id is higher than the
> > highest node id
> > c) small performance tweaks
> >
> > ciao,
> > Gerd
> >
> 
> Hi Gerd,
> 
> commited it to r187.
> 
> Have fun!
> WanMil
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20111112/20018365/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memory_patch_v4.patch
Type: application/octet-stream
Size: 19414 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20111112/20018365/attachment.obj 