[mkgmap-dev] [PATCH]splitter memory usage

Wed Nov 9 18:24:37 GMT 2011

Thanks Gerd!

I didn't yet review the patch but I have commited (r184) it so it's 
available at http://www.mkgmap.org.uk/splitter/

I am looking forward that your patches will improve splitter for several 
use cases.

WanMil

> Hello WanMil,
>
> attached is the larger patch, together with a modified fastutil.jar. The
> patch must be applied to the memory_optimization branch.
>
> Added features compared to r183:
> - Allows Node Ids up to 2^37 - 1
> I (hopefully) changed all places where node ids were stored in int or
> Integer to long or Long. I tested it with a modified OSM.gz file.
>
> - Is a bit faster when default methods are used and enough heap is available
> During the 1st phase, the highest used Node Id is saved, this value is
> used to allocate the needeed arrays. The previous version used
> ArrayLists, which were resized quite often because data from osm.pbf
> delivers the node ids in order.
>
> - With new parm optimize-mem it uses a hash map instead of the huge
> arrays to store chunk data. This allows to split files on machines with
> small available java heap, but is usually slower.
>
> remarks:
> a) as mentioned before I am a newbe to Java, so I am pretty sure that
> experts will find some nonsense in my code, esp. regarding coding style.
> b) Someone should review the way how I collect the information regarding
> the highest node id and the number of nodes
> c) Reg. the new parm: I would prefer to decide the best algorithm within
> the program, but found it very hard to do, so this is the simle solution.
> d) If the limit of 2^37-1 is a problem I'd suggest to change the code
> that stores the data in the chunks. I played with larger CHUNK_SIZE
> values combined with a (very simple) run length encoding to compress the
> chunks. I'll continue analysing that way because it might be faster as well.
>
> regards,
> Gerd
>
>
>  > Date: Sun, 6 Nov 2011 14:13:18 +0100
>  > From: wmgcnfg at web.de
>  > To: mkgmap-dev at lists.mkgmap.org.uk
>  > Subject: Re: [mkgmap-dev] [PATCH]splitter memory usage
>  >
>  > Thanks for your analysis and your patch.
>  >
>  > I do not have enough time to check if the patch produces the same valid
>  > results. But to me it sounds like a good aproach.
>  >
>  > Therefore I created a branch memory_optimization including your patch so
>  > that your changes might be tested more easily by a bigger group of
> people.
>  > The compiled release of the branch should appear automatically at
>  > http://www.mkgmap.org.uk/splitter/. If that's not the case Steve might
>  > has to do some configuration stuff on the webserver?
>  >
>  > WanMil
>  >
>
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev