[mkgmap-dev] [PATCH]splitter memory usage

Wed Nov 9 11:43:21 GMT 2011

Hello WanMil,

attached is the larger patch, together with a modified fastutil.jar. The patch must be applied to the memory_optimization branch.

Added features compared to r183:
- Allows Node Ids up to 2^37 - 1
I (hopefully) changed all places where node ids were stored in int or Integer to long or Long. I tested it with  a modified OSM.gz file. 

- Is a bit faster when default methods are used and enough heap is available
During the 1st phase, the highest used Node Id is saved, this value is used to allocate the needeed arrays. The previous version used ArrayLists, which were resized quite often because data from osm.pbf delivers the node ids in order.

- With new parm optimize-mem it uses a hash map instead of the huge arrays to store chunk data. This allows to split files on machines with small available java heap, but is usually slower.

remarks:
a) as mentioned before I am a newbe to Java, so I am pretty sure that experts will find some nonsense in my code, esp. regarding coding style. 
b) Someone should review the way how I collect the information regarding the highest node id and the number of nodes
c) Reg. the new parm: I would prefer to decide the best algorithm within the program, but found it very hard to do, so this is the simle solution.
d) If the limit of 2^37-1 is a problem I'd suggest to change the code that stores the data in the chunks. I played with larger CHUNK_SIZE values combined with a (very simple) run length encoding to compress the chunks. I'll continue analysing that way because it might be faster as well.

regards,
Gerd

> Date: Sun, 6 Nov 2011 14:13:18 +0100
> From: wmgcnfg at web.de
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] [PATCH]splitter memory usage
> 
> Thanks for your analysis and your patch.
> 
> I do not have enough time to check if the patch produces the same valid 
> results. But to me it sounds like a good aproach.
> 
> Therefore I created a branch memory_optimization including your patch so 
> that your changes might be tested more easily by a bigger group of people.
> The compiled release of the branch should appear automatically at 
> http://www.mkgmap.org.uk/splitter/. If that's not the case Steve might 
> has to do some configuration stuff on the webserver?
> 
> WanMil
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20111109/9695e662/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memory_patch_v2.patch
Type: application/octet-stream
Size: 43209 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20111109/9695e662/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fastutil.jar
Type: application/x-jar
Size: 305457 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20111109/9695e662/attachment.bin 