[mkgmap-dev] splitter - performance and memory

Wed Jul 22 12:17:43 BST 2009

Hi Steve,

I have some questions about what you think should be the correct behaviour 
of the splitter in certain situations:

1) If a way covers more than one area, should ALL nodes for the way belong 
to all of those areas, or should each area only contain the nodes that lie 
within it, regardless of ways? Currently the latter seems to be the case 
but that feels wrong to me.

2) Does the XML have to contain nodes first, then ways, then relations, or 
can they be intermingled? (I think osmcut produced files like this) If it 
is permitted, what are the consequences of intermingling them for other tools 
etc? I know that as it stands splitter.jar expects them to be in node/way/relation 
order and wouldn't work very well with intermingled input, especially forwards 
references, but if it's not a big deal then it may help the performance of 
the splitter (especially if we should be dealing with 1 above).

Chris

> On 19/07/09 23:58, Chris Miller wrote:
> 
>> Good news - with my changes the areas.list file was generated fine,
>> peaking at maybe a 3GB heap.
>> 
> Cool.
> 
> It is also worth noting something that osmcut (an other map splitter)
> does. It has two modes; one where a map is used for when the input
> file is small and another where a list is used and the node-id is the
> index into the list. The later is used for splitting the whole planet
> file.
> 
> The second method uses less memory where the memory wasted by node-ids
> that are not in use is less than the amount of memory used by storing
> the node-id plus the memory wasted in free space in the hash maps.
> As currently most nodes are used (about 88%) this is a big win
> on the whole planet file, while still giving access to the node ids.
> I've committed the patch.
> 
> ..Steve
> 