[mkgmap-dev] New splitter version, big memory savings

Thu Sep 3 13:52:41 BST 2009

Hi Valentijn,

It depends... how many areas does your osm file get split into, and what 
value (if any) are you using for --max-areas? Are you providing your own 
areas.list file via the --split-file parameter?

It's still noticably slower to parse an osm file (even an uncompressed one) 
than it is to read the data from the cache. Offsetting this is the overhead 
required to create the cache in the first place. So it all comes down to 
the number of passes over the data that are required to perform the split. 
If it's only a single pass, the cache is a hinderance (even with a compressed 
osm file). If two passes are required, the cache is a definite win if your 
osm file is compressed, and about break-even if it is uncompressed. If three 
or more passes are required, the cache is a definite win regardless of whether 
your osm files are compressed or not.

The first stage of the split always requires one (and only one) pass. The 
only time the first stage is skipped is when you specify the --split-file 
parameter.

The second stage of the split always requires at least one additional pass, 
possibly more. The number of passes required in stage two equals:

T / M (rounded up to the nearest whole number)

where:

T = the total number of areas generated by the split
M = the --max-areas value (default = 255)

Note also that if you intend to run the splitter more than once against the 
same osm file (eg because you are experimenting with different parameters 
for --max-nodes), then it's also going to be a big win if you use the cache 
since it can be reused across runs and the osm file never needs reparsing.

Hope that helps clarify things for you.

Chris

VS> Chris, thanks for al the good work. Just a small and unrelated
VS> remark. The script that builds my map first unpacks the osm.bz2
VS> file, then runs splitter. Still, Splitter complains about no --cache
VS> being used, while as far as I understand, there's no real advantage
VS> using --cache if you're using uncompressed files, or is there?
VS> 
VS> Best regards,
VS> 
VS> Valentijn
VS> 
VS> Chris Miller schreef:
VS> 
>> I've just checked in a new version of the splitter (r84) that
>> requires far less memory and also performs slightly better during the
>> first stage of the split.