[mkgmap-dev] New, faster splitter

Tue Jun 8 00:02:14 BST 2010

WanMil's patch has inspired me to add some additional threading to the splitter. 
In r110 the loading of osm files is now performed in a separate thread from 
the parsing. This provides an especially big benefit when reading in a .bz2 
compressed osm file since the decompression happens in parallel. Any .gz 
or .zip osm files should also benefit significantly. Uncompressed osm files 
see some speedup but it's much much smaller.

You don't need to do anything to enable this feature. As long as --max-threads 
is set to a value greater than 1 it will be enabled automatically. I'd recommend 
you just leave out the --max-threads parameter altogether and the splitter 
will use an appropriate number of threads for your CPU.

Here are some benchmarks I've run when splitting an osm file of the UK on 
a Core i7 to give you some idea of the impact of the recent changes:

bz2 compressed file, no cache:
No threading: 238s
r109 threading: 193s
r110 threading: 136s

bz2 compressed file, cache:
No threading: 170s
r109 threading: 123s
r110 threading: 92s

Uncompressed osm file, no cache:
No threading: 132s
r109 threading: 80s
r110 threading: 76s

Uncompressed osm file, cache:
No threading: 107s
r109 threading: 63s
r110 threading: 61s

Pre-existing cache (so no osm file parsing required):
No threading: 69s
r109 threading: 23s
r110 threading: 23s (the r110 threading isn't used in this situation)

(for the record, I tried loading the cache in a background thread too but 
that didn't make any difference to performance)

Enjoy,
Chris