[mkgmap-dev] [PATCH v5]splitter memory usage

Thu Nov 17 18:30:48 GMT 2011

Europe max-nodes 900.000 (resulting in 975 tiles, extract is about 4 
weeks old already - hence 4 passes down to 1 pass )

old splitter: 52min 38 seconds (3158 seconds)
splitter_patched: 29min 45 seconds (1785 seconds)
splitter patched with optimize-mem: 30min 41seconds (1841 seconds)

So I would support optimize-mem to stay off by default. It seems to 
produce worse results the more data it has to handle. The general 
improvement on splitting time is however great, especially on bigger 
files (mind mkgmap itself needs about 5x as much time as the patched 
splitter). Also the estimates taken by optimize-mem seem to be plain 
underestimated (end of the output):

/

MAP occupancy: 56'000'000, number of area dictionary entries: 10530
Chunk map details: used 1'447'751 (~ 4418%) of the estimated 32'768 entries.
56'000'000 ways processed... id=111312902
MAP occupancy: 57'000'000, number of area dictionary entries: 10596
Chunk map details: used 1'469'838 (~ 4486%) of the estimated 32'768 entries.
57'000'000 ways processed... id=112919265
MAP occupancy: 58'000'000, number of area dictionary entries: 10633
Chunk map details: used 1'493'839 (~ 4559%) of the estimated 32'768 entries.
58'000'000 ways processed... id=114728311
MAP occupancy: 59'000'000, number of area dictionary entries: 10660
Chunk map details: used 1'516'986 (~ 4629%) of the estimated 32'768 entries.
59'000'000 ways processed... id=116413634
MAP occupancy: 60'000'000, number of area dictionary entries: 10700
Chunk map details: used 1'541'052 (~ 4703%) of the estimated 32'768 entries.
60'000'000 ways processed... id=118147644
MAP occupancy: 61'000'000, number of area dictionary entries: 10750
Chunk map details: used 1'564'456 (~ 4774%) of the estimated 32'768 entries.
61'000'000 ways processed... id=119857968
MAP occupancy: 62'000'000, number of area dictionary entries: 10789
Chunk map details: used 1'589'073 (~ 4849%) of the estimated 32'768 entries.
62'000'000 ways processed... id=121660891
MAP occupancy: 63'000'000, number of area dictionary entries: 10828
Chunk map details: used 1'610'303 (~ 4914%) of the estimated 32'768 entries.
63'000'000 ways processed... id=123132157
MAP occupancy: 64'000'000, number of area dictionary entries: 10838
Chunk map details: used 1'632'875 (~ 4983%) of the estimated 32'768 entries.
64'000'000 ways processed... id=124780404
MAP occupancy: 65'000'000, number of area dictionary entries: 10847
Chunk map details: used 1'656'390 (~ 5055%) of the estimated 32'768 entries.
65'000'000 ways processed... id=126458865
MAP occupancy: 66'000'000, number of area dictionary entries: 10907
Chunk map details: used 1'679'991 (~ 5127%) of the estimated 32'768 entries.
66'000'000 ways processed... id=128180713
MAP occupancy: 67'000'000, number of area dictionary entries: 11004
Chunk map details: used 1'702'474 (~ 5196%) of the estimated 32'768 entries.
67'000'000 ways processed... id=129790909
MAP occupancy: 68'000'000, number of area dictionary entries: 11036
Chunk map details: used 1'724'908 (~ 5264%) of the estimated 32'768 entries.
68'000'000 ways processed... id=131429900
Writing relations Thu Nov 17 19:22:19 CET 2011
Elapsed time: 30m 6s   Memory: Current 5574MB (4285MB used, 1289MB free) 
Max 6044MB
100'000 relations processed... id=190988
200'000 relations processed... id=349055
300'000 relations processed... id=485393
400'000 relations processed... id=952259
500'000 relations processed... id=1198900
600'000 relations processed... id=1376675
700'000 relations processed... id=1573951
800'000 relations processed... id=1743479
***********************************************************
Final statistics
***********************************************************
coords occupancy
MAP occupancy: 579'162'124, number of area dictionary entries: 6248
Chunk map details: used 15'886'784 (~ 48483%) of the estimated 32'768 
entries.
ways occupancy
MAP occupancy: 68'744'281, number of area dictionary entries: 11058
Chunk map details: used 1'741'394 (~ 5314%) of the estimated 32'768 entries.

Thread worker-1 has finished
Thread worker-0 has finished
Thread worker-2 has finished
Time finished: Thu Nov 17 19:22:55 CET 2011
Total time taken: 1841s
/
On 17.11.2011 16:48, Felix Hartmann wrote:
> some more speed observations:
> country germany.pbf from Geofabrik
>
> splitter_patched with optimize-mem: 3 min 6 seconds
> splitter_patched without optimize-mem: 3min 2 seconds
> old splitter: 3 min 42 seconds
>
> I am currently running against europe.pbf, will post the results 
> later. But it seems that the patched mkgmap is consistently a bit 
> faster. I think optimize-mem is better of by default though (most 
> people using mkgmap/splitter have powerful computers anyhow).
> I'm interested if the 1024 max areas vs the old 255 and subsequent 
> less runs make for a bigger speedup than on small(er) extracts.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20111117/276be27f/attachment.html 