[mkgmap-dev] Splitter pbf vs o5m processing

Thu Dec 13 14:49:49 GMT 2012

Hi WanMil,

I can't say for sure what the pbf reader is doing in detail, but it is for sure creating a
lot more temporary objects which have to be GCed. In the o5m reader I tried
to avoid that. In fact, in the current implementation, the o5m reader still
reads and saves the tags to the internal string table, so that is similar to the
pbf reader.

I'll look at your logs soon. I am working on a tuning guide for splitter, because
I found a lot of nonsense in the net searching for splitter.

Ciio,
Gerd

> Date: Thu, 13 Dec 2012 15:27:07 +0100
> From: wmgcnfg at web.de
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: [mkgmap-dev] Splitter pbf vs o5m processing
> 
> > Hi Steve,
> >
> >
> > Steve Ratcliffe wrote
> >> Hello Gerd
> >>
> >>> no, it is not (yet). I plan to add o5m support to mkgmap soon. With my
> >>> patch you can use splitter
> >>
> >> As an aside, what do you think it is about the o5m format that makes
> >> it quicker than pbf?
> >
> > Well, not easy to say. I think it's a combination of many small points:
> > 1) pbf uses (by default) compressied blocks, so you have to unzip a complete
> > block before you can
> > use any information in the block.
> > 2) pbf read routines create a lot of temporary objects, this seems to stress
> > GC
> > 3) pbf doesn't allow to skip processing of node tags or way tags, but
> > splitters' read passes often don't need them. So, with pbf we create lists
> > of tags and return them to GC, with o5m we can simply skip them.
> >
> > To be fair, using the --drop-version parm in osmconvert removes a lot of
> > info which is ignored by splitter and mkgmap. I did never try what effect is
> > has to use pbf input that was created with this parm.
> >
> > When writing, o5m is probably only faster because it doesn't zip the data.
> > As long as mkgmap doesn't understand o5m I see no benefit in using this.
> >
> > Maybe other computers show different results, esp. if the CPU is much faster
> > than mine and the Disk access is slower.
> > By the way: my patch also speeds up pbf reading a little bit.
> >
> > Ciao,
> > Gerd
> 
> Hi Gerd
> 
> I've done some tests with the latest splitter version r255.
> I have split the geofabrics europe extract in pbf and o5m format.
> 
> As you pointed out o5m processing is much quicker (8528s vs. 12939s).
> I also observed that pbf seemed to use more memory than o5m and 
> therefore I activated gc logging and checked it with garbagecat.
> 
> The interesting values are
> Throughput
> o5m: 94%
> pbf: 61%
> So 3400m seems to be too small for pbf processing to workout the europe 
> extract so that the GC runs permanently.
> 
> Total Pause:
> o5m:  527816ms =  528s
> pbf: 5093916ms = 5094s
> Wow, so for pbf GC requires 4566s more time.
> 
> Subtracting the GC time from the total processing time o5m and pbf need 
> quite the same time:
> o5m:  8528s -  528s = 8000s
> pbf: 12939s - 5094s = 7845s
> 
> Obviously a part of the difference in GC time can be explained with your 
> thoughts (pbf must extract all parts and must read tags which are thrown 
> away directly afterwards). But do you think that the whole difference 
> can be explained with that?
> 
> I will post my logfiles directly to you because they are too big to be 
> posted on the mailing list.
> 
> WanMil
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20121213/28561a23/attachment.html 