[mkgmap-dev] splitter r325: improved split algo and new option

Wed May 7 15:06:27 BST 2014

Well, yes - I mainly tried on Austria now, so results may differ a bit 
by country - but for me the trend is clear.

Starting with a .pbf file.
Fastest is osmcovert it to o5m (about 15seconds) then splitter output to 
o5m. Total time to split 1:11
Second fastest os osmconvert it to o5m - then splitter output osm.pbf =1:14
Slowest is splitter osm.pbf to osm.pbf = 2:04min..

For mkgmap austria:
osm.pbf (and creating mapset files twice): 3:32
o5m (and creating mapset files twice): 3:30...

So for right now, I will output splitter to osm.pbf, but convert first 
to o5m with osmconvert. After successful split, I directly delete the 
o5m file.
This is using a conventional hdd, maybe with an SSD the o5m advantage is 
a bit bigger (but then you might be more space bound on the SSD too)
The output to osm.pbf by splitter is simply more effective - because o5m 
is 50% bigger. And for Austria the overall time it is faster is about 5 
seconds...

Of course osmconvert also needs some time (about 15seconds of the 1:11) 
- so If I started with an o5m file instead of osm.pbf - it would save 
that 15seconds...
I only update my maps once a week (and europe continent only every ~6 
weeks) - so even if I download the planet - I will have to see if I 
maybe cut it to osm.pbf instead of o5m files to save space... I don't 
have time right now to set up all that toolchain however. My server is 
on a 1000Mbit connection - so downloads are faster than the HDD if the 
source is also fast...

For sure converting to o5m before feeding tiles to splitter - makes a 
lot of sense. Even if all else stays osm.pbf..

On 07.05.2014 15:28, Gerd Petermann wrote:
> Hi Felix,
>
> yes, confirmed. The output files are much bigger because splitter 
> never writes the
> version info (timestamp,uid,user,changeset, version).
> In mkgmap we read the file only once, not multiple times as in splitter,
> so the advantage is rather small.
>
> Gerd
>
> > Date: Wed, 7 May 2014 15:20:51 +0200
> > From: extremecarver at gmail.com
> > To: mkgmap-dev at lists.mkgmap.org.uk
> > Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new 
> option
> >
> > Well, I just ran splitter on the .osm.pbf file for Asia and austria.
> > True the split is a bit faster to o5m - but the difference is not so 
> big.
> >
> > Compiling from osm.pbf or o5m however - was more or less the same 
> speed...
> > on the other hand the o5m splitted files use much more space than
> > osm.pbf (for Austria 293MB vs 450MB - that's about 50% more).
> >
> > I will do some further test with input o5m and output osm.pbf vs output
> > o5m. However if there is also no significant time difference (right now
> > it's like maybe 10% faster output to o5m instead of osm.pbf) - then I
> > think I'm gonna stay splitter output at osm.pbf - simply because on my
> > server I'm a bit harddisk bound (got a large network drive for saving
> > stuff, but with about 20MB/s vs 150MB/s for the local drive, it's 
> really
> > only useful to save stuff there - not for working with anything on the
> > network drive). Splitter input according to the above time measures
> > however - really seems to profit from o5m...
> > On 07.05.2014 12:56, Felix Hartmann wrote:
> > > okay, well the numbers are convincing. I will change to o5m and
> > > osmupdate too - as soon as I find the time (I do think I need 5-10
> > > hours changing my scripts and making sure everything is neatly cut,
> > > but right now I'm too timelimited to do so)..
> > > On 07.05.2014 12:15, Bernd Weigelt wrote:
> > >> Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann:
> > >>> Well I still use pbf and not o5m.
> > >>> First pbf is smaller..
> > >>> Second - Geofabrik only offers pbf - that's why I stayed with it.
> > >>>
> > >>> I don't think I can cut a lot of time by first converting to 
> 05m, then
> > >>> hand it over to splitter...
> > >>> Actually I also let splitter output pbf... Maybe I could change 
> that in
> > >>> future to 05m..
> > >> I download a planet.pbf, convert it to o5m, time ~50 mins,and cut 
> the
> > >> needed
> > >> polies from that planet.o5m. an update of my germany.o5m, 3,2 GB.
> > >> takes now 3
> > >> minutes. I update the planet once a month, my extracts everytime i
> > >> need them.
> > >>
> > >> thats much faster then download a new pbf everyday or update this 
> pbf.
> > >> and with the local planet it is easier to create special maps for
> > >> friends like
> > >> bonn+100 or dach+
> > >>
> > >> Bernd
> > >>
> > >>
> > >
> >
> > --
> > keep on biking and discovering new trails
> >
> > Felix
> > openmtbmap.org & www.velomap.org
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-- 
keep on biking and discovering new trails

Felix
openmtbmap.org & www.velomap.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140507/87a2accc/attachment-0001.html>