<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Hi Felix,<br><br>I seem to remember that the disadvantage of pbf for output of splitter was higher,<br>but maybe that was before I modified splitter to use a smaller "BatchLimit"<br>in the pbf write routine:<br>configBatchLimit(1000);<br><br>The default value produced slightly better compression but required much more heap.<br>The original code was optimized for one write process, while splitter may start many<br>hundrets, and in some situations splitter and pbf writer were fighting for the heap,<br>causing heavy work for GC. Of course this only happened when splitting large files<br>like Europe or planet.<br><br>Gerd<br><br><div><hr id="stopSpelling">Date: Wed, 7 May 2014 16:06:27 +0200<br>From: extremecarver@gmail.com<br>To: mkgmap-dev@lists.mkgmap.org.uk<br>Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option<br><br>
Well, yes - I mainly tried on Austria now, so results may differ a
bit by country - but for me the trend is clear.<br>
<br>
Starting with a .pbf file.<br>
Fastest is osmcovert it to o5m (about 15seconds) then splitter
output to o5m. Total time to split 1:11<br>
Second fastest os osmconvert it to o5m - then splitter output
osm.pbf =1:14<br>
Slowest is splitter osm.pbf to osm.pbf = 2:04min..<br>
<br>
For mkgmap austria:<br>
osm.pbf (and creating mapset files twice): 3:32<br>
o5m (and creating mapset files twice): 3:30...<br>
<br>
So for right now, I will output splitter to osm.pbf, but convert
first to o5m with osmconvert. After successful split, I directly
delete the o5m file.<br>
This is using a conventional hdd, maybe with an SSD the o5m
advantage is a bit bigger (but then you might be more space bound on
the SSD too)<br>
The output to osm.pbf by splitter is simply more effective - because
o5m is 50% bigger. And for Austria the overall time it is faster is
about 5 seconds...<br>
<br>
Of course osmconvert also needs some time (about 15seconds of the
1:11) - so If I started with an o5m file instead of osm.pbf - it
would save that 15seconds...<br>
I only update my maps once a week (and europe continent only every
~6 weeks) - so even if I download the planet - I will have to see if
I maybe cut it to osm.pbf instead of o5m files to save space... I
don't have time right now to set up all that toolchain however. My
server is on a 1000Mbit connection - so downloads are faster than
the HDD if the source is also fast...<br>
<br>
For sure converting to o5m before feeding tiles to splitter - makes
a lot of sense. Even if all else stays osm.pbf..<br>
<br>
<div class="ecxmoz-cite-prefix">On 07.05.2014 15:28, Gerd Petermann
wrote:<br>
</div>
<blockquote cite="mid:DUB112-W139A4B9035B919A9AF74D619E4E0@phx.gbl">
<style><!--
.ExternalClass .ecxhmmessage P {
padding:0px;
}
.ExternalClass body.ecxhmmessage {
font-size:12pt;
font-family:Calibri;
}
--></style>
<div dir="ltr">Hi Felix,<br>
<br>
yes, confirmed. The output files are much bigger because
splitter never writes the<br>
version info (timestamp,uid,user,changeset, version).<br>
In mkgmap we read the file only once, not multiple times as in
splitter,<br>
so the advantage is rather small.<br>
<br>
Gerd<br>
<br>
<div>> Date: Wed, 7 May 2014 15:20:51 +0200<br>
> From: <a class="ecxmoz-txt-link-abbreviated" href="mailto:extremecarver@gmail.com">extremecarver@gmail.com</a><br>
> To: <a class="ecxmoz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a><br>
> Subject: Re: [mkgmap-dev] splitter r325: improved split
algo and new option<br>
> <br>
> Well, I just ran splitter on the .osm.pbf file for Asia
and austria.<br>
> True the split is a bit faster to o5m - but the
difference is not so big.<br>
> <br>
> Compiling from osm.pbf or o5m however - was more or less
the same speed...<br>
> on the other hand the o5m splitted files use much more
space than <br>
> osm.pbf (for Austria 293MB vs 450MB - that's about 50%
more).<br>
> <br>
> I will do some further test with input o5m and output
osm.pbf vs output <br>
> o5m. However if there is also no significant time
difference (right now <br>
> it's like maybe 10% faster output to o5m instead of
osm.pbf) - then I <br>
> think I'm gonna stay splitter output at osm.pbf - simply
because on my <br>
> server I'm a bit harddisk bound (got a large network
drive for saving <br>
> stuff, but with about 20MB/s vs 150MB/s for the local
drive, it's really <br>
> only useful to save stuff there - not for working with
anything on the <br>
> network drive). Splitter input according to the above
time measures <br>
> however - really seems to profit from o5m...<br>
> On 07.05.2014 12:56, Felix Hartmann wrote:<br>
> > okay, well the numbers are convincing. I will change
to o5m and <br>
> > osmupdate too - as soon as I find the time (I do
think I need 5-10 <br>
> > hours changing my scripts and making sure everything
is neatly cut, <br>
> > but right now I'm too timelimited to do so)..<br>
> > On 07.05.2014 12:15, Bernd Weigelt wrote:<br>
> >> Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix
Hartmann:<br>
> >>> Well I still use pbf and not o5m.<br>
> >>> First pbf is smaller..<br>
> >>> Second - Geofabrik only offers pbf - that's
why I stayed with it.<br>
> >>><br>
> >>> I don't think I can cut a lot of time by
first converting to 05m, then<br>
> >>> hand it over to splitter...<br>
> >>> Actually I also let splitter output pbf...
Maybe I could change that in<br>
> >>> future to 05m..<br>
> >> I download a planet.pbf, convert it to o5m, time
~50 mins,and cut the <br>
> >> needed<br>
> >> polies from that planet.o5m. an update of my
germany.o5m, 3,2 GB. <br>
> >> takes now 3<br>
> >> minutes. I update the planet once a month, my
extracts everytime i <br>
> >> need them.<br>
> >><br>
> >> thats much faster then download a new pbf
everyday or update this pbf.<br>
> >> and with the local planet it is easier to create
special maps for <br>
> >> friends like<br>
> >> bonn+100 or dach+<br>
> >><br>
> >> Bernd<br>
> >><br>
> >><br>
> ><br>
> <br>
> -- <br>
> keep on biking and discovering new trails<br>
> <br>
> Felix<br>
> openmtbmap.org & <a class="ecxmoz-txt-link-abbreviated" href="http://www.velomap.org" target="_blank">www.velomap.org</a><br>
> <br>
> _______________________________________________<br>
> mkgmap-dev mailing list<br>
> <a class="ecxmoz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a><br>
> <a class="ecxmoz-txt-link-freetext" href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev" target="_blank">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a><br>
</div>
</div>
<br>
<fieldset class="ecxmimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
mkgmap-dev mailing list
<a class="ecxmoz-txt-link-abbreviated" href="mailto:mkgmap-dev@lists.mkgmap.org.uk">mkgmap-dev@lists.mkgmap.org.uk</a>
<a class="ecxmoz-txt-link-freetext" href="http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev" target="_blank">http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</a></pre>
</blockquote>
<br>
<pre class="ecxmoz-signature">--
keep on biking and discovering new trails
Felix
openmtbmap.org & <a class="ecxmoz-txt-link-abbreviated" href="http://www.velomap.org" target="_blank">www.velomap.org</a></pre>
<br>_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev</div> </div></body>
</html>