[mkgmap-dev] mkgmap ToDo list

Sun May 4 20:21:59 BST 2014

No - I use splitter like this - with maxnodes depending on area - I 
usually just reduced it after failures to compile a tile.:
java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar 
--max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete 
--max-areas=1524 --geonames-file=cities5000 --description=%country% 
--mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf
would using the precomp-sea parameter for splitter improve results? It's 
not that tiles with loads of sea actually failed.

On 03.05.2014 09:00, Gerd Petermann wrote:
> Hi all,
>
> I've coded a quick hack which seems to improve the ratios.
> On the other hand, I don't see these large differences between
> smallest and largest img file.
> What part of the world should I try?
> Do you use the precomp-sea parameter in splitter?
>
> Gerd
>
> ------------------------------------------------------------------------
> Date: Wed, 30 Apr 2014 14:59:32 +0200
> From: extremecarver at gmail.com
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>
> if that doesn't seriously (more than 30-40%) slow down the splitter, I 
> assume it would be much better...
> On 30.04.2014 14:06, Gerd Petermann wrote:
>
>     Hi,
>
>     if I got that right the number of nodes is not
>     highly correlated to the img size, so the max-nodes
>     value  is not a good estimate.
>
>     I assume the reason is that nodes which belong to
>     roads produce a lot more bytes
>     in the img file compared to nodes which are parts
>     of shapes or other non-routable ways, not talking
>     about nodes which are simply ignored by the style.
>
>     So, a possible solution in splitter could be to parse
>     the ways before reading the nodes and save all nodeids
>     which belong to ways with highway=*.
>     If these nodes are refered by more than one way with highway=*
>     we assume that they will be routing nodes.
>     These special nodes could be counted e.g. 10 times to
>     give a better estimate.
>
>     Gerd
>
>     > Date: Wed, 30 Apr 2014 13:36:19 +0200
>     > From: osm at na1400.info <mailto:osm at na1400.info>
>     > To: mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     > Subject: Re: [mkgmap-dev] mkgmap ToDo list
>     >
>     > Multithreading the tile rendering for a single map is indeed a bit
>     > difficult and I gave it up because you need to keep track which
>     image
>     > id's are already in use. Since I provide multiple maps the
>     work-around
>     > is running a few scripts parallel, which is also a crude form of
>     > multithreading.
>     >
>     > The script language is PHP and it doesn't run on Windows without
>     some
>     > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff).
>     Never tried it.
>     >
>     > To get a better optimum in file size, using the process I described
>     > earlier, you could start off with a huge --max-nodes setting and
>     then
>     > 'search' for the highest --max-nodes that works for each
>     specific area.
>     >
>     > On 30/04/2014 11:49, Felix Hartmann wrote:
>     > > I would love if there was a possibility that you pass the used
>     max-nodes
>     > > value to mkgmap.
>     > > When mkgmap is compiling the maps, then after the .img is
>     created it
>     > > should check
>     > > a) did it crash due to too many max-nodes
>     > > b) for me not important - but for others with very old GPS,
>     etrex 10,
>     > > ---> is tile bigger than X (usually 8) MB.
>     > >
>     > > if a) or b) true, then pass the file back to splitter and
>     split with 60%
>     > > of maxnodes - and compile the resulting .img files again.
>     Should it fail
>     > > again, use 40%, again 25%... Sometimes there are awful tiles,
>     that need
>     > > supersmall max-nodes till they compile, however lately (last
>     1-2 years)
>     > > I never encountered them anymore. I think that happened rather
>     due to a
>     > > but in splitter/mgkmap that is fixed now.
>     > >
>     > > okay, you could also do this with a script, but it gets rather
>     > > complicated to multithread it (you need to wait till mgkmap
>     finished
>     > > compiling all .img files - and run mkgmap first without
>     address index to
>     > > save time) and do some clever routines on making sure that the
>     map id
>     > > (e.g. 6340????.img) stay correct. Even more complicated to have
>     > > consequent map id...
>     > >
>     > > For europe with a fixed max-node I get tiles from 1.9MB up to
>     18MB.
>     > > That's factor 9 - so it's huge...
>     > > If I could narrows that down easily to 8-18MB - without
>     getting tiles
>     > > crashing due to too high max-nodes values, that would be sweet.
>     > >
>     > >
>     > >
>     > > As for the scripts - would they run on windows too? - What
>     programming
>     > > language are they in?
>     > >
>     > >
>     > > On 29.04.2014 21:39, osm at na1400.info <mailto:osm at na1400.info>
>     wrote:
>     > >> Oh, and ofcourse anyone interested can get my scripts, send
>     an email.
>     > >> They'll be on Github someday anyway.
>     > >>
>     > >>
>     > >> On 2014-04-29 20:37, Gerd Petermann wrote:
>     > >>> Hi Lambertus,
>     > >>>
>     > >>> okay, if I got that right you finally get *.img files with a
>     size
>     > >>> near (but below) 8MB, so maybe Henning can use that script, too.
>     > >>>
>     > >>> If you do that for e.g. Germany, how small is tpically the
>     smallest
>     > >>> *.img file ?
>     > >>> Is it probably near 4 MB?
>     > >>>
>     > >>> Gerd
>     > >>>
>     > >>>> To: mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200
>     > >>>> From: osm at na1400.info <mailto:osm at na1400.info>
>     > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>     > >>>>
>     > >>>> These are the direct results from Splitter. The format is
>     o5m, both
>     > >>>> input as output. Splitter version is: r321.
>     > >>>>
>     > >>>> For this test I split the original source with
>     --max-nodes=8000000.
>     > >>> Then
>     > >>>> I render the initial tiles, when the result is larger than
>     8MB it's
>     > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The
>     > >>> initial
>     > >>>> source files are ~70MB (o5m) and after several subsplits
>     the two
>     > >>> *.img
>     > >>>> are < 8MB. During this process --max-nodes has been reduced
>     to e.g.
>     > >>>> 7500000 and the source file is split up in two o5m files of
>     about
>     > >>> 37MB.
>     > >>>>
>     > >>>> I can upload an example source file and it's two subsplit
>     siblings
>     > >>> if
>     > >>>> you want.
>     > >>>>
>     > >>>>
>     > >>>> On 2014-04-29 19:38, GerdP wrote:
>     > >>>> > Hi Lambertus,
>     > >>>> >
>     > >>>> > that's interesting. Are these the img file sizes or the
>     osm file
>     > >>> sizes?
>     > >>>> >
>     > >>>> > Gerd
>     > >>>> >
>     > >>>> >
>     > >>>> > Lambertus wrote
>     > >>>> >> Unfortunately I cannot confirm that. Below is a bit of
>     logging
>     > >>> from my
>     > >>>> >> script:
>     > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1
>     > >>> (36852845)
>     > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1
>     > >>> (37222739)
>     > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1
>     > >>> (38098627)
>     > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1
>     > >>> (39261197)
>     > >>>> >>
>     > >>>> >> The original files above contain contour data, the
>     filesize is
>     > >>> between
>     > >>>> >> brackets. As you can see both resulting file are
>     approximately
>     > >>> the
>     > >>>> >> same
>     > >>>> >> size.
>     > >>>> >>
>     > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote:
>     > >>>> >>> Hi Lambertus,
>     > >>>> >>>
>     > >>>> >>> and I guess that even after this optimization you will
>     > >>>> >>> see a factor 3 or higher between the largest tile and the
>     > >>> smallest.
>     > >>>> >>> Can you confirm that?
>     > >>>> >>>
>     > >>>> >>> Gerd
>     > >>>> >>>
>     > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200
>     > >>>> >>>> From:
>     > >>>> >
>     > >>>> >> osm@
>     > >>>> >
>     > >>>> >>>> To:
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>     > >>>> >>>>
>     > >>>> >>>> Num-tiles=x would indeed be better for this specific need.
>     > >>>> >>>>
>     > >>>> >>>> It is my experience that it regularly takes multiple
>     calls to
>     > >>>> >>> Splitter
>     > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by
>     100k for
>     > >>> each
>     > >>>> >>>> sub-split attempt. This is what I currently do to get an
>     > >>> optimum in
>     > >>>> >>>> tile-size vs total number of tiles.
>     > >>>> >>>>
>     > >>>> >>>>
>     > >>>> >>>>
>     > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote:
>     > >>>> >>>> > Hi Lambertus,
>     > >>>> >>>> >
>     > >>>> >>>> > that sounds like a possible change in splitter:
>     > >>>> >>>> > Instead of specifying max-nodes you may specify
>     --num-tiles=x
>     > >>>> >>>> > and splitter will try to find a split that produces
>     excactly
>     > >>> x
>     > >>>> >>> tiles
>     > >>>> >>>> > which are not too narrow and have a node number
>     which is not
>     > >>>> >>>> > too far from the average (but still aligned to a
>     multiple of
>     > >>> map
>     > >>>> >>> units
>     > >>>> >>>> > as now).
>     > >>>> >>>> > So, for your script that means you don't have to
>     find the
>     > >>>> >>> max-nodes
>     > >>>> >>>> > value.
>     > >>>> >>>> >
>     > >>>> >>>> > I'll think about this again...
>     > >>>> >>>> >
>     > >>>> >>>> > Gerd
>     > >>>> >>>> >
>     > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200
>     > >>>> >>>> > > From:
>     > >>>> >
>     > >>>> >> osm@
>     > >>>> >
>     > >>>> >>>> > > To:
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list
>     > >>>> >>>> > >
>     > >>>> >>>> > > While this possibly can be solved in Splitter or
>     Mkgmap, it
>     > >>>> >>> could also
>     > >>>> >>>> > > be solved by your build-script when you add a
>     maximum tile
>     > >>> size
>     > >>>> >>> check
>     > >>>> >>>> > > and re-split (with a lower number of max-nodes)
>     until you
>     > >>> get
>     > >>>> >>> two or
>     > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to
>     the script
>     > >>> but
>     > >>>> >>> it works
>     > >>>> >>>> > > well for me.
>     > >>>> >>>> > >
>     > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote:
>     > >>>> >>>> > > > Hi Gerd,
>     > >>>> >>>> > > >
>     > >>>> >>>> > > > I would like to have img-tiles which have
>     globally nearly
>     > >>> the
>     > >>>> >>> same
>     > >>>> >>>> > > > filesize, so that they use the space of devices like
>     > >>> eTrex 10.
>     > >>>> >>>> > > >
>     > >>>> >>>> > > > With my actual map I use globally the same value for
>     > >>>> >>> max-nodes. But the
>     > >>>> >>>> > > > size of the img-tiles differ more then factor 2.
>     Eg. a
>     > >>> tile in
>     > >>>> >>> Germany
>     > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is
>     about 10
>     > >>> mb. If
>     > >>>> >>> I remove
>     > >>>> >>>> > > > details, this difference will increase, because in
>     > >>> Germany
>     > >>>> >>> more objects
>     > >>>> >>>> > > > will be removed from the img-tile then in China.
>     > >>>> >>>> > > >
>     > >>>> >>>> > > > Henning
>     > >>>> >>>> > > >
>     > >>>> >>>> > > > _______________________________________________
>     > >>>> >>>> > > > mkgmap-dev mailing list
>     > >>>> >>>> > > >
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>> >>>> > >
>     > >>>> >>>> > > _______________________________________________
>     > >>>> >>>> > > mkgmap-dev mailing list
>     > >>>> >>>> > >
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>> >>>> >
>     > >>>> >>>> >
>     > >>>> >>>> > _______________________________________________
>     > >>>> >>>> > mkgmap-dev mailing list
>     > >>>> >>>> >
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>> >>>> >
>     > >>>> >>>>
>     > >>>> >>>> _______________________________________________
>     > >>>> >>>> mkgmap-dev mailing list
>     > >>>> >>>>
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>> >>>
>     > >>>> >>> _______________________________________________
>     > >>>> >>> mkgmap-dev mailing list
>     > >>>> >>>
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>> >> _______________________________________________
>     > >>>> >> mkgmap-dev mailing list
>     > >>>> >
>     > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org>
>     > >>>> >
>     > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>> >
>     > >>>> >
>     > >>>> >
>     > >>>> >
>     > >>>> >
>     > >>>> > --
>     > >>>> > View this message in context:
>     > >>>> >
>     > >>>
>     http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
>     > >>>> > Sent from the Mkgmap Development mailing list archive at
>     > >>> Nabble.com.
>     > >>>> > _______________________________________________
>     > >>>> > mkgmap-dev mailing list
>     > >>>> > mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>> _______________________________________________
>     > >>>> mkgmap-dev mailing list
>     > >>>> mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >>>
>     > >>> _______________________________________________
>     > >>> mkgmap-dev mailing list
>     > >>> mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >> _______________________________________________
>     > >> mkgmap-dev mailing list
>     > >> mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>     > >
>     >
>     > _______________________________________________
>     > mkgmap-dev mailing list
>     > mkgmap-dev at lists.mkgmap.org.uk
>     <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
>     _______________________________________________
>     mkgmap-dev mailing list
>     mkgmap-dev at lists.mkgmap.org.uk  <mailto:mkgmap-dev at lists.mkgmap.org.uk>
>     http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> -- 
> keep on biking and discovering new trails
>
> Felix
> openmtbmap.org &www.velomap.org  <http://www.velomap.org>
>
> _______________________________________________ mkgmap-dev mailing 
> list mkgmap-dev at lists.mkgmap.org.uk 
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-- 
keep on biking and discovering new trails

Felix
openmtbmap.org & www.velomap.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140504/69f24ad1/attachment-0001.html>