[mkgmap-dev] mkgmap ToDo list

Mon May 5 19:47:45 BST 2014

Thanks Gerd, seems to be working great!

Two questions:
- How does this interact with the --max-nodes setting?
- Could it be used for an initial split of the entire planet? I.e., set 
the number of tiles I want and have splitter output equally sized tiles? 
(I know I could just try, but the servers are busy)

On 2014-05-05 09:28, Gerd Petermann wrote:
> Hi Lambertus,
> 
> attached is a patch that implements the new option:
>  --num-tiles A target value that is used when no split-file is given.
>  Splitting is done so that the given number of tiles is
>  produced. The max-nodes value is ignored if this option is
>  given.
> A compiled binary is here:
> 
> http://files.mkgmap.org.uk/download/206/splitter.jar [4]
> 
> Please let me know if it works as expected.
> 
> Gerd
> 
> -------------------------
> Date: Sun, 4 May 2014 22:20:20 +0200
> From: osm at na1400.info
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] mkgmap ToDo list
> 
>  Great, many thanks. I'm currently trying to create as large as
> (automatically) possible contour tiles and a lot of time is put in
> subsplitting a tile again and again until two subtiles appear. This
> new option would save much time.
> 
> On 05/04/2014 10:13 PM, Gerd Petermann wrote:
> 
>> Hi Lambertus,
>> 
>> yes, I want to code that tomorrow. I prefer to make it "give me n
>> tiles", but if that turns out to be
>> difficult I try the simple version. I can think of scenarios where
>> it is not possible to create exactly
>> n tiles.
>> 
>> Gerd
>> 
>> -------------------------
>> Date: Sun, 4 May 2014 22:08:32 +0200
>> From: osm at na1400.info
>> To: mkgmap-dev at lists.mkgmap.org.uk
>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>> 
>> Is the 'just give me two tiles' option for splitter on the to-do
>> list? I'd really appreciate such a function.
>> 
>> On 05/04/2014 09:36 PM, Gerd Petermann wrote:
>> 
>> Hi Felix,
>> 
>>> would using the precomp-sea parameter for splitter improve
>> results? It's not that tiles with loads of sea actually failed.
>> 
>> good question. I've coded that parameter in splitter before we did
>> some important changes in mkgmap.
>> I think it is needed when you use a polygon file in
>> combination with an imput file that doesn't cover the polygon.
>> If the polygon covers a costline with a lot of islands, but the
>> input file doesn't contain the data,
>> splitter might create large tiles covering these "empty" areas.
>> Later, in mkgmap, the empty
>> areas are filled with the precompiled-sea data, and that can cause
>> too large files.
>> 
>> BTW: The quick hack turned out to be useless. It worked better in
>> some areas and worse in others.
>> I'll look again at this later next week.
>> 
>> Gerd
>> 
>> -------------------------
>> Date: Sun, 4 May 2014 21:21:59 +0200
>> From: extremecarver at gmail.com
>> To: mkgmap-dev at lists.mkgmap.org.uk
>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>> 
>> No - I use splitter like this - with maxnodes depending on area - I
>> usually just reduced it after failures to compile a tile.:
>> java -Xms4000m -Xmx10400m -jar c:openmtbmapsplitter.jar
>> --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete
>> --max-areas=1524 --geonames-file=cities5000 --description=%country%
>> --mapid=%FID%0000
>> c:OpenMTBMaposmpbf_geofabrik%country%-latest.osm.pbf
>> would using the precomp-sea parameter for splitter improve results?
>> It's not that tiles with loads of sea actually failed.
>> 
>> On 03.05.2014 09:00, Gerd Petermann wrote:
>> 
>> Hi all,
>> 
>> I've coded a quick hack which seems to improve the ratios.
>> On the other hand, I don't see these large differences between
>> smallest and largest img file.
>> What part of the world should I try?
>> Do you use the precomp-sea parameter in splitter?
>> 
>> Gerd
>> 
>> -------------------------
>> Date: Wed, 30 Apr 2014 14:59:32 +0200
>> From: extremecarver at gmail.com
>> To: mkgmap-dev at lists.mkgmap.org.uk
>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>> 
>> if that doesn't seriously (more than 30-40%) slow down the
>> splitter, I assume it would be much better...
>> 
>> On 30.04.2014 14:06, Gerd Petermann wrote:
>> 
>> Hi,
>> 
>> if I got that right the number of nodes is not
>> highly correlated to the img size, so the max-nodes
>> value is not a good estimate.
>> 
>> I assume the reason is that nodes which belong to
>> roads produce a lot more bytes
>> in the img file compared to nodes which are parts
>> of shapes or other non-routable ways, not talking
>> about nodes which are simply ignored by the style.
>> 
>> So, a possible solution in splitter could be to parse
>> the ways before reading the nodes and save all nodeids
>> which belong to ways with highway=*.
>> If these nodes are refered by more than one way with highway=*
>> we assume that they will be routing nodes.
>> These special nodes could be counted e.g. 10 times to
>> give a better estimate.
>> 
>> Gerd
>> 
>>> Date: Wed, 30 Apr 2014 13:36:19 +0200
>>> From: osm at na1400.info
>>> To: mkgmap-dev at lists.mkgmap.org.uk
>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>> 
>>> Multithreading the tile rendering for a single map is indeed a
>> bit
>>> difficult and I gave it up because you need to keep track which
>> image
>>> id's are already in use. Since I provide multiple maps the
>> work-around
>>> is running a few scripts parallel, which is also a crude form of
>>> multithreading.
>>> 
>>> The script language is PHP and it doesn't run on Windows without
>> some
>>> changes ('/' vs '' in paths, 'rm -rf', that sort of stuff). Never
>> tried it.
>>> 
>>> To get a better optimum in file size, using the process I
>> described
>>> earlier, you could start off with a huge --max-nodes setting and
>> then
>>> 'search' for the highest --max-nodes that works for each specific
>> area.
>>> 
>>> On 30/04/2014 11:49, Felix Hartmann wrote:
>>> > I would love if there was a possibility that you pass the used
>> max-nodes
>>> > value to mkgmap.
>>> > When mkgmap is compiling the maps, then after the .img is
>> created it
>>> > should check
>>> > a) did it crash due to too many max-nodes
>>> > b) for me not important - but for others with very old GPS,
>> etrex 10,
>>> > ---> is tile bigger than X (usually 8) MB.
>>> >
>>> > if a) or b) true, then pass the file back to splitter and split
>> with 60%
>>> > of maxnodes - and compile the resulting .img files again.
>> Should it fail
>>> > again, use 40%, again 25%... Sometimes there are awful tiles,
>> that need
>>> > supersmall max-nodes till they compile, however lately (last
>> 1-2 years)
>>> > I never encountered them anymore. I think that happened rather
>> due to a
>>> > but in splitter/mgkmap that is fixed now.
>>> >
>>> > okay, you could also do this with a script, but it gets rather
>>> > complicated to multithread it (you need to wait till mgkmap
>> finished
>>> > compiling all .img files - and run mkgmap first without address
>> index to
>>> > save time) and do some clever routines on making sure that the
>> map id
>>> > (e.g. 6340????.img) stay correct. Even more complicated to have
>>> > consequent map id...
>>> >
>>> > For europe with a fixed max-node I get tiles from 1.9MB up to
>> 18MB.
>>> > That's factor 9 - so it's huge...
>>> > If I could narrows that down easily to 8-18MB - without getting
>> tiles
>>> > crashing due to too high max-nodes values, that would be sweet.
>>> >
>>> >
>>> >
>>> > As for the scripts - would they run on windows too? - What
>> programming
>>> > language are they in?
>>> >
>>> >
>>> > On 29.04.2014 21:39, osm at na1400.info wrote:
>>> >> Oh, and ofcourse anyone interested can get my scripts, send an
>> email.
>>> >> They'll be on Github someday anyway.
>>> >>
>>> >>
>>> >> On 2014-04-29 20:37, Gerd Petermann wrote:
>>> >>> Hi Lambertus,
>>> >>>
>>> >>> okay, if I got that right you finally get *.img files with a
>> size
>>> >>> near (but below) 8MB, so maybe Henning can use that script,
>> too.
>>> >>>
>>> >>> If you do that for e.g. Germany, how small is tpically the
>> smallest
>>> >>> *.img file ?
>>> >>> Is it probably near 4 MB?
>>> >>>
>>> >>> Gerd
>>> >>>
>>> >>>> To: mkgmap-dev at lists.mkgmap.org.uk
>>> >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200
>>> >>>> From: osm at na1400.info
>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>> >>>>
>>> >>>> These are the direct results from Splitter. The format is
>> o5m, both
>>> >>>> input as output. Splitter version is: r321.
>>> >>>>
>>> >>>> For this test I split the original source with
>> --max-nodes=8000000.
>>> >>> Then
>>> >>>> I render the initial tiles, when the result is larger than
>> 8MB it's
>>> >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)).
>> The
>>> >>> initial
>>> >>>> source files are ~70MB (o5m) and after several subsplits the
>> two
>>> >>> *.img
>>> >>>> are < 8MB. During this process --max-nodes has been reduced
>> to e.g.
>>> >>>> 7500000 and the source file is split up in two o5m files of
>> about
>>> >>> 37MB.
>>> >>>>
>>> >>>> I can upload an example source file and it's two subsplit
>> siblings
>>> >>> if
>>> >>>> you want.
>>> >>>>
>>> >>>>
>>> >>>> On 2014-04-29 19:38, GerdP wrote:
>>> >>>> > Hi Lambertus,
>>> >>>> >
>>> >>>> > that's interesting. Are these the img file sizes or the
>> osm file
>>> >>> sizes?
>>> >>>> >
>>> >>>> > Gerd
>>> >>>> >
>>> >>>> >
>>> >>>> > Lambertus wrote
>>> >>>> >> Unfortunately I cannot confirm that. Below is a bit of
>> logging
>>> >>> from my
>>> >>>> >> script:
>>> >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1
>>> >>> (36852845)
>>> >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1
>>> >>> (37222739)
>>> >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1
>>> >>> (38098627)
>>> >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1
>>> >>> (39261197)
>>> >>>> >>
>>> >>>> >> The original files above contain contour data, the
>> filesize is
>>> >>> between
>>> >>>> >> brackets. As you can see both resulting file are
>> approximately
>>> >>> the
>>> >>>> >> same
>>> >>>> >> size.
>>> >>>> >>
>>> >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote:
>>> >>>> >>> Hi Lambertus,
>>> >>>> >>>
>>> >>>> >>> and I guess that even after this optimization you will
>>> >>>> >>> see a factor 3 or higher between the largest tile and
>> the
>>> >>> smallest.
>>> >>>> >>> Can you confirm that?
>>> >>>> >>>
>>> >>>> >>> Gerd
>>> >>>> >>>
>>> >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200
>>> >>>> >>>> From:
>>> >>>> >
>>> >>>> >> osm@
>>> >>>> >
>>> >>>> >>>> To:
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>> >>>> >>>>
>>> >>>> >>>> Num-tiles=x would indeed be better for this specific
>> need.
>>> >>>> >>>>
>>> >>>> >>>> It is my experience that it regularly takes multiple
>> calls to
>>> >>>> >>> Splitter
>>> >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by
>> 100k for
>>> >>> each
>>> >>>> >>>> sub-split attempt. This is what I currently do to get
>> an
>>> >>> optimum in
>>> >>>> >>>> tile-size vs total number of tiles.
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote:
>>> >>>> >>>> > Hi Lambertus,
>>> >>>> >>>> >
>>> >>>> >>>> > that sounds like a possible change in splitter:
>>> >>>> >>>> > Instead of specifying max-nodes you may specify
>> --num-tiles=x
>>> >>>> >>>> > and splitter will try to find a split that produces
>> excactly
>>> >>> x
>>> >>>> >>> tiles
>>> >>>> >>>> > which are not too narrow and have a node number which
>> is not
>>> >>>> >>>> > too far from the average (but still aligned to a
>> multiple of
>>> >>> map
>>> >>>> >>> units
>>> >>>> >>>> > as now).
>>> >>>> >>>> > So, for your script that means you don't have to find
>> the
>>> >>>> >>> max-nodes
>>> >>>> >>>> > value.
>>> >>>> >>>> >
>>> >>>> >>>> > I'll think about this again...
>>> >>>> >>>> >
>>> >>>> >>>> > Gerd
>>> >>>> >>>> >
>>> >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200
>>> >>>> >>>> > > From:
>>> >>>> >
>>> >>>> >> osm@
>>> >>>> >
>>> >>>> >>>> > > To:
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>> >>>> >>>> > >
>>> >>>> >>>> > > While this possibly can be solved in Splitter or
>> Mkgmap, it
>>> >>>> >>> could also
>>> >>>> >>>> > > be solved by your build-script when you add a
>> maximum tile
>>> >>> size
>>> >>>> >>> check
>>> >>>> >>>> > > and re-split (with a lower number of max-nodes)
>> until you
>>> >>> get
>>> >>>> >>> two or
>>> >>>> >>>> > > more sub-tiles. Granted, this adds complexity to
>> the script
>>> >>> but
>>> >>>> >>> it works
>>> >>>> >>>> > > well for me.
>>> >>>> >>>> > >
>>> >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote:
>>> >>>> >>>> > > > Hi Gerd,
>>> >>>> >>>> > > >
>>> >>>> >>>> > > > I would like to have img-tiles which have
>> globally nearly
>>> >>> the
>>> >>>> >>> same
>>> >>>> >>>> > > > filesize, so that they use the space of devices
>> like
>>> >>> eTrex 10.
>>> >>>> >>>> > > >
>>> >>>> >>>> > > > With my actual map I use globally the same value
>> for
>>> >>>> >>> max-nodes. But the
>>> >>>> >>>> > > > size of the img-tiles differ more then factor 2.
>> Eg. a
>>> >>> tile in
>>> >>>> >>> Germany
>>> >>>> >>>> > > > is between 2 and 5 mb where a tile in China is
>> about 10
>>> >>> mb. If
>>> >>>> >>> I remove
>>> >>>> >>>> > > > details, this difference will increase, because
>> in
>>> >>> Germany
>>> >>>> >>> more objects
>>> >>>> >>>> > > > will be removed from the img-tile then in China.
>>> >>>> >>>> > > >
>>> >>>> >>>> > > > Henning
>>> >>>> >>>> > > >
>>> >>>> >>>> > > > _______________________________________________
>>> >>>> >>>> > > > mkgmap-dev mailing list
>>> >>>> >>>> > > >
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >>>> > > >
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >>>> >>>> > >
>>> >>>> >>>> > > _______________________________________________
>>> >>>> >>>> > > mkgmap-dev mailing list
>>> >>>> >>>> > >
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >>>> > >
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >>>> >>>> >
>>> >>>> >>>> >
>>> >>>> >>>> > _______________________________________________
>>> >>>> >>>> > mkgmap-dev mailing list
>>> >>>> >>>> >
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> [1]
>>> >>>> >>>> >
>>> >>>> >>>>
>>> >>>> >>>> _______________________________________________
>>> >>>> >>>> mkgmap-dev mailing list
>>> >>>> >>>>
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> [1]
>>> >>>> >>>
>>> >>>> >>> _______________________________________________
>>> >>>> >>> mkgmap-dev mailing list
>>> >>>> >>>
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >>>> >> _______________________________________________
>>> >>>> >> mkgmap-dev mailing list
>>> >>>> >
>>> >>>> >> mkgmap-dev at .org
>>> >>>> >
>>> >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > --
>>> >>>> > View this message in context:
>>> >>>> >
>>> >>>
>> 
> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
>> [3]
>>> >>>> > Sent from the Mkgmap Development mailing list archive at
>>> >>> Nabble.com.
>>> >>>> > _______________________________________________
>>> >>>> > mkgmap-dev mailing list
>>> >>>> > mkgmap-dev at lists.mkgmap.org.uk
>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >>>> _______________________________________________
>>> >>>> mkgmap-dev mailing list
>>> >>>> mkgmap-dev at lists.mkgmap.org.uk
>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >>>
>>> >>> _______________________________________________
>>> >>> mkgmap-dev mailing list
>>> >>> mkgmap-dev at lists.mkgmap.org.uk
>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >> _______________________________________________
>>> >> mkgmap-dev mailing list
>>> >> mkgmap-dev at lists.mkgmap.org.uk
>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> >
>>> 
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>> --
>> keep on biking and discovering new trails
>> 
>> Felix
>> openmtbmap.org & www.velomap.org [2]
>> 
>> _______________________________________________ mkgmap-dev mailing
>> list mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
> 
> --
> keep on biking and discovering new trails
> 
> Felix
> openmtbmap.org & www.velomap.org [2]
> 
>  _______________________________________________ mkgmap-dev mailing
> list mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
> 
>  _______________________________________________ mkgmap-dev mailing
> list mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
> 
> _______________________________________________ mkgmap-dev mailing
> list mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> 
> Links:
> ------
> [1] http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> [2] http://www.velomap.org
> [3] 
> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
> [4] http://files.mkgmap.org.uk/download/206/splitter.jar
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev