[mkgmap-dev] mkgmap ToDo list

Mon May 5 20:47:03 BST 2014

Hi Lambertus,

the algo is quite simple. The max-nodes parameter is ignored, splitter
calculates an initial max-nodes value by dividing the total number of nodes
by
the wanted number of tiles. Then the normal split algo is used. If the
result has too many tiles, the max-nodes value is increased, if it is too
low,
it is decreased. This is repeated until either a solution is found or 
no max-nodes value is found that results in the number of tiles.
In other words: the tiles are as "equally sized" as before. 

Gerd

Lambertus wrote
> Thanks Gerd, seems to be working great!
> 
> Two questions:
> - How does this interact with the --max-nodes setting?
> - Could it be used for an initial split of the entire planet? I.e., set 
> the number of tiles I want and have splitter output equally sized tiles? 
> (I know I could just try, but the servers are busy)
> 
> 
> On 2014-05-05 09:28, Gerd Petermann wrote:
>> Hi Lambertus,
>> 
>> attached is a patch that implements the new option:
>>  --num-tiles A target value that is used when no split-file is given.
>>  Splitting is done so that the given number of tiles is
>>  produced. The max-nodes value is ignored if this option is
>>  given.
>> A compiled binary is here:
>> 
>> http://files.mkgmap.org.uk/download/206/splitter.jar [4]
>> 
>> Please let me know if it works as expected.
>> 
>> Gerd
>> 
>> -------------------------
>> Date: Sun, 4 May 2014 22:20:20 +0200
>> From: 

> osm@

>> To: 

> mkgmap-dev at .org

>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>> 
>>  Great, many thanks. I'm currently trying to create as large as
>> (automatically) possible contour tiles and a lot of time is put in
>> subsplitting a tile again and again until two subtiles appear. This
>> new option would save much time.
>> 
>> On 05/04/2014 10:13 PM, Gerd Petermann wrote:
>> 
>>> Hi Lambertus,
>>> 
>>> yes, I want to code that tomorrow. I prefer to make it "give me n
>>> tiles", but if that turns out to be
>>> difficult I try the simple version. I can think of scenarios where
>>> it is not possible to create exactly
>>> n tiles.
>>> 
>>> Gerd
>>> 
>>> -------------------------
>>> Date: Sun, 4 May 2014 22:08:32 +0200
>>> From: 

> osm@

>>> To: 

> mkgmap-dev at .org

>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>> 
>>> Is the 'just give me two tiles' option for splitter on the to-do
>>> list? I'd really appreciate such a function.
>>> 
>>> On 05/04/2014 09:36 PM, Gerd Petermann wrote:
>>> 
>>> Hi Felix,
>>> 
>>>> would using the precomp-sea parameter for splitter improve
>>> results? It's not that tiles with loads of sea actually failed.
>>> 
>>> good question. I've coded that parameter in splitter before we did
>>> some important changes in mkgmap.
>>> I think it is needed when you use a polygon file in
>>> combination with an imput file that doesn't cover the polygon.
>>> If the polygon covers a costline with a lot of islands, but the
>>> input file doesn't contain the data,
>>> splitter might create large tiles covering these "empty" areas.
>>> Later, in mkgmap, the empty
>>> areas are filled with the precompiled-sea data, and that can cause
>>> too large files.
>>> 
>>> BTW: The quick hack turned out to be useless. It worked better in
>>> some areas and worse in others.
>>> I'll look again at this later next week.
>>> 
>>> Gerd
>>> 
>>> -------------------------
>>> Date: Sun, 4 May 2014 21:21:59 +0200
>>> From: 

> extremecarver@

>>> To: 

> mkgmap-dev at .org

>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>> 
>>> No - I use splitter like this - with maxnodes depending on area - I
>>> usually just reduced it after failures to compile a tile.:
>>> java -Xms4000m -Xmx10400m -jar c:openmtbmapsplitter.jar
>>> --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete
>>> --max-areas=1524 --geonames-file=cities5000 --description=%country%
>>> --mapid=%FID%0000
>>> c:OpenMTBMaposmpbf_geofabrik%country%-latest.osm.pbf
>>> would using the precomp-sea parameter for splitter improve results?
>>> It's not that tiles with loads of sea actually failed.
>>> 
>>> On 03.05.2014 09:00, Gerd Petermann wrote:
>>> 
>>> Hi all,
>>> 
>>> I've coded a quick hack which seems to improve the ratios.
>>> On the other hand, I don't see these large differences between
>>> smallest and largest img file.
>>> What part of the world should I try?
>>> Do you use the precomp-sea parameter in splitter?
>>> 
>>> Gerd
>>> 
>>> -------------------------
>>> Date: Wed, 30 Apr 2014 14:59:32 +0200
>>> From: 

> extremecarver@

>>> To: 

> mkgmap-dev at .org

>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>> 
>>> if that doesn't seriously (more than 30-40%) slow down the
>>> splitter, I assume it would be much better...
>>> 
>>> On 30.04.2014 14:06, Gerd Petermann wrote:
>>> 
>>> Hi,
>>> 
>>> if I got that right the number of nodes is not
>>> highly correlated to the img size, so the max-nodes
>>> value is not a good estimate.
>>> 
>>> I assume the reason is that nodes which belong to
>>> roads produce a lot more bytes
>>> in the img file compared to nodes which are parts
>>> of shapes or other non-routable ways, not talking
>>> about nodes which are simply ignored by the style.
>>> 
>>> So, a possible solution in splitter could be to parse
>>> the ways before reading the nodes and save all nodeids
>>> which belong to ways with highway=*.
>>> If these nodes are refered by more than one way with highway=*
>>> we assume that they will be routing nodes.
>>> These special nodes could be counted e.g. 10 times to
>>> give a better estimate.
>>> 
>>> Gerd
>>> 
>>>> Date: Wed, 30 Apr 2014 13:36:19 +0200
>>>> From: 

> osm@

>>>> To: 

> mkgmap-dev at .org

>>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>>> 
>>>> Multithreading the tile rendering for a single map is indeed a
>>> bit
>>>> difficult and I gave it up because you need to keep track which
>>> image
>>>> id's are already in use. Since I provide multiple maps the
>>> work-around
>>>> is running a few scripts parallel, which is also a crude form of
>>>> multithreading.
>>>> 
>>>> The script language is PHP and it doesn't run on Windows without
>>> some
>>>> changes ('/' vs '' in paths, 'rm -rf', that sort of stuff). Never
>>> tried it.
>>>> 
>>>> To get a better optimum in file size, using the process I
>>> described
>>>> earlier, you could start off with a huge --max-nodes setting and
>>> then
>>>> 'search' for the highest --max-nodes that works for each specific
>>> area.
>>>> 
>>>> On 30/04/2014 11:49, Felix Hartmann wrote:
>>>> > I would love if there was a possibility that you pass the used
>>> max-nodes
>>>> > value to mkgmap.
>>>> > When mkgmap is compiling the maps, then after the .img is
>>> created it
>>>> > should check
>>>> > a) did it crash due to too many max-nodes
>>>> > b) for me not important - but for others with very old GPS,
>>> etrex 10,
>>>> > ---> is tile bigger than X (usually 8) MB.
>>>> >
>>>> > if a) or b) true, then pass the file back to splitter and split
>>> with 60%
>>>> > of maxnodes - and compile the resulting .img files again.
>>> Should it fail
>>>> > again, use 40%, again 25%... Sometimes there are awful tiles,
>>> that need
>>>> > supersmall max-nodes till they compile, however lately (last
>>> 1-2 years)
>>>> > I never encountered them anymore. I think that happened rather
>>> due to a
>>>> > but in splitter/mgkmap that is fixed now.
>>>> >
>>>> > okay, you could also do this with a script, but it gets rather
>>>> > complicated to multithread it (you need to wait till mgkmap
>>> finished
>>>> > compiling all .img files - and run mkgmap first without address
>>> index to
>>>> > save time) and do some clever routines on making sure that the
>>> map id
>>>> > (e.g. 6340????.img) stay correct. Even more complicated to have
>>>> > consequent map id...
>>>> >
>>>> > For europe with a fixed max-node I get tiles from 1.9MB up to
>>> 18MB.
>>>> > That's factor 9 - so it's huge...
>>>> > If I could narrows that down easily to 8-18MB - without getting
>>> tiles
>>>> > crashing due to too high max-nodes values, that would be sweet.
>>>> >
>>>> >
>>>> >
>>>> > As for the scripts - would they run on windows too? - What
>>> programming
>>>> > language are they in?
>>>> >
>>>> >
>>>> > On 29.04.2014 21:39, 

> osm@

>  wrote:
>>>> >> Oh, and ofcourse anyone interested can get my scripts, send an
>>> email.
>>>> >> They'll be on Github someday anyway.
>>>> >>
>>>> >>
>>>> >> On 2014-04-29 20:37, Gerd Petermann wrote:
>>>> >>> Hi Lambertus,
>>>> >>>
>>>> >>> okay, if I got that right you finally get *.img files with a
>>> size
>>>> >>> near (but below) 8MB, so maybe Henning can use that script,
>>> too.
>>>> >>>
>>>> >>> If you do that for e.g. Germany, how small is tpically the
>>> smallest
>>>> >>> *.img file ?
>>>> >>> Is it probably near 4 MB?
>>>> >>>
>>>> >>> Gerd
>>>> >>>
>>>> >>>> To: 

> mkgmap-dev at .org

>>>> >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200
>>>> >>>> From: 

> osm@

>>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>>> >>>>
>>>> >>>> These are the direct results from Splitter. The format is
>>> o5m, both
>>>> >>>> input as output. Splitter version is: r321.
>>>> >>>>
>>>> >>>> For this test I split the original source with
>>> --max-nodes=8000000.
>>>> >>> Then
>>>> >>>> I render the initial tiles, when the result is larger than
>>> 8MB it's
>>>> >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)).
>>> The
>>>> >>> initial
>>>> >>>> source files are ~70MB (o5m) and after several subsplits the
>>> two
>>>> >>> *.img
>>>> >>>> are < 8MB. During this process --max-nodes has been reduced
>>> to e.g.
>>>> >>>> 7500000 and the source file is split up in two o5m files of
>>> about
>>>> >>> 37MB.
>>>> >>>>
>>>> >>>> I can upload an example source file and it's two subsplit
>>> siblings
>>>> >>> if
>>>> >>>> you want.
>>>> >>>>
>>>> >>>>
>>>> >>>> On 2014-04-29 19:38, GerdP wrote:
>>>> >>>> > Hi Lambertus,
>>>> >>>> >
>>>> >>>> > that's interesting. Are these the img file sizes or the
>>> osm file
>>>> >>> sizes?
>>>> >>>> >
>>>> >>>> > Gerd
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > Lambertus wrote
>>>> >>>> >> Unfortunately I cannot confirm that. Below is a bit of
>>> logging
>>>> >>> from my
>>>> >>>> >> script:
>>>> >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1
>>>> >>> (36852845)
>>>> >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1
>>>> >>> (37222739)
>>>> >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1
>>>> >>> (38098627)
>>>> >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1
>>>> >>> (39261197)
>>>> >>>> >>
>>>> >>>> >> The original files above contain contour data, the
>>> filesize is
>>>> >>> between
>>>> >>>> >> brackets. As you can see both resulting file are
>>> approximately
>>>> >>> the
>>>> >>>> >> same
>>>> >>>> >> size.
>>>> >>>> >>
>>>> >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote:
>>>> >>>> >>> Hi Lambertus,
>>>> >>>> >>>
>>>> >>>> >>> and I guess that even after this optimization you will
>>>> >>>> >>> see a factor 3 or higher between the largest tile and
>>> the
>>>> >>> smallest.
>>>> >>>> >>> Can you confirm that?
>>>> >>>> >>>
>>>> >>>> >>> Gerd
>>>> >>>> >>>
>>>> >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200
>>>> >>>> >>>> From:
>>>> >>>> >
>>>> >>>> >> osm@
>>>> >>>> >
>>>> >>>> >>>> To:
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>>> >>>> >>>>
>>>> >>>> >>>> Num-tiles=x would indeed be better for this specific
>>> need.
>>>> >>>> >>>>
>>>> >>>> >>>> It is my experience that it regularly takes multiple
>>> calls to
>>>> >>>> >>> Splitter
>>>> >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by
>>> 100k for
>>>> >>> each
>>>> >>>> >>>> sub-split attempt. This is what I currently do to get
>>> an
>>>> >>> optimum in
>>>> >>>> >>>> tile-size vs total number of tiles.
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote:
>>>> >>>> >>>> > Hi Lambertus,
>>>> >>>> >>>> >
>>>> >>>> >>>> > that sounds like a possible change in splitter:
>>>> >>>> >>>> > Instead of specifying max-nodes you may specify
>>> --num-tiles=x
>>>> >>>> >>>> > and splitter will try to find a split that produces
>>> excactly
>>>> >>> x
>>>> >>>> >>> tiles
>>>> >>>> >>>> > which are not too narrow and have a node number which
>>> is not
>>>> >>>> >>>> > too far from the average (but still aligned to a
>>> multiple of
>>>> >>> map
>>>> >>>> >>> units
>>>> >>>> >>>> > as now).
>>>> >>>> >>>> > So, for your script that means you don't have to find
>>> the
>>>> >>>> >>> max-nodes
>>>> >>>> >>>> > value.
>>>> >>>> >>>> >
>>>> >>>> >>>> > I'll think about this again...
>>>> >>>> >>>> >
>>>> >>>> >>>> > Gerd
>>>> >>>> >>>> >
>>>> >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200
>>>> >>>> >>>> > > From:
>>>> >>>> >
>>>> >>>> >> osm@
>>>> >>>> >
>>>> >>>> >>>> > > To:
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list
>>>> >>>> >>>> > >
>>>> >>>> >>>> > > While this possibly can be solved in Splitter or
>>> Mkgmap, it
>>>> >>>> >>> could also
>>>> >>>> >>>> > > be solved by your build-script when you add a
>>> maximum tile
>>>> >>> size
>>>> >>>> >>> check
>>>> >>>> >>>> > > and re-split (with a lower number of max-nodes)
>>> until you
>>>> >>> get
>>>> >>>> >>> two or
>>>> >>>> >>>> > > more sub-tiles. Granted, this adds complexity to
>>> the script
>>>> >>> but
>>>> >>>> >>> it works
>>>> >>>> >>>> > > well for me.
>>>> >>>> >>>> > >
>>>> >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote:
>>>> >>>> >>>> > > > Hi Gerd,
>>>> >>>> >>>> > > >
>>>> >>>> >>>> > > > I would like to have img-tiles which have
>>> globally nearly
>>>> >>> the
>>>> >>>> >>> same
>>>> >>>> >>>> > > > filesize, so that they use the space of devices
>>> like
>>>> >>> eTrex 10.
>>>> >>>> >>>> > > >
>>>> >>>> >>>> > > > With my actual map I use globally the same value
>>> for
>>>> >>>> >>> max-nodes. But the
>>>> >>>> >>>> > > > size of the img-tiles differ more then factor 2.
>>> Eg. a
>>>> >>> tile in
>>>> >>>> >>> Germany
>>>> >>>> >>>> > > > is between 2 and 5 mb where a tile in China is
>>> about 10
>>>> >>> mb. If
>>>> >>>> >>> I remove
>>>> >>>> >>>> > > > details, this difference will increase, because
>>> in
>>>> >>> Germany
>>>> >>>> >>> more objects
>>>> >>>> >>>> > > > will be removed from the img-tile then in China.
>>>> >>>> >>>> > > >
>>>> >>>> >>>> > > > Henning
>>>> >>>> >>>> > > >
>>>> >>>> >>>> > > > _______________________________________________
>>>> >>>> >>>> > > > mkgmap-dev mailing list
>>>> >>>> >>>> > > >
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >>>> > > >
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >>>> >>>> > >
>>>> >>>> >>>> > > _______________________________________________
>>>> >>>> >>>> > > mkgmap-dev mailing list
>>>> >>>> >>>> > >
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >>>> > >
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >>>> >>>> >
>>>> >>>> >>>> >
>>>> >>>> >>>> > _______________________________________________
>>>> >>>> >>>> > mkgmap-dev mailing list
>>>> >>>> >>>> >
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>> [1]
>>>> >>>> >>>> >
>>>> >>>> >>>>
>>>> >>>> >>>> _______________________________________________
>>>> >>>> >>>> mkgmap-dev mailing list
>>>> >>>> >>>>
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>> [1]
>>>> >>>> >>>
>>>> >>>> >>> _______________________________________________
>>>> >>>> >>> mkgmap-dev mailing list
>>>> >>>> >>>
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >>>> >> _______________________________________________
>>>> >>>> >> mkgmap-dev mailing list
>>>> >>>> >
>>>> >>>> >> mkgmap-dev at .org
>>>> >>>> >
>>>> >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > --
>>>> >>>> > View this message in context:
>>>> >>>> >
>>>> >>>
>>> 
>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
>>> [3]
>>>> >>>> > Sent from the Mkgmap Development mailing list archive at
>>>> >>> Nabble.com.
>>>> >>>> > _______________________________________________
>>>> >>>> > mkgmap-dev mailing list
>>>> >>>> > 

> mkgmap-dev at .org

>>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >>>> _______________________________________________
>>>> >>>> mkgmap-dev mailing list
>>>> >>>> 

> mkgmap-dev at .org

>>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> mkgmap-dev mailing list
>>>> >>> 

> mkgmap-dev at .org

>>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >> _______________________________________________
>>>> >> mkgmap-dev mailing list
>>>> >> 

> mkgmap-dev at .org

>>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>>> >
>>>> 
>>>> _______________________________________________
>>>> mkgmap-dev mailing list
>>>> 

> mkgmap-dev at .org

>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> 
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> 

> mkgmap-dev at .org

>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> 
>>> --
>>> keep on biking and discovering new trails
>>> 
>>> Felix
>>> openmtbmap.org & www.velomap.org [2]
>>> 
>>> _______________________________________________ mkgmap-dev mailing
>>> list 

> mkgmap-dev at .org

>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>>> 
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> 

> mkgmap-dev at .org

>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>> --
>> keep on biking and discovering new trails
>> 
>> Felix
>> openmtbmap.org & www.velomap.org [2]
>> 
>>  _______________________________________________ mkgmap-dev mailing
>> list 

> mkgmap-dev at .org

>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>> _______________________________________________
>> mkgmap-dev mailing list
>> 

> mkgmap-dev at .org

>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>>  _______________________________________________ mkgmap-dev mailing
>> list 

> mkgmap-dev at .org

>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>> _______________________________________________
>> mkgmap-dev mailing list
>> 

> mkgmap-dev at .org

>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
>> 
>> _______________________________________________ mkgmap-dev mailing
>> list 

> mkgmap-dev at .org

>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> 
>> Links:
>> ------
>> [1] http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> [2] http://www.velomap.org
>> [3] 
>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
>> [4] http://files.mkgmap.org.uk/download/206/splitter.jar
>> 
>> _______________________________________________
>> mkgmap-dev mailing list
>> 

> mkgmap-dev at .org

>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list

> mkgmap-dev at .org

> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--
View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805281.html
Sent from the Mkgmap Development mailing list archive at Nabble.com.