[mkgmap-dev] mkgmap ToDo list

Wed Apr 30 13:59:32 BST 2014

if that doesn't seriously (more than 30-40%) slow down the splitter, I 
assume it would be much better...
On 30.04.2014 14:06, Gerd Petermann wrote:
> Hi,
>
> if I got that right the number of nodes is not
> highly correlated to the img size, so the max-nodes
> value  is not a good estimate.
>
> I assume the reason is that nodes which belong to
> roads produce a lot more bytes
> in the img file compared to nodes which are parts
> of shapes or other non-routable ways, not talking
> about nodes which are simply ignored by the style.
>
> So, a possible solution in splitter could be to parse
> the ways before reading the nodes and save all nodeids
> which belong to ways with highway=*.
> If these nodes are refered by more than one way with highway=*
> we assume that they will be routing nodes.
> These special nodes could be counted e.g. 10 times to
> give a better estimate.
>
> Gerd
>
> > Date: Wed, 30 Apr 2014 13:36:19 +0200
> > From: osm at na1400.info
> > To: mkgmap-dev at lists.mkgmap.org.uk
> > Subject: Re: [mkgmap-dev] mkgmap ToDo list
> >
> > Multithreading the tile rendering for a single map is indeed a bit
> > difficult and I gave it up because you need to keep track which image
> > id's are already in use. Since I provide multiple maps the work-around
> > is running a few scripts parallel, which is also a crude form of
> > multithreading.
> >
> > The script language is PHP and it doesn't run on Windows without some
> > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never 
> tried it.
> >
> > To get a better optimum in file size, using the process I described
> > earlier, you could start off with a huge --max-nodes setting and then
> > 'search' for the highest --max-nodes that works for each specific area.
> >
> > On 30/04/2014 11:49, Felix Hartmann wrote:
> > > I would love if there was a possibility that you pass the used 
> max-nodes
> > > value to mkgmap.
> > > When mkgmap is compiling the maps, then after the .img is created it
> > > should check
> > > a) did it crash due to too many max-nodes
> > > b) for me not important - but for others with very old GPS, etrex 10,
> > > ---> is tile bigger than X (usually 8) MB.
> > >
> > > if a) or b) true, then pass the file back to splitter and split 
> with 60%
> > > of maxnodes - and compile the resulting .img files again. Should 
> it fail
> > > again, use 40%, again 25%... Sometimes there are awful tiles, that 
> need
> > > supersmall max-nodes till they compile, however lately (last 1-2 
> years)
> > > I never encountered them anymore. I think that happened rather due 
> to a
> > > but in splitter/mgkmap that is fixed now.
> > >
> > > okay, you could also do this with a script, but it gets rather
> > > complicated to multithread it (you need to wait till mgkmap finished
> > > compiling all .img files - and run mkgmap first without address 
> index to
> > > save time) and do some clever routines on making sure that the map id
> > > (e.g. 6340????.img) stay correct. Even more complicated to have
> > > consequent map id...
> > >
> > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB.
> > > That's factor 9 - so it's huge...
> > > If I could narrows that down easily to 8-18MB - without getting tiles
> > > crashing due to too high max-nodes values, that would be sweet.
> > >
> > >
> > >
> > > As for the scripts - would they run on windows too? - What programming
> > > language are they in?
> > >
> > >
> > > On 29.04.2014 21:39, osm at na1400.info wrote:
> > >> Oh, and ofcourse anyone interested can get my scripts, send an email.
> > >> They'll be on Github someday anyway.
> > >>
> > >>
> > >> On 2014-04-29 20:37, Gerd Petermann wrote:
> > >>> Hi Lambertus,
> > >>>
> > >>> okay, if I got that right you finally get *.img files with a size
> > >>> near (but below) 8MB, so maybe Henning can use that script, too.
> > >>>
> > >>> If you do that for e.g. Germany, how small is tpically the smallest
> > >>> *.img file ?
> > >>> Is it probably near 4 MB?
> > >>>
> > >>> Gerd
> > >>>
> > >>>> To: mkgmap-dev at lists.mkgmap.org.uk
> > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200
> > >>>> From: osm at na1400.info
> > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
> > >>>>
> > >>>> These are the direct results from Splitter. The format is o5m, both
> > >>>> input as output. Splitter version is: r321.
> > >>>>
> > >>>> For this test I split the original source with --max-nodes=8000000.
> > >>> Then
> > >>>> I render the initial tiles, when the result is larger than 8MB it's
> > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The
> > >>> initial
> > >>>> source files are ~70MB (o5m) and after several subsplits the two
> > >>> *.img
> > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g.
> > >>>> 7500000 and the source file is split up in two o5m files of about
> > >>> 37MB.
> > >>>>
> > >>>> I can upload an example source file and it's two subsplit siblings
> > >>> if
> > >>>> you want.
> > >>>>
> > >>>>
> > >>>> On 2014-04-29 19:38, GerdP wrote:
> > >>>> > Hi Lambertus,
> > >>>> >
> > >>>> > that's interesting. Are these the img file sizes or the osm file
> > >>> sizes?
> > >>>> >
> > >>>> > Gerd
> > >>>> >
> > >>>> >
> > >>>> > Lambertus wrote
> > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging
> > >>> from my
> > >>>> >> script:
> > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1
> > >>> (36852845)
> > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1
> > >>> (37222739)
> > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1
> > >>> (38098627)
> > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1
> > >>> (39261197)
> > >>>> >>
> > >>>> >> The original files above contain contour data, the filesize is
> > >>> between
> > >>>> >> brackets. As you can see both resulting file are approximately
> > >>> the
> > >>>> >> same
> > >>>> >> size.
> > >>>> >>
> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote:
> > >>>> >>> Hi Lambertus,
> > >>>> >>>
> > >>>> >>> and I guess that even after this optimization you will
> > >>>> >>> see a factor 3 or higher between the largest tile and the
> > >>> smallest.
> > >>>> >>> Can you confirm that?
> > >>>> >>>
> > >>>> >>> Gerd
> > >>>> >>>
> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200
> > >>>> >>>> From:
> > >>>> >
> > >>>> >> osm@
> > >>>> >
> > >>>> >>>> To:
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list
> > >>>> >>>>
> > >>>> >>>> Num-tiles=x would indeed be better for this specific need.
> > >>>> >>>>
> > >>>> >>>> It is my experience that it regularly takes multiple calls to
> > >>>> >>> Splitter
> > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for
> > >>> each
> > >>>> >>>> sub-split attempt. This is what I currently do to get an
> > >>> optimum in
> > >>>> >>>> tile-size vs total number of tiles.
> > >>>> >>>>
> > >>>> >>>>
> > >>>> >>>>
> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote:
> > >>>> >>>> > Hi Lambertus,
> > >>>> >>>> >
> > >>>> >>>> > that sounds like a possible change in splitter:
> > >>>> >>>> > Instead of specifying max-nodes you may specify 
> --num-tiles=x
> > >>>> >>>> > and splitter will try to find a split that produces excactly
> > >>> x
> > >>>> >>> tiles
> > >>>> >>>> > which are not too narrow and have a node number which is not
> > >>>> >>>> > too far from the average (but still aligned to a multiple of
> > >>> map
> > >>>> >>> units
> > >>>> >>>> > as now).
> > >>>> >>>> > So, for your script that means you don't have to find the
> > >>>> >>> max-nodes
> > >>>> >>>> > value.
> > >>>> >>>> >
> > >>>> >>>> > I'll think about this again...
> > >>>> >>>> >
> > >>>> >>>> > Gerd
> > >>>> >>>> >
> > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200
> > >>>> >>>> > > From:
> > >>>> >
> > >>>> >> osm@
> > >>>> >
> > >>>> >>>> > > To:
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list
> > >>>> >>>> > >
> > >>>> >>>> > > While this possibly can be solved in Splitter or 
> Mkgmap, it
> > >>>> >>> could also
> > >>>> >>>> > > be solved by your build-script when you add a maximum tile
> > >>> size
> > >>>> >>> check
> > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you
> > >>> get
> > >>>> >>> two or
> > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the 
> script
> > >>> but
> > >>>> >>> it works
> > >>>> >>>> > > well for me.
> > >>>> >>>> > >
> > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote:
> > >>>> >>>> > > > Hi Gerd,
> > >>>> >>>> > > >
> > >>>> >>>> > > > I would like to have img-tiles which have globally 
> nearly
> > >>> the
> > >>>> >>> same
> > >>>> >>>> > > > filesize, so that they use the space of devices like
> > >>> eTrex 10.
> > >>>> >>>> > > >
> > >>>> >>>> > > > With my actual map I use globally the same value for
> > >>>> >>> max-nodes. But the
> > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a
> > >>> tile in
> > >>>> >>> Germany
> > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10
> > >>> mb. If
> > >>>> >>> I remove
> > >>>> >>>> > > > details, this difference will increase, because in
> > >>> Germany
> > >>>> >>> more objects
> > >>>> >>>> > > > will be removed from the img-tile then in China.
> > >>>> >>>> > > >
> > >>>> >>>> > > > Henning
> > >>>> >>>> > > >
> > >>>> >>>> > > > _______________________________________________
> > >>>> >>>> > > > mkgmap-dev mailing list
> > >>>> >>>> > > >
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>> >>>> > >
> > >>>> >>>> > > _______________________________________________
> > >>>> >>>> > > mkgmap-dev mailing list
> > >>>> >>>> > >
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>> >>>> >
> > >>>> >>>> >
> > >>>> >>>> > _______________________________________________
> > >>>> >>>> > mkgmap-dev mailing list
> > >>>> >>>> >
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>> >>>> >
> > >>>> >>>>
> > >>>> >>>> _______________________________________________
> > >>>> >>>> mkgmap-dev mailing list
> > >>>> >>>>
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>> >>>
> > >>>> >>> _______________________________________________
> > >>>> >>> mkgmap-dev mailing list
> > >>>> >>>
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>> >> _______________________________________________
> > >>>> >> mkgmap-dev mailing list
> > >>>> >
> > >>>> >> mkgmap-dev at .org
> > >>>> >
> > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> > --
> > >>>> > View this message in context:
> > >>>> >
> > >>> 
> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
> > >>>> > Sent from the Mkgmap Development mailing list archive at
> > >>> Nabble.com.
> > >>>> > _______________________________________________
> > >>>> > mkgmap-dev mailing list
> > >>>> > mkgmap-dev at lists.mkgmap.org.uk
> > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>> _______________________________________________
> > >>>> mkgmap-dev mailing list
> > >>>> mkgmap-dev at lists.mkgmap.org.uk
> > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >>>
> > >>> _______________________________________________
> > >>> mkgmap-dev mailing list
> > >>> mkgmap-dev at lists.mkgmap.org.uk
> > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >> _______________________________________________
> > >> mkgmap-dev mailing list
> > >> mkgmap-dev at lists.mkgmap.org.uk
> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-- 
keep on biking and discovering new trails

Felix
openmtbmap.org & www.velomap.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140430/042886da/attachment-0001.html>