[mkgmap-dev] mkgmap ToDo list
From Gerd Petermann gpetermann_muenchen at hotmail.com on Sat May 3 08:00:47 BST 2014
Hi all, I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter? Gerd Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver at gmail.com To: mkgmap-dev at lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote: Hi, if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate. I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style. So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate. Gerd > Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm at na1400.info > To: mkgmap-dev at lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm at na1400.info wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev at lists.mkgmap.org.uk > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm at na1400.info > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev at .org > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev at lists.mkgmap.org.uk > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev at lists.mkgmap.org.uk > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev at lists.mkgmap.org.uk > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev at lists.mkgmap.org.uk > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140503/6fb5923e/attachment-0001.html>
- Previous message: [mkgmap-dev] mkgmap ToDo list
- Next message: [mkgmap-dev] mkgmap ToDo list
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list