[mkgmap-dev] mkgmap ToDo list
From Lambertus osm at na1400.info on Sun May 4 21:08:32 BST 2014
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function. On 05/04/2014 09:36 PM, Gerd Petermann wrote: > Hi Felix, > > > would using the precomp-sea parameter for splitter improve results? > It's not that tiles with loads of sea actually failed. > > good question. I've coded that parameter in splitter before we did > some important changes in mkgmap. > I think it is needed when you use a polygon file in > combination with an imput file that doesn't cover the polygon. > If the polygon covers a costline with a lot of islands, but the input > file doesn't contain the data, > splitter might create large tiles covering these "empty" areas. Later, > in mkgmap, the empty > areas are filled with the precompiled-sea data, and that can cause too > large files. > > BTW: The quick hack turned out to be useless. It worked better in some > areas and worse in others. > I'll look again at this later next week. > > Gerd > > > > ------------------------------------------------------------------------ > Date: Sun, 4 May 2014 21:21:59 +0200 > From: extremecarver at gmail.com > To: mkgmap-dev at lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > No - I use splitter like this - with maxnodes depending on area - I > usually just reduced it after failures to compile a tile.: > java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar > --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete > --max-areas=1524 --geonames-file=cities5000 --description=%country% > --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf > would using the precomp-sea parameter for splitter improve results? > It's not that tiles with loads of sea actually failed. > > On 03.05.2014 09:00, Gerd Petermann wrote: > > Hi all, > > I've coded a quick hack which seems to improve the ratios. > On the other hand, I don't see these large differences between > smallest and largest img file. > What part of the world should I try? > Do you use the precomp-sea parameter in splitter? > > Gerd > > ------------------------------------------------------------------------ > Date: Wed, 30 Apr 2014 14:59:32 +0200 > From: extremecarver at gmail.com <mailto:extremecarver at gmail.com> > To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > if that doesn't seriously (more than 30-40%) slow down the > splitter, I assume it would be much better... > On 30.04.2014 14:06, Gerd Petermann wrote: > > Hi, > > if I got that right the number of nodes is not > highly correlated to the img size, so the max-nodes > value is not a good estimate. > > I assume the reason is that nodes which belong to > roads produce a lot more bytes > in the img file compared to nodes which are parts > of shapes or other non-routable ways, not talking > about nodes which are simply ignored by the style. > > So, a possible solution in splitter could be to parse > the ways before reading the nodes and save all nodeids > which belong to ways with highway=*. > If these nodes are refered by more than one way with highway=* > we assume that they will be routing nodes. > These special nodes could be counted e.g. 10 times to > give a better estimate. > > Gerd > > > Date: Wed, 30 Apr 2014 13:36:19 +0200 > > From: osm at na1400.info <mailto:osm at na1400.info> > > To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > > > Multithreading the tile rendering for a single map is indeed > a bit > > difficult and I gave it up because you need to keep track > which image > > id's are already in use. Since I provide multiple maps the > work-around > > is running a few scripts parallel, which is also a crude > form of > > multithreading. > > > > The script language is PHP and it doesn't run on Windows > without some > > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). > Never tried it. > > > > To get a better optimum in file size, using the process I > described > > earlier, you could start off with a huge --max-nodes setting > and then > > 'search' for the highest --max-nodes that works for each > specific area. > > > > On 30/04/2014 11:49, Felix Hartmann wrote: > > > I would love if there was a possibility that you pass the > used max-nodes > > > value to mkgmap. > > > When mkgmap is compiling the maps, then after the .img is > created it > > > should check > > > a) did it crash due to too many max-nodes > > > b) for me not important - but for others with very old > GPS, etrex 10, > > > ---> is tile bigger than X (usually 8) MB. > > > > > > if a) or b) true, then pass the file back to splitter and > split with 60% > > > of maxnodes - and compile the resulting .img files again. > Should it fail > > > again, use 40%, again 25%... Sometimes there are awful > tiles, that need > > > supersmall max-nodes till they compile, however lately > (last 1-2 years) > > > I never encountered them anymore. I think that happened > rather due to a > > > but in splitter/mgkmap that is fixed now. > > > > > > okay, you could also do this with a script, but it gets rather > > > complicated to multithread it (you need to wait till > mgkmap finished > > > compiling all .img files - and run mkgmap first without > address index to > > > save time) and do some clever routines on making sure that > the map id > > > (e.g. 6340????.img) stay correct. Even more complicated to > have > > > consequent map id... > > > > > > For europe with a fixed max-node I get tiles from 1.9MB up > to 18MB. > > > That's factor 9 - so it's huge... > > > If I could narrows that down easily to 8-18MB - without > getting tiles > > > crashing due to too high max-nodes values, that would be > sweet. > > > > > > > > > > > > As for the scripts - would they run on windows too? - What > programming > > > language are they in? > > > > > > > > > On 29.04.2014 21:39, osm at na1400.info > <mailto:osm at na1400.info> wrote: > > >> Oh, and ofcourse anyone interested can get my scripts, > send an email. > > >> They'll be on Github someday anyway. > > >> > > >> > > >> On 2014-04-29 20:37, Gerd Petermann wrote: > > >>> Hi Lambertus, > > >>> > > >>> okay, if I got that right you finally get *.img files > with a size > > >>> near (but below) 8MB, so maybe Henning can use that > script, too. > > >>> > > >>> If you do that for e.g. Germany, how small is tpically > the smallest > > >>> *.img file ? > > >>> Is it probably near 4 MB? > > >>> > > >>> Gerd > > >>> > > >>>> To: mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > > >>>> From: osm at na1400.info <mailto:osm at na1400.info> > > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > >>>> > > >>>> These are the direct results from Splitter. The format > is o5m, both > > >>>> input as output. Splitter version is: r321. > > >>>> > > >>>> For this test I split the original source with > --max-nodes=8000000. > > >>> Then > > >>>> I render the initial tiles, when the result is larger > than 8MB it's > > >>>> subsplit again with > --max-nodes=(8000000-(attempt*100000)). The > > >>> initial > > >>>> source files are ~70MB (o5m) and after several > subsplits the two > > >>> *.img > > >>>> are < 8MB. During this process --max-nodes has been > reduced to e.g. > > >>>> 7500000 and the source file is split up in two o5m > files of about > > >>> 37MB. > > >>>> > > >>>> I can upload an example source file and it's two > subsplit siblings > > >>> if > > >>>> you want. > > >>>> > > >>>> > > >>>> On 2014-04-29 19:38, GerdP wrote: > > >>>> > Hi Lambertus, > > >>>> > > > >>>> > that's interesting. Are these the img file sizes or > the osm file > > >>> sizes? > > >>>> > > > >>>> > Gerd > > >>>> > > > >>>> > > > >>>> > Lambertus wrote > > >>>> >> Unfortunately I cannot confirm that. Below is a bit > of logging > > >>> from my > > >>>> >> script: > > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > > >>> (36852845) > > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > > >>> (37222739) > > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > > >>> (38098627) > > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > > >>> (39261197) > > >>>> >> > > >>>> >> The original files above contain contour data, the > filesize is > > >>> between > > >>>> >> brackets. As you can see both resulting file are > approximately > > >>> the > > >>>> >> same > > >>>> >> size. > > >>>> >> > > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > > >>>> >>> Hi Lambertus, > > >>>> >>> > > >>>> >>> and I guess that even after this optimization you will > > >>>> >>> see a factor 3 or higher between the largest tile > and the > > >>> smallest. > > >>>> >>> Can you confirm that? > > >>>> >>> > > >>>> >>> Gerd > > >>>> >>> > > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > > >>>> >>>> From: > > >>>> > > > >>>> >> osm@ > > >>>> > > > >>>> >>>> To: > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > >>>> >>>> > > >>>> >>>> Num-tiles=x would indeed be better for this > specific need. > > >>>> >>>> > > >>>> >>>> It is my experience that it regularly takes > multiple calls to > > >>>> >>> Splitter > > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes > by 100k for > > >>> each > > >>>> >>>> sub-split attempt. This is what I currently do to > get an > > >>> optimum in > > >>>> >>>> tile-size vs total number of tiles. > > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > > >>>> >>>> > Hi Lambertus, > > >>>> >>>> > > > >>>> >>>> > that sounds like a possible change in splitter: > > >>>> >>>> > Instead of specifying max-nodes you may specify > --num-tiles=x > > >>>> >>>> > and splitter will try to find a split that > produces excactly > > >>> x > > >>>> >>> tiles > > >>>> >>>> > which are not too narrow and have a node number > which is not > > >>>> >>>> > too far from the average (but still aligned to a > multiple of > > >>> map > > >>>> >>> units > > >>>> >>>> > as now). > > >>>> >>>> > So, for your script that means you don't have to > find the > > >>>> >>> max-nodes > > >>>> >>>> > value. > > >>>> >>>> > > > >>>> >>>> > I'll think about this again... > > >>>> >>>> > > > >>>> >>>> > Gerd > > >>>> >>>> > > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > > >>>> >>>> > > From: > > >>>> > > > >>>> >> osm@ > > >>>> > > > >>>> >>>> > > To: > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > >>>> >>>> > > > > >>>> >>>> > > While this possibly can be solved in Splitter > or Mkgmap, it > > >>>> >>> could also > > >>>> >>>> > > be solved by your build-script when you add a > maximum tile > > >>> size > > >>>> >>> check > > >>>> >>>> > > and re-split (with a lower number of > max-nodes) until you > > >>> get > > >>>> >>> two or > > >>>> >>>> > > more sub-tiles. Granted, this adds complexity > to the script > > >>> but > > >>>> >>> it works > > >>>> >>>> > > well for me. > > >>>> >>>> > > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > > >>>> >>>> > > > Hi Gerd, > > >>>> >>>> > > > > > >>>> >>>> > > > I would like to have img-tiles which have > globally nearly > > >>> the > > >>>> >>> same > > >>>> >>>> > > > filesize, so that they use the space of > devices like > > >>> eTrex 10. > > >>>> >>>> > > > > > >>>> >>>> > > > With my actual map I use globally the same > value for > > >>>> >>> max-nodes. But the > > >>>> >>>> > > > size of the img-tiles differ more then > factor 2. Eg. a > > >>> tile in > > >>>> >>> Germany > > >>>> >>>> > > > is between 2 and 5 mb where a tile in China > is about 10 > > >>> mb. If > > >>>> >>> I remove > > >>>> >>>> > > > details, this difference will increase, > because in > > >>> Germany > > >>>> >>> more objects > > >>>> >>>> > > > will be removed from the img-tile then in China. > > >>>> >>>> > > > > > >>>> >>>> > > > Henning > > >>>> >>>> > > > > > >>>> >>>> > > > _______________________________________________ > > >>>> >>>> > > > mkgmap-dev mailing list > > >>>> >>>> > > > > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>>> > > > > >>>> >>>> > > _______________________________________________ > > >>>> >>>> > > mkgmap-dev mailing list > > >>>> >>>> > > > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>>> > > > >>>> >>>> > > > >>>> >>>> > _______________________________________________ > > >>>> >>>> > mkgmap-dev mailing list > > >>>> >>>> > > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>>> > > > >>>> >>>> > > >>>> >>>> _______________________________________________ > > >>>> >>>> mkgmap-dev mailing list > > >>>> >>>> > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >>> > > >>>> >>> _______________________________________________ > > >>>> >>> mkgmap-dev mailing list > > >>>> >>> > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> >> _______________________________________________ > > >>>> >> mkgmap-dev mailing list > > >>>> > > > >>>> >> mkgmap-dev at .org <mailto:mkgmap-dev at .org> > > >>>> > > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > -- > > >>>> > View this message in context: > > >>>> > > > >>> > http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > > >>>> > Sent from the Mkgmap Development mailing list archive at > > >>> Nabble.com. > > >>>> > _______________________________________________ > > >>>> > mkgmap-dev mailing list > > >>>> > mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>>> _______________________________________________ > > >>>> mkgmap-dev mailing list > > >>>> mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >>> > > >>> _______________________________________________ > > >>> mkgmap-dev mailing list > > >>> mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > >> _______________________________________________ > > >> mkgmap-dev mailing list > > >> mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > -- > keep on biking and discovering new trails > > Felix > openmtbmap.org &www.velomap.org <http://www.velomap.org> > > > _______________________________________________ mkgmap-dev mailing > list mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk <mailto:mkgmap-dev at lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > -- > keep on biking and discovering new trails > > Felix > openmtbmap.org &www.velomap.org <http://www.velomap.org> > > _______________________________________________ mkgmap-dev mailing > list mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140504/47441e7e/attachment-0001.html>
- Previous message: [mkgmap-dev] mkgmap ToDo list
- Next message: [mkgmap-dev] mkgmap ToDo list
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list