[mkgmap-dev] mkgmap ToDo list
From Gerd Petermann gpetermann_muenchen at hotmail.com on Sun May 4 20:36:21 BST 2014
Hi Felix,
> would using the precomp-sea parameter for splitter improve results?
It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap.
I think it is needed when you use a polygon file in
combination with an imput file that doesn't cover the polygon.
If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data,
splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty
areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others.
I'll look again at this later next week.
Gerd
Date: Sun, 4 May 2014 21:21:59 +0200
From: extremecarver at gmail.com
To: mkgmap-dev at lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I
usually just reduced it after failures to compile a tile.:
java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar
--max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete
--max-areas=1524 --geonames-file=cities5000 --description=%country%
--mapid=%FID%0000
c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf
would using the precomp-sea parameter for splitter improve results?
It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann
wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios.
On the other hand, I don't see these large differences between
smallest and largest img file.
What part of the world should I try?
Do you use the precomp-sea parameter in splitter?
Gerd
Date: Wed, 30 Apr 2014 14:59:32 +0200
From: extremecarver at gmail.com
To: mkgmap-dev at lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the
splitter, I assume it would be much better...
On 30.04.2014 14:06, Gerd
Petermann wrote:
Hi,
if I got that right the number of nodes is not
highly correlated to the img size, so the max-nodes
value is not a good estimate.
I assume the reason is that nodes which belong to
roads produce a lot more bytes
in the img file compared to nodes which are parts
of shapes or other non-routable ways, not talking
about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse
the ways before reading the nodes and save all nodeids
which belong to ways with highway=*.
If these nodes are refered by more than one way with
highway=*
we assume that they will be routing nodes.
These special nodes could be counted e.g. 10 times to
give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200
> From: osm at na1400.info
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] mkgmap ToDo list
>
> Multithreading the tile rendering for a single
map is indeed a bit
> difficult and I gave it up because you need to
keep track which image
> id's are already in use. Since I provide multiple
maps the work-around
> is running a few scripts parallel, which is also
a crude form of
> multithreading.
>
> The script language is PHP and it doesn't run on
Windows without some
> changes ('/' vs '\' in paths, 'rm -rf', that sort
of stuff). Never tried it.
>
> To get a better optimum in file size, using the
process I described
> earlier, you could start off with a huge
--max-nodes setting and then
> 'search' for the highest --max-nodes that works
for each specific area.
>
> On 30/04/2014 11:49, Felix Hartmann wrote:
> > I would love if there was a possibility that
you pass the used max-nodes
> > value to mkgmap.
> > When mkgmap is compiling the maps, then
after the .img is created it
> > should check
> > a) did it crash due to too many max-nodes
> > b) for me not important - but for others
with very old GPS, etrex 10,
> > ---> is tile bigger than X (usually 8)
MB.
> >
> > if a) or b) true, then pass the file back to
splitter and split with 60%
> > of maxnodes - and compile the resulting .img
files again. Should it fail
> > again, use 40%, again 25%... Sometimes there
are awful tiles, that need
> > supersmall max-nodes till they compile,
however lately (last 1-2 years)
> > I never encountered them anymore. I think
that happened rather due to a
> > but in splitter/mgkmap that is fixed now.
> >
> > okay, you could also do this with a script,
but it gets rather
> > complicated to multithread it (you need to
wait till mgkmap finished
> > compiling all .img files - and run mkgmap
first without address index to
> > save time) and do some clever routines on
making sure that the map id
> > (e.g. 6340????.img) stay correct. Even more
complicated to have
> > consequent map id...
> >
> > For europe with a fixed max-node I get tiles
from 1.9MB up to 18MB.
> > That's factor 9 - so it's huge...
> > If I could narrows that down easily to
8-18MB - without getting tiles
> > crashing due to too high max-nodes values,
that would be sweet.
> >
> >
> >
> > As for the scripts - would they run on
windows too? - What programming
> > language are they in?
> >
> >
> > On 29.04.2014 21:39, osm at na1400.info
wrote:
> >> Oh, and ofcourse anyone interested can
get my scripts, send an email.
> >> They'll be on Github someday anyway.
> >>
> >>
> >> On 2014-04-29 20:37, Gerd Petermann
wrote:
> >>> Hi Lambertus,
> >>>
> >>> okay, if I got that right you
finally get *.img files with a size
> >>> near (but below) 8MB, so maybe
Henning can use that script, too.
> >>>
> >>> If you do that for e.g. Germany, how
small is tpically the smallest
> >>> *.img file ?
> >>> Is it probably near 4 MB?
> >>>
> >>> Gerd
> >>>
> >>>> To: mkgmap-dev at lists.mkgmap.org.uk
> >>>> Date: Tue, 29 Apr 2014 20:30:27
+0200
> >>>> From: osm at na1400.info
> >>>> Subject: Re: [mkgmap-dev] mkgmap
ToDo list
> >>>>
> >>>> These are the direct results
from Splitter. The format is o5m, both
> >>>> input as output. Splitter
version is: r321.
> >>>>
> >>>> For this test I split the
original source with --max-nodes=8000000.
> >>> Then
> >>>> I render the initial tiles, when
the result is larger than 8MB it's
> >>>> subsplit again with
--max-nodes=(8000000-(attempt*100000)). The
> >>> initial
> >>>> source files are ~70MB (o5m) and
after several subsplits the two
> >>> *.img
> >>>> are < 8MB. During this
process --max-nodes has been reduced to e.g.
> >>>> 7500000 and the source file is
split up in two o5m files of about
> >>> 37MB.
> >>>>
> >>>> I can upload an example source
file and it's two subsplit siblings
> >>> if
> >>>> you want.
> >>>>
> >>>>
> >>>> On 2014-04-29 19:38, GerdP
wrote:
> >>>> > Hi Lambertus,
> >>>> >
> >>>> > that's interesting. Are
these the img file sizes or the osm file
> >>> sizes?
> >>>> >
> >>>> > Gerd
> >>>> >
> >>>> >
> >>>> > Lambertus wrote
> >>>> >> Unfortunately I cannot
confirm that. Below is a bit of logging
> >>> from my
> >>>> >> script:
> >>>> >> Original: 97000020
(70551453), New: 0 (35684445), New: 1
> >>> (36852845)
> >>>> >> Original: 97000001
(74621042), New: 0 (37522992), New: 1
> >>> (37222739)
> >>>> >> Original: 97000002
(73391358), New: 0 (37679505), New: 1
> >>> (38098627)
> >>>> >> Original: 97000003
(77862567), New: 0 (39075311), New: 1
> >>> (39261197)
> >>>> >>
> >>>> >> The original files
above contain contour data, the filesize is
> >>> between
> >>>> >> brackets. As you can
see both resulting file are approximately
> >>> the
> >>>> >> same
> >>>> >> size.
> >>>> >>
> >>>> >> On 2014-04-29 15:39,
Gerd Petermann wrote:
> >>>> >>> Hi Lambertus,
> >>>> >>>
> >>>> >>> and I guess that
even after this optimization you will
> >>>> >>> see a factor 3 or
higher between the largest tile and the
> >>> smallest.
> >>>> >>> Can you confirm
that?
> >>>> >>>
> >>>> >>> Gerd
> >>>> >>>
> >>>> >>>> Date: Tue, 29
Apr 2014 15:32:38 +0200
> >>>> >>>> From:
> >>>> >
> >>>> >> osm@
> >>>> >
> >>>> >>>> To:
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> Subject: Re:
[mkgmap-dev] mkgmap ToDo list
> >>>> >>>>
> >>>> >>>> Num-tiles=x
would indeed be better for this specific need.
> >>>> >>>>
> >>>> >>>> It is my
experience that it regularly takes multiple calls to
> >>>> >>> Splitter
> >>>> >>>> to get 2+
sub-tiles when you reduce the max-nodes by 100k for
> >>> each
> >>>> >>>> sub-split
attempt. This is what I currently do to get an
> >>> optimum in
> >>>> >>>> tile-size vs
total number of tiles.
> >>>> >>>>
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> On 29/04/2014
15:09, Gerd Petermann wrote:
> >>>> >>>> > Hi
Lambertus,
> >>>> >>>> >
> >>>> >>>> > that
sounds like a possible change in splitter:
> >>>> >>>> > Instead of
specifying max-nodes you may specify --num-tiles=x
> >>>> >>>> > and
splitter will try to find a split that produces
excactly
> >>> x
> >>>> >>> tiles
> >>>> >>>> > which are
not too narrow and have a node number which is not
> >>>> >>>> > too far
from the average (but still aligned to a multiple of
> >>> map
> >>>> >>> units
> >>>> >>>> > as now).
> >>>> >>>> > So, for
your script that means you don't have to find the
> >>>> >>> max-nodes
> >>>> >>>> > value.
> >>>> >>>> >
> >>>> >>>> > I'll think
about this again...
> >>>> >>>> >
> >>>> >>>> > Gerd
> >>>> >>>> >
> >>>> >>>> > > Date:
Tue, 29 Apr 2014 14:59:36 +0200
> >>>> >>>> > > From:
> >>>> >
> >>>> >> osm@
> >>>> >
> >>>> >>>> > > To:
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > >
Subject: Re: [mkgmap-dev] mkgmap ToDo list
> >>>> >>>> > >
> >>>> >>>> > > While
this possibly can be solved in Splitter or Mkgmap, it
> >>>> >>> could also
> >>>> >>>> > > be
solved by your build-script when you add a maximum
tile
> >>> size
> >>>> >>> check
> >>>> >>>> > > and
re-split (with a lower number of max-nodes) until you
> >>> get
> >>>> >>> two or
> >>>> >>>> > > more
sub-tiles. Granted, this adds complexity to the script
> >>> but
> >>>> >>> it works
> >>>> >>>> > > well
for me.
> >>>> >>>> > >
> >>>> >>>> > > On
25/04/2014 21:54, Henning Scholland wrote:
> >>>> >>>> > > >
Hi Gerd,
> >>>> >>>> > > >
> >>>> >>>> > > >
I would like to have img-tiles which have globally
nearly
> >>> the
> >>>> >>> same
> >>>> >>>> > > >
filesize, so that they use the space of devices like
> >>> eTrex 10.
> >>>> >>>> > > >
> >>>> >>>> > > >
With my actual map I use globally the same value for
> >>>> >>> max-nodes. But the
> >>>> >>>> > > >
size of the img-tiles differ more then factor 2. Eg. a
> >>> tile in
> >>>> >>> Germany
> >>>> >>>> > > >
is between 2 and 5 mb where a tile in China is about
10
> >>> mb. If
> >>>> >>> I remove
> >>>> >>>> > > >
details, this difference will increase, because in
> >>> Germany
> >>>> >>> more objects
> >>>> >>>> > > >
will be removed from the img-tile then in China.
> >>>> >>>> > > >
> >>>> >>>> > > >
Henning
> >>>> >>>> > > >
> >>>> >>>> > > >
_______________________________________________
> >>>> >>>> > > >
mkgmap-dev mailing list
> >>>> >>>> > > >
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > > >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>> > >
> >>>> >>>> > >
_______________________________________________
> >>>> >>>> > >
mkgmap-dev mailing list
> >>>> >>>> > >
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> >
_______________________________________________
> >>>> >>>> > mkgmap-dev
mailing list
> >>>> >>>> >
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>> >
> >>>> >>>>
> >>>> >>>>
_______________________________________________
> >>>> >>>> mkgmap-dev
mailing list
> >>>> >>>>
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>>
> >>>> >>>
_______________________________________________
> >>>> >>> mkgmap-dev mailing
list
> >>>> >>>
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >>
_______________________________________________
> >>>> >> mkgmap-dev mailing list
> >>>> >
> >>>> >> mkgmap-dev at .org
> >>>> >
> >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > View this message in
context:
> >>>> >
> >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
> >>>> > Sent from the Mkgmap
Development mailing list archive at
> >>> Nabble.com.
> >>>> >
_______________________________________________
> >>>> > mkgmap-dev mailing list
> >>>> > mkgmap-dev at lists.mkgmap.org.uk
> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>>
_______________________________________________
> >>>> mkgmap-dev mailing list
> >>>> mkgmap-dev at lists.mkgmap.org.uk
> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>>
> >>>
_______________________________________________
> >>> mkgmap-dev mailing list
> >>> mkgmap-dev at lists.mkgmap.org.uk
> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>
_______________________________________________
> >> mkgmap-dev mailing list
> >> mkgmap-dev at lists.mkgmap.org.uk
> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
keep on biking and discovering new trails
Felix
openmtbmap.org & www.velomap.org
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
keep on biking and discovering new trails
Felix
openmtbmap.org & www.velomap.org
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20140504/2202dc7c/attachment-0001.html>
- Previous message: [mkgmap-dev] mkgmap ToDo list
- Next message: [mkgmap-dev] mkgmap ToDo list
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list