[mkgmap-dev] tile takes very long time to generate

Wed Mar 10 15:04:20 GMT 2021

Hi Gerd

Having looked more closely at current OSM rules and what you said
earlier in the thread about "old style" multipolygons and:

https://wiki.openstreetmap.org/wiki/Old_style_multipolygons

I'd agree that support for these should be removed.

Then, with the appropriate --delete-tags-file=... the problem goes
away.

Ticker

On Wed, 2021-03-10 at 12:47 +0000, Gerd Petermann wrote:
> Hi Ticker,
> 
> reg. thetag handling I forgot to mention the --delete-tags-file
> option. This would allow to remove tags like place=sea or
> place=island while the input file is read, but the strange handling
> of MP tags makes this useless.
> 
> Gerd
> 
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Gerd Petermann <gpetermann_muenchen at hotmail.com>
> Gesendet: Mittwoch, 10. März 2021 13:42
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] tile takes very long time to generate
> 
> Hi Ticker,
> 
> I think it would be good if we could defer the actions in method
> MultiPolygonRelation.processElements() until the relation style rules
> were used. This would allow to remove rather useless relations like
> the place=sea or place=island MPs or maybe add a tag to tell mkgmap
> that only a POI is needed. No idea how much work that would be or
> what side effects it would have.
> 
> I've not yet checked why the mentioned MP takes soooo long, maybe
> it's because the MP contains some errors.
> 
> Gerd
> 
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> Gesendet: Mittwoch, 10. März 2021 13:13
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] tile takes very long time to generate
> 
> Hi
> 
> I'm not sure of all the rules regarding relations and resultant
> polygons and some of the MultiPolygonRelation.java code defeat me but
> some thoughts:
> 
> Even though the relation might not have any relevant tags, it is what
> causes all the significant inner and/or outer polygons to be create
> by
> mkgmap. It might have been necessary to use a relation because the
> closed way has too many points, etc.
> 
> mkgmap's OSM loading process drops any tags not referenced by the
> style. For relations it sets a flag to say if any tags have been
> dropped. Later, it doesn't bother processing the relation (building
> the
> multipolygons) if it *HAD* no tags. So, a relation tagged with just:
>   [notUsedInStyle=someValue]
> will cause full processing, whereas without this, nothing will
> happen.
> 
> Much of the cost of MultiPolygon processing is the assembling of bits
> of way to make up the polygons. The rest of the cost is cutting inner
> polygons out of outer polygons. If the outer polygons really don't
> have
> tags used in the style, maybe this phase can be skipped.
> 
> In this example, the tag in question is "place"; this will occur in
> most styles.
> 
> In the above, I'm ignoring the special relation tag "type" and
> internal
> mkgmap:* tags.
> 
> Ticker
> 
> On Tue, 2021-03-09 at 15:27 +0000, Gerd Petermann wrote:
> > Hi all,
> > 
> > I sea no easy way to fix this. We'll have more and more monster
> > multipolygons like r9488835. The current code in mkgmap r4607
> > processes a type=multipolygon before any rule from relations is
> > used,
> > so there is no easy way to tell mkgmap to ignore any MP with
> > place=sea.
> > Using a program like osmfilter on complete countries or continents
> > also takes time, so maybe the best place to filter these killers
> > would be in splitter, but we probably need some flexible syntax
> > similar to the style rules.
> > Another option would be a new mkgmap rule file for mkgmap that
> > would
> > be processed after reading the MP but before processing its
> > geometry.
> > This new rule file could allow actions like those in the finalize
> > section of points/polygons.
> > 
> > Any ideas besides hard coded "quick&dirty" solutions?
> > 
> > Gerd
> > 
> > 
> > 
> > ________________________________________
> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> > von Bernhard Hiller <bhil at gmx.de>
> > Gesendet: Dienstag, 9. März 2021 09:50
> > An: mkgmap-dev at lists.mkgmap.org.uk
> > Betreff: Re: [mkgmap-dev] tile takes very long time to generate
> > 
> > Hi Karl,
> > some time ago, I stumbled upon a tile which took 2 hours(!) on my
> > machine.
> > With the "--polygon-file" option, the time could be reduced to less
> > than
> > 100 s.
> > See the old message in the archive:
> > http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2017q1/026490.html
> > and the thread "Performance with large files":
> > http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2017q1/thread.html#26
> > 46
> > 3
> > Perhaps the trick helps in your case, too.
> > Kind regards,
> > Bernhard
> > 
> > Am 08.03.2021 um 06:53 schrieb 7770:
> > > Hi.
> > > 
> > > I observe a situation that one of the very first tiles (often the
> > > first or the
> > > third) that mkgmap generates takes 20 - 30 minutes to generate,
> > > wheres the
> > > others take about 30 seconds each.
> > > 
> > > The map data produced by splitter is a total of around 700 files
> > > with splitter
> > > option --max-nodes=1275000.
> > > 
> > > sea and bouds are used for mkgmap.
> > > 
> > > At first i thought i am running low on memory but changing to max
> > > -jobs=1
> > > (instead of the possible max of 2) did not make any change.
> > > 
> > > Is mkgmap doing something specific in the beginning that can
> > > explain this long
> > > time generating one of the very first tiles?
> > > 
> > > I can provide more details, let me know how i can best collect
> > > those details
> > > in case i need to set some parameters to have mkgmap to say more
> > > about what it
> > > is doing.
> > > 
> > > Regards
> > > Karl
> > > 
> > > 
> > > 
> > > 
> > 
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev