[mkgmap-dev] tile takes very long time to generate

Wed Mar 10 12:42:55 GMT 2021

Hi Ticker,

I think it would be good if we could defer the actions in method MultiPolygonRelation.processElements() until the relation style rules were used. This would allow to remove rather useless relations like the place=sea or place=island MPs or maybe add a tag to tell mkgmap that only a POI is needed. No idea how much work that would be or what side effects it would have.

I've not yet checked why the mentioned MP takes soooo long, maybe it's because the MP contains some errors.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
Gesendet: Mittwoch, 10. März 2021 13:13
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] tile takes very long time to generate

Hi

I'm not sure of all the rules regarding relations and resultant
polygons and some of the MultiPolygonRelation.java code defeat me but
some thoughts:

Even though the relation might not have any relevant tags, it is what
causes all the significant inner and/or outer polygons to be create by
mkgmap. It might have been necessary to use a relation because the
closed way has too many points, etc.

mkgmap's OSM loading process drops any tags not referenced by the
style. For relations it sets a flag to say if any tags have been
dropped. Later, it doesn't bother processing the relation (building the
multipolygons) if it *HAD* no tags. So, a relation tagged with just:
  [notUsedInStyle=someValue]
will cause full processing, whereas without this, nothing will happen.

Much of the cost of MultiPolygon processing is the assembling of bits
of way to make up the polygons. The rest of the cost is cutting inner
polygons out of outer polygons. If the outer polygons really don't have
tags used in the style, maybe this phase can be skipped.

In this example, the tag in question is "place"; this will occur in
most styles.

In the above, I'm ignoring the special relation tag "type" and internal
mkgmap:* tags.

Ticker

On Tue, 2021-03-09 at 15:27 +0000, Gerd Petermann wrote:
> Hi all,
>
> I sea no easy way to fix this. We'll have more and more monster
> multipolygons like r9488835. The current code in mkgmap r4607
> processes a type=multipolygon before any rule from relations is used,
> so there is no easy way to tell mkgmap to ignore any MP with
> place=sea.
> Using a program like osmfilter on complete countries or continents
> also takes time, so maybe the best place to filter these killers
> would be in splitter, but we probably need some flexible syntax
> similar to the style rules.
> Another option would be a new mkgmap rule file for mkgmap that would
> be processed after reading the MP but before processing its geometry.
> This new rule file could allow actions like those in the finalize
> section of points/polygons.
>
> Any ideas besides hard coded "quick&dirty" solutions?
>
> Gerd
>
>
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Bernhard Hiller <bhil at gmx.de>
> Gesendet: Dienstag, 9. März 2021 09:50
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] tile takes very long time to generate
>
> Hi Karl,
> some time ago, I stumbled upon a tile which took 2 hours(!) on my
> machine.
> With the "--polygon-file" option, the time could be reduced to less
> than
> 100 s.
> See the old message in the archive:
> http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2017q1/026490.html
> and the thread "Performance with large files":
> http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2017q1/thread.html#2646
> 3
> Perhaps the trick helps in your case, too.
> Kind regards,
> Bernhard
>
> Am 08.03.2021 um 06:53 schrieb 7770:
> > Hi.
> >
> > I observe a situation that one of the very first tiles (often the
> > first or the
> > third) that mkgmap generates takes 20 - 30 minutes to generate,
> > wheres the
> > others take about 30 seconds each.
> >
> > The map data produced by splitter is a total of around 700 files
> > with splitter
> > option --max-nodes=1275000.
> >
> > sea and bouds are used for mkgmap.
> >
> > At first i thought i am running low on memory but changing to max
> > -jobs=1
> > (instead of the possible max of 2) did not make any change.
> >
> > Is mkgmap doing something specific in the beginning that can
> > explain this long
> > time generating one of the very first tiles?
> >
> > I can provide more details, let me know how i can best collect
> > those details
> > in case i need to set some parameters to have mkgmap to say more
> > about what it
> > is doing.
> >
> > Regards
> > Karl
> >
> >
> >
> >
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev