[mkgmap-dev] tile takes very long time to generate

Sat Mar 13 08:32:42 GMT 2021

Hi all,

most of the time that is needed to process complex multipolygons (MP) is spent in compex tests which try to detect invalid geometries like "inner" ring is not inside outer or overlapping /crossing rings. I wonder if it really makes sense to perform those tests in mkgmap. In JOSM, the strategy is like this:
The renderer is optimistic and assumes that the geometry is correct, for invalid geometries "garbage in -> garbage out" is used. The validator performs a lot more tests and should find all the special cases mkgmap is looking for. This test is slow but still much faster than that in mkgmap (maybe 30 secs in JOSM, many minutes in mkgmap). 
If mkgmap finds invalid geometries the behaviour is still rather unpredictable. I've used JOSM to create some test cases with "inner" overlapping "outer" and sometimes the inner is completely ignores, sometimes not. So, I really wonder what all the complex code is doing.

I've attached my test file 
I am not sure if I should try to improve the test code 
1) to be faster or 
2) to be more predictable or
3) both 1+2 or
4) if I should just remove all code that isn't needed to produce good results with correct data 

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen at hotmail.com>
Gesendet: Donnerstag, 11. März 2021 07:58
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] tile takes very long time to generate

Hi Ticker,

I fear the problem will not go away because the relation still has tags like name:en .
Anyway, I work on a patch to remove the old-style support.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
Gesendet: Mittwoch, 10. März 2021 16:04
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] tile takes very long time to generate

Hi Gerd

Having looked more closely at current OSM rules and what you said
earlier in the thread about "old style" multipolygons and:

https://wiki.openstreetmap.org/wiki/Old_style_multipolygons

I'd agree that support for these should be removed.

Then, with the appropriate --delete-tags-file=... the problem goes
away.

Ticker

On Wed, 2021-03-10 at 12:47 +0000, Gerd Petermann wrote:
> Hi Ticker,
>
> reg. thetag handling I forgot to mention the --delete-tags-file
> option. This would allow to remove tags like place=sea or
> place=island while the input file is read, but the strange handling
> of MP tags makes this useless.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Gerd Petermann <gpetermann_muenchen at hotmail.com>
> Gesendet: Mittwoch, 10. März 2021 13:42
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] tile takes very long time to generate
>
> Hi Ticker,
>
> I think it would be good if we could defer the actions in method
> MultiPolygonRelation.processElements() until the relation style rules
> were used. This would allow to remove rather useless relations like
> the place=sea or place=island MPs or maybe add a tag to tell mkgmap
> that only a POI is needed. No idea how much work that would be or
> what side effects it would have.
>
> I've not yet checked why the mentioned MP takes soooo long, maybe
> it's because the MP contains some errors.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> Gesendet: Mittwoch, 10. März 2021 13:13
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] tile takes very long time to generate
>
> Hi
>
> I'm not sure of all the rules regarding relations and resultant
> polygons and some of the MultiPolygonRelation.java code defeat me but
> some thoughts:
>
> Even though the relation might not have any relevant tags, it is what
> causes all the significant inner and/or outer polygons to be create
> by
> mkgmap. It might have been necessary to use a relation because the
> closed way has too many points, etc.
>
> mkgmap's OSM loading process drops any tags not referenced by the
> style. For relations it sets a flag to say if any tags have been
> dropped. Later, it doesn't bother processing the relation (building
> the
> multipolygons) if it *HAD* no tags. So, a relation tagged with just:
>   [notUsedInStyle=someValue]
> will cause full processing, whereas without this, nothing will
> happen.
>
> Much of the cost of MultiPolygon processing is the assembling of bits
> of way to make up the polygons. The rest of the cost is cutting inner
> polygons out of outer polygons. If the outer polygons really don't
> have
> tags used in the style, maybe this phase can be skipped.
>
> In this example, the tag in question is "place"; this will occur in
> most styles.
>
> In the above, I'm ignoring the special relation tag "type" and
> internal
> mkgmap:* tags.
>
> Ticker
>
> On Tue, 2021-03-09 at 15:27 +0000, Gerd Petermann wrote:
> > Hi all,
> >
> > I sea no easy way to fix this. We'll have more and more monster
> > multipolygons like r9488835. The current code in mkgmap r4607
> > processes a type=multipolygon before any rule from relations is
> > used,
> > so there is no easy way to tell mkgmap to ignore any MP with
> > place=sea.
> > Using a program like osmfilter on complete countries or continents
> > also takes time, so maybe the best place to filter these killers
> > would be in splitter, but we probably need some flexible syntax
> > similar to the style rules.
> > Another option would be a new mkgmap rule file for mkgmap that
> > would
> > be processed after reading the MP but before processing its
> > geometry.
> > This new rule file could allow actions like those in the finalize
> > section of points/polygons.
> >
> > Any ideas besides hard coded "quick&dirty" solutions?
> >
> > Gerd
> >
> >
> >
> > ________________________________________
> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> > von Bernhard Hiller <bhil at gmx.de>
> > Gesendet: Dienstag, 9. März 2021 09:50
> > An: mkgmap-dev at lists.mkgmap.org.uk
> > Betreff: Re: [mkgmap-dev] tile takes very long time to generate
> >
> > Hi Karl,
> > some time ago, I stumbled upon a tile which took 2 hours(!) on my
> > machine.
> > With the "--polygon-file" option, the time could be reduced to less
> > than
> > 100 s.
> > See the old message in the archive:
> > http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2017q1/026490.html
> > and the thread "Performance with large files":
> > http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2017q1/thread.html#26
> > 46
> > 3
> > Perhaps the trick helps in your case, too.
> > Kind regards,
> > Bernhard
> >
> > Am 08.03.2021 um 06:53 schrieb 7770:
> > > Hi.
> > >
> > > I observe a situation that one of the very first tiles (often the
> > > first or the
> > > third) that mkgmap generates takes 20 - 30 minutes to generate,
> > > wheres the
> > > others take about 30 seconds each.
> > >
> > > The map data produced by splitter is a total of around 700 files
> > > with splitter
> > > option --max-nodes=1275000.
> > >
> > > sea and bouds are used for mkgmap.
> > >
> > > At first i thought i am running low on memory but changing to max
> > > -jobs=1
> > > (instead of the possible max of 2) did not make any change.
> > >
> > > Is mkgmap doing something specific in the beginning that can
> > > explain this long
> > > time generating one of the very first tiles?
> > >
> > > I can provide more details, let me know how i can best collect
> > > those details
> > > in case i need to set some parameters to have mkgmap to say more
> > > about what it
> > > is doing.
> > >
> > > Regards
> > > Karl
> > >
> > >
> > >
> > >
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mp-overlap.osm
Type: application/octet-stream
Size: 7294 bytes
Desc: mp-overlap.osm
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210313/fe689b37/attachment.obj>