[mkgmap-dev] [PATCH v2] Performance improvement by removing unused elements before the style processing

Wed Jan 4 19:32:45 GMT 2012

Hello WanMil,

I can reproduce the problem with the attached files like this:
1) download saarland.osm.pbf from geofabrik created 04-Jan-2012 04:36 

2) Use splitter r198 with --max-nodes=200000

3) compile r2160 with my identical_output.patch 
4) execute mkgmap with the parms in test.bat 
5) copy the output files to somewhwere else
6) execute mkgmap again with the same commands, verify that the new output files are identical to the copy
7) compile r2160 with my identical_output.patch + c:\TEMP\remove_unused_elements_v2.patch 
8) execute mkgmap again with the same commands. I would again expect identical output, but I see this
C:\temp\prove_patch>diff -qb ..\prove .
Files ..\prove\63240003.img and .\63240003.img differ
Files ..\prove\gmapsupp.img and .\gmapsupp.img differ
Files ..\prove\mkgmap.jar and .\mkgmap.jar differ
Files ..\prove\osmmap.tdb and .\osmmap.tdb differ
Files ..\prove\osmmap_mdr.img and .\osmmap_mdr.img differ

Ciao,
Gerd

> Date: Wed, 4 Jan 2012 19:10:28 +0100
> From: wmgcnfg at web.de
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] [PATCH v2] Performance improvement by removing unused elements before the style processing
> 
> Hi Gerd,
> 
> yes of course I am interested, Send me all information you have.
> 
> Thanks
> WanMil
> 
> > Hi WanMil,
> >
> > I tested the patch with my tiles for Saarland. I can confirm a reduction of
> > processing time ~ 6% compared to r2160, and also a reduction for the peek
> > value of heap memory usage (238M ->  210M), so that's both good!
> > BUT I also see a difference in one of the seven output img files (plus the
> > resulting gmapsupp.img), and I think this is not intended. Both files have
> > the same size, but are different in many bytes.
> > Let me know when I should send details, maybe the new result is better than
> > the old ;-)
> >
> > Ciao,
> > Gerd
> >
> >
> > WanMil wrote
> >>
> >> The 2nd patch fixes the remaining problems:
> >> 1. Ways without tags but referenced by relations are no longer removed
> >> (they might be tagged by the relation style file and could therefore
> >> appear in the map)
> >> 2. Intersection of ways with the tile bounding box is now checked
> >> instead of only checking that at least one point is contained in the
> >> bounding box. Only very few ways are affected by this but otherwise
> >> routing problems are possible.
> >>
> >> The performance improvement seems to be good (measured with my test map):
> >> r2159: ~250s
> >> patched: ~225s
> >>
> >> I also expect (although I haven't tested) that the max memory
> >> requirement of mkgmap is decreased. I think max mem is used when the
> >> style file is just processed. At this stage all raw OSM elements and all
> >> style file processed elements are kept in memory. With the patch the
> >> number of OSM elements is noticeably reduced.
> >>
> >> WanMil
> >>
> >>> Hi,
> >>>
> >>> this is another performance improvement:
> >>>
> >>> Usually the mkgmap input tiles are larger than the processed bounding
> >>> box (splitter parameter overlap). So there are much many elements which
> >>> are processed but thrown away at a late step in mkgmap.
> >>>
> >>> The patch tries to remove them much earlier before the style files are
> >>> processed and before the LocationHook starts (which ignores them but
> >>> that must also be calculated).
> >>>
> >>> The patch contains one drawback:
> >>> Ways which have all its points outside the bounding box of the tile but
> >>> which cross the tile are also removed. If that's a point the patch must
> >>> be improved.
> >>>
> >>> Have fun!
> >>> WanMil
> >>>
> >>>
> >>> _______________________________________________
> >>> mkgmap-dev mailing list
> >>> mkgmap-dev at .org
> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>
> >>
> >> _______________________________________________
> >> mkgmap-dev mailing list
> >> mkgmap-dev at .org
> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >>
> >
> >
> > --
> > View this message in context: http://gis.638310.n2.nabble.com/PATCH-v1-Performance-improvement-by-removing-unused-elements-before-the-style-processing-tp7144978p7149735.html
> > Sent from the Mkgmap Development mailing list archive at Nabble.com.
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20120104/514dbbda/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: prove.zip
Type: application/zip
Size: 1741 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20120104/514dbbda/attachment.zip 