[mkgmap-dev] Error writing overviewmap - continue at failure

Sun Feb 28 10:39:29 GMT 2021

Hi

I can't remember all the details of this message, but what has happened
is a tile needs 3-byte references to a city index and this will force
the full MDR for all tiles to use 3-byte city references, so growing
the index structure a lot. So, probably not relevant for overview.

Ticker

On Sun, 2021-02-28 at 10:23 +0000, Gerd Petermann wrote:
> Hi Joris,
> 
> the message is printed when the LBL file contains so many citie
> names. Those names are are extracted from mkgmap:city (maybe more),
> so maybe addr:city is a problem. Probably a mapping error in that
> area.
> 
> Gerd
> 
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Joris Bo <jorisbo at hotmail.com>
> Gesendet: Sonntag, 28. Februar 2021 11:11
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Error writing overviewmap - continue at
> failure
> 
> Hi
> 
> When reading my error log again there was another error. Too many
> cities... Could it have caused my overview map too fail?
> 
> SEVERE (MapBuilder): C:\Garmin\Temp\8004 - JBM - Planet\Output
> Splitter\80041630.osm.pbf: Map contains 94484 Cities - more than
> 65535 might cause indexing problems and excess size. Suggest splitter
> with lower --max-nodes
> 
> I Investigated the area (which is more or less a dessert in Jemen :-)
> Overpass does not count a lot place = city.   Could somebody help to
> explain?
> Could this halfway-error also be a reason to fail the overview map
> being created at the end?
> 
> Kind regards
> Joris
> 
> KML - file of the area
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <kml xmlns="http://www.opengis.net/kml/2.2">;
> <Document>
>   <Style id="transWhitePoly">
>     <LineStyle>
>       <width>1.5</width>
>     </LineStyle>
>     <PolyStyle>
>       <color>00ffffff</color>
>       <colorMode>normal</colorMode>
>     </PolyStyle>
>   </Style>
> 
>   <Placemark>
>     <name>80041630</name>
>     <styleUrl>#transWhitePoly</styleUrl>
>       <description>
>         <![CDATA[YE-Sanaa]]>
>       </description>
>     <Polygon>
>       <outerBoundaryIs>
>         <LinearRing>
>           <coordinates>
>             43.637695,13.666992
>             43.637695,16.962891
>             45.571289,16.962891
>             45.571289,13.666992
>             43.637695,13.666992
>           </coordinates>
>         </LinearRing>
>       </outerBoundaryIs>
>     </Polygon>
>   </Placemark>
> 
>   </Document>
> </kml>
> 
> 
> 
> 
> Met vriendelijke groeten,
> 
> Joris Bo
> jorisbo at hotmail.com
> 
> -----Oorspronkelijk bericht-----
> Van: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> Namens Joris
> Bo
> Verzonden: zaterdag 27 februari 2021 18:40
> Aan: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk>
> Onderwerp: Re: [mkgmap-dev] Error writing overviewmap - continue at
> failure
> 
> Hi Mike & Gerd ,
> 
> My europe map (Europe-latest.osm.pbf  split of 24Gb) which works
> fine, has these img- files
> 
> 80010001.img to 80011554.img (1554 map files) Osmmap.img of 7Mb
> Osmmap_mdr.img of 760Mb
> 
> I assumed the 7Mb file is the overview map? Not the Osmmap_mdr.img
> file?
> For planet.osm (56Gb in stead of 24Gb) would then make an overview
> map of lets say approximately 20Mb My planet ovm work files together
> are 1.2Gb 😊 this is as pointed out, not a reference So I assume th
> e real problem is that the overview map is loaded with too many
> objects.
> 
> I will study Gerd's proposal first (for now I get the concept but
> don't understand the exact procedure) But with a good backup not much
> can go wrong.
> 
> Thank you for the solutions and the effort off course, Kind regards
> Joris
> 
> 
> Met vriendelijke groeten,
> 
> Joris Bo
> jorisbo at hotmail.com
> 
> -----Oorspronkelijk bericht-----
> Van: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> Namens Gerd
> Petermann
> Verzonden: zaterdag 27 februari 2021 17:58
> Aan: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk>
> Onderwerp: Re: [mkgmap-dev] Error writing overviewmap - continue at
> failure
> 
> Hi Mike and Joris,
> 
> as Ticker pointer out before the overall sum of file sizes for the
> ovm*.img files is typically much larger than the resultiing overview
> map. Reason is that the ovm* files contain the full LBL section (all
> Strings that appear in the detailed map), only a few of them are used
> in the overview map.
> 
> Gerd
> 
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Mike Baggaley <mike at tvage.co.uk>
> Gesendet: Samstag, 27. Februar 2021 17:38
> An: 'Development list for mkgmap'
> Betreff: Re: [mkgmap-dev] Error writing overviewmap - continue at
> failure
> 
> Hi Joris,
> 
> I will soon be passing an updated patch to Gerd to improve this, as I
> had also been experiencing similar problems. The overview map is not
> built until all the tiles have been created (assuming the error is
> not in an overview tile), and I don’t think it uses the style file at
> that point, just merges all the ovm data, so using a new style will
> not help without restarting your run. The message produced by the
> patch when the overview file is too big is “The overview map is too
> big. The maximum size is 16777215. Try reducing the highest overview
> resolution or reducing the amount of information included in the
> overview.” If you build your own version of mkgmap, you can use the
> version of the patch I posted a few days ago. For the whole world, I
> expect the maximum resolution of overview to be pretty low. If I
> build a single UK county, I can have an overview at resolution 22,
> but only 20 for the UK. I haven’t tried anything bigger. You may get
> some idea how close you are to the limit by totalling the sizes of
> the ovm files. The maximum size may not necessarily be the max for
> the whole file, it may be the limit of a sub file though. That is
> certainly the case for ordinary map tiles, but I’m not sure on the
> overview.
> 
> I also have some code that I’m working on that adds support for a
> mkgmap:nooverview tag in the style file, so that if you have
> overlapping main and overview resolutions, you can direct your style
> to omit some of the information from the overview. Will be passing
> this patch over shortly as well.
> 
> I don’t think allowing a continuation after a failure in the final
> overview map would be helpful, as there is very little left to do at
> that point. There are some issues with the keep going handling and
> I’m also looking at that.
> 
> Cheers,
> Mike
> 
> From: Joris Bo [mailto:jorisbo at hotmail.com]
> Sent: 27 February 2021 14:49
> To: Development list for mkgmap <mkgmap-dev at lists.mkgmap.org.uk>
> Subject: [mkgmap-dev] Error writing overviewmap - continue at failure
> 
> Hi,
> 
> I think the below error message is for writing the overview map.
> Of which I would not be surprised, it was a test on splitting and
> building a full-planet.osm map. So… no worries.
> 
> But now I’m wondering if there is a way in which I could continue
> testing with the already generated 4.426 img-files and 4.426 ovm_work
> files to obtain a .tdb file, mdx files, nsis installer etc.
> It took 21 hours for mkgmap sofar, so it would be nice if that time
> spent was not useless.
> I used splitter with maxnodes 1.700.000 which took another 9 hours
> and did not give errors when building a single .img file
> 
> I would like to simplify the style-file run mkgmap again on the
> previous output and try the create an overview map only (So without
> building all the img files again) and this way iterate until I found
> the breakpoint.
> 1)     Would that make sense?
> 2)     Could I enable some more error/verbose logging to obtain info
> about sizes/counts or whatever indicators that would help to see if
> my changes help in my search to a smaller overviewmap.
> 3)     Would It make sense to not generate a MapFailedException but a
> warning only? So mkgmap would continue finishing the last millimeters
> to the finish (off course without a working overviewmap)
> 
> Kind regards
> Joris
> 
> SEVERE (MapFailedException): (thrown in
> BufferedImgFileWriter.ensureSize()) There is not enough room in a
> single garmin map for all the input data. The .osm file should be
> split into smaller pieces first.
> uk.me.parabola.imgfmt.MapFailedException: There is not enough room in
> a single garmin map for all the input data. The .osm file should be
> split into smaller pieces first.
>             at
> uk.me.parabola.imgfmt.app.BufferedImgFileWriter.ensureSize(BufferedIm
> gFileWriter.java:275)
>             at
> uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put(BufferedImgFileWr
> iter.java:238)
>             at
> uk.me.parabola.imgfmt.app.trergn.Polyline.write(Polyline.java:142)
>             at
> uk.me.parabola.imgfmt.app.trergn.RGNFile.addMapObject(RGNFile.java:14
> 0)
>             at
> uk.me.parabola.imgfmt.app.map.Map.addMapObject(Map.java:273)
>             at
> uk.me.parabola.mkgmap.build.MapBuilder$ShapeAddFilter.doFilter(MapBui
> lder.java:1459)
>             at
> uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChai
> n.java:58)
>             at
> uk.me.parabola.mkgmap.filters.LinePreparerFilter.doFilter(LinePrepare
> rFilter.java:143)
>             at
> uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChai
> n.java:58)
>             at
> uk.me.parabola.mkgmap.filters.RemoveEmpty.doFilter(RemoveEmpty.java:5
> 9)
>             at
> uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChai
> n.java:58)
>             at
> uk.me.parabola.mkgmap.filters.RemoveObsoletePointsFilter.doFilter(Rem
> oveObsoletePointsFilter.java:142)
>             at
> uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChai
> n.java:58)
>             at
> uk.me.parabola.mkgmap.filters.PolygonSplitterFilter.doFilter(PolygonS
> plitterFilter.java:81)
>             at
> uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChai
> n.java:58)
>             at
> uk.me.parabola.mkgmap.build.LayerFilterChain.startFilter(LayerFilterC
> hain.java:73)
>             at
> uk.me.parabola.mkgmap.build.MapBuilder.processShapes(MapBuilder.java:
> 1260)
>             at
> uk.me.parabola.mkgmap.build.MapBuilder.makeSubdivision(MapBuilder.jav
> a:901)
>             at
> uk.me.parabola.mkgmap.build.MapBuilder.makeMapAreas(MapBuilder.java:8
> 13)
>             at
> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:312)
>             at
> uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(Over
> viewBuilder.java:192)
>             at
> uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuil
> der.java:105)
>             at
> uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:675)
>             at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja
> va:125)
>             at
> uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:144)
>             at uk.me.parabola.mkgmap.main.Main.main(Main.java:115)
> 
> 
> 
> Met vriendelijke groeten,
> 
> Joris Bo
> jorisbo at hotmail.com<mailto:jorisbo at hotmail.com>
> 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev