[mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile

Mon May 28 07:55:04 BST 2018

Hi Randolph,

sorry, I've checked again the stack trace and noticed that the problem is in the overview map, not in a single file. So, for this case the solution is probably to change the style so that less data is written to the overview map.

Gerd

________________________________________
Von: Gerd Petermann <gpetermann_muenchen at hotmail.com>
Gesendet: Montag, 28. Mai 2018 07:46:11
An: Randolph J. Herber
Betreff: AW: AW: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile

Hi Randolph,

The --max-nodes option is just a way to estimate the size of the RGN file (which has the 16M limit). The actual size of the file depends on your style, the mkgmap options
and the way how a given area is mapped. It doesn't depend on the number of ways.

I think the common method to solve this problem is to modify the split-file (areas.list) created by splitter. You create the file with the wanted --max-nodes value
and try to run mkgmap. For those tiles that are  too big you edit the areas.list so that the tile is divided into two parts.  The resulting areas.list is used with the --split-file
option and should work for quite a while.
I think I also added the --num-tiles option to assist with this:
You feed splitter with the too-large tile and option --num-tiles=2 and it creates two new tiles.

Not sure how to do that in a script, but I think someone might have that.
Hope that helps?

Gerd

________________________________________
Von: Randolph J. Herber <army.bronze.star at gmail.com>
Gesendet: Mittwoch, 23. Mai 2018 15:00:01
An: Gerd Petermann
Betreff: Re: AW: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile

Hi, Gerd,

That is correct. It is also a gross simplification.

I have read the code enough to learn what the error message actually means.  A tile is written like a MS-DOS FAT16 file system within the .img file.  If any component exceeds a limit on the size for that type of component, then this error occurs. By far, the most common reason is too many nodes, where the limit is a little less than 16MiB.  But, it can occur for too many ways or relations as well. It is hard to reach too many relations without exceeding too much node data or too much way data as well. The condition arises when an area has a large fraction of its ways with 2 or 3 nodes in a way. An example is in the 5 millennia old areas near Cairo, Egypt.

The too many ways condition can be addressed by forcing down the number of nodes using --maxnodes. This, though, can cause another problem of too many tiles in a Garmin map set as Mapsource itself has a limit on the number of tiles it can handle in a single mapset of about 2025 tiles.

With a --maxways parameter, the few tiles with the short ways can be divided by that parameter without requiring a --maxnodes=500000 parameter with a resultant too many tiles when making a mapset for Africa as a whole.

Randolph J. Herber

On 5/23/2018 3:25 AM, Gerd Petermann wrote:

Hi Randolph,

the message just means that the map is too large. This means too many bytes, not too many lines or points.

Gerd

________________________________________
Von: Randolph J. Herber <army.bronze.star at gmail.com><mailto:army.bronze.star at gmail.com>
Gesendet: Dienstag, 22. Mai 2018 14:48:28
An: Gerd Petermann
Betreff: Re: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile

Hi, Gerd,

You are correct. The actual failure occurs in mkgmap. The output of one apparently working program can cause problems in another following program.

In the map data, the problem occurs in the Cairo-Port Said area while making a full world map.

If I remember correctly, the message resembles:

uk.me.parabola.imgfmt.MapFailedException: There is not enough room in a single garmin map for all the input data. The .osm file should be split into smaller pieces first.
        at uk.me.parabola.imgfmt.app.BufferedImgFileWriter.ensureSize(BufferedImgFileWriter.java:275)
        at uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put(BufferedImgFileWriter.java:238)
        at uk.me.parabola.imgfmt.app.trergn.Polyline.write(Polyline.java:144)
        at uk.me.parabola.imgfmt.app.trergn.RGNFile.addMapObject(RGNFile.java:140)
        at uk.me.parabola.imgfmt.app.map.Map.addMapObject(Map.java:265)
        at uk.me.parabola.mkgmap.build.MapBuilder$ShapeAddFilter.doFilter(MapBuilder.java:1497)
        at uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChain.java:58)
        at uk.me.parabola.mkgmap.filters.LinePreparerFilter.doFilter(LinePreparerFilter.java:148)
        at uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChain.java:58)
        at uk.me.parabola.mkgmap.filters.RemoveEmpty.doFilter(RemoveEmpty.java:61)
        at uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChain.java:58)
        at uk.me.parabola.mkgmap.filters.RemoveObsoletePointsFilter.doFilter(RemoveObsoletePointsFilter.java:138)
        at uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChain.java:58)
        at uk.me.parabola.mkgmap.filters.PolygonSplitterFilter.doFilter(PolygonSplitterFilter.java:50)
        at uk.me.parabola.mkgmap.build.LayerFilterChain.doFilter(LayerFilterChain.java:58)
        at uk.me.parabola.mkgmap.build.LayerFilterChain.startFilter(LayerFilterChain.java:73)
        at uk.me.parabola.mkgmap.build.MapBuilder.processShapes(MapBuilder.java:1256)
        at uk.me.parabola.mkgmap.build.MapBuilder.makeSubdivision(MapBuilder.java:891)
        at uk.me.parabola.mkgmap.build.MapBuilder.makeMapAreas(MapBuilder.java:802)
        at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:286)
        at uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(OverviewBuilder.java:191)
        at uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuilder.java:104)
        at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:678)
        at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Note that the error occurs in PolyLine and not in anything about nodes.

Randolph

On 5/22/2018 2:39 AM, Gerd Petermann wrote:

Hi Randolph,

I think it would be a lot of work to change splitter like that, and probably it would require a lot more resources.
I'd be interested in the original error message from MapSource and the tile that produces it, maybe the error is somewhere
in mkgmap.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Randolph J. Herber <army.bronze.star at gmail.com><mailto:army.bronze.star at gmail.com><mailto:army.bronze.star at gmail.com><mailto:army.bronze.star at gmail.com>
Gesendet: Dienstag, 22. Mai 2018 00:45:49
An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
Betreff: Re: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile

Dear Sirs:

My definition of "ways" was requested.

Splitter first scans nodes (points of various kinds), then ways (forming lines, paths and polygons) and finally relations (container polygons for any or all of nodes, ways or relations). I was using "ways" in the same sense that I understand that splitter uses "ways" in its messages. I hope this helps.

--maxnodes sets a soft upper limit on the number of nodes in a map tile.

I am requesting a similar in concept --maxways to set a soft limit on the number of ways in a map tile.

Randolph J. Herber

On 5/21/2018 12:27 PM, Andrzej Popowski wrote:
Hi Randolph,

what do you mean by "ways"? Any line, road with an address, road used for routing? I don't remember if I ever spotted similar problem.

On the other side, I think splitter can be better tuned for current mkgmap. I get impression, that splitter is designed to make a good work with simple maps and doesn't account for addresses, routing nodes od DEM data. It would be nice, if there were more options, than maxnodes.

For my maps, I split OSM data in two stages. First I prepare artificial data, that I believe represent better actual size of compiled tiles.

I extract addresses as points and highways as simple 2-points lines (I do it with modified osmfilter). Then I make splitter to calculate all tiles using as an input following data:
- OSM source
- 0-2 times extracted addresses
- 6-10 times extracted highways
- 1-2 times contour lines

I believe, that additional points form address account for some NET size, points form highways for additional NOD size and duplicated contour lines for DEM size. It needs some tuning for different areas, but I think I get a bit more uniform sizes of img than with direct splitting.

On second stage I use prepared areas.list to split real data.

Basically I do something like this:

osmfilter.exe data.o5m --keep= --keep-ways=highway=* -o=net.o5m
osmfilter.exe data.o5m --keep= --keep-nodes=addr:housenumber=* -o=adr.o5m

splitter.jar --stop-after=split --max-nodes=3000000 data.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m adr.o5m adr.o5m contour.pbf contour.pbf

splitter.jar --split-file=areas.list data.o5m contour.pbf

It would be helpful, if I could do it in a single pass. Maybe splitter could filter internally specific nodes - these which would end as routing nodes or address information and add some multiplier when counting these nodes for splitting? Assuming this is easy to implement, otherwise my procedure is good enough.

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev