[mkgmap-dev] MapFailedException when generating map for Italy

Thu Aug 17 20:57:22 BST 2017

Hi Gerd,

thanks for your deep analysis.
I will keep an eye on my next splitter runs if this problem reappears.
Also thanks for the update of mkgmap to avoid the NULL pointer exception.

Walter

-----Ursprüngliche Nachricht----- 
From: Gerd Petermann
Sent: Thursday, August 17, 2017 8:16 AM
To: Development list for mkgmap
Subject: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Walter,

I think these messages in the log will not help much unless you already know 
that something went wrong.
Looking at the wrong file I assume that an I/O error occured while reading 
the cropped file.
The enormous increase in the number of "problem-ways" and "problem-rels" 
tells us that splitter
got this wrong data multiple times (maybe from cache), because it already 
appeared in the pass which detects
the objects that cross tile borders.
Since the O5M format uses delta encoding and has no checksums a single I/O 
error can cause lots of wrong values.

I see no efficient way to detect this problem in splitter, but mkgmap should 
probably complain about
building polygons that span > 10km, maybe also about class 0 highways with 
very long segments.

Gerd
________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von 
Walter Schlögl <Walter.Schloegl-Resch at aon.at>
Gesendet: Mittwoch, 16. August 2017 22:29:51
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Gerd,

I think I should have looked at the logfiles earlier,
but since I do not really understand the splitter log, I tend not to read it
carefully.

I found out, that the problem was gone even with the old parameter for
max-nodes.
The solution was just to splitt the area again.

I checked the original europe.o5m, it was correct,
I checked the old 50100031.o5m and it was wrong as you said.
I compared the 2 splitter logfiles that have generated the wrong and the
correct version
(I have also copied them for your reference)
and I found the first importand difference at line 564

correct run:
dictionary contains now 1.000 entries
...
Number of stored area combis for nodes: 154.673.904
Number of stored area combis for ways: 16.877.672
Number of stored Integers for rels: 0
Number of stored combis in dictionary: 1.449
Number of detected problem ways: 64.970
Number of detected problem rels: 6.311
JVM Memory Info: Current 2558MB (1865MB used, 693MB free) Max 5461MB

wrong run:
dictionary contains now 1.000 entries
dictionary contains now 2.000 entries
...
dictionary contains now 34.000 entries
...
Number of stored area combis for nodes: 154.672.668
Number of stored area combis for ways: 11.580.442
Number of stored Integers for rels: 0
Number of stored combis in dictionary: 34.620
Number of detected problem ways: 608.980
Number of detected problem rels: 25.075
JVM Memory Info: Current 2555MB (1954MB used, 601MB free) Max 5461MB

Of course I don't know what this means.

Since I am executing all this via Scripts that I did not change between
these 2 runs,
my only explanation is, that splitter (or my computer) did not had the best
day last saturday.

Walter

-----Ursprüngliche Nachricht-----
From: Gerd Petermann
Sent: Wednesday, August 16, 2017 10:11 AM
To: Development list for mkgmap
Subject: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Walter,

another possible way to verify the data in your europe.o5m:
extract a node that is corrupted in 50100031.o5m, e.g.
osmfilter europe.o5m --keep="@id=1965123604" > test.osm

When I do this with     50100031.o5m I see

        <node id="1965123604" lat="44.933073" lon="17.5296559" version="1"/>

while the correct data should look like this:
        <node id="1965123604" lat="37.6469658" lon="14.0636559"
version="1"/>

Gerd
________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd
Petermann <GPetermann_muenchen at hotmail.com>
Gesendet: Mittwoch, 16. August 2017 08:12:18
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Walter,

The data in 50100031.o5m  looks very strange, maybe there is a special case
in splitter.
So if you still have the same europe.o5m file, it would be great if you
could try to reproduce the
problem. If you can, please send the splitter log and the areas.list.

Gerd
________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
Walter Schlögl <Walter.Schloegl-Resch at aon.at>
Gesendet: Dienstag, 15. August 2017 22:29:33
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Gerd,

I did not try to reproduce the corrupted file,
but I did not update the original italy.o5m file
that I used for splitting it with the lower maxnodes value.

My workflow has 4 steps.

1. update europe.o5m with daily diffs
2. cropping a country or region out of europe
3. splitter
4. mkgmap

Since I only repeated step 3 and 4,
I think that my splitter run has generated the corruption.
But since I remember also a corruption in step 1 some weeks ago,
this assumption could be wrong.

Walter

-----Ursprüngliche Nachricht-----
From: Gerd Petermann
Sent: Tuesday, August 15, 2017 7:19 PM
To: Development list for mkgmap
Subject: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Walter,

does that mean that you can still reproduce the corrupted file when using
the old maxnodes value?

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
Walter Schlögl <Walter.Schloegl-Resch at aon.at>
Gesendet: Dienstag, 15. August 2017 18:31:44
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Gerd,

thanks for this info.
This could happen by merging the daily diffs into europe.o5m
But since the working version was built out of the same file,
I could also happen during the splitting.

Very strange, but now I know what to look at if it happens again.

Walter

-----Ursprüngliche Nachricht-----
From: Gerd Petermann
Sent: Tuesday, August 15, 2017 3:07 PM
To: Development list for mkgmap
Subject: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Walter,

something is wrong with the data in 50100031.o5m
It contains a lot of very long lines which do not appear in OSM. I guess you
have merged different data sources
and they use similar OSM ids for different objects (typically this happens
with SRTM data). Please check your tool chain
and make sure that generated ids are much higher than those in real OSM
data. The node ids recently jumped over 5.000.000.000, see
https://osmstats.neis-one.org/?item=nodes

The tile in 50100025.o5m confirms that you have a tile that contains data
from italy and from Malta (which is a drive-on-left country).

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd
Petermann <GPetermann_muenchen at hotmail.com>
Gesendet: Dienstag, 15. August 2017 14:01:15
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Walter,

I'll see if I can reproduce the crash. Reg. the remaining message:
I guess the tile contains also Malta?

Gerd
________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
Walter Schlögl <Walter.Schloegl-Resch at aon.at>
Gesendet: Dienstag, 15. August 2017 12:36:31
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Gerd,

many thanks for your hints, with this I could solve all issues.

I have changed the splitter parameter from --max-nodes=800000 to 600000
but I have not updated the osm data to see what happens.
The message "There is not enough room" is gone now as expected,
but also the NullPointerException is gone.

I have copied the tile anyway if you still see a need for analysis.

https://1drv.ms/f/s!AqiYxUeSl56qgnCes1X0pcV4NIQ9

The only message that I still get for Italy is this.
SCHWERWIEGEND (StyledConverter): 50100025.o5m: Attention: Tile contains both
drive-on-left (22469) and drive-on-right roads (25540)

I would expect such a message for other countries, but not for Italy.
If this message is harmless, I will just ignore it.

Walter

-----Ursprüngliche Nachricht-----
From: Gerd Petermann
Sent: Monday, August 14, 2017 10:02 AM
To: Development list for mkgmap
Subject: Re: [mkgmap-dev] MapFailedException when generating map for Italy

Hi Walter,

I think you have several problems here.
1) The message "There is not enough room in a single garmin map for all the
input data. The .osm file should be split into smaller pieces first."
means that you should use a smaller maxnodes value  in splitter (or change
the style to add fewer data)
2) in 50100031.o5m there seems to be a data error. If I got that right the
error was already fixed in the OSM database (way 195550142 was changed
recently),
so I probably cannot reproduce the problem. Maybe post a link to your input
file.
3) Finally we have the NullPointerException. I can add a check to avoid
that, but I'd like to find out why this happens.
It might be related to the problems in 2) , so again I'd like to have the
input file 50100031.o5m.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
Walter Schlögl <Walter.Schloegl-Resch at aon.at>
Gesendet: Montag, 14. August 2017 09:41:44
An: Development list for mkgmap
Betreff: [mkgmap-dev] MapFailedException when generating map for Italy

I am generating many maps without any error,
only the area of Italy is throwing an error which I cannot solve.

I hope, anybody is having an idea, what could be the solution.

The coordinates are: 4.4°O – 21.1°O and 35°N - 45°N

The error message:

SCHWERWIEGEND (MapFailedException): 50100004.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (MapFailedException): 50100009.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (MapFailedException): 50100017.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (MapFailedException): 50100018.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (MapFailedException): 50100021.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (MapFailedException): 50100023.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (StyledConverter): 50100025.o5m: Attention: Tile contains both
drive-on-left (16159) and drive-on-right roads (21405)
SCHWERWIEGEND (MapFailedException): 50100025.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (MapFailedException): 50100028.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5717225 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5717225 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5717225 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5717225 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5327027 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5327027 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5327027 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (RouteArc): 50100031.o5m: Way SS116 Strada Statale 116
Randazzo - Capo d'Orlando (id 195550142) contains an arc whose length
(5327027 units) is too big to be encoded, using length 4194303
SCHWERWIEGEND (MapFailedException): 50100034.o5m: (thrown in
BufferedImgFileWriter.ensureSize()) There is not enough room in a single
garmin map for all the input data. The .osm file should be split into
smaller pieces first.
java.lang.NullPointerException
    at
uk.me.parabola.imgfmt.app.Coord.distanceInDegreesSquared(Coord.java:425)
    at uk.me.parabola.imgfmt.app.Coord.distance(Coord.java:409)
    at
uk.me.parabola.mkgmap.osmstyle.housenumber.HousenumberGroup.findSegment(HousenumberGroup.java:151)
    at
uk.me.parabola.mkgmap.osmstyle.housenumber.HousenumberRoad.detectGroups(HousenumberRoad.java:205)
    at
uk.me.parabola.mkgmap.osmstyle.housenumber.HousenumberRoad.buildIntervals(HousenumberRoad.java:104)
    at
uk.me.parabola.mkgmap.osmstyle.housenumber.HousenumberGenerator.generate(HousenumberGenerator.java:706)
    at
uk.me.parabola.mkgmap.osmstyle.StyledConverter.end(StyledConverter.java:624)
    at
uk.me.parabola.mkgmap.reader.osm.ElementSaver.convert(ElementSaver.java:246)
    at
uk.me.parabola.mkgmap.reader.osm.o5m.O5mBinMapDataSource.load(O5mBinMapDataSource.java:53)
    at
uk.me.parabola.mkgmap.reader.osm.OsmMapDataSource.load(OsmMapDataSource.java:138)
    at uk.me.parabola.mkgmap.main.MapMaker.loadFromFile(MapMaker.java:154)
    at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:52)
    at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:265)
    at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:261)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Exiting - if you want to carry on regardless, use the --keep-going option

Walter Schlögl
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev 