[mkgmap-dev] [PATCH] Fix crash when parsing invalid UTF-8 files (WanMil)

Tue Oct 25 19:47:09 BST 2011

>> Thanks for the patch.
>> Can you post the exception you want to catch in your patch? The patch covers a problem which I think should be fixed at its origin. So a stacktrace would be very helpful.
>
>> Thanks!
>> WanMil
>
> The origin is either JVM6 or the benelux dataset, depending on how you
> look at it :)
>
> The complete trace and original mail message I send some time ago:
>
> mkgmap r2023 (and many older version) crash when running with --road-name-pois
> on file like in http://planet.openstreetmap.nl/benelux
>
> ++ /usr/lib/jvm/java-6-sun/bin/java -ea -Xmx4000m -jar
> ../mkgmap-r2023/mkgmap.jar --style-file=../aiostyles/basemap_style
> --tdbfile --gmapsupp --max-jobs --adjust-turn-headings
> --reduce-point-density=5 --merge-lines --route --remove-short-arcs=1
> --link-pois-to-ways --family-id=4 --product-id=45
> --family-name=OpenStreetmap --draw-priority=29 --add-pois-to-areas
> --road-name-pois 63240001.osm.pbf [...snip...] 63240075.osm.pbf
> ../aiostyles/basemap.TYP
> java.lang.IllegalStateException: Current state = FLUSHED, new state = CODING_END
>         at java.nio.charset.CharsetEncoder.throwIllegalStateException(CharsetEncoder.java:951)
>         at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:537)
>         at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:766)
>         at uk.me.parabola.imgfmt.app.srt.Sort.createSortKey(Sort.java:96)
>         at uk.me.parabola.imgfmt.app.srt.Sort.createSortKey(Sort.java:150)
>         at uk.me.parabola.mkgmap.main.MapMaker.makeRoadNamePOIS(MapMaker.java:266)
>         at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:64)
>         at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:222)
>         at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:219)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
>
> Regards,
> Rutger.

Thanks Rutger,

I think the problem is that the mkgmap Sort object is unsynchronized but 
used by several threads. If they use the internal encoder at the same 
time the given exception occurs. I fear that some other problems could 
also be caused by this (e.g. sorting problems with the index?!?)

You can use --max-jobs=1 as workaround (although map creation will take 
much longer :-(.)

The problem should be fixed by using a separate Sort object for each 
thread. One solution would be if CommandArgs.getSort() returns a copy. I 
am not sure how to create a valid copy of a Sort object.
Steve, can you help?

WanMil