[mkgmap-dev] splitter crashes on Windows x64

Fri Jul 12 13:13:06 BST 2013

Hi,

well, thanks. I'm a programmer too ;)

I suffer these problems for over a half year, since I have started to 
publish OSM maps for Garmin fenix. I have used several spliter versions 
and updated java a few times. I think my installation is correct and 
while it's java that crashes, problem is connected to splitter.

Just have run 2 tests more, splitting Europe on Windows with smaller 
Xmx2000m size and splitting Europe on Linux Mint 13 with Xmx3000m.

On Windows java 1.7.0_21.
On Linux java 1.6.0_24.
On both system splitter 306 and europe-lates.osm.pbf from Geofabric.

Both java VM crashed at the same stage of processing, see logs. 
Consequently I would exclude an error in my installation.

While Java VM is crashing, the problem can have its trigger in HashMap 
class or in splitter code. For now I assume that there is some kind of 
incompatibility. I think you should put at least a warning in documentation.

This is end of the log from Windows 7, I have checked smaller heap 
limit, Xms2000m:

Stats for MultiTileProcessor pass 4
   SparseBitSet neededWays contains now 716 846 Ids.
   Number of stored tile combinations in multiTileDictionary: 7 689
   nodeWriterMap contains 37 325 218 pairs.
   wayWriterMap contains 716 551 pairs.
   relWriterMap contains 20 159 pairs.
Status: Finished processing problem lists.
Multi-tile analyses phase 4 took 323338 ms
-----------------------------------
Writing results of MultiTileAnalyser to temp files ...
Wrote 37325218 node pairs to F:\OSM\_Europa\pbf\node6306497458120114834.tmp
Wrote 716551 way pairs to F:\OSM\_Europa\pbf\way983025392980897089.tmp
Wrote 20159 rel pairs to F:\OSM\_Europa\pbf\rel6950999521421056268.tmp
Writing temp files took 7153 ms
Distributing data Fri Jul 12 13:26:09 CEST 2013
Processing 874 areas in 2 passes, 437 areas at a time
SparseLong2ShortMapInline: Allocating three-tier structure to save area 
info (HashMap->vector->chunkvector)
SparseLong2ShortMapInline: Allocating three-tier structure to save area 
info (HashMap->vector->chunkvector)
Compressed buffers are too short, causing extra copy
Compressed buffers are too short, causing extra copy
Compressed buffers are too short, causing extra copy
Compressed buffers are too short, causing extra copy
Starting distribution pass 1 of 2, processing 437 areas (8910000 to 8910436)
Processing h:\download\europe-latest.osm.pbf
Bounding box -34.492960000000004 30.81659 45.0 81.47299000000001
10 000 000 nodes processed... id=47476220
MAP occupancy: 10 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 10 640 032 / 17 147 808, overhead includes 2 
arrays with 8 MB
20 000 000 nodes processed... id=250740052
30 000 000 nodes processed... id=273134731
MAP occupancy: 20 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 20 950 432 / 25 617 944, overhead includes 3 
arrays with 8 MB
40 000 000 nodes processed... id=292210307
50 000 000 nodes processed... id=313151665
MAP occupancy: 30 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 32 320 480 / 25 663 976, overhead includes 3 
arrays with 8 MB
Elapsed time: 1h 2m 0s   Memory: Current 1627MB (696MB used, 931MB free) 
Max 1777MB
60 000 000 nodes processed... id=337198150
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000002833dc9, 
pid=1068, tid=1124
#
# JRE version: 7.0_21-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode 
windows-amd64 compressed oops)
# Problematic frame:
# J  java.util.HashMap.get(Ljava/lang/Object;)Ljava/lang/Object;
#
# Failed to write core dump. Minidumps are not enabled by default on 
client versions of Windows
#
# An error report file with more information is saved as:
# F:\OSM\_Europa\hs_err_pid1068.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

Here is the end of the log from Linux, -Xmx3000m:

Stats for MultiTileProcessor pass 4
   SparseBitSet neededWays contains now 716 846 Ids.
   Number of stored tile combinations in multiTileDictionary: 7 689
   nodeWriterMap contains 37 325 218 pairs.
   wayWriterMap contains 716 551 pairs.
   relWriterMap contains 20 159 pairs.
Status: Finished processing problem lists.
Multi-tile analyses phase 4 took 548398 ms
-----------------------------------
Writing results of MultiTileAnalyser to temp files ...
Elapsed time: 1h 42m 1s   Memory: Current 2127MB (806MB used, 1321MB 
free) Max 2666MB
Wrote 37325218 node pairs to 
/mnt/hgfs/wymiana/OSM/./node6451152352372778660.tmp
Wrote 716551 way pairs to /mnt/hgfs/wymiana/OSM/./way4965940815694883700.tmp
Wrote 20159 rel pairs to /mnt/hgfs/wymiana/OSM/./rel6910153637011523738.tmp
Writing temp files took 22516 ms
Distributing data Fri Jul 12 13:21:57 CEST 2013
Processing 874 areas in 2 passes, 437 areas at a time
SparseLong2ShortMapHuge: Allocating three-tier structure to save area 
info (HashMap->vector->chunkvector)
SparseLong2ShortMapHuge: Allocating three-tier structure to save area 
info (HashMap->vector->chunkvector)
Compressed buffers are too short, causing extra copy
Compressed buffers are too short, causing extra copy
Compressed buffers are too short, causing extra copy
Compressed buffers are too short, causing extra copy
Starting distribution pass 1 of 2, processing 437 areas (63240001 to 
63240437)
Processing europe-latest.osm.pbf
Bounding box -34.492960000000004 30.81659 45.0 81.47299000000001
10 000 000 nodes processed... id=47476220
MAP occupancy: 10 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 10 640 032 / 33 925 024, overhead includes 2 
arrays with 16 MB
20 000 000 nodes processed... id=250740052
Elapsed time: 1h 44m 1s   Memory: Current 2054MB (462MB used, 1592MB 
free) Max 2666MB
30 000 000 nodes processed... id=273134731
MAP occupancy: 20 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 20 950 432 / 50 783 768, overhead includes 3 
arrays with 16 MB
40 000 000 nodes processed... id=292210307
Elapsed time: 1h 46m 1s   Memory: Current 1969MB (492MB used, 1477MB 
free) Max 2666MB
50 000 000 nodes processed... id=313151665
Elapsed time: 1h 48m 1s   Memory: Current 2023MB (642MB used, 1381MB 
free) Max 2666MB
MAP occupancy: 30 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 32 320 480 / 50 829 800, overhead includes 3 
arrays with 16 MB
60 000 000 nodes processed... id=337198150
70 000 000 nodes processed... id=367030346
MAP occupancy: 40 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 44 741 700 / 51 105 244, overhead includes 3 
arrays with 16 MB
80 000 000 nodes processed... id=410489134
MAP occupancy: 50 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 51 385 724 / 67 898 164, overhead includes 4 
arrays with 16 MB
90 000 000 nodes processed... id=429362632
Elapsed time: 1h 50m 1s   Memory: Current 2068MB (823MB used, 1245MB 
free) Max 2666MB
100 000 000 nodes processed... id=473148396
110 000 000 nodes processed... id=515484334
MAP occupancy: 60 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 61 227 140 / 67 975 396, overhead includes 4 
arrays with 16 MB
120 000 000 nodes processed... id=531539659
130 000 000 nodes processed... id=575184969
MAP occupancy: 70 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 70 304 880 / 84 777 064, overhead includes 5 
arrays with 16 MB
140 000 000 nodes processed... id=602379096
Elapsed time: 1h 52m 1s   Memory: Current 2065MB (1179MB used, 886MB 
free) Max 2666MB
150 000 000 nodes processed... id=631641570
MAP occupancy: 80 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 78 655 398 / 85 181 402, overhead includes 5 
arrays with 16 MB
160 000 000 nodes processed... id=663150825
170 000 000 nodes processed... id=681033288
180 000 000 nodes processed... id=703767929
MAP occupancy: 90 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 86 859 152 / 101 976 440, overhead includes 
6 arrays with 16 MB
190 000 000 nodes processed... id=731035466
200 000 000 nodes processed... id=765822488
Elapsed time: 1h 54m 1s   Memory: Current 2040MB (1280MB used, 760MB 
free) Max 2666MB
210 000 000 nodes processed... id=785850597
MAP occupancy: 100 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 97 240 266 / 102 060 982, overhead includes 
6 arrays with 16 MB
220 000 000 nodes processed... id=798873240
230 000 000 nodes processed... id=819276167
MAP occupancy: 110 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 109 325 090 / 118 883 582, overhead includes 
7 arrays with 16 MB
240 000 000 nodes processed... id=838353048
250 000 000 nodes processed... id=858533136
260 000 000 nodes processed... id=879165465
MAP occupancy: 120 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 122 094 530 / 118 953 390, overhead includes 
7 arrays with 16 MB
Elapsed time: 1h 56m 1s   Memory: Current 2067MB (1470MB used, 597MB 
free) Max 2666MB
270 000 000 nodes processed... id=899391839
280 000 000 nodes processed... id=920069157
290 000 000 nodes processed... id=940588715
MAP occupancy: 130 000 000, number of area dictionary entries: 6310 of 65535
Map details: bytes/overhead 133 942 274 / 135 757 830, overhead includes 
8 arrays with 16 MB
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f255d15c4f7, pid=2600, tid=139797293168384
#
# JRE version: 6.0_24-b24
# Java VM: OpenJDK 64-Bit Server VM (20.0-b12 mixed mode linux-amd64 
compressed oops)
# Derivative: IcedTea6 1.11.5
# Distribution: Ubuntu 12.04 LTS, package 6b24-1.11.5-0ubuntu1~12.04.1
# Problematic frame:
# J  java.util.HashMap.containsKey(Ljava/lang/Object;)Z
#
# An error report file with more information is saved as:
# /mnt/hgfs/wymiana/OSM/hs_err_pid2600.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-6/
#
Aborted

-- 
Best regards
popej

 >Hi,
 >
 >AFAIK there should be no way for a java program to crash the JVM. So
 >in any case a crash of the JVM is a problem of the JVM and not of the
 >java program.
 >
 >So there is no chance for splitter to avoid such a crash.
 >
 >Please check if your 64bit JVM is installed correctly.