[mkgmap-dev] Splitting srtm.osm fails

Thu Jul 28 01:27:18 BST 2011

On Wed, 27 Jul 2011 19:39:38 +0200, WanMil wrote:

>It is a regression if r161 works and r179 not so in this case the 
>problem should be tracked down.

This is a known limitation of the current splitter. It can handle node
IDs only up to something less than 2e09 because Java allows only 31 bit
numbers as index for an array.

Even if this could be solved, you would need huge amounts of memory to
process data with node IDs starting at 2e09 (like the srtm data),
because the amount of memory the splitter needs primilary depends on
the max. node ID (and not on the number of nodes) in the data. If there
are node IDs larger than about 1.7e09 the splitter needs even more
memory than a 32 bit JVM can provide.

Scott has explained this in
http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010598.html and
http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010629.html

Michael

>> Hi WanMil,
>>
>> the easiest way for me would be 'splitter-r161'. Do you know, where to get?
>>
>> BTW, the file with the srtm data is about 4,2 GB and has over 45 million
>> nodes.
>>
>> Josef
>>
>>
>> Am 26.07.2011 21:43, schrieb WanMil:
>>> Hi Josef,
>>>
>>> I have googled for "splitter mkgmap IndexOutOfBoundsException" and found
>>> a similar problem description:
>>> http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010627.html
>>>
>>> Maybe that helps?
>>>
>>> WanMil
>>>
>>>> No hints?
>>>>
>>>> Am 21.07.2011 08:36, schrieb Josef Latt:
>>>>> Hi,
>>>>>
>>>>> I get srtm data with 'mono Srtm2Osm.exe -bounds1 47.2 5.8 55.2 15.1
>>>>> -step 20 -cat 100 20 -large -corrxy 0.0005 0.0005 -o srtm.osm'
>>>>>
>>>>>
>>>>> Splitting (splitter-r179) the data with 'java -ea -Xmx2048M -jar
>>>>> /home/jolatt/OSM/splitter/splitter.jar --mapid=21000001 --mixed=yes
>>>>> --max-nodes=5000000 srtm.osm' into tiles I get the following error:
>>>>>
>>>>>
>>>>> **** WARNING: the default output type has changed to pbf, use
>>>>> --output=xml for .osm.gz files
>>>>>
>>>>> Elapsed time: 0s   Memory: Current 59MB (1MB used, 58MB free) Max 1820MB
>>>>> Time started: Thu Jul 21 08:22:09 CEST 2011
>>>>> Map is being split for resolution 13:
>>>>>    - area boundaries are aligned to 0x800 map units
>>>>>    - areas are multiples of 0x1000 map units wide and high
>>>>> Processing srtm.osm
>>>>> in 1 file
>>>>> Time: Thu Jul 21 08:22:09 CEST 2011
>>>>> Exact map coverage is (53.99924039840698,5.800073146820068) to
>>>>> (55.20007610321045,8.0000638961792)
>>>>> Trimmed and rounded map coverage is (54.0087890625,6.9873046875) to
>>>>> (55.2392578125,8.0419921875)
>>>>> Splitting nodes into areas containing a maximum of 5.000.000 nodes each...
>>>>> Area (54.0087890625,6.9873046875) to (55.2392578125,8.0419921875)
>>>>> contains 2.629 nodes. DONE!
>>>>> 1 areas:
>>>>> Area 21000001 covers (0x266800,0x4f800) to (0x274800,0x5b800)
>>>>> Writing out split osm files Thu Jul 21 08:22:09 CEST 2011
>>>>> Processing 1 areas in a single pass
>>>>> (54.0087890625,6.9873046875) to (55.2392578125,8.0419921875)
>>>>> Starting pass 1 of 1, processing 1 areas (21000001 to 21000001)
>>>>> Making SparseMultiMap
>>>>> Making SparseMultiMap
>>>>> Processing srtm.osm
>>>>> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index
>>>>> (31250001) is greater than or equal to list size (31250001)
>>>>> 	at
>>>>> it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList.java:258)
>>>>> 	at
>>>>> uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortMapInline.java:128)
>>>>> 	at
>>>>> uk.me.parabola.splitter.SparseInt2ShortMultiMap$Inner.put(SparseInt2ShortMultiMap.java:81)
>>>>> 	at
>>>>> uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMultiMap.java:31)
>>>>> 	at
>>>>> uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:209)
>>>>> 	at
>>>>> uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.java:118)
>>>>> 	at uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:243)
>>>>> 	at
>>>>> uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:57)
>>>>> 	at uk.me.parabola.splitter.Main.processMap(Main.java:412)
>>>>> 	at uk.me.parabola.splitter.Main.writeAreas(Main.java:368)
>>>>> 	at uk.me.parabola.splitter.Main.split(Main.java:190)
>>>>> 	at uk.me.parabola.splitter.Main.start(Main.java:118)
>>>>> 	at uk.me.parabola.splitter.Main.main(Main.java:107)
>>>>>
>>>>>
>>>>>
>>>>> I would try an older version of splitter, but where to download, or is
>>>>> there another idea?
>>>>>
>>>>> Greetings
>>>>> Josef Latt
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>
>>
>
>_______________________________________________
>mkgmap-dev mailing list
>mkgmap-dev at lists.mkgmap.org.uk
>http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev