[mkgmap-dev] splitter exception with SRTM data for USA

Sat Apr 13 09:22:28 BST 2013

Hi all,

I see a few possible solutions for the problem:
1) simple but slow: use a smaller max-areas value so that splitter need one
more pass.
The only needed change in the code would be a test for the situation and an
appropriate error message.
2) I can add an option like --allow-huge-data which tells splitter to use a
different data structure
that can hold much more data, but requires more memory to hold the same
amount of data.
3) I can change splitter to always use this structure .

I'd prefer option 3) 

Attached is a patch that implements the "huge-data" structure.
huge_data_v1.patch
<http://gis.19327.n5.nabble.com/file/n5756885/huge_data_v1.patch>  

The compiled binary is here:
http://files.mkgmap.org.uk/download/104/splitter.jar

Details:
The "huge-data" structure uses arrays of longs (8 bytes) instead of ints (4
bytes) to
store information. The number of these arrays depends on the distribution of
(OSM) ids in the 
input file, not on the amount of nodes/ways etc.
For most users, the effect of the change should be rather small, with
current OSM data you will 
need ~180MB more memory in the "gen-problem-list" phase and in the "dist"
phase.
Worst case would be data with OSM ids that are generated with a random()
function and a huge range of numbers, say between 1 and 2 ^63. So, I hope
that no one codes such a generator ;-)

With the "huge-data" structure, splitter uses 30 instead of 19 bits to
address "chunk stores" of the same length,and there is room left to allow
larger "chunk stores", so these limits should not be reached within many
years.

Gerd

Thorsten Kukuk wrote
> Hi,
> 
> I generated SRTM data for the USA with phygthmap and wanted to try
> to split it now for compiling with mkgmap. But while this worked fine
> for all other countries, in this case I get always the following
> exception:
> 
> MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of
> 65535
> Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes
> 19 arrays with 8 MB
> 2.350.000.000 nodes processed... id=2526831732
> MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of
> 65535
> Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes
> 19 arrays with 8 MB
> 2.360.000.000 nodes processed... id=2536831732
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException:
> 524288
>         at
> uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448)
>         at
> uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191)
>         at
> uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218)
>         at
> uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167)
>         at
> uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113)
>         at crosby.binary.BinaryParser.parse(Unknown Source)
>         at crosby.binary.BinaryParser.handleBlock(Unknown Source)
>         at crosby.binary.file.FileBlock.process(Unknown Source)
>         at crosby.binary.file.BlockInputStream.process(Unknown Source)
>         at uk.me.parabola.splitter.Main.processMap(Main.java:792)
>         at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582)
>         at
> uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613)
>         at uk.me.parabola.splitter.Main.split(Main.java:244)
>         at uk.me.parabola.splitter.Main.start(Main.java:157)
>         at uk.me.parabola.splitter.Main.main(Main.java:146)
> 
> Any ideas? It's splitter r300.
> 
> Splitter version unknown compiled 2013-04-10T21:31:05+0000
> boundary-tags=use-exclude-list
> cache=
> description=TK-USA-Tile
> geonames-file=osmmaps/scripts/cities/USA.txt
> keep-complete=true
> mapid=71510001
> max-areas=1024
> max-nodes=5000000
> max-threads=4 (auto)
> mixed=false
> no-trim=false
> output=pbf
> output-dir=build/usa/tiles-srtm
> overlap=0
> polygon-file=
> precomp-sea=data/sea/current
> problem-file=
> problem-report=
> resolution=13
> split-file=
> status-freq=120
> stop-after=dist
> write-kml=
> 
> 
> -- 
> Thorsten Kukuk, Project Manager/Release Manager SLES
> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
> _______________________________________________
> mkgmap-dev mailing list

> mkgmap-dev at .org

> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--
View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5756629p5756885.html
Sent from the Mkgmap Development mailing list archive at Nabble.com.