[mkgmap-dev] splitter exception with SRTM data for USA

Thu Apr 11 16:59:49 BST 2013

Hi Thorsten,

I fear my idea regarding the numering of node ids will not help. 
The problem is the amount of nodes and the fact that the generated data probably 
is distributed in a way that the nodes with id x,x+1,x+2,..,x+n are very close
together. This makes it very likely that splitter reaches a limit of 2.147.483.648.
With normal OSM data, this is very unlikely.

I fear I have to revert the latest optimizations in SparseLong2ShortMapInline to remove that limit.

Gerd

> Date: Thu, 11 Apr 2013 16:54:13 +0200
> From: kukuk at suse.de
> To: mkgmap-dev at lists.mkgmap.org.uk
> Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
> 
> 
> Hi Gerd,
> 
> On Thu, Apr 11, GerdP wrote:
> 
> > Hi Thorsten,
> > 
> > ok, so I assume that you can patch pyghtmap to generate the ids as I
> > requested.
> 
> Yes, I will try that over the weekend, generating such big data needs
> some time.
> 
>    Thorsten
> 
> > In the mean time I will write a small generator which creates OSM data with
> > a large number of nodes. Maybe it is a bug in splitter, not a limit.
> > 
> > Gerd
> > 
> > 
> > Thorsten Kukuk wrote
> > > Hi,
> > > 
> > > On Thu, Apr 11, Felix Hartmann wrote:
> > > 
> > >> are you using the newest version of pyghtmap, newest splitter, and no 
> > >> older than 3-4 weeks mkgmap?
> > >> 
> > >> I had a similar problem with Asia. mkgmap splitter wouldn't split the 
> > >> pyghtmap extract - no matter what I tried (splitter didn't split but 
> > >> simply output one huge file instead). (about 6 month ago) - 1 month ago 
> > >> I redid everything using newest versions, and it worked...
> > > 
> > > Since I wrote a lot of the patches for the last phyghtmap versions,
> > > I use of course the newest one.
> > > 
> > > And it is only with the US. Canada, but that's a little bit
> > > smaller, doesn't create any problems.
> > > 
> > >   Thorsten
> > > 
> > >> On 11.04.2013 16:33, Thorsten Kukuk wrote:
> > >> > Hi Gerd,
> > >> >
> > >> > On Thu, Apr 11, Gerd Petermann wrote:
> > >> >
> > >> >> Hi Thorsten,
> > >> >>
> > >> >> another question: The wiki of phygthmap says that you don't need
> > >> splitting:
> > >> >> http://wiki.openstreetmap.org/wiki/Phyghtmap
> > >> >>
> > >> >> Why do you do it?
> > >> > Because I create one big data file with phyghtmap and then extract
> > >> > the parts I need. That's much easier and simpler then to let phyghtmap
> > >> > create a big amount of files and try to find and cut the ones you need.
> > >> >
> > >> >    Thorsten
> > >> >
> > >> >> Gerd
> > >> >>
> > >> >>> Date: Thu, 11 Apr 2013 14:06:30 +0200
> > >> >>> From: 
> > 
> > > kukuk@
> > 
> > >> >>> To: 
> > 
> > > mkgmap-dev at .org
> > 
> > >> >>> Subject: [mkgmap-dev] splitter exception with SRTM data for USA
> > >> >>>
> > >> >>>
> > >> >>> Hi,
> > >> >>>
> > >> >>> I generated SRTM data for the USA with phygthmap and wanted to try
> > >> >>> to split it now for compiling with mkgmap. But while this worked fine
> > >> >>> for all other countries, in this case I get always the following
> > >> >>> exception:
> > >> >>>
> > >> >>> MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994
> > >> of 65535
> > >> >>> Map details: bytes/overhead 401.218.630 / 166.809.090, overhead
> > >> includes 19 arrays with 8 MB
> > >> >>> 2.350.000.000 nodes processed... id=2526831732
> > >> >>> MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000
> > >> of 65535
> > >> >>> Map details: bytes/overhead 402.878.606 / 166.808.202, overhead
> > >> includes 19 arrays with 8 MB
> > >> >>> 2.360.000.000 nodes processed... id=2536831732
> > >> >>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException:
> > >> 524288
> > >> >>>          at
> > >> uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448)
> > >> >>>          at
> > >> uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191)
> > >> >>>          at
> > >> uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218)
> > >> >>>          at
> > >> uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167)
> > >> >>>          at
> > >> uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113)
> > >> >>>          at crosby.binary.BinaryParser.parse(Unknown Source)
> > >> >>>          at crosby.binary.BinaryParser.handleBlock(Unknown Source)
> > >> >>>          at crosby.binary.file.FileBlock.process(Unknown Source)
> > >> >>>          at crosby.binary.file.BlockInputStream.process(Unknown
> > >> Source)
> > >> >>>          at uk.me.parabola.splitter.Main.processMap(Main.java:792)
> > >> >>>          at
> > >> uk.me.parabola.splitter.Main.genProblemLists(Main.java:582)
> > >> >>>          at
> > >> uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613)
> > >> >>>          at uk.me.parabola.splitter.Main.split(Main.java:244)
> > >> >>>          at uk.me.parabola.splitter.Main.start(Main.java:157)
> > >> >>>          at uk.me.parabola.splitter.Main.main(Main.java:146)
> > >> >>>
> > >> >>> Any ideas? It's splitter r300.
> > >> >>>
> > >> >>> Splitter version unknown compiled 2013-04-10T21:31:05+0000
> > >> >>> boundary-tags=use-exclude-list
> > >> >>> cache=
> > >> >>> description=TK-USA-Tile
> > >> >>> geonames-file=osmmaps/scripts/cities/USA.txt
> > >> >>> keep-complete=true
> > >> >>> mapid=71510001
> > >> >>> max-areas=1024
> > >> >>> max-nodes=5000000
> > >> >>> max-threads=4 (auto)
> > >> >>> mixed=false
> > >> >>> no-trim=false
> > >> >>> output=pbf
> > >> >>> output-dir=build/usa/tiles-srtm
> > >> >>> overlap=0
> > >> >>> polygon-file=
> > >> >>> precomp-sea=data/sea/current
> > >> >>> problem-file=
> > >> >>> problem-report=
> > >> >>> resolution=13
> > >> >>> split-file=
> > >> >>> status-freq=120
> > >> >>> stop-after=dist
> > >> >>> write-kml=
> > >> >>>
> > >> >>>
> > >> >>> -- 
> > >> >>> Thorsten Kukuk, Project Manager/Release Manager SLES
> > >> >>> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg
> > >> >>> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG
> > >> Nürnberg)
> > >> >>> _______________________________________________
> > >> >>> mkgmap-dev mailing list
> > >> >>> 
> > 
> > > mkgmap-dev at .org
> > 
> > >> >>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >> >>   		 	   		
> > >> >> _______________________________________________
> > >> >> mkgmap-dev mailing list
> > >> >> 
> > 
> > > mkgmap-dev at .org
> > 
> > >> >> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > >> 
> > >> -- 
> > >> keep on biking and discovering new trails
> > >> 
> > >> Felix
> > >> openmtbmap.org & www.velomap.org
> > >> 
> > >> 
> > >> _______________________________________________
> > >> mkgmap-dev mailing list
> > >> 
> > 
> > > mkgmap-dev at .org
> > 
> > >> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > 
> > > -- 
> > > Thorsten Kukuk, Project Manager/Release Manager SLES
> > > SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg
> > > GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
> > > _______________________________________________
> > > mkgmap-dev mailing list
> > 
> > > mkgmap-dev at .org
> > 
> > > http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > 
> > 
> > 
> > 
> > 
> > --
> > View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5756629p5756654.html
> > Sent from the Mkgmap Development mailing list archive at Nabble.com.
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> -- 
> Thorsten Kukuk, Project Manager/Release Manager SLES
> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20130411/31b6b3a2/attachment-0001.html 