[mkgmap-dev] Splitter issue

Sat Feb 19 03:47:33 GMT 2011

On Wed, Feb 16, 2011 at 9:57 AM, Jeffrey Ollie <jeff at ocjtech.us> wrote:
> On Sun, Nov 7, 2010 at 3:05 PM, Nakor <nakor.osm at gmail.com> wrote:
>>
>> I was splitting the file from
>> http://download.geofabrik.de/osm/north-america/us-west.osm.pbf and got
>> the following error a couple times:
>>
>> Exception in thread "worker-0"
>> java.lang.StringIndexOutOfBoundsException: String index out of range: -2
>>     at java.lang.String.getChars(String.java:860)
>>     at uk.me.parabola.splitter.OSMWriter.writeString(OSMWriter.java:203)
>>     at uk.me.parabola.splitter.OSMWriter.write(OSMWriter.java:98)
>>     at
>> uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.processElement(SplitProcessor.java:294)
>>     at
>> uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.run(SplitProcessor.java:327)
>>     at java.lang.Thread.run(Thread.java:636)
>> Exception in thread "worker-1"
>> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>>     at java.lang.String.getChars(String.java:860)
>>     at uk.me.parabola.splitter.OSMWriter.writeString(OSMWriter.java:203)
>>     at uk.me.parabola.splitter.OSMWriter.write(OSMWriter.java:105)
>>     at
>> uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.processElement(SplitProcessor.java:294)
>>     at
>> uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.run(SplitProcessor.java:327)
>>     at java.lang.Thread.run(Thread.java:636)
>
> Has anyone had a chance to look into this?  I'm running into a similar
> issue when trying to split a .osm.pbf extract of the continental US
> generated by osmosis 0.38 from last week's planet file.  I'm going to
> try and look into this a bit more but it might take a while as each
> run of the splitter takes 90+ minutes for me.
>
> I'm using splitter rev 161 from the crosby_integration branch.

OK, I think I have this one solved, or at least worked around.  The
original code used a character array to buffer up output before
feeding it to the gzip compression routines.  I think that there was
an edge case in there that wasn't taken care of.  I solved it by
switching to the standard BufferedWriter to buffer the data before
sending it to be compressed.  I also had to change things around so
that only the complete XML for nodes, ways, and relations was sent to
the BufferedWriter because multiple threads are writing to the same
file.  My changes are slightly slower than the original code I think.
I didn't do a lot of performance testing because the server that I do
my map processing has a lot of background activity that would make any
timing unpredictable.

-- 
Jeff Ollie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: StringIndexOutOfBoundsException.patch
Type: text/x-patch
Size: 11615 bytes
Desc: not available
Url : http://lists.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20110218/d3b25a55/attachment.bin 