[mkgmap-dev] splitter r261: performance improvements

Sun Dec 16 18:03:51 GMT 2012

Hi Carlos,

I was able to reproduce the error message, and it disappeared when 
I removed the BufferedInputStream.
So, it seems that the pbf read routines have problems with buffered input. 
I'll look at that during the next days, for now I've just reverted the
change. For pbf files,
the performance gain wasn't that big anyway. For o5m, the
BufferedInputStream was 
always used, it was just made larger with r261.

Please try again with r262.

Ciao,
Gerd

Carlos Dávila-2 wrote
> I'm getting the error below using r261 on south-america.osm.pbf. r257 
> works fine for the same input file and parameters
> Starting multi-tile analyses pass 1
> Processing south-america.osm.pbf
> Bounding box -107.03999990000001 -61.829999900000004 -23.229999900000003 
> 13.25
> Exception in thread "main" java.lang.NegativeArraySizeException
>          at
> crosby.binary.file.FileBlockHead.readHead(FileBlockHead.java:53)
>          at crosby.binary.file.FileBlock.process(FileBlock.java:130)
>          at 
> crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
>          at uk.me.parabola.splitter.Main.processMap(Main.java:679)
>          at uk.me.parabola.splitter.Main.writeAreas(Main.java:599)
>          at uk.me.parabola.splitter.Main.split(Main.java:248)
>          at uk.me.parabola.splitter.Main.start(Main.java:154)
>          at uk.me.parabola.splitter.Main.main(Main.java:143)
> El 16/12/12 16:51, GerdP escribió:
>> Hi,
>>
>> I've committed r261: Changes
>> -improve throughput:
>>   + use BufferedInputStream with 4MB buffer for pbf and o5m input
>>   + reduced (oversized) o5m writer hashtable (saves around 635KB for each
>> open output file),
>>   + improved hash function to avoid collisions
>>   + avoid GC because of obsolete new String()
>>
>> On my machine, the effect is great:
>> When splitting germany.o5m with r260, the hard disk is rattling and
>> rattling, and it takes
>> between 75 and 110 secs to split the file (yes, very random results)
>> With r261, the disk sound is normal again and it takes more less
>> constantly 60 secs to split the same file with the same parms.
>>
>> I did some other tests regarding performance and found out that
>> splitter works best when input file is on a different disk (even a slow
>> USB 2.0 drive) and heap (-Xmx) is big enough to avoid massive GC.
>> It has no positive effect if you specify -Xmx7000m when -Xmx1600m is
>> already good enough.
>>
>> Ciao,
>> Gerd
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://gis.19327.n5.nabble.com/splitter-r261-performance-improvements-tp5740592.html
>> Sent from the Mkgmap Development mailing list archive at Nabble.com.
>> _______________________________________________
>> mkgmap-dev mailing list
>> 

> mkgmap-dev at .org

>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>>    
> 
> 
> -- 
> Por favor, no me envíe documentos con extensiones .doc, .docx, .xls,
> .xlsx, .ppt, .pptx, .mdb, mdbx
> Instale LibreOffice desde http://es.libreoffice.org/descarga/
> LibreOffice es libre: se puede copiar, modificar y redistribuir
> libremente. Gratis y totalmente legal.
> LibreOffice está en continuo desarrollo y no tendrá que pagar por las
> nuevas versiones.
> 
> _______________________________________________
> mkgmap-dev mailing list

> mkgmap-dev at .org

> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--
View this message in context: http://gis.19327.n5.nabble.com/splitter-r261-performance-improvements-tp5740592p5740604.html
Sent from the Mkgmap Development mailing list archive at Nabble.com.