[mkgmap-dev] Performance with zipped hgt files

Mon Jan 8 10:19:11 GMT 2018

Hi all,

I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 
1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions)
2) reduce run time be storing more data in buffers (risking OutOfMemoryException)
The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files,
and I see no simple way to change that now.  

I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution.
The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%,
when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.

I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000.
This gave me 174 tiles.
I used the following command to create a map: 
java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip   --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args

I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files.
All worked fine, java heap memory was no problem, but run time was different:
953 secs for the uncompressed files
1084 secs for the NTFS compressed files
1538 secs for the zipped files

I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems.
I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data.  

Gerd