[mkgmap-dev] Performance with zipped hgt files

Mon Jan 8 11:52:05 GMT 2018

Hi Gerd,
how about using a compressed temp-format for DEM information, which is
more suitable for DEM-calculation?
Or even store DEM-information in temp folder <output-dir>/DEM and just
reuse them next time if possible. I think (at least for me) they don't
change that often. So this will have a high impact and should be quite
easy to implement.

These DEM-precompiled data mkgmap could calculate in a separate run
after user changed map areas.

I guess whole DEM needs to be recalculated if areas change, am I right?
That's why we can't store pre-calculated DEM-1°-Tiles.

Henning

On 08.01.2018 18:19, Gerd Petermann wrote:
> Hi all,
>
> I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 
> 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions)
> 2) reduce run time be storing more data in buffers (risking OutOfMemoryException)
> The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files,
> and I see no simple way to change that now.  
>
> I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution.
> The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%,
> when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.
>
> I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000.
> This gave me 174 tiles.
> I used the following command to create a map: 
> java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip   --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
>
> I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files.
> All worked fine, java heap memory was no problem, but run time was different:
> 953 secs for the uncompressed files
> 1084 secs for the NTFS compressed files
> 1538 secs for the zipped files
>
> I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems.
> I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data.  
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>