[mkgmap-dev] Performance with zipped hgt files

Mon Jan 8 13:16:31 GMT 2018

Hi Gerd,
I was thinking of writing the 'final' DEM not only in img-container but
also in a temp-file. This temp file can be used for all updates of map
data without changes in areas.list.

More or less the work flow can be: splitter calculate areas.list, mkgmap
can calculate temporary DEM for areas.list. As long as I use the same
areas.list for the map, I can reuse the finished DEM. If I need to
change areas.list of course I need to let mkgmap calculate a new DEM.
Overview-DEM will only change, if total map area is changing. So
probably only needs to be calculated once per map. If we work like that
at least for me compilation time wouldn't be that important.

Basically the idea is to keep the static map-data and only update the
dynamic part of the data. The other advantage would be, that we can use
full amount of RAM during DEM-calculation, as there is no RAM used for
map- data and so on...

I don't know if it's really possible to store the 'final' DEM but at
least some pre-compiled file should be possible I assume. I'm not sure,
how img-files are written.

Henning

On 08.01.2018 20:42, Gerd Petermann wrote:
> Hi Henning,
> I also thought about that and I think it could work using a similar algo as for the DEM format in IMG files.
> We may have an index as with precomp sea data and rather small files compared to zip. On the other hand that format
> would not be used by any other application, so in the end you might  need the data twice.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm at hscholland.de>
> Gesendet: Montag, 8. Januar 2018 12:52:05
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Performance with zipped hgt files
>
> Hi Gerd,
> how about using a compressed temp-format for DEM information, which is
> more suitable for DEM-calculation?
> Or even store DEM-information in temp folder <output-dir>/DEM and just
> reuse them next time if possible. I think (at least for me) they don't
> change that often. So this will have a high impact and should be quite
> easy to implement.
>
> These DEM-precompiled data mkgmap could calculate in a separate run
> after user changed map areas.
>
> I guess whole DEM needs to be recalculated if areas change, am I right?
> That's why we can't store pre-calculated DEM-1°-Tiles.
>
> Henning
>
> On 08.01.2018 18:19, Gerd Petermann wrote:
>> Hi all,
>>
>> I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here:
>> 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions)
>> 2) reduce run time be storing more data in buffers (risking OutOfMemoryException)
>> The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files,
>> and I see no simple way to change that now.
>>
>> I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution.
>> The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%,
>> when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.
>>
>> I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000.
>> This gave me 174 tiles.
>> I used the following command to create a map:
>> java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip   --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
>>
>> I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files.
>> All worked fine, java heap memory was no problem, but run time was different:
>> 953 secs for the uncompressed files
>> 1084 secs for the NTFS compressed files
>> 1538 secs for the zipped files
>>
>> I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems.
>> I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data.
>>
>> Gerd
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>