[mkgmap-dev] DEM File Format and mkgmap

Mon Dec 4 13:49:28 GMT 2017

Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk>
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: mkgmap-dev at lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick

On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could  compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
>> Hi all,
>>
>> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
>> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
>> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
>> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>>
>> Reg. the format:
>> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
>> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
>> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
>> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
>> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
>> A 2nd trick to save space is to use run length encoding for repeated values.
>> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
>> Frank has done really great work here.
>>
>> For those who want to try the display tool:
>> You have to compile on your own and you also need an installed mgkmap.jar.
>> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
>> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>>
>> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>>           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>>           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>>           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>>           at test.display.DemDisplay.print(DemDisplay.java:845)
>>           at test.display.CommonDisplay.display(CommonDisplay.java:171)
>>           at test.display.CommonDisplay.display(CommonDisplay.java:196)
>>           at test.display.DemDisplay.main(DemDisplay.java:903)
>>
>> Gerd
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev