[mkgmap-dev] DEM File Format and mkgmap

Mon Dec 4 15:27:20 GMT 2017

Hi Gerd

I think all the dem files from the very old GBTopo can only be parsed up 
to a certain point

I've included 3 excmples with tre

http://files.mkgmap.org.uk/download/362/00145336.zip

Nick


On 04/12/2017 15:02, osm at pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped   with repeated plateau len 
> 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go 
> using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
>> Hi Nick,
>>
>> I just found out that display tool can not yet read DEM tiles with 
>> multiple zoom levels. Working on it now...
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag 
>> von osm at pinns <osm at pinns.co.uk>
>> Gesendet: Montag, 4. Dezember 2017 14:35:15
>> An: mkgmap-dev at lists.mkgmap.org.uk
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Many thanks Gerd
>>
>> Will have a go at it today and let you know my findings, if any !
>>
>> Nick
>>
>>
>> On 04/12/2017 13:27, Gerd Petermann wrote:
>>> Hi Nick,
>>>
>>> My understanding is that the prediction starts with the values from 
>>> the header (min and max height in the tile)
>>> The first row is indeed rather easy to decode. Frank has described 
>>> many more details in his pdf, and his program
>>> seems to create valid DEM data since Basecamp displays it, I just 
>>> try to reduce the number of special cases now.
>>>
>>> A compiled display.jar r505 is here:
>>> http://files.mkgmap.org.uk/download/361/display.jar
>>> If you also download and unzip this mkgmap package 
>>> http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
>>> you can run e.g.
>>> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 
>>> 12345678.DEM > log.txt
>>>
>>> I am interested in those files created by BuildDemFile which create 
>>> a run time exception. If you find one or more, I need the original 
>>> *.TRE file(s)
>>> which were the input for BuildDemFile .
>>>
>>> Gerd
>>> ________________________________________
>>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag 
>>> von osm at pinns <osm at pinns.co.uk>
>>> Gesendet: Montag, 4. Dezember 2017 13:00:27
>>> An: mkgmap-dev at lists.mkgmap.org.uk
>>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>>
>>> Hi Gerd
>>>
>>> Presumably some hgt data gets stored in a tile from which other data is
>>> predicted? Perhaps the first 'row'?
>>>
>>> Would be great if you could  compile the DemDisplay.java so we can do
>>> ome testing as well.
>>>
>>> I'm glad you are almost 'getting there'!
>>>
>>> r
>>>
>>> Nick
>>>
>>>
>>> On 03/12/2017 11:16, Gerd Petermann wrote:
>>>> Hi all,
>>>>
>>>> I think I understand now much more about this format. I used Frank 
>>>> Stinners code for BuildDEMFile and his pdf to write a decoder in 
>>>> the display tool.
>>>> I still fails to read some Garmin DEM files but seems to work with 
>>>> the files produced by  BuildDEMFile.
>>>> My current work flow is to generate DEM files with BuildDEMFile and 
>>>> read them with my decoder. If the decoder fails I can find out why 
>>>> because I can trace step by step what BuildDEMFile does.
>>>> I guess that I did not yet hit all special cases identified by 
>>>> Frank, but I am making good progress now, so I hope I'll be able to 
>>>> start implementing a DEM encoder in mkgmap soon.
>>>>
>>>> Reg. the format:
>>>> It is quite different to those used in other GARMIN sub files like 
>>>> NOD, NET, RGN etc.  In short: The data is organized in tiles, 
>>>> typically 64*64 values. Instead of storing each value
>>>> Garmin uses some statistics to predict the next value based on the 
>>>> already processed values. This predicted value is subtracted from 
>>>> the real value and only the delta is stored.
>>>> The better the prediction, the closer to 0 is the delta. The 
>>>> prediction also calculates the error-probability of the prediction 
>>>> and uses this to select the number of bits which are
>>>> needed to store the value.  If the delta is larger than expected 
>>>> the so called BigBin encoding is used.
>>>> The format to store the delta is optimized for values close to 0, 
>>>> so that many deltas are stored with only 1 or 2 bits.
>>>> A 2nd trick to save space is to use run length encoding for 
>>>> repeated values.
>>>> So, the tricky part is to find out exactly how the prediction algo 
>>>> by Garmin works.
>>>> Frank has done really great work here.
>>>>
>>>> For those who want to try the display tool:
>>>> You have to compile on your own and you also need an installed 
>>>> mgkmap.jar.
>>>> If that is available a command like this (replace d:\... with your 
>>>> paths) should produce a large txt file:
>>>> java -ea -Xmx1000m -cp 
>>>> d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar 
>>>> test.display.DemDisplay test.DEM > dem_decode.txt
>>>>
>>>> I am now searching for DEM files produced by BuildDEMFile which are 
>>>> not properly processed by display tool. When it fails you'll see a 
>>>> traceback like this
>>>> Exception in thread "main" java.lang.RuntimeException: did not read 
>>>> all avalaible bytes
>>>>            at 
>>>> test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532) 
>>>>
>>>>            at 
>>>> test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831) 
>>>>
>>>>            at 
>>>> test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>>>>            at test.display.DemDisplay.print(DemDisplay.java:845)
>>>>            at 
>>>> test.display.CommonDisplay.display(CommonDisplay.java:171)
>>>>            at 
>>>> test.display.CommonDisplay.display(CommonDisplay.java:196)
>>>>            at test.display.DemDisplay.main(DemDisplay.java:903)
>>>>
>>>> Gerd
>>>> _______________________________________________
>>>> mkgmap-dev mailing list
>>>> mkgmap-dev at lists.mkgmap.org.uk
>>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20171204/1bddd3e0/attachment-0001.html>