[mkgmap-dev] DEM File Format and mkgmap

Tue Dec 5 13:33:14 GMT 2017

Hi Gerd

Excellent, will do so more hunting!

Nick

On 05/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk>
> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Gert
>
> I think I may have a dem that causes a problem
>
> http://files.mkgmap.org.uk/download/364/12345684.zip
>
> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
> .DemDisplay 12345684.DEM  1>log12345684.txt
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
> e bytes
>          at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
> 98)
>          at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
> 02)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
>          at test.display.DemDisplay.print(DemDisplay.java:816)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:875)
>
> It is part of OSM Greece
>
> r
>
> Nick
>
>
> On 05/12/2017 10:17, Gerd Petermann wrote:
>
> Hi all,
>
> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen at hotmail.com><mailto:gpetermann_muenchen at hotmail.com>
> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Nick,
> yes, I know that.
>
> I've fixed some problem in DemDisplay, so here is r507:
> http://files.mkgmap.org.uk/download/363/display.jar
>
> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
> the problem with that data is that I don't know for sure what values they contain.
> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
> reproduce the DEM file and finally trace what values Franks program creates.
> With a bit of luck this will help to fix the problems with the demo data.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk><mailto:osm at pinns.co.uk>
> Gesendet: Montag, 4. Dezember 2017 16:27:20
> An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>
> I've included 3 excmples with tre
>
> http://files.mkgmap.org.uk/download/362/00145336.zip
>
> Nick
>
> On 04/12/2017 15:02, osm at pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk><mailto:osm at pinns.co.uk><mailto:osm at pinns.co.uk><mailto:osm at pinns.co.uk>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk><mailto:osm at pinns.co.uk><mailto:osm at pinns.co.uk><mailto:osm at pinns.co.uk>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could  compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>             at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>             at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>             at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>             at test.display.DemDisplay.print(DemDisplay.java:845)
>             at test.display.CommonDisplay.display(CommonDisplay.java:171)
>             at test.display.CommonDisplay.display(CommonDisplay.java:196)
>             at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev