[mkgmap-dev] DEM File Format and mkgmap
From Gerd Petermann gpetermann_muenchen at hotmail.com on Tue Dec 5 10:17:54 GMT 2017
Hi all, I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen at hotmail.com> Gesendet: Dienstag, 5. Dezember 2017 09:26:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap Hi Nick, yes, I know that. I've fixed some problem in DemDisplay, so here is r507: http://files.mkgmap.org.uk/download/363/display.jar It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse. Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but the problem with that data is that I don't know for sure what values they contain. If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can reproduce the DEM file and finally trace what values Franks program creates. With a bit of luck this will help to fix the problems with the demo data. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk> Gesendet: Montag, 4. Dezember 2017 16:27:20 An: mkgmap-dev at lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap Hi Gerd I think all the dem files from the very old GBTopo can only be parsed up to a certain point I've included 3 excmples with tre http://files.mkgmap.org.uk/download/362/00145336.zip Nick On 04/12/2017 15:02, osm at pinns wrote: Hi Gerd Thanks for letting me know That is probably why eventually it stopped with repeated plateau len 64 messages on one of the dems builddemfile created I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar r Nick On 04/12/2017 13:49, Gerd Petermann wrote: Hi Nick, I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now... Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk><mailto:osm at pinns.co.uk> Gesendet: Montag, 4. Dezember 2017 14:35:15 An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap Many thanks Gerd Will have a go at it today and let you know my findings, if any ! Nick On 04/12/2017 13:27, Gerd Petermann wrote: Hi Nick, My understanding is that the prediction starts with the values from the header (min and max height in the tile) The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now. A compiled display.jar r505 is here: http://files.mkgmap.org.uk/download/361/display.jar If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip you can run e.g. java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s) which were the input for BuildDemFile . Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von osm at pinns <osm at pinns.co.uk><mailto:osm at pinns.co.uk> Gesendet: Montag, 4. Dezember 2017 13:00:27 An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap Hi Gerd Presumably some hgt data gets stored in a tile from which other data is predicted? Perhaps the first 'row'? Would be great if you could compile the DemDisplay.java so we can do ome testing as well. I'm glad you are almost 'getting there'! r Nick On 03/12/2017 11:16, Gerd Petermann wrote: Hi all, I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool. I still fails to read some Garmin DEM files but seems to work with the files produced by BuildDEMFile. My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does. I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon. Reg. the format: It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc. In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored. The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are needed to store the value. If the delta is larger than expected the so called BigBin encoding is used. The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits. A 2nd trick to save space is to use run length encoding for repeated values. So, the tricky part is to find out exactly how the prediction algo by Garmin works. Frank has done really great work here. For those who want to try the display tool: You have to compile on your own and you also need an installed mgkmap.jar. If that is available a command like this (replace d:\... with your paths) should produce a large txt file: java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt I am now searching for DEM files produced by BuildDEMFile which are not properly processed by display tool. When it fails you'll see a traceback like this Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532) at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831) at test.display.DemDisplay.readFileHeader(DemDisplay.java:887) at test.display.DemDisplay.print(DemDisplay.java:845) at test.display.CommonDisplay.display(CommonDisplay.java:171) at test.display.CommonDisplay.display(CommonDisplay.java:196) at test.display.DemDisplay.main(DemDisplay.java:903) Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
- Previous message: [mkgmap-dev] DEM File Format and mkgmap
- Next message: [mkgmap-dev] DEM File Format and mkgmap
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list