[mkgmap-dev] Please try and report results of Huffman compression
From Gerd Petermann gpetermann_muenchen at hotmail.com on Sun Jan 16 17:14:35 GMT 2022
Hi Arndt, I don't see anything wrong in the diagnostic data. This seems to be a very small map. Please post a link to the tile(s) so that I can try to reproduce the crash here. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt at speichenkarte.de> Gesendet: Sonntag, 16. Januar 2022 17:16 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Please try and report results of Huffman compression Hi Gerd, i forget to activate the diagnostic function. I build the maps new, but it takes time. Fabrik and NRWplusare running now...... Here are the information from "Fabrik", the other map send i later. Greetz Arndt Mkgmap version 4867 Time started: Sun Jan 16 17:11:03 CET 2022 Number of MapFailedExceptions: 0 building Huffman tree for the following frequencies: [24620, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 648, 2, 0, 959, 47594, 3, 12, 4, 0, 303, 5763, 12, 2261, 2254, 4, 16, 52, 4086, 261, 356, 2610, 5213, 5125, 3989, 2721, 2168, 1694, 2113, 1539, 1641, 1840, 84, 1245, 170, 1320, 0, 3, 6207, 4855, 618, 1680, 2499, 1767, 3949, 3152, 784, 808, 2849, 2951, 1127, 3473, 976, 2938, 294, 4674, 7662, 1941, 809, 908, 6439, 2179, 5, 633, 2732, 0, 2732, 29, 607, 0, 24393, 5179, 9548, 8380, 44728, 3449, 9432, 12777, 11171, 21, 5132, 12721, 6529, 18350, 9267, 6653, 105, 31110, 14401, 20263, 8101, 2806, 2674, 465, 1074, 1629, 0, 870, 0, 38, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16, 0, 0, 0, 0, 0, 11, 0, 0, 6179, 0, 0, 1, 0, 477, 0, 0, 0, 3, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1200, 0, 0, 0, 0, 0, 1808, 0, 0, 0] depth:7 '4' 0x34 freq: 2721 depth:7 '[' 0x5b freq: 2732 depth:7 ']' 0x5d freq: 2732 depth:7 'v' 0x76 freq: 2806 depth:5 'i' 0x69 freq: 11171 depth:8 '>' 0x3e freq: 1320 depth:10 '/' 0x2f freq: 356 depth:11 '=' 0x3d freq: 170 depth:12 ';' 0x3b freq: 84 depth:12 'q' 0x71 freq: 105 depth:9 'I' 0x49 freq: 784 depth:7 'K' 0x4b freq: 2849 depth:6 '&' 0x26 freq: 5763 depth:7 'P' 0x50 freq: 2938 depth:7 'L' 0x4c freq: 2951 depth:6 'ß' 0xdf freq: 6179 depth:3 ' ' 0x20 freq: 47594 depth:4 'a' 0x61 freq: 24393 depth:4 '.' 0x00 freq: 24620 depth:6 'A' 0x41 freq: 6207 depth:7 'H' 0x48 freq: 3152 depth:8 '8' 0x38 freq: 1539 depth:9 'J' 0x4a freq: 808 depth:9 'U' 0x55 freq: 809 depth:5 'l' 0x6c freq: 12721 depth:5 'h' 0x68 freq: 12777 depth:6 'W' 0x57 freq: 6439 depth:6 'm' 0x6d freq: 6529 depth:8 'z' 0x7a freq: 1629 depth:8 '9' 0x39 freq: 1641 depth:8 'D' 0x44 freq: 1680 depth:8 '6' 0x36 freq: 1694 depth:6 'p' 0x70 freq: 6653 depth:7 'f' 0x66 freq: 3449 depth:7 'N' 0x4e freq: 3473 depth:8 'F' 0x46 freq: 1767 depth:9 '|' 0x7c freq: 870 depth:9 'V' 0x56 freq: 908 depth:8 'ü' 0xfc freq: 1808 depth:8 ':' 0x3a freq: 1840 depth:5 's' 0x73 freq: 14401 depth:6 'S' 0x53 freq: 7662 depth:10 'x' 0x78 freq: 465 depth:10 'ä' 0xe4 freq: 477 depth:9 ' ' 0x1f freq: 959 depth:8 'T' 0x54 freq: 1941 depth:7 'G' 0x47 freq: 3949 depth:4 'r' 0x72 freq: 31110 depth:7 '3' 0x33 freq: 3989 depth:7 '-' 0x2d freq: 4086 depth:6 'u' 0x75 freq: 8101 depth:6 'd' 0x64 freq: 8380 depth:9 'O' 0x4f freq: 976 depth:9 'y' 0x79 freq: 1074 depth:8 '7' 0x37 freq: 2113 depth:8 '5' 0x35 freq: 2168 depth:8 'X' 0x58 freq: 2179 depth:5 'n' 0x6e freq: 18350 depth:13 ',' 0x2c freq: 52 depth:15 '"' 0x22 freq: 12 depth:17 'è' 0xe8 freq: 3 depth:17 '#' 0x23 freq: 4 depth:16 'é' 0xe9 freq: 7 depth:14 '^' 0x5e freq: 29 depth:16 'Ä' 0xc4 freq: 8 depth:18 ' ' 0x1d freq: 2 depth:18 '' 0xad freq: 2 depth:19 '„' 0x84 freq: 1 depth:19 '°' 0xb0 freq: 1 depth:19 'â' 0xe2 freq: 1 depth:19 'ô' 0xf4 freq: 1 depth:15 'Ö' 0xd6 freq: 16 depth:15 '+' 0x2b freq: 16 depth:17 '*' 0x2a freq: 4 depth:17 'Y' 0x59 freq: 5 depth:16 'Ü' 0xdc freq: 11 depth:14 '~' 0x7e freq: 38 depth:15 'j' 0x6a freq: 21 depth:16 ''' 0x27 freq: 12 depth:18 'š' 0x9a freq: 3 depth:18 '!' 0x21 freq: 3 depth:18 '@' 0x40 freq: 3 depth:18 '–' 0x96 freq: 3 depth:11 '.' 0x2e freq: 261 depth:11 'Q' 0x51 freq: 294 depth:11 '%' 0x25 freq: 303 depth:9 'M' 0x4d freq: 1127 depth:8 ')' 0x29 freq: 2254 depth:7 'R' 0x52 freq: 4674 depth:6 'o' 0x6f freq: 9267 depth:6 'g' 0x67 freq: 9432 depth:8 '(' 0x28 freq: 2261 depth:9 'ö' 0xf6 freq: 1200 depth:10 '_' 0x5f freq: 607 depth:10 'C' 0x43 freq: 618 depth:7 'B' 0x42 freq: 4855 depth:6 'c' 0x63 freq: 9548 depth:8 'E' 0x45 freq: 2499 depth:9 '<' 0x3c freq: 1245 depth:10 'Z' 0x5a freq: 633 depth:10 ' ' 0x1c freq: 648 depth:7 '2' 0x32 freq: 5125 depth:5 't' 0x74 freq: 20263 depth:7 'k' 0x6b freq: 5132 depth:7 'b' 0x62 freq: 5179 depth:7 '1' 0x31 freq: 5213 depth:8 '0' 0x30 freq: 2610 depth:8 'w' 0x77 freq: 2674 depth:4 'e' 0x65 freq: 44728 Huffman code: 0000000000000000000 '„' 0x84 Huffman code: 0000000000000000001 '°' 0xb0 Huffman code: 0000000000000000010 'â' 0xe2 Huffman code: 0000000000000000011 'ô' 0xf4 Huffman code: 000000000000000010 ' ' 0x1d Huffman code: 000000000000000011 '' 0xad Huffman code: 000000000000000100 'š' 0x9a Huffman code: 000000000000000101 '!' 0x21 Huffman code: 000000000000000110 '@' 0x40 Huffman code: 000000000000000111 '–' 0x96 Huffman code: 00000000000000100 'è' 0xe8 Huffman code: 00000000000000101 '#' 0x23 Huffman code: 00000000000000110 '*' 0x2a Huffman code: 00000000000000111 'Y' 0x59 Huffman code: 0000000000000100 'é' 0xe9 Huffman code: 0000000000000101 'Ä' 0xc4 Huffman code: 0000000000000110 'Ü' 0xdc Huffman code: 0000000000000111 ''' 0x27 Huffman code: 000000000000100 '"' 0x22 Huffman code: 000000000000101 'Ö' 0xd6 Huffman code: 000000000000110 '+' 0x2b Huffman code: 000000000000111 'j' 0x6a Huffman code: 00000000000100 '^' 0x5e Huffman code: 00000000000101 '~' 0x7e Huffman code: 0000000000011 ',' 0x2c Huffman code: 000000000010 ';' 0x3b Huffman code: 000000000011 'q' 0x71 Huffman code: 00000000010 '=' 0x3d Huffman code: 00000000011 '.' 0x2e Huffman code: 00000000100 'Q' 0x51 Huffman code: 00000000101 '%' 0x25 Huffman code: 0000000011 '/' 0x2f Huffman code: 0000000100 'x' 0x78 Huffman code: 0000000101 'ä' 0xe4 Huffman code: 0000000110 '_' 0x5f Huffman code: 0000000111 'C' 0x43 Huffman code: 0000001000 'Z' 0x5a Huffman code: 0000001001 ' ' 0x1c Huffman code: 000000101 'I' 0x49 Huffman code: 000000110 'J' 0x4a Huffman code: 000000111 'U' 0x55 Huffman code: 000001000 '|' 0x7c Huffman code: 000001001 'V' 0x56 Huffman code: 000001010 ' ' 0x1f Huffman code: 000001011 'O' 0x4f Huffman code: 000001100 'y' 0x79 Huffman code: 000001101 'M' 0x4d Huffman code: 000001110 'ö' 0xf6 Huffman code: 000001111 '<' 0x3c Huffman code: 00001000 '>' 0x3e Huffman code: 00001001 '8' 0x38 Huffman code: 00001010 'z' 0x7a Huffman code: 00001011 '9' 0x39 Huffman code: 00001100 'D' 0x44 Huffman code: 00001101 '6' 0x36 Huffman code: 00001110 'F' 0x46 Huffman code: 00001111 'ü' 0xfc Huffman code: 00010000 ':' 0x3a Huffman code: 00010001 'T' 0x54 Huffman code: 00010010 '7' 0x37 Huffman code: 00010011 '5' 0x35 Huffman code: 00010100 'X' 0x58 Huffman code: 00010101 ')' 0x29 Huffman code: 00010110 '(' 0x28 Huffman code: 00010111 'E' 0x45 Huffman code: 00011000 '0' 0x30 Huffman code: 00011001 'w' 0x77 Huffman code: 0001101 '4' 0x34 Huffman code: 0001110 '[' 0x5b Huffman code: 0001111 ']' 0x5d Huffman code: 0010000 'v' 0x76 Huffman code: 0010001 'K' 0x4b Huffman code: 0010010 'P' 0x50 Huffman code: 0010011 'L' 0x4c Huffman code: 0010100 'H' 0x48 Huffman code: 0010101 'f' 0x66 Huffman code: 0010110 'N' 0x4e Huffman code: 0010111 'G' 0x47 Huffman code: 0011000 '3' 0x33 Huffman code: 0011001 '-' 0x2d Huffman code: 0011010 'R' 0x52 Huffman code: 0011011 'B' 0x42 Huffman code: 0011100 '2' 0x32 Huffman code: 0011101 'k' 0x6b Huffman code: 0011110 'b' 0x62 Huffman code: 0011111 '1' 0x31 Huffman code: 010000 '&' 0x26 Huffman code: 010001 'ß' 0xdf Huffman code: 010010 'A' 0x41 Huffman code: 010011 'W' 0x57 Huffman code: 010100 'm' 0x6d Huffman code: 010101 'p' 0x70 Huffman code: 010110 'S' 0x53 Huffman code: 010111 'u' 0x75 Huffman code: 011000 'd' 0x64 Huffman code: 011001 'o' 0x6f Huffman code: 011010 'g' 0x67 Huffman code: 011011 'c' 0x63 Huffman code: 111 ' ' 0x20 tab2: 31 11111 7 ' ' 0x20 tab2: 30 11110 7 ' ' 0x20 tab2: 29 11101 7 ' ' 0x20 tab2: 28 11100 7 ' ' 0x20 Huffman code: 1101 'a' 0x61 tab2: 27 11011 9 'a' 0x61 tab2: 26 11010 9 'a' 0x61 Huffman code: 1100 '.' 0x00 tab2: 25 11001 9 '.' 0x00 tab2: 24 11000 9 '.' 0x00 Huffman code: 1011 'r' 0x72 tab2: 23 10111 9 'r' 0x72 tab2: 22 10110 9 'r' 0x72 Huffman code: 1010 'e' 0x65 tab2: 21 10101 9 'e' 0x65 tab2: 20 10100 9 'e' 0x65 Huffman code: 10011 'i' 0x69 tab2: 19 10011 11 'i' 0x69 Huffman code: 10010 'l' 0x6c tab2: 18 10010 11 'l' 0x6c Huffman code: 10001 'h' 0x68 tab2: 17 10001 11 'h' 0x68 Huffman code: 10000 's' 0x73 tab2: 16 10000 11 's' 0x73 Huffman code: 01111 'n' 0x6e tab2: 15 01111 11 'n' 0x6e Huffman code: 01110 't' 0x74 tab2: 14 01110 11 't' 0x74 tab2: 13 01101 26 13 tab2: 12 01100 26 13 tab2: 11 01011 26 13 tab2: 10 01010 26 13 tab2: 9 01001 26 13 tab2: 8 01000 26 13 tab2: 7 00111 24 12 tab2: 6 00110 24 12 tab2: 5 00101 24 12 tab2: 4 00100 24 12 tab2: 3 00011 22 12 tab2: 2 00010 22 11 tab2: 1 00001 22 11 tab2: 0 00000 0 10 compressed/uncompressed MDR 15 size: 332289/475949 ratio ~0,698 Number of ExitExceptions: 0 Time finished: Sun Jan 16 17:12:07 CET 2022 Total time taken: 1 minute 4 seconds Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>> hat am 16.01.2022 16:53 geschrieben: Hi Arndt, please post the diagnostic information from mkgmap for the map(s) that causes the crash. Esp. I need the data following the text "following frequencies:" Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Arndt Röhrig <arndt at speichenkarte.de<mailto:arndt at speichenkarte.de>> Gesendet: Sonntag, 16. Januar 2022 16:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Please try and report results of Huffman compression Hi Gerd, i run a test with Speiche Deutschland, NRW+, Kanaren, Balearen, Alpen and "Fabrik"-a little cut around my home. size mkgmap-r4855 vs mkgmap-mdr2-r4867 (gmap) Alpen: 3,68 GB (3.954.902.039 Bytes) 3,67 GB (3.944.957.454 Bytes) Deutschland: 3,92 GB (4.215.119.117 Bytes) 3,91 GB (4.209.139.554 Bytes) Kanaren: 74,4 MB (78.077.953 Bytes) 74,2 MB (77.873.649 Bytes) NRW+ 3,06 GB (3.291.110.031 Bytes) 3,05 GB (3.284.340.528 Bytes) Balearen: 41,7 MB (43.772.910 Bytes) 41,5 MB (43.606.904 Bytes) The adress search in BaseCamp ist not OK: Balearen, Kanaren and Fabrik seems to be OK. Alpen: if you type a name from a city, there is no choice-list NRW+ and Germany: BaseCamp crash, when type a adress -> Fehler erkannt Es ist ein Problem mit BaseCamp aufgetreten App: BaseCamp At: 16.01.2022 16:35:40 (UTC1) OS: Windows 10 (64-bit) Processor: x86, Processor Level: 23, Processors:4, Model: 17 Stepping: 0, RAM: 14606744 BC.ADDRESSFINDCOMPONENTFINDERBASE_T.GETADDRESSFINDERRESULTS-1064-4.7.4.0 Es wurde versucht, im geschützten Speicher zu lesen oder zu schreiben. Dies ist häufig ein Hinweis darauf, dass anderer Speicher beschädigt ist. bei bc.AddressFindComponentFinderBase_t.GetAddressFinderResults(IList`1 A_0, Int32 A_1, String A_2) bei bc.AddressFindAutoCompleteComponentController_t.HandleAutoCompleteCitySearchRequest(AutoCompleteSearchRequestArgs_t aArgs) bei bc.AutoCompleteController_t.RequestItems() bei bc.AutoCompleteController_t.HandleTimerTick(Object sender, EventArgs e) bei System.Windows.Forms.Timer.OnTick(EventArgs e) bei System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m) bei System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam) bei System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg) bei System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData) bei System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context) bei System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context) bei System.Windows.Forms.Application.Run(Form mainForm) bei main(String[] aArgs) Active Map Product: Speiche_Deutschland_220116 Version: 1.00 Installed Map Products: Map Name: Globale Karte (Garmin Product) Map Id: 133758977 Map Version: 2.0 Map Type: Bundle Custom Drawn Map: No Digital Elevation Model: No Map Name: Speiche_Fabrik_220116 Map Id: 517734401 Map Version: 1.0 Map Type: Bundle Custom Drawn Map: Yes Digital Elevation Model: Yes Map Name: Speiche_Kanaren_220116 Map Id: 518848513 Map Version: 1.0 Map Type: Bundle Custom Drawn Map: Yes Digital Elevation Model: Yes Map Name: Speiche_NRWplus_220116 Map Id: 518914049 Map Version: 1.0 Map Type: Bundle Custom Drawn Map: Yes Digital Elevation Model: Yes Map Name: Speiche_Balearen_220116 Map Id: 518979585 Map Version: 1.0 Map Type: Bundle Custom Drawn Map: Yes Digital Elevation Model: Yes Map Name: Speiche_Deutschland_220116 Map Id: 519045121 Map Version: 1.0 Map Type: Bundle Custom Drawn Map: Yes Digital Elevation Model: Yes Map Name: Speiche_Alpen_220116 Map Id: 519110657 Map Version: 1.0 Map Type: Bundle Custom Drawn Map: Yes Digital Elevation Model: Yes Map Name: Worldwide Autoroute DEM Basemap,NR (Garmin Product) Map Id: 3302470335 Map Version: 5.1 Map Type: Card Custom Drawn Map: No Digital Elevation Model: Yes .NET Versions: .NET FW 1.0 not installed. .NET FW 1.1 not installed. .NET FW 2.0 with SP2 installed .NET FW 3.0 with SP2 installed .NET FW 3.5 with SP1 installed OpenGL Version 2.1 Attached File: MiniDump Greetz Arndt Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com><mailto:gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>> hat am 16.01.2022 10:38 geschrieben: Hi all, see https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4866 and https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4867 Please use https://www.mkgmap.org.uk/download/mkgmap-mdr2-r4867.zip for further tests. Gerd ________________________________________ Von: 7770 <7770 at foskan.eu<mailto:7770 at foskan.eu><mailto:7770 at foskan.eu<mailto:7770 at foskan.eu>>> Gesendet: Samstag, 15. Januar 2022 18:53 An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>> Cc: Gerd Petermann Betreff: Re: [mkgmap-dev] Please try and report results of Huffman compression Hi. https://files.mkgmap.org.uk/detail/544 ALLVARLIG (MultiPolygonRelation): folder/77700046.osm.pbf: Internal error: Failed to render MultiPolygonRelation 2970267 that was with --latin1 mkgmap version 4839 works well. Regards Karl On lördag 15 januari 2022 16:59:24 CET Gerd Petermann wrote: Hi all, during the last weeks Ticker and I decoded Garmin's String compression of the global index for the PC (the *_mdr.img) . In the mdr2 branch I've coded support for that in mkgmap and I think its ready for a first beta test. It works with codepages 1250 ..1258 or 65001. I've no idea how the 2-byte pages work, but Garmin seems to support that, too. Maybe I'll work on this later. The branch version produces around 200 lines of diagnostic output to show what happens when compression is enabled. This will be changed to debug level later. The encoding is enabled by default, special option --x-no-mdr15-compression can be used to turn it off. Note that the size of the *_mdr.img file will be smaller (which is esp. intersting for those who get close to the 2G limit) Note also that a zipped *_mdr.img can and will be larger since zip cannot remove as much redundancy. Note also that this branch includes the changes from the faster-mp branch and introduces a few changes which are relavant for --lower-case maps. How to test? Install the map in MapSource and check if you see any unexpected characters in the result lists or maybe even crashes while searching. If so, please try the same input files with the --x-no-mdr15-compression to verify if this is caused by compression or maybe other changes. Please don't yet publish the maps, but let me know if you see anything unexpected. Some changes reg. lower-case in this branch are still experimental. The binary is here: https://www.mkgmap.org.uk/download/mkgmap-mdr2-r4865.zip If you see errors which are caused by compression, please attach the diagnostic output. @Ticker: Please review the source. Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
- Previous message: [mkgmap-dev] Please try and report results of Huffman compression
- Next message: [mkgmap-dev] Please try and report results of Huffman compression
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list