[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

Wed Sep 8 20:36:26 BST 2021

so continued..
33659 GB
After Step 1:
new 33663 GB
now set all files in Product1 folder to read only. Only applies to files
not to the actual folder...
attrib +r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6* /s /d
attrib -r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1 /d

Step2:
Time started: Wed Sep 08 20:54:20 CEST 2021
Number of MapFailedExceptions: 0
SEVERE (global): Cannot write file java.nio.file.AccessDeniedException:
.\mtb_alp_08.09.2021.gmap\Product1\65280000\65280000.RGN
Number of ExitExceptions: 1
Time finished: Wed Sep 08 20:54:22 CEST 2021
Total time taken: 1 second

so this is not working bad idea....
lets try moving the files to a temp directory on the same drive (so it's
actually just a rename) and then run mkgmap again:
Starting with 33663GB
do Step 1 and move the files - and symlink them back...

-- for /D %%f in
(C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6*) do move %%f
C:\openmtbmap\maps\temp >NUL
SET SrcRoot=C:\openmtbmap\maps\temp\
SET TargetRoot=C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\

FOR /D %%A IN ("%SrcRoot%\*") DO (
    MKLINK /D "%TargetRoot%\%%~NA" "%%~A" ) 1>NUL 2>NUL
FOR %%A IN ("%SrcRoot%\*") DO (
    MKLINK "%TargetRoot%\%%~NXA" "%%~A" ) 1>NUL 2>NUL

After Step 1
33668 GB

After step 2
33669 GB

After step 3
33669 GB

After step 4:
33671 GB

So now we're down to 8GB for actual 7GB of maps created. Bingo we have
found a solution but it's really a lot of scripting to go there. I don't
know why setting files to read only does not work - but creating symlinks
does work with the tiny patch to mkgmap. I do think it would be in
everyones interest to implement a better strategy for mkgmap... And with
the loads of memory mkmap is taking up (25GB for me) - writing those .tmp
files concerning the address search into memory instead of to disk would
gain back the last ~1GB.

The alternative being of course to write everything to RAMDISK - if you
have loads of RAM - say 128GB this will even work for Europe or Asia
continent map (and I guess in this scenario ECC RAM really makes sense?) -
if you have 64GB you could try for all but Asia and Europe. Actually it
will work for Asia using my above hacks of symlinking the contourlines.
Because Asia alone is like 10GB contourlines on 20m. So if they are out of
the way - the rest will do on the RAMDISK. Again would be much easier if
those symlinks are not needed because mkgmap behaves SSD friendly and does
not write those files in first place. I have come down from 14GB to 8GB of
writes for the Alps. Maybe only 7.5GB (sound more plausible).

On Wed, 8 Sept 2021 at 21:44, Felix Hartmann <extremecarver at gmail.com>
wrote:

> Uncommenting line 102  from /combiners/gmapibuilder.java and then messing
> by making folder read only works (even though this is really like using
> crutches while being healthy)! And yeah it's great mkgmap code is so
> cleanly written that even someone at a loss with java can write a
> workaround. I really just believe no-one ever really noticed or minded
> enough to write here. There was a lot of optimization in making mkgmap
> faster or use less memory - but no developper so far cared for hdd/ssd
> writes. Actually I am pretty sure not writing those temp files or
> attempting to write the gmap files if they exist will be an easy speed up
> for mkgmap. Thanks Gerd and everyone else of the developper for writing
> such great software"
>
> as in the following example.
>
>>   } catch (IOException e) {
>> // throw new ExitException("Error saving gmapi data", e);
>> }
>
>
>
> So let's detail the writes on Alps step by step - I guess the best
> solution is to use a Ram Disk if mkgmap cannot be optimized - and then only
> write Asia and Europe Continent maps to NVME disk because I cannot create a
> big enough Ram Disk for them....
>
> So the log is only in full GB - so here it what happens:
> start at: 63648GB written to C:
> *1.* 7*.img are the 20m contourlines. They and the 9*.img contourlines
> are already present in gmap format and in the relavant gmap folder....
> mkgmap.jar is patched as above in order to not stop when it cannot write
> the 7* and 9* folders as they are read only. I'm thinking of making the 6*
> folder read only for the further steps?
> start /belownormal /b /wait java -jar -XX:+AggressiveHeap
> -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar
> --max-jobs=12 --order-by-decreasing-area  "--generate-sea" --code-page=1252
> "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip"
> "--style-file=C:\openmtbmap\openmtbmap_style"
> --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction
> --improve-overview --drive-on=detect --allow-reverse-merge
> --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d
>  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18"
> --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas
> --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance
> --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6
>  --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4
> --add-boundary-nodes-at-admin-boundaries=2
> --poi-excl-index=0x6405,0x4316,0x2f00
> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --cycle-map
> --ignore-fixme-values --housenumbers
> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
> --split-name-index --link-pois-to-ways --ignore-turn-restrictions
> --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20,
> 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_alp --show-profiles=1
>  --location-autofill=bounds,is_in,nearest
> --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip  --route
> --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528
> --product-id=1 --series-name=omtb_alps_08.09.2021
> --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi
> --overview-mapname=mapsetc --keep-going --area-name="alps_08.09.2021_omtb"
> -c D:\openmtbmap\maps\template.alps
> E:\OpenMTBMap\contourlines20\alps\7*.img typalp.TYP  1>NUL
> Mkgmap version 4806M
> Time started: Wed Sep 08 19:44:09 CEST 2021
> ...
> Time taken 10 minutes
>
> Now written
> 63652GB
> While it has actually written 2.12GB of .img files and 2.1GB for mac OSx
> files. Everything going pretty smooth here. Yes there have been some
> temporary files which I do not see why they are written if the max memory
> used was 25GB with 43GB heap (and never less than 37GB of RAM available)
>
> *2.*
> C:\openmtbmap\maps>start /belownormal /b /wait java -jar
> -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m
> C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252
> "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip"
> "--style-file=C:\openmtbmap\openmtbmap_style"
> --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction
> --improve-overview --drive-on=detect --allow-reverse-merge
> --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d
>  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18"
> --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas
> --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance
> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
> --split-name-index --housenumbers --description=omtb_alp --show-profiles=0
> --location-autofill=bounds,is_in,nearest
> --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route
> --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map
> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
> --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528
> --product-id=1 --series-name=omtb_alps_08.09.2021
> --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi
> --overview-mapname=mapsetx --keep-going --area-name="alps_08.09.2021_omtb"
> 6528*.img typalp.TYP  1>NUL 2>NUL
>
> Time taken - 30 seconds or so
> Written 460MB (twice 230MB for each mapset_mdr file)
> However new counter:
> 63655GB
> Waste - minimum 2GB. maybe 3GB..
>
> 3.
> C:\openmtbmap\maps>start /belownormal /b /wait java -jar
> -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m
> C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252
> "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip"
> "--style-file=C:\openmtbmap\openmtbmap_style"
> --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction
> --improve-overview --drive-on=detect --allow-reverse-merge
> --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d
>  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18"
> --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas
> --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance
> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
> --split-name-index --housenumbers --description=omtb_alp --show-profiles=1
> --location-autofill=bounds,is_in,nearest
> --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route
> --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map
> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
> --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528
> --product-id=1 --series-name=omtb_alps_08.09.2021
> --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi
> --overview-mapname=mapset10 --keep-going --area-name="alps_08.09.2021_omtb"
> 6528*.img typalp.TYP E:\OpenMTBMap\contourlines10\alps\9*.img  1>NUL 2>NUL
>
> Time taken 1 minute or so
> Written 460M (same as above)
> However new counter:
> 33657 GB
> (I guess the waste is the same as above - it cannot write the 9*.img - so
> each time around 2GB of wasted Write commands)
>
> 4.
> Let's create the gmapsupp.img - there is no more wasted writes here
> start /belownormal /b /wait java -jar -XX:+AggressiveHeap
> -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar
> --max-jobs=12 --hide-gmapsupp-on-pc
> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
> --code-page=1252 --family-id=6528  --housenumbers
> --description="mtb_alp_08.09.2021" --series-name=mtbmap_alps_08.09.2021
> --family-name=mtb_alp_08.09.2021 --product-id=1 --gmapsupp 6528*.img
> C:\openmtbmap\maps\widealp.TYP  1>NUL 2>NUL
> New counter:
> 33659 GB
>
> Overall summing it up:
> 2.57GB of .img mapdata supposed to write. 1.87GB for the gmapsupp.img ,
> 2.53GB for the gmap folder containing everything written (i moved the
> mapset files and the info.xml into subdirectories).
> So 7GB of actual mapdata needed 11GB of writes. And I already saved 2GB of
> writes by making the contourline gmapi folder read only... Before this I
> had an additional 2GB of writes.
>
> --------------------------------
> ------------------------------
>
> So let's try something new - we move the gmap 6* folder after the first
> compile and put a hardlink (mklink) back so mkgmap cannot fuss around with
> them anymore. Yeah we're using croutches so to say. I will post again the
> results if I can improve by moving stuff and putting symlinks or by setting
> read only attribute.
>
>
> On Wed, 8 Sept 2021 at 14:10, Felix Hartmann <extremecarver at gmail.com>
> wrote:
>
>> Quite a few people do not compile the contourlines each time they update
>> a map, but reuse the old contourlines. This works fine if you insert them
>> as say 1234*img (and they are named 12340000.img 12340001.img and so on).
>>
>> However this does not work when providing the --gmapi option. Actually
>> mkgmap tries to rewrite all those files again/convert them.
>>
>>
>> Could it please be possible for mkgmap to change this?
>> There could be the status quo with --gmapi
>> and there could be a new mode called
>> --gmapiminimal that only converts files input as .o5m or .osm.pbf or .osm
>> but not as .img
>>
>> --gpapiminimal would then work the same way as it does with .img files.
>> This has two use cases.
>>
>> a) if you have to resplit a tile because some tiles were breaking - and
>> directly gave the --gmapiminimal option - no need to redo the whole .gmap
>> file.
>>
>> b) you symlink all gmap folders with the contourlines (or other layers
>> you do not recreate on each update) into the gmap folder and mkgmap only
>> writes out the new stuff.
>>
>>
>> Yes on an nvme disk you barely notice the conversion - it's really quick.
>> BUT it is not needed if you have the files and even more - it burns your
>> NVME SSD disk.
>>
>> To my horror I noticed that one worldwide update of maps wrote 5TB of
>> data to my disk (thanks to NVME SSD and fast processor this all took 28
>> hours). At that rate even datacenter NVME ssd disks are quickly turned into
>> trash., consumer grade NVME disk will not make it half a year for me... In
>> general maybe mkgmap could have an option to write temporary files to
>> another disk? Then you use a RAMDisk for those files. Or mkgmap should keep
>> them in memory. Modern servers often have 64 or 128GB of RAM so enough RAM
>> for mkgmap not to waste writes to disk...
>> Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than
>> continous SSD writes....
>>
>>
>> Write now mkgmap crashes if you symlink already converted files to it:
>> e.g. message:
>> Number of MapFailedExceptions: 0
>> SEVERE (global): Error saving gmapi data
>> java.nio.file.FileAlreadyExistsException:
>> .\mtb_alp_08.09.2021.gmap\Product1\75280000
>>
>> While if it would be allowed to write out the data it would overwrite
>> existing files (which is fine or should do so - but same way as for .img
>> files)
>>
>>
>>
>> For my worldwide maps 10m contourlines are 200GB, 20m contourlines are
>> 100GB in size. Rendering the map in two styles and each time needing to
>> convert them even though I already have them converted creates 600GB of
>> useless writes per update (as well as useless time spend converting those
>> 300GB at least twice) - and that is only if I use the --gmapi option as a
>> separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile
>> failed to write - and I just parse it again.
>>
>> And that is only for the contourlines. As I need three sets of mapset
>> files - once without contourlines, once with 10m contourlines, and once
>> with 20m contourlines the actual map gmapi folders are written three times
>> instead of once... Well over 1TB of useless conversion time and useless
>> data written to disk (but I do not know another way to create the needed
>> mapset_mdr.img, mapset.img and mapset.tdb files)
>>
>> I don't know how to change that myself, but I hope this is not much work
>> to change.
>>
>>
>> --
>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>
>>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210908/b93e70c9/attachment-0001.html>