[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

Wed Sep 8 20:52:50 BST 2021

oh sorry - made a typo at the first number. (there was no error in the
SMART data I just somehow wrote 3 instead of 6. Those 30TB of writes did
not suddenly disappear).

On Wed, 8 Sept 2021 at 22:36, Felix Hartmann <extremecarver at gmail.com>
wrote:

> so continued..
> 33659 GB
> After Step 1:
> new 33663 GB
> now set all files in Product1 folder to read only. Only applies to files
> not to the actual folder...
> attrib +r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6* /s /d
> attrib -r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1 /d
>
> Step2:
> Time started: Wed Sep 08 20:54:20 CEST 2021
> Number of MapFailedExceptions: 0
> SEVERE (global): Cannot write file java.nio.file.AccessDeniedException:
> .\mtb_alp_08.09.2021.gmap\Product1\65280000\65280000.RGN
> Number of ExitExceptions: 1
> Time finished: Wed Sep 08 20:54:22 CEST 2021
> Total time taken: 1 second
>
>
> so this is not working bad idea....
> lets try moving the files to a temp directory on the same drive (so it's
> actually just a rename) and then run mkgmap again:
> Starting with 33663GB
> do Step 1 and move the files - and symlink them back...
>
> -- for /D %%f in
> (C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6*) do move %%f
> C:\openmtbmap\maps\temp >NUL
> SET SrcRoot=C:\openmtbmap\maps\temp\
> SET TargetRoot=C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\
>
> FOR /D %%A IN ("%SrcRoot%\*") DO (
>     MKLINK /D "%TargetRoot%\%%~NA" "%%~A" ) 1>NUL 2>NUL
> FOR %%A IN ("%SrcRoot%\*") DO (
>     MKLINK "%TargetRoot%\%%~NXA" "%%~A" ) 1>NUL 2>NUL
>
> After Step 1
> 33668 GB
>
> After step 2
> 33669 GB
>
> After step 3
> 33669 GB
>
> After step 4:
> 33671 GB
>
> So now we're down to 8GB for actual 7GB of maps created. Bingo we have
> found a solution but it's really a lot of scripting to go there. I don't
> know why setting files to read only does not work - but creating symlinks
> does work with the tiny patch to mkgmap. I do think it would be in
> everyones interest to implement a better strategy for mkgmap... And with
> the loads of memory mkmap is taking up (25GB for me) - writing those .tmp
> files concerning the address search into memory instead of to disk would
> gain back the last ~1GB.
>
>
> The alternative being of course to write everything to RAMDISK - if you
> have loads of RAM - say 128GB this will even work for Europe or Asia
> continent map (and I guess in this scenario ECC RAM really makes sense?) -
> if you have 64GB you could try for all but Asia and Europe. Actually it
> will work for Asia using my above hacks of symlinking the contourlines.
> Because Asia alone is like 10GB contourlines on 20m. So if they are out of
> the way - the rest will do on the RAMDISK. Again would be much easier if
> those symlinks are not needed because mkgmap behaves SSD friendly and does
> not write those files in first place. I have come down from 14GB to 8GB of
> writes for the Alps. Maybe only 7.5GB (sound more plausible).
>
>
> On Wed, 8 Sept 2021 at 21:44, Felix Hartmann <extremecarver at gmail.com>
> wrote:
>
>> Uncommenting line 102  from /combiners/gmapibuilder.java and then messing
>> by making folder read only works (even though this is really like using
>> crutches while being healthy)! And yeah it's great mkgmap code is so
>> cleanly written that even someone at a loss with java can write a
>> workaround. I really just believe no-one ever really noticed or minded
>> enough to write here. There was a lot of optimization in making mkgmap
>> faster or use less memory - but no developper so far cared for hdd/ssd
>> writes. Actually I am pretty sure not writing those temp files or
>> attempting to write the gmap files if they exist will be an easy speed up
>> for mkgmap. Thanks Gerd and everyone else of the developper for writing
>> such great software"
>>
>> as in the following example.
>>
>>>   } catch (IOException e) {
>>> // throw new ExitException("Error saving gmapi data", e);
>>> }
>>
>>
>>
>> So let's detail the writes on Alps step by step - I guess the best
>> solution is to use a Ram Disk if mkgmap cannot be optimized - and then only
>> write Asia and Europe Continent maps to NVME disk because I cannot create a
>> big enough Ram Disk for them....
>>
>> So the log is only in full GB - so here it what happens:
>> start at: 63648GB written to C:
>> *1.* 7*.img are the 20m contourlines. They and the 9*.img contourlines
>> are already present in gmap format and in the relavant gmap folder....
>> mkgmap.jar is patched as above in order to not stop when it cannot write
>> the 7* and 9* folders as they are read only. I'm thinking of making the 6*
>> folder read only for the further steps?
>> start /belownormal /b /wait java -jar -XX:+AggressiveHeap
>> -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar
>> --max-jobs=12 --order-by-decreasing-area  "--generate-sea" --code-page=1252
>> "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip"
>> "--style-file=C:\openmtbmap\openmtbmap_style"
>> --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction
>> --improve-overview --drive-on=detect --allow-reverse-merge
>> --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d
>>  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18"
>> --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas
>> --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance
>> --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6
>>  --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4
>> --add-boundary-nodes-at-admin-boundaries=2
>> --poi-excl-index=0x6405,0x4316,0x2f00
>> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
>> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --cycle-map
>> --ignore-fixme-values --housenumbers
>> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
>> --split-name-index --link-pois-to-ways --ignore-turn-restrictions
>> --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20,
>> 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_alp --show-profiles=1
>>  --location-autofill=bounds,is_in,nearest
>> --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip  --route
>> --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528
>> --product-id=1 --series-name=omtb_alps_08.09.2021
>> --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi
>> --overview-mapname=mapsetc --keep-going --area-name="alps_08.09.2021_omtb"
>> -c D:\openmtbmap\maps\template.alps
>> E:\OpenMTBMap\contourlines20\alps\7*.img typalp.TYP  1>NUL
>> Mkgmap version 4806M
>> Time started: Wed Sep 08 19:44:09 CEST 2021
>> ...
>> Time taken 10 minutes
>>
>> Now written
>> 63652GB
>> While it has actually written 2.12GB of .img files and 2.1GB for mac OSx
>> files. Everything going pretty smooth here. Yes there have been some
>> temporary files which I do not see why they are written if the max memory
>> used was 25GB with 43GB heap (and never less than 37GB of RAM available)
>>
>> *2.*
>> C:\openmtbmap\maps>start /belownormal /b /wait java -jar
>> -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m
>> C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252
>> "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip"
>> "--style-file=C:\openmtbmap\openmtbmap_style"
>> --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction
>> --improve-overview --drive-on=detect --allow-reverse-merge
>> --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d
>>  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18"
>> --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas
>> --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance
>> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
>> --split-name-index --housenumbers --description=omtb_alp --show-profiles=0
>> --location-autofill=bounds,is_in,nearest
>> --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route
>> --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map
>> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
>> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
>> --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528
>> --product-id=1 --series-name=omtb_alps_08.09.2021
>> --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi
>> --overview-mapname=mapsetx --keep-going --area-name="alps_08.09.2021_omtb"
>> 6528*.img typalp.TYP  1>NUL 2>NUL
>>
>> Time taken - 30 seconds or so
>> Written 460MB (twice 230MB for each mapset_mdr file)
>> However new counter:
>> 63655GB
>> Waste - minimum 2GB. maybe 3GB..
>>
>> 3.
>> C:\openmtbmap\maps>start /belownormal /b /wait java -jar
>> -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m
>> C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252
>> "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip"
>> "--style-file=C:\openmtbmap\openmtbmap_style"
>> --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction
>> --improve-overview --drive-on=detect --allow-reverse-merge
>> --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d
>>  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18"
>> --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas
>> --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance
>> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
>> --split-name-index --housenumbers --description=omtb_alp --show-profiles=1
>> --location-autofill=bounds,is_in,nearest
>> --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route
>> --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map
>> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
>> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
>> --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528
>> --product-id=1 --series-name=omtb_alps_08.09.2021
>> --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi
>> --overview-mapname=mapset10 --keep-going --area-name="alps_08.09.2021_omtb"
>> 6528*.img typalp.TYP E:\OpenMTBMap\contourlines10\alps\9*.img  1>NUL 2>NUL
>>
>> Time taken 1 minute or so
>> Written 460M (same as above)
>> However new counter:
>> 33657 GB
>> (I guess the waste is the same as above - it cannot write the 9*.img - so
>> each time around 2GB of wasted Write commands)
>>
>> 4.
>> Let's create the gmapsupp.img - there is no more wasted writes here
>> start /belownormal /b /wait java -jar -XX:+AggressiveHeap
>> -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar
>> --max-jobs=12 --hide-gmapsupp-on-pc
>> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
>> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
>> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
>> --code-page=1252 --family-id=6528  --housenumbers
>> --description="mtb_alp_08.09.2021" --series-name=mtbmap_alps_08.09.2021
>> --family-name=mtb_alp_08.09.2021 --product-id=1 --gmapsupp 6528*.img
>> C:\openmtbmap\maps\widealp.TYP  1>NUL 2>NUL
>> New counter:
>> 33659 GB
>>
>> Overall summing it up:
>> 2.57GB of .img mapdata supposed to write. 1.87GB for the gmapsupp.img ,
>> 2.53GB for the gmap folder containing everything written (i moved the
>> mapset files and the info.xml into subdirectories).
>> So 7GB of actual mapdata needed 11GB of writes. And I already saved 2GB
>> of writes by making the contourline gmapi folder read only... Before this I
>> had an additional 2GB of writes.
>>
>> --------------------------------
>> ------------------------------
>>
>> So let's try something new - we move the gmap 6* folder after the first
>> compile and put a hardlink (mklink) back so mkgmap cannot fuss around with
>> them anymore. Yeah we're using croutches so to say. I will post again the
>> results if I can improve by moving stuff and putting symlinks or by setting
>> read only attribute.
>>
>>
>> On Wed, 8 Sept 2021 at 14:10, Felix Hartmann <extremecarver at gmail.com>
>> wrote:
>>
>>> Quite a few people do not compile the contourlines each time they update
>>> a map, but reuse the old contourlines. This works fine if you insert them
>>> as say 1234*img (and they are named 12340000.img 12340001.img and so on).
>>>
>>> However this does not work when providing the --gmapi option. Actually
>>> mkgmap tries to rewrite all those files again/convert them.
>>>
>>>
>>> Could it please be possible for mkgmap to change this?
>>> There could be the status quo with --gmapi
>>> and there could be a new mode called
>>> --gmapiminimal that only converts files input as .o5m or .osm.pbf or
>>> .osm but not as .img
>>>
>>> --gpapiminimal would then work the same way as it does with .img files.
>>> This has two use cases.
>>>
>>> a) if you have to resplit a tile because some tiles were breaking - and
>>> directly gave the --gmapiminimal option - no need to redo the whole .gmap
>>> file.
>>>
>>> b) you symlink all gmap folders with the contourlines (or other layers
>>> you do not recreate on each update) into the gmap folder and mkgmap only
>>> writes out the new stuff.
>>>
>>>
>>> Yes on an nvme disk you barely notice the conversion - it's really
>>> quick. BUT it is not needed if you have the files and even more - it burns
>>> your NVME SSD disk.
>>>
>>> To my horror I noticed that one worldwide update of maps wrote 5TB of
>>> data to my disk (thanks to NVME SSD and fast processor this all took 28
>>> hours). At that rate even datacenter NVME ssd disks are quickly turned into
>>> trash., consumer grade NVME disk will not make it half a year for me... In
>>> general maybe mkgmap could have an option to write temporary files to
>>> another disk? Then you use a RAMDisk for those files. Or mkgmap should keep
>>> them in memory. Modern servers often have 64 or 128GB of RAM so enough RAM
>>> for mkgmap not to waste writes to disk...
>>> Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than
>>> continous SSD writes....
>>>
>>>
>>> Write now mkgmap crashes if you symlink already converted files to it:
>>> e.g. message:
>>> Number of MapFailedExceptions: 0
>>> SEVERE (global): Error saving gmapi data
>>> java.nio.file.FileAlreadyExistsException:
>>> .\mtb_alp_08.09.2021.gmap\Product1\75280000
>>>
>>> While if it would be allowed to write out the data it would overwrite
>>> existing files (which is fine or should do so - but same way as for .img
>>> files)
>>>
>>>
>>>
>>> For my worldwide maps 10m contourlines are 200GB, 20m contourlines are
>>> 100GB in size. Rendering the map in two styles and each time needing to
>>> convert them even though I already have them converted creates 600GB of
>>> useless writes per update (as well as useless time spend converting those
>>> 300GB at least twice) - and that is only if I use the --gmapi option as a
>>> separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile
>>> failed to write - and I just parse it again.
>>>
>>> And that is only for the contourlines. As I need three sets of mapset
>>> files - once without contourlines, once with 10m contourlines, and once
>>> with 20m contourlines the actual map gmapi folders are written three times
>>> instead of once... Well over 1TB of useless conversion time and useless
>>> data written to disk (but I do not know another way to create the needed
>>> mapset_mdr.img, mapset.img and mapset.tdb files)
>>>
>>> I don't know how to change that myself, but I hope this is not much work
>>> to change.
>>>
>>>
>>> --
>>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>>
>>>
>>
>> --
>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>
>>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210908/043b3f14/attachment-0001.html>