[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

Wed Sep 8 12:10:52 BST 2021

Quite a few people do not compile the contourlines each time they update a
map, but reuse the old contourlines. This works fine if you insert them as
say 1234*img (and they are named 12340000.img 12340001.img and so on).

However this does not work when providing the --gmapi option. Actually
mkgmap tries to rewrite all those files again/convert them.

Could it please be possible for mkgmap to change this?
There could be the status quo with --gmapi
and there could be a new mode called
--gmapiminimal that only converts files input as .o5m or .osm.pbf or .osm
but not as .img

--gpapiminimal would then work the same way as it does with .img files.
This has two use cases.

a) if you have to resplit a tile because some tiles were breaking - and
directly gave the --gmapiminimal option - no need to redo the whole .gmap
file.

b) you symlink all gmap folders with the contourlines (or other layers you
do not recreate on each update) into the gmap folder and mkgmap only writes
out the new stuff.

Yes on an nvme disk you barely notice the conversion - it's really quick.
BUT it is not needed if you have the files and even more - it burns your
NVME SSD disk.

To my horror I noticed that one worldwide update of maps wrote 5TB of data
to my disk (thanks to NVME SSD and fast processor this all took 28 hours).
At that rate even datacenter NVME ssd disks are quickly turned into trash.,
consumer grade NVME disk will not make it half a year for me... In general
maybe mkgmap could have an option to write temporary files to another disk?
Then you use a RAMDisk for those files. Or mkgmap should keep them in
memory. Modern servers often have 64 or 128GB of RAM so enough RAM for
mkgmap not to waste writes to disk...
Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than
continous SSD writes....

Write now mkgmap crashes if you symlink already converted files to it:
e.g. message:
Number of MapFailedExceptions: 0
SEVERE (global): Error saving gmapi data
java.nio.file.FileAlreadyExistsException:
.\mtb_alp_08.09.2021.gmap\Product1\75280000

While if it would be allowed to write out the data it would overwrite
existing files (which is fine or should do so - but same way as for .img
files)

For my worldwide maps 10m contourlines are 200GB, 20m contourlines are
100GB in size. Rendering the map in two styles and each time needing to
convert them even though I already have them converted creates 600GB of
useless writes per update (as well as useless time spend converting those
300GB at least twice) - and that is only if I use the --gmapi option as a
separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile
failed to write - and I just parse it again.

And that is only for the contourlines. As I need three sets of mapset files
- once without contourlines, once with 10m contourlines, and once with 20m
contourlines the actual map gmapi folders are written three times instead
of once... Well over 1TB of useless conversion time and useless data
written to disk (but I do not know another way to create the needed
mapset_mdr.img, mapset.img and mapset.tdb files)

I don't know how to change that myself, but I hope this is not much work to
change.

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210908/c6725873/attachment.html>