[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
From Felix Hartmann extremecarver at gmail.com on Mon Sep 13 11:28:36 BST 2021
I move away the info.xml - and use a different name for the mapset files and then just have mkgmap any file that is input in osm.pbf / osm / o5m format. Gmapi-minimal should not convert any file that is supplied as .img - as it can be assumed that those exist already (if they do not - then create them with --gmapi). That is in my opinion the best approach. So mkgmap does not even try to convert them. Afterwards I distribute a gmapi folder that includes all the data - and by replacing the info.xml you can enable or disable contourlines. For big countries the contourlines would be a separate download anyhow - so the user only needs to download the maps (including mapset files) but not redownload the contourlines. Same principle as in offering the contourlines as a separate gmapsupp.img file. so you have maps.img contourlines10m.img contourlines20m.img buildings.img .... On Mon, 13 Sept 2021 at 13:17, Gerd Petermann < gpetermann_muenchen at hotmail.com> wrote: > Hi Felix, > > sorry, the Data Deduplication as implemented in Windows Server would not > help here. It works after data was written. > > And yes, files which are not just copies of the *.img are written as > before. My understanding is that you have different product directories in > the gmapi folder and that you write protect those files which should be > kept. > If you have a script to zip the gmapi directory you have to exclude the > unwanted folders. > Didn't try it. Does it make sense? > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von > Felix Hartmann <extremecarver at gmail.com> > Gesendet: Montag, 13. September 2021 12:09 > An: Development list for mkgmap > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to > disk if used with --gmapi option > > Oh - but data was certainly written - A rename will not show as data > written in both Task manager on Windows, as well as in the smart data (I'm > using Windows 10 pro not Windows server however - maybe that functionality > is limited to windows server?) > > On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver at gmail.com > <mailto:extremecarver at gmail.com>> wrote: > Not sure if it makes it possible to use read only attribute instead of > moving and mklink. Maybe yes - because that was not possible before. So it > then would be set read only - instead of of move & mklink - and at the end > remove read only instead of moving back. > > On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver at gmail.com > <mailto:extremecarver at gmail.com>> wrote: > Thanks Gerd, > > but that is just removing the warning if it cannot overwrite, correct? > If it can overwrite the gmap file it will still overwrite it as I see.. So > in essence just a bit more intelligent then my disabling that line. > > I think it should not overwrite at all if present and --x-gmapi-minimal > (then you don't have to move away the files and link them back to the > original folder). > > On Mon, 13 Sept 2021 at 10:43, Gerd Petermann < > gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>> > wrote: > Hi all, > > attached is a quick patch that implements experimaltal option > --x-gmapi-minimal. > > If used instead of --gmapi mkgmap will not fail if a write-protected > output file exists in the gmapi output folder and mkgmap copies data from > *.img. It should still crash when other files like Info.xml are written. > > BTW: no conversion is done when those files are written. I think mkgmap > simply copies data from sub files in *.img into single files. So, the same > sequence of Bytes exists two or more times on the Computer. Windows Server > seems to support automatic data deduplication (1). I am sure Linux offers > similar support. No idea how effective or reliable it is, but it might be > worth trying. > > Gerd > (1) > > https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto: > mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila < > carlos at alternativaslibres.org<mailto:carlos at alternativaslibres.org>> > Gesendet: Sonntag, 12. September 2021 22:45 > An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to > disk if used with --gmapi option > > I think it's a good idea if mkgmap checks the required files are present. > > El 12/9/21 a las 21:16, Gerd Petermann escribió: > > Hi Felix, > > > > so you just want a new option so that mkgmap doesn't fail if it cannot > overwrite (write-protected) files in the output directory, right? No need > to verify the content of the exiting file(s)? > > > > Gerd > > > > ________________________________________ > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto: > mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < > extremecarver at gmail.com<mailto:extremecarver at gmail.com>> > > Gesendet: Sonntag, 12. September 2021 19:10 > > An: Development list for mkgmap > > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting > to disk if used with --gmapi option > > > > And well - it is burning SSD for the contourlines currently even if not > calling multiple times. 130GB minimum per worldwide map update for all > geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple > times and 10m and 20m I had about 2TB of useless writes per weekly map > update. I've got rid of all of them with my uncommenting the line, plus > saved about 500GB of writes by now doing all the gmapsupp.img and gmap > stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm > down to 1TB.. (and yeah the resplitting of tiles added another 5TB of > writes - but that could have been fixed easily by changing order of > commands too). > > > > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver at gmail.com > <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto: > extremecarver at gmail.com>>> wrote: > > because you need to do it if you want to show contourlines. Now > compiling the worldwide contourlines each time again - would burn SSD as > well - besides taking longer than just compiling the maps. So everyone who > offers maps for many countries as download puts the contourlines separate. > If you want to offer the choice of showing contourlines or not - mkmgap > will write the gmap contourlines once not needed, and once the maps not > needed. If you don't want to offer that option - it's only the contourlines > being written without need. Contourlines for all geofabrik extracts 20m > equidistance are about 130GB, 10m would be 260GB. > > If you offer 10m and 20m it will be even more not needed writes. > > > > The only solution is to go for a Ramdisk instead - However you likely > will need 128GB of RAM to do that - because for Asia or Europe you need a > 64GB Ramdisk. Same for North America extract (but few people offer that and > just have Canada and the US divided into the 4-5 zones). For other maps you > will get by with a 15-20GB Ramdisk (which I have resorted to now for all > but the windows installers because I don't want to let the server wait for > ages for the single thread NSIS wrapping up the data and instead start the > next country in parallel). And yes going RAMDISK already using my patch and > symlinking those files so mkgmap cannot overwrite them (would still be > faster if mkgmap didn't try in first place I think). If you want to write > out the contourlines as well besides the additional time - for Asia > continent you may then go for a 90GB Ramdisk minimum if offering windows > and gmap installers. In this case gmapsupp.img donwnloads aren't possible > anyhow due to the huge data amounts. For sm > > aller countries I have them too.... > > > > I coded around this now by using symlinks - but yeah that will be quite > a lot of work and is prone to break - while I guess it's only 10 lines or > so to add to mkgmap code to have a mode that does not write them out if > they are present - or if you tell on commandline they are present already. > For .img files they aren't overwritten either... > > > > On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < > gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com > ><mailto:gpetermann_muenchen at hotmail.com<mailto: > gpetermann_muenchen at hotmail.com>>> wrote: > > Hi Felix, > > > > did not read all the posts in detail. I understand that mkgmap is > neither burning SSDs nor doing any excessive writing unless you call it > multiple times for the same input files. So the question is: Why do you do > that? > > > > Gerd > > > > ________________________________________ > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto: > mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto: > mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto: > mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann < > extremecarver at gmail.com<mailto:extremecarver at gmail.com><mailto: > extremecarver at gmail.com<mailto:extremecarver at gmail.com>>> > > Gesendet: Freitag, 10. September 2021 00:02 > > An: Development list for mkgmap > > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting > to disk if used with --gmapi option > > > > Well I think the .tmp files are just building up - and the renamed. So > they are not causing any actual excessive write. > > As for the gmap - it would be cool if there is a mode to not write them. > > > > Actually it would be great if mkgmap could write all in one go. Because > the thing that takes so much time - is the address search - and that is > always the same. The differences are tiny (just because MapInstall is > crashing when files are missing) you need to compile them separately. > > > > but maybe there could be a mode where mkgamp writes all in one go. > > So family-name / family-name1 / family-name2 > > description / description1 / description2 > > input input1 input2 > > family-name.. > > show-profiles > > overview-mapname > > product-id > > (and maybe I missed some options are those that would need to be given > for each set of input tiles. And then just an option where you tell mkgmap > files starting with which first 4 numbers are relevant for address search. > No need to analyze if those other supplied .img (e.g. buildings or > contourlines) need to be added to address search.). I know coded around the > problem of the gmap files causing excessive writes. But yeah that is > actually really complicated be it on windows or linux... > > > > On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver at gmail.com > <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto: > extremecarver at gmail.com>><mailto:extremecarver at gmail.com<mailto: > extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto: > extremecarver at gmail.com>>>> wrote: > > Well I could give it 20 GB ram disk, maybe 32 but then I need to render > on less than 12 processes 64GB ram available). But that is not enough for > Asia continent map, and I guess super tight for Europe... > > > > Mkgmap could definitely keep those .tmp files in memory. But the > important bit is the gmap files not needeed to be written.... Also would > save quite some CPU time. > > > > On Wed, 8 Sep 2021, 14:31 ael <witwall3 at disroot.org<mailto: > witwall3 at disroot.org><mailto:witwall3 at disroot.org<mailto: > witwall3 at disroot.org>><mailto:witwall3 at disroot.org<mailto: > witwall3 at disroot.org><mailto:witwall3 at disroot.org<mailto: > witwall3 at disroot.org>>>> wrote: > > On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote: > >> Yes on an nvme disk you barely notice the conversion - it's really > quick. > >> BUT it is not needed if you have the files and even more - it burns your > >> NVME SSD disk. > > +1. I always use an old spinning rust disk when using mkgmap to save ssd > > write cycles, even without contours and such. It seems to be profligate > > in its use of disk cycles. I did try using RAM disk, but even with 16GB > > on a laptop, that was soon exhausted. > > > > ael > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk > ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto: > mkgmap-dev at lists.mkgmap.org.uk>><mailto:mkgmap-dev at lists.mkgmap.org.uk > <mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto: > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>>> > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk > ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto: > mkgmap-dev at lists.mkgmap.org.uk>> > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > > > > > -- > > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev at lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > -- Felix Hartman - Openmtbmap.org & VeloMap.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210913/2d87857d/attachment-0001.html>
- Previous message: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
- Next message: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the mkgmap-dev mailing list