[mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

Mon Sep 13 11:28:36 BST 2021

I move away the info.xml - and use a different name for the mapset files
and then just have mkgmap any file that is input in osm.pbf / osm / o5m
format. Gmapi-minimal should not convert any file that is supplied as .img
- as it can be assumed that those exist already (if they do not - then
create them with --gmapi). That is in my opinion the best approach. So
mkgmap does not even try to convert them.

Afterwards I distribute a gmapi folder that includes all the data - and by
replacing the info.xml you can enable or disable contourlines. For big
countries the contourlines would be a separate download anyhow - so the
user only needs to download the maps (including mapset files) but not
redownload the contourlines.  Same principle as in offering the
contourlines as a separate gmapsupp.img file. so you have maps.img
contourlines10m.img contourlines20m.img buildings.img ....

On Mon, 13 Sept 2021 at 13:17, Gerd Petermann <
gpetermann_muenchen at hotmail.com> wrote:

> Hi Felix,
>
> sorry, the Data Deduplication as implemented in Windows Server would not
> help here. It works after data was written.
>
> And yes, files which are not just copies of the *.img are written as
> before. My understanding is that you have different product directories in
> the gmapi folder and that you write protect those files which should be
> kept.
> If you have a script to zip the gmapi directory you have to exclude the
> unwanted folders.
> Didn't try it. Does it make sense?
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
> Felix Hartmann <extremecarver at gmail.com>
> Gesendet: Montag, 13. September 2021 12:09
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to
> disk if used with --gmapi option
>
> Oh - but data was certainly written - A rename will not show as data
> written in both Task manager on Windows, as well as in the smart data (I'm
> using Windows 10 pro not Windows server however - maybe that functionality
> is limited to windows server?)
>
> On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com>> wrote:
> Not sure if it makes it possible to use read only attribute instead of
> moving and mklink. Maybe yes - because that was not possible before. So it
> then would be set read only - instead of of move & mklink - and at the end
> remove read only instead of moving back.
>
> On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com>> wrote:
> Thanks Gerd,
>
> but that is just removing the warning if it cannot overwrite, correct?
> If it can overwrite the gmap file it will still overwrite it as I see.. So
> in essence just a bit more intelligent then my disabling that line.
>
> I think it should not overwrite at all if present and --x-gmapi-minimal
> (then you don't have to move away the files and link them back to the
> original folder).
>
> On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>
> wrote:
> Hi all,
>
> attached is a quick patch that implements experimaltal option
> --x-gmapi-minimal.
>
> If used instead of --gmapi mkgmap will not fail if a write-protected
> output file exists in the gmapi output folder and mkgmap copies data from
> *.img. It should still crash when other files like Info.xml are written.
>
> BTW: no conversion is done when those files are written. I think mkgmap
> simply copies data from sub files in *.img into single files. So, the same
> sequence of Bytes exists two or more times on the Computer. Windows Server
> seems to support automatic data deduplication (1). I am sure Linux offers
> similar support. No idea how effective or reliable it is, but it might be
> worth trying.
>
> Gerd
> (1)
>
> https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila <
> carlos at alternativaslibres.org<mailto:carlos at alternativaslibres.org>>
> Gesendet: Sonntag, 12. September 2021 22:45
> An: mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to
> disk if used with --gmapi option
>
> I think it's a good idea if mkgmap checks the required files are present.
>
> El 12/9/21 a las 21:16, Gerd Petermann escribió:
> > Hi Felix,
> >
> > so you just want a new option so that mkgmap doesn't fail if it cannot
> overwrite (write-protected) files in the output directory, right? No need
> to verify the content of the exiting file(s)?
> >
> > Gerd
> >
> > ________________________________________
> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <
> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>
> > Gesendet: Sonntag, 12. September 2021 19:10
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
> to disk if used with --gmapi option
> >
> > And well - it is burning SSD for the contourlines currently even if not
> calling multiple times. 130GB minimum per worldwide map update for all
> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple
> times and 10m and 20m I had about 2TB of useless writes per weekly map
> update. I've got rid of all of them with my uncommenting the line, plus
> saved about 500GB of writes by now doing all the gmapsupp.img and gmap
> stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm
> down to 1TB.. (and yeah the resplitting of tiles added another 5TB of
> writes - but that could have been fixed easily by changing order of
> commands too).
> >
> > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto:
> extremecarver at gmail.com>>> wrote:
> > because you need to do it if you want to show contourlines. Now
> compiling the worldwide contourlines each time again - would burn SSD as
> well - besides taking longer than just compiling the maps. So everyone who
> offers maps for many countries as download puts the contourlines separate.
> If you want to offer the choice of showing contourlines or not - mkmgap
> will write the gmap contourlines once not needed, and once the maps not
> needed. If you don't want to offer that option - it's only the contourlines
> being written without need. Contourlines for all geofabrik extracts 20m
> equidistance are about 130GB, 10m would be 260GB.
> > If you offer 10m and 20m it will be even more not needed writes.
> >
> > The only solution is to go for a Ramdisk instead - However you likely
> will need 128GB of RAM to do that - because for Asia or Europe you need a
> 64GB Ramdisk. Same for North America extract (but few people offer that and
> just have Canada and the US divided into the 4-5 zones). For other maps you
> will get by with a 15-20GB Ramdisk (which I have resorted to now for all
> but the windows installers because I don't want to let the server wait for
> ages for the single thread NSIS wrapping up the data and instead start the
> next country in parallel). And yes going RAMDISK already using my patch and
> symlinking those files so mkgmap cannot overwrite them (would still be
> faster if mkgmap didn't try in first place I think). If you want to write
> out the contourlines as well besides the additional time - for Asia
> continent you may then go for a 90GB Ramdisk minimum if offering windows
> and gmap installers. In this case gmapsupp.img donwnloads aren't possible
> anyhow due to the huge data amounts. For sm
> >   aller countries I have them too....
> >
> > I coded around this now by using symlinks - but yeah that will be quite
> a lot of work and is prone to break - while I guess it's only 10 lines or
> so to add to mkgmap code to have a mode that does not write them out if
> they are present - or if you tell on commandline they are present already.
> For .img files they aren't overwritten either...
> >
> > On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com
> ><mailto:gpetermann_muenchen at hotmail.com<mailto:
> gpetermann_muenchen at hotmail.com>>> wrote:
> > Hi Felix,
> >
> > did not read all the posts in detail. I understand that mkgmap is
> neither burning SSDs nor doing any excessive writing unless you call it
> multiple times for the same input files. So the question is: Why do you do
> that?
> >
> > Gerd
> >
> > ________________________________________
> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann <
> extremecarver at gmail.com<mailto:extremecarver at gmail.com><mailto:
> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>>
> > Gesendet: Freitag, 10. September 2021 00:02
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
> to disk if used with --gmapi option
> >
> > Well I think the .tmp files are just building up - and the renamed. So
> they are not causing any actual excessive write.
> > As for the gmap - it would be cool if there is a mode to not write them.
> >
> > Actually it would be great if mkgmap could write all in one go. Because
> the thing that takes so much time - is the address search - and that is
> always the same. The differences are tiny (just because MapInstall is
> crashing when files are missing) you need to compile them separately.
> >
> > but maybe there could be a mode where mkgamp writes all in one go.
> > So family-name / family-name1 / family-name2
> > description / description1 / description2
> > input input1 input2
> > family-name..
> > show-profiles
> > overview-mapname
> > product-id
> > (and maybe I missed some options are those that would need to be given
> for each set of input tiles. And then just an option where you tell mkgmap
> files starting with which first 4 numbers are relevant for address search.
> No need to analyze if those other supplied .img (e.g. buildings or
> contourlines) need to be added to address search.). I know coded around the
> problem of the gmap files causing excessive writes. But yeah that is
> actually really complicated be it on windows or linux...
> >
> > On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto:
> extremecarver at gmail.com>><mailto:extremecarver at gmail.com<mailto:
> extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto:
> extremecarver at gmail.com>>>> wrote:
> > Well I could give it 20 GB ram disk, maybe 32 but then I need to render
> on less than 12 processes 64GB ram available). But that is not enough for
> Asia continent map, and I guess super tight for Europe...
> >
> > Mkgmap could definitely keep those .tmp files in memory. But the
> important bit is the gmap files not needeed to be written.... Also would
> save quite some CPU time.
> >
> > On Wed, 8 Sep 2021, 14:31 ael <witwall3 at disroot.org<mailto:
> witwall3 at disroot.org><mailto:witwall3 at disroot.org<mailto:
> witwall3 at disroot.org>><mailto:witwall3 at disroot.org<mailto:
> witwall3 at disroot.org><mailto:witwall3 at disroot.org<mailto:
> witwall3 at disroot.org>>>> wrote:
> > On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
> >> Yes on an nvme disk you barely notice the conversion - it's really
> quick.
> >> BUT it is not needed if you have the files and even more - it burns your
> >> NVME SSD disk.
> > +1. I always use an old spinning rust disk when using mkgmap to save ssd
> > write cycles, even without contours and such. It seems to be profligate
> > in its use of disk cycles. I did try using RAM disk, but even with 16GB
> > on a laptop, that was soon exhausted.
> >
> > ael
> >
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk
> ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:
> mkgmap-dev at lists.mkgmap.org.uk>><mailto:mkgmap-dev at lists.mkgmap.org.uk
> <mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>>>
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >
> >
> > --
> > Felix Hartman - Openmtbmap.org & VeloMap.org
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk
> ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:
> mkgmap-dev at lists.mkgmap.org.uk>>
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> >
> >
> > --
> > Felix Hartman - Openmtbmap.org & VeloMap.org
> >
> >
> >
> > --
> > Felix Hartman - Openmtbmap.org & VeloMap.org
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210913/2d87857d/attachment-0001.html>