[mkgmap-dev] max-jobs patch

Sat Feb 10 15:19:04 GMT 2018

Hi Gerd,

these are my runtimes of several compilations of my Turkey map, splitted
with 1.500.000 nodes, compiled witout index, gmapi and gmappsupp, but
including DEM and routing etc.

RAM in GB    Threads     Time1 in ms    Time2 in ms
12                    8                328106            323605
8                      8                323135            341242
4                      8                378460            408714
4                      7                364123            361501
4                     6                 349048            343866
4                      4                371786            366581
4                       1                845778            797376
2                       4                527267            509698
2                        1               804117            853805

It seems a lack of RAM slows more down if there are two many jobs. For
4GB Ram, 6 threads are faster than 8 threads, even using only 4 threads
is faster than using 8.

Henning

On 10.02.2018 16:41, Gerd Petermann wrote:
On 10.02.2018 16:41, Gerd Petermann wrote:
> Hi Henning,
>
> thanks for the numbers. I assume that your JRE sets xmx=4G because you probably have 16G.
> My machine has 4 cores and 8G and the JRE sets xmx=2G when I don't specify a value.
> When you compile a large number of tiles peak memory is reached when building the index.
>
> Maybe you can do some more tests with a smaller set of tiles?
> I'd like to know if the run time changes much when you compile e.g. 32 tiles with
> a) -Xmx8GB and --max-jobs=8
> b) -Xmx4GB and --max-jobs=8
> c) -Xmx4GB and --max-jobs=7
> d) -Xmx4GB and --max-jobs=6
>
> I suggest to do each test at least 2 times and start with one unmeasured test to reduce
> IO caching effects. You should also not use --index, --gmapi or --gmapsupp for these tests.
>
> Gerd
>
>
> ________________________________________
> Von: Henning Scholland <osm at hscholland.de>
> Gesendet: Samstag, 10. Februar 2018 07:23
> An: Gerd Petermann; Development list for mkgmap
> Betreff: Re: AW: [mkgmap-dev] max-jobs patch
>
> Hi Gerd,
> Hi Mike,
>
> I gave it a try on all my maps (splitted with max-nodes=1.600.000).
> Removed xmx and max jobs. mkgmap used all 8 logical cores and about 4GB
> of memory. So it seems, Gerd was correct with the 500mb per job. So I
> would suggest to limit max-jobs with 32bit java to 3 jobs and to
> RAM/500mb on 64bit java. The used jobs should be minimum of both CPU and
> RAM.
>
> Regarding to higher amount of RAM I was mentioning before, I was calling
> mkgmap with xmx=12G, maybe java is not freeing the heap in that case. It
> was during dem-branch. Maybe during that time mkgmap wasn't also that
> efficient.
>
> Henning
>