[mkgmap-dev] max-jobs patch

Thu Feb 8 08:56:56 GMT 2018

Hi Gerd,
Maybe my map is somehow special, but mkgmap definitely need more than 1gb heap per job. I will check on the weekend. Can you please post a patched mkgmap.jar? 

How about doing it the other way round and suggest the user to use max-job option if mkgmap think it is useful?

Henning 

On 8 Feb 2018, 16:38, at 16:38, Gerd Petermann <gpetermann_muenchen at hotmail.com> wrote:
>Hi Manfred,
>
>thanks for testing. Just to make this clear: I always use --max-jobs,
>because I also
>always use -Xmx whenever I compile many tiles.
>
>In this case a large tile is one that results in a large *.img
>(one with many MB). Typically this depends on the --max-nodes option
>for splitter
>and the style and the mkgmap options (routable or not, without or
>without DEM etc).
>
>The max-jobs option allows to specify the number of parallel jobs,
>e.g. --max-jobs=5 would start 5 parallel jobs (threads). Whenever one
>input file is done,
>the next one is started until all are finished. The patch only changes
>the default from 1
>to "number of cores". You can also use max-jobs=10 on a 2-core cpu, but
>this will slow down
>the run time.
>
>What I tried to point out is that a user who doesn't know about the
>--max--jobs option
>might as well not know about the meaning of the -Xmx JRE option.
>
>What happens if you start mkgmap on an 8-core machine with max-jobs=8
>and
>without -Xmx option? I think it will crash, and with the 2nd version of
>the patch it will tell
>the user to look at the -Xmx option. So far so good.
>What happens on a 4 core machine with max-jobs=4 and no -Xmx option?
>What if the OS is 32 bit only? This would not allow much more than
>-Xmx1700m.
>What if the option is used but set to a rather low value, e.g.
>-Xmx=1000m?
>
>My current thinking is this:
>If --max-jobs is not specified mkgmap should check the number of cores
>and the available heap
>and assume that one thread needs ~500M. 
>
>So, I would use something like num =
>Math.min(Runtime.getRuntime().maxMemory() / (512 * 1024 * 1024),
>Runtime.getRuntime().availableProcessors()) as default, maybe also for
>the case that one
>specifies --max-jobs .
>
>How does that sound?
>
>Gerd
>
>
>
>
>
>
>________________________________________
>Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
>Manfred Haiduk <mhaiduk at t-online.de>
>Gesendet: Mittwoch, 7. Februar 2018 22:28
>An: mkgmap-dev at lists.mkgmap.org.uk
>Betreff: Re: [mkgmap-dev] max-jobs patch
>
>I don't know, if Nordrhein-Westfalen is that kind of large tiles you
>meant. Splitter ended up with 43 files and i tried to make mapsource
>maps with and without --max-jobs option and mkgmap r4000. My computer
>does the job with 43 files about 12minutes faster than without this
>option. My laptop is a core i5 with 8Gb of memory. The command i use is
>like this line :
>
>java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar
>-Dlog.config=logging.properties
>C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar
>--location-autofill=is_in,nearest --mapname=70060001 --family-id=7006
>.\styles\mystyle.txt --series-name="STYLE TEST"
>--family-name="OpenStreetmap" --country-name=STYLETEST
>--country-abbr=STY --area-name=STY --overview-mapname="overview"
>--latin1 --style-file=.\styles --style=myland --max-jobs --keep-going
>--check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE
>--index --bounds=mybounds --route
>--name-tag-list=name,place_name,loc_name --housenumbers
>--x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip
>--tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args
>--description=MYSTYLE --remove-ovm-work-files=true
>
>Hope this gave you some useful hints
>
>
>Am 07.02.2018 um 20:06 schrieb Gerd Petermann:
>> Hi Mike,
>>
>> I did not yet try it. My understanding is that the Garbage Collection
>will try hard
>> to avoid an out of heap exception. In other words: A user might see
>better throughput
>> with fewer parallel jobs. What are your experiences? Did you try with
>> large maps, say > 40 large tiles ?
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
>von Mike Baggaley <mike at tvage.co.uk>
>> Gesendet: Mittwoch, 7. Februar 2018 01:58
>> An: 'Development list for mkgmap'
>> Betreff: Re: [mkgmap-dev] max-jobs patch
>>
>> Hi Gerd, please find attached v2 of the max-jobs patch.
>>
>> This version additionally catches out of heap memory exceptions and
>displays
>> a message suggesting either the use of the Java -Xmx option to
>increase the
>> available heap memory or the mkgmap --max-jobs option with a smaller
>value
>> to reduce the memory requirement (the latter is suggested only if
>using more
>> than one thread).
>>
>> How does that seem?
>> Regards,
>> Mike
>>
>> -----Original Message-----
>> From: Gerd Petermann [mailto:gpetermann_muenchen at hotmail.com]
>> Sent: 05 February 2018 07:28
>> To: Mike Baggaley <mike at tvage.co.uk>; 'Development list for mkgmap'
>> <mkgmap-dev at lists.mkgmap.org.uk>
>> Subject: Re: [mkgmap-dev] max-jobs patch
>>
>> Hi Mike,
>>
>> thought about this again. Maybe this change is too simple. With
>multiple
>> jobs one also
>> needs more heap (-Xmx JRE option). If you create rather large tiles
>with
>> splitter (max-nodes=1600000)
>> you need 0.5 - 1 GB for each job. Not sure what happens when a user
>creates
>> a map with
>> 10 tiles on an 8-core machine without any -Xmx option.
>>
>> I fear this will result in OutOfMemory exception, so better check the
>> available heap as well.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
>von Mike
>> Baggaley <mike at tvage.co.uk>
>> Gesendet: Sonntag, 4. Februar 2018 15:14
>> An: 'Development list for mkgmap'
>> Betreff: [mkgmap-dev] max-jobs patch
>>
>> Hi Gerd,
>>
>> Please find attached a patch that amends the default behaviour if the
>> --max-jobs option is not specified, to using a value equal to the
>number of
>> CPU cores, as suggested in a previous post. The documentation is also
>> amended to reflect the change. This halves the execution time of
>mkgmap for
>> building a map of Staffordshire on my 8-core PC when --max-jobs is
>not
>> specified (I didn't know about this option previously and was unaware
>the
>> performance could have been improved).
>>
>> Cheers,
>> Mike
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>--
>
>  Mit freundlichen Grüßen
>
>#####################################################
>Manfred Haiduk,
>Zum Fischbach 9,
>52393 Hürtgenwald
>e-mail mhaiduk at t-online.de
>#####################################################
>
>_______________________________________________
>mkgmap-dev mailing list
>mkgmap-dev at lists.mkgmap.org.uk
>http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>_______________________________________________
>mkgmap-dev mailing list
>mkgmap-dev at lists.mkgmap.org.uk
>http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20180208/773dd412/attachment.html>