[mkgmap-dev] Slowness processing specific tile

Thu May 19 20:52:31 BST 2011

>>>> I noticed a big hang/delay processing one tile in NC. Here's the
>>>> generated IMG files for each tile:
>>>>
>>>> 05/19/2011  09:33 AM         4,230,144 63240001.img
>>>> 05/19/2011  09:33 AM         2,450,432 63240002.img
>>>> 05/19/2011  09:33 AM         2,893,824 63240003.img
>>>> 05/19/2011  09:33 AM         2,170,880 63240004.img
>>>> 05/19/2011  09:33 AM         2,173,952 63240005.img
>>>> 05/19/2011  09:34 AM         2,518,016 63240006.img
>>>> 05/19/2011  09:34 AM         2,553,856 63240007.img
>>>> 05/19/2011  09:34 AM         1,730,560 63240008.img
>>>> 05/19/2011  09:34 AM         2,041,856 63240009.img
>>>> 05/19/2011  09:34 AM         3,491,328 63240010.img
>>>> 05/19/2011  09:34 AM         6,999,552 63240011.img
>>>> 05/19/2011  09:35 AM         4,713,984 63240012.img
>>>> 05/19/2011  09:35 AM         2,435,072 63240013.img
>>>> 05/19/2011  09:35 AM         3,771,392 63240014.img
>>>> 05/19/2011  09:35 AM         6,385,152 63240015.img
>>>> 05/19/2011  09:35 AM         3,237,376 63240016.img
>>>> 05/19/2011  09:35 AM         3,420,160 63240017.img
>>>> 05/19/2011  09:35 AM         2,976,768 63240018.img
>>>> 05/19/2011  09:35 AM         3,629,056 63240019.img
>>>> 05/19/2011  09:36 AM         3,863,552 63240020.img
>>>> 05/19/2011  09:36 AM         3,583,488 63240021.img
>>>> 05/19/2011  09:36 AM         1,553,408 63240022.img
>>>> 05/19/2011  09:36 AM         3,411,456 63240023.img
>>>> 05/19/2011  09:36 AM         3,038,208 63240024.img
>>>> 05/19/2011  09:36 AM         7,901,696 63240025.img
>>>> 05/19/2011  09:36 AM         3,146,752 63240026.img
>>>> 05/19/2011  09:36 AM         4,070,400 63240027.img
>>>> 05/19/2011  09:37 AM         2,481,664 63240028.img
>>>> 05/19/2011  09:37 AM         3,462,656 63240029.img
>>>> 05/19/2011  09:37 AM         4,033,536 63240030.img
>>>> 05/19/2011  09:47 AM         8,536,576 63240031.img
>>>> 05/19/2011  09:47 AM         1,745,408 63240032.img
>>>> 05/19/2011  09:47 AM         4,779,008 63240033.img
>>>> 05/19/2011  09:47 AM         3,905,024 63240034.img
>>>> 05/19/2011  09:47 AM         5,238,272 63240035.img
>>>> 05/19/2011  09:47 AM         3,605,504 63240036.img
>>>>
>>>> You can see that tile 31 took about 10 minutes to be generated while
>>>> the
>>>> rest was much quicker. Anyone seen anything similar? Here's the tile
>>>> boundaries from the splitter output:
>>>>
>>>> 63240031: 1576960,-3932160 to 1597440,-3891200
>>>> #       : 33.837891,-84.375000 to 34.277344,-83.496094
>>>>
>>>> Just curious. Not a big deal but I thought I had caused it somehow with
>>>> the PBF patch.
>>> Noticed the same thing on Turkey (11min for one tile). I have to recheck
>>> if on Iceland and Serbia the same problem appears - and mkgmap is not
>>> completly choking. This happened also before the pbf patches/ or without
>>> locator branch, but now more often??
>>>> Francisco
>>
>> Don't know if that's the reason for your case but you might check. If
>> Java has too few memory the garbage collector needs so much time that
>> it seems as if mkgmap blocks. I have seen such cases.
>>
>> You can check that either by connecting with jvisualvm and watch the
>> CPU time used by the garbage collector or you can add -verbose:gc as
>> java parameter:
>> java -verbose:gc -jar mkgmap.jar ...
>> In this case you see a log message each time the gc is running. In
>> case mkgmap seems to block and low memory is the reason you should get
>> tons of such messages.
>>
>> By the way: I suppose the mkgmap code still has some potential to
>> reduce the memory footprint. All nodes, ways and relations are stored
>> in HashMaps which might be changed to a less memory consuming
>> implementation. It would be great if anyone tries to patch that.
>>
>> WanMil
>>
> In my case with Turkey that is definitely not the problem. It happens on
> small maps with 8GB RAM attributed (the server got 12GB, so RAM shortage
> is highly unlikely). I'll try jvisualvm on Saturday or Sunday.

The tile that took so long is the biggest one of all. Important for 
memory considerations is the size of tiles and the number of threads. So 
8GB *might* also be too low if you use big tiles and a high number of 
threads.