[mkgmap-dev] Slowness processing specific tile

Thu May 19 21:05:27 BST 2011


On 19.05.2011 21:52, WanMil wrote:
>>>>> I noticed a big hang/delay processing one tile in NC. Here's the
>>>>> generated IMG files for each tile:
>>>>>
>>>>> 05/19/2011  09:33 AM         4,230,144 63240001.img
>>>>> 05/19/2011  09:33 AM         2,450,432 63240002.img
>>>>> 05/19/2011  09:33 AM         2,893,824 63240003.img
>>>>> 05/19/2011  09:33 AM         2,170,880 63240004.img
>>>>> 05/19/2011  09:33 AM         2,173,952 63240005.img
>>>>> 05/19/2011  09:34 AM         2,518,016 63240006.img
>>>>> 05/19/2011  09:34 AM         2,553,856 63240007.img
>>>>> 05/19/2011  09:34 AM         1,730,560 63240008.img
>>>>> 05/19/2011  09:34 AM         2,041,856 63240009.img
>>>>> 05/19/2011  09:34 AM         3,491,328 63240010.img
>>>>> 05/19/2011  09:34 AM         6,999,552 63240011.img
>>>>> 05/19/2011  09:35 AM         4,713,984 63240012.img
>>>>> 05/19/2011  09:35 AM         2,435,072 63240013.img
>>>>> 05/19/2011  09:35 AM         3,771,392 63240014.img
>>>>> 05/19/2011  09:35 AM         6,385,152 63240015.img
>>>>> 05/19/2011  09:35 AM         3,237,376 63240016.img
>>>>> 05/19/2011  09:35 AM         3,420,160 63240017.img
>>>>> 05/19/2011  09:35 AM         2,976,768 63240018.img
>>>>> 05/19/2011  09:35 AM         3,629,056 63240019.img
>>>>> 05/19/2011  09:36 AM         3,863,552 63240020.img
>>>>> 05/19/2011  09:36 AM         3,583,488 63240021.img
>>>>> 05/19/2011  09:36 AM         1,553,408 63240022.img
>>>>> 05/19/2011  09:36 AM         3,411,456 63240023.img
>>>>> 05/19/2011  09:36 AM         3,038,208 63240024.img
>>>>> 05/19/2011  09:36 AM         7,901,696 63240025.img
>>>>> 05/19/2011  09:36 AM         3,146,752 63240026.img
>>>>> 05/19/2011  09:36 AM         4,070,400 63240027.img
>>>>> 05/19/2011  09:37 AM         2,481,664 63240028.img
>>>>> 05/19/2011  09:37 AM         3,462,656 63240029.img
>>>>> 05/19/2011  09:37 AM         4,033,536 63240030.img
>>>>> 05/19/2011  09:47 AM         8,536,576 63240031.img
>>>>> 05/19/2011  09:47 AM         1,745,408 63240032.img
>>>>> 05/19/2011  09:47 AM         4,779,008 63240033.img
>>>>> 05/19/2011  09:47 AM         3,905,024 63240034.img
>>>>> 05/19/2011  09:47 AM         5,238,272 63240035.img
>>>>> 05/19/2011  09:47 AM         3,605,504 63240036.img
>>>>>
>>>>> You can see that tile 31 took about 10 minutes to be generated while
>>>>> the
>>>>> rest was much quicker. Anyone seen anything similar? Here's the tile
>>>>> boundaries from the splitter output:
>>>>>
>>>>> 63240031: 1576960,-3932160 to 1597440,-3891200
>>>>> #       : 33.837891,-84.375000 to 34.277344,-83.496094
>>>>>
>>>>> Just curious. Not a big deal but I thought I had caused it somehow 
>>>>> with
>>>>> the PBF patch.
>>>> Noticed the same thing on Turkey (11min for one tile). I have to 
>>>> recheck
>>>> if on Iceland and Serbia the same problem appears - and mkgmap is not
>>>> completly choking. This happened also before the pbf patches/ or 
>>>> without
>>>> locator branch, but now more often??
>>>>> Francisco
>>>
>>> Don't know if that's the reason for your case but you might check. If
>>> Java has too few memory the garbage collector needs so much time that
>>> it seems as if mkgmap blocks. I have seen such cases.
>>>
>>> You can check that either by connecting with jvisualvm and watch the
>>> CPU time used by the garbage collector or you can add -verbose:gc as
>>> java parameter:
>>> java -verbose:gc -jar mkgmap.jar ...
>>> In this case you see a log message each time the gc is running. In
>>> case mkgmap seems to block and low memory is the reason you should get
>>> tons of such messages.
>>>
>>> By the way: I suppose the mkgmap code still has some potential to
>>> reduce the memory footprint. All nodes, ways and relations are stored
>>> in HashMaps which might be changed to a less memory consuming
>>> implementation. It would be great if anyone tries to patch that.
>>>
>>> WanMil
>>>
>> In my case with Turkey that is definitely not the problem. It happens on
>> small maps with 8GB RAM attributed (the server got 12GB, so RAM shortage
>> is highly unlikely). I'll try jvisualvm on Saturday or Sunday.
>
> The tile that took so long is the biggest one of all. Important for 
> memory considerations is the size of tiles and the number of threads. 
> So 8GB *might* also be too low if you use big tiles and a high number 
> of threads.
Well I use small tiles (max-nodes 900.000 on Turkey) plus only 4 
threads. (I could use 8 due to Hyperthreading, but kept it at 4). I even 
have this problem on single tiles where the osm.gz is like 10MB. There 
shouldn't be a slow down due to memory in that case, should there? (I 
cannot check for memory use, as my user rights are too low).