[mkgmap-dev] splitter: option for maximum tile area?

Sun Nov 4 10:19:59 GMT 2018

Hi Gerd,
I've uploaded some files:
splitter log: 
https://drive.google.com/open?id=12wsnncNkfwKruEw-4UIgmvddK3E_TxtE
kml: https://drive.google.com/open?id=1UOv_FTtl_pK_Nj126WDFnirhaswnvyfZ
smallest tile: 
https://drive.google.com/open?id=1_yDKKpALKSfiRTiicCrEOYgH1vKmnm4w
largest tile: 
https://drive.google.com/open?id=1aidryuJuhixrJIHHou3sibbBNBpRWJeN
Kind regards,
Bernhard

Am 03.11.2018 um 11:38 schrieb Gerd Petermann:
> Hi Bernhard,
>
> please post a link to one of the files produced by splitter. If the sum of file sizes is much larger than the input file that means that each file
> contains a lot of data that was written to keep ways complete. Maybe srtm2osm creates lots of very long ways ?
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil at gmx.de>
> Gesendet: Samstag, 3. November 2018 11:25
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
>
> Hi Gerd,
> contour lines were created with srtm2osm  -step 25 -cat 500 100 25
> -large with 3" elevation data downloaded from viewfinderpanoramas.
> The mkgmap parameters contain "--add-pois-to-areas", but no
> "--add-pois-to-lines" (the splitter parameters contain neither - as Ralf
> said).
> The style for contour lines is simple:
> contour=evation & ele<=0 { delete contour; }
> contour=evation & contour_ext=elevation_major  { name
> '${ele|conv:m=t}'; } [0x22 level 4]
> contour=evation & contour_ext=elevation_medium { name
> '${ele|conv:m=t}'; } [0x21 level 2]
> contour=evation & contour_ext=elevation_minor  { name
> '${ele|conv:m=t}'; } [0x20 level 0]
> There are no rules for contours in the points and polygons file. The
> options file defines the levels only.
>
> Running splitter on the contour lines file only, creates several pdb
> files summing up to 4.5 GB
> =the problem is with the contour lines.
>
> Next, I'll try to cut Laos from the contour lines file and create a
> contour lines map of Laos only...
>
> Kind regards,
> Bernhard
>
> Am 02.11.2018 um 19:31 schrieb Gerd Petermann:
>> Hi Bernhard,
>>
>> just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI
>> you may see something like that.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen at hotmail.com>
>> Gesendet: Freitag, 2. November 2018 19:24
>> An: Bernhard Hiller; Development list for mkgmap
>> Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
>>
>> Hi Bernhard,
>>
>> what are your rules for the contour lines?
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil at gmx.de>
>> Gesendet: Freitag, 2. November 2018 19:18
>> An: mkgmap-dev at lists.mkgmap.org.uk
>> Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
>>
>> Hi all,
>> still struggling with that strange issue. But I guess I found some hint
>> to the cause: inconsistent file sizes.
>> - extracted OSM data: 400 MB pbf
>> - elevation contour lines: 600 MB pbf
>> - merged file: 1 GB pbf
>> So far, sizes are consistent.
>>
>> When I run splitter on the OSM data only file, it produces many tiles
>> summing up to some 400 MB. Consistent, too. When I run mkgmap on this
>> output, I get a map within about half an hour, and it looks OK (tested
>> with QLandkarte).
>>
>> When I run splitter on the merged file, the tiles sum up to some 9 GB.
>> That is 9 times the size I'd expect. mkgmap can render several of those
>> tiles (but very slowly); and then crashes on one of them with an
>> OutOfMemory exception.
>>
>> So I think that the problem is somewhere in those contour lines, either
>> when merged or alone.
>> I'll try to create a contour lines only map as the next step to test
>> this hypothesis.
>>
>> Kind regards,
>> Bernhard
>>
>> Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
>>> Hi Bernhard,
>>>
>>> I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc.
>>> My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=our data?
>>>
>>> Gerd
>>>
>>> ________________________________________
>>> Von: Gerd Petermann <gpetermann_muenchen at hotmail.com>
>>> Gesendet: Mittwoch, 31. Oktober 2018 22:49
>>> An: Gerd Petermann; Development list for mkgmap
>>> Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
>>>
>>> Hi Bernhard,
>>>
>>> looked again at the splitter command in your last post. You also use a rather high max-nodes value.
>>> Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile
>>> compared to the default 1400000. Many users use a value near 1200000.
>>>
>>> Gerd
>>>
>>> ________________________________________
>>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen at hotmail.com>
>>> Gesendet: Mittwoch, 31. Oktober 2018 22:20
>>> An: Development list for mkgmap
>>> Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
>>>
>>> Hi Bernhard,
>>> there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ?
>>> If that doesn't help you can try to change the values for
>>> private static final int MAX_LAT_DEGREES >> private static final int MAX_LON_DEGREES >> in SplittableDensityArea.java and compile your own version of splitter.
>>>
>>> Gerd
>>>
>>> ________________________________________
>>> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil at gmx.de>
>>> Gesendet: Mittwoch, 31. Oktober 2018 22:03
>>> An: mkgmap-dev at lists.mkgmap.org.uk
>>> Betreff: [mkgmap-dev] splitter: option for maximum tile area?
>>>
>>> Hi all,
>>> currently a Java OutOfMemory exception prevents me from creating a map.
>>> I already use option --max-jobs= machine has 4 physical cores) and
>>> -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and
>>> elevation contour lines.
>>>     From the tiles finished and those with a new timestamp but about 0
>>> bytes length, I can see that mkgmap was rendering tiles 47120005,
>>> 47120006, 47120007 at the time of crash.
>>> Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see
>>> attached file), covering a lot of the south chinese sea, i.e. there are
>>> not many actual data in that area.
>>> I guess that the problem arises with that tile. I remember some case in
>>> the past where a single tile covering such a large area of mainly sea
>>> caused mkgmap to take an enormous amount of time for rendering - also
>>> here, mkgmap already spent about 1 hour before crashing.
>>> So I'd like to ask: is there some possibility for limiting the area of a
>>> tile among the splitter options?
>>> Kind regards,
>>> Bernhard
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>