[mkgmap-dev] Idea: map layers (multiple output tiles per input tile)

Sun Apr 10 20:18:00 BST 2011

> On Sun, Apr 10, 2011 at 07:42:53PM +0200, WanMil wrote:
>> * The advantage to create multiple img files in one mkgmap run is that
>> parsing and preparing of the OSM data must happen once only. Do you
>> have numbers how many percent of the time is used for these steps?
>
> No, I haven't collected any profiling data yet. Which tool would you
> recommend? For C and C++, which is what I have mainly been developing
> in, I have been using OProfile and before that, gprof. Which Java tools
> would come closest?

I would add some time logging to the mkgmap source code. There are 
profiles for Java but I think they are not useful for such a job.

>
>> Please be aware that mkgmap is optimized in such a way that it loads
>> only the tags which are needed in the style file.
>
> Based on the log files, multipolygon relations seem to be processed even
> though the style does not contain any rules for polygons. Could you fix
> this?

Yes that's an additional optimization. I can add an easy solution: Do 
not process a multipolygon if the mp does not contain any additional tag 
(other than type=multipolygon) and if no member way contains any tag.

>
>> * I think the layer concept will be too complicated. I would prefer to
>> have multiple styles (as Torsten Leistikow proposed in a seperate
>> mail).
>
> Yes, doing it with styles would be better from the compatibility point
> of view. My example of "ski routes", "bus routes", "hiking routes", "mtb
> routes" could be implemented with a parameterized style, something like
> -style=routes --style-param route=bus.
>
>> * From my point of view it won't be possible to run mkgmap with
>> different options for the different styles (or layers in your concept).
>> Will this remove the timing advantages?
>
> Why wouldn't it be possible to use style-specific options? I think that
> there could be a command line option for associating subsequent options
> with an output file. Something like this:
>
> java -jar mkgmap.jar \
> --output-file 10000001.img -c family1.args \
> --output-file 20000001.img -c family2.args \
> --output-file 30000001.img -c family3.args \
> --input-file 10000001.osm.gz \
> --output-file 10000002.img -c family1.args \
> --output-file 20000002.img -c family2.args \
> --output-file 30000002.img -c family3.args \
> --input-file 10000002.osm.gz \
> and so on, for every input file.

The problem arises with options that change the processing before the 
style come into play. Maybe that's a small number of options only:
family1.args: generate-sea=multipolygon
family2.args: generate-sea=polygon
or
family1.args: remove-short-arcs=3
family2.args: remove-short-arcs=-1

One need to check the complete list of options which will cause a problem.

>
> Basically, we would have a producer (input file parser) and several
> consumers (img file generators). Each img file generator would have its
> own set of options.
>
> Currently, another problem with multiple styles (apart from the alleged
> parsing overhead) is that the processing cannot be parallelized in an
> optimal way. For example, if I have an N-core machine, --max-jobs does
> the right thing for splitter and each mkgmap run, but my script would
> invoke mkgmap several times in succession, once for each output style.
> If I have M output tiles so that M is not an integer multiple of N, some
> cores would be sitting idle when a mkgmap run is close to completion.

I usually have more problems with memory limitations than CPU 
limitations (of course probably that's due to my hardware configuration).

Your example with the N-core machine is not completely correct. You 
assume that each tile needs a fixed time T for compilation, so that you 
have some cores sitting idle at the end. But that's not a real example. 
Each tile needs a different time T(x) for compilation and so the CPU 
consumption decreases from 100% to 100%/N while compiling the last N 
tiles. After this the index and the other stuff (gmapsupp, nsis file 
etc.) is created with a single CPU only. I assume your new idea would 
not improve this point.

>
> Best regards,
>
> 	Marko

Have fun!
WanMil