[mkgmap-dev] Error when running splitter with several input files, could there be keep-complete=single mode?

Thu Sep 2 01:08:20 BST 2021

In general I now got it fully working - the problem is if those merged
tiles are split - and not split in "pairs" but say from two tiles that
failed - which are not adjacent - 3 tiles are created (or 5) there
sometimes will be a tile spanning another tile. This works as far as I can
see in Basecamp/MapInstall/Mapsource - but gives strange impression when
selecting the tiles in MapInstall/Mapsource.
Have not yet tested it on device.

So if there is no possibility for splitter to implement a multiple single
input tile mode - I will have to resort to writing a big for loop calling
splitter separately for each crashed tile on mkgmap.jar processing. It
often can be prevented by using exactly 50% maxnodes size on resplit - but
that is not reliable of course. Basically for each tile that is 0 byte, you
need to split the original o5m, remove the mention to the input-file from
the template.args file, after each run check for a new highest mapid (or if
splitting with 50% maxnodes just add 3), always delete all 0byte files -
and after all tiles are split run mkgmap.jar again (either on the existing
.img files plus the new templates.args file (from joining all the
templates.args now newly created - also required quite a bit of logic).
It's scriptable - but really requires a lot of work. Would be great if
splitter.jar has a multiple tile input method (even more if it could
multithread that one as usual). If not the multithreading of course can be
achieved by piping each of these broken tiles into a separate process for
splitting.

On Tue, 31 Aug 2021 at 12:23, Felix Hartmann <extremecarver at gmail.com>
wrote:

> That step is very easy.
>
> You feed %FID% (if you use fid for first 4 numbers of mapid) and the new
> template.args to mkgmap.jar. No need to recreate the .img files that are
> okay. The broken ones - all 0byte ones - just delete them. If you wanna run
> mkgmap again on the same o5m input files - you join the template.args file
> from both runs - I always replace mapname with #notused so the
> template.args only contains description and the actual input file. It
> throws a warning that input files are missing - but that is allright. Of
> course you need to add in your script to delete those input files that were
> resplit.
>
> I don't know what your "app" does - but the process is pretty simple. As
> long as splitter can handle the several input files with --keep-complete
> (and just runs keep-complete on each input file but not on the totality of
> input files, plus does not try to join data from two files). And yes as
> said - if you want to run splitter on each input file separately, that
> creates loads of work - I don't know how to script that efficiently. Oh
> yeah - and your script needs to read the highest number of the maps created
> - add 1, and run mkgmap again with that number.  But yeah - scripting this
> takes some time. But it is understandable that mkgmap cannot call
> splitter.jar on it's own....The main thing in this is getting to run
> splitter on several input files with keep-complete valid for each tile on
> it's own.
> But yeah I think osmconvert 1000000.o5m 100001.o5m 100010.o5m
> -o=inputfile.o5m
> and then calling mkgmap with the higher mapid, and *already_created.img
> inputfile.o5m would suffice. So far I cannot see problems (as long as using
> keep-complete=true)  - you can even do a third pass. You cannot do a third
> pass if you use keep-complete=false overlap=2000
> because the overlap would create a huge mess (or better objects ending up
> twice in your map)
>
> On Tue, 31 Aug 2021 at 11:33, Eric Internet <ankeric.osm at gmail.com> wrote:
>
>> Also do not understand and on vacation now, so my options for
>> understanding are very limited.
>>
>> But fwiw...
>>
>> Mkgmap should read files collection from contents files. One o5m file
>> missing or 1 file extra (having different FID) is a show stopper. (Missing
>> files is not tested, assumption only).
>>
>> I create 2-6 different maps each overnight from same dataset by simply:
>> - Rename all o5m files by FID
>> - Rename Reference in two content files by filename (changed by FID).
>> This procedure takes seconds to run.
>>
>> All scripts (power management, splitter, OsmConvert, mkgmap, NSI) are
>> executed by shell.run from within a "wrapper application".
>> VB exe is: flexible, configurable, unlimited procedures and language
>> elements.
>>
>> App UI checkboxes will select which datasets to download.
>> App UI checkboxes will select which maps to be created.
>> UI settings are saved in iNI file for Unattended execution.
>> App will decide which datasets need to be merged by OsmConvert.
>> App will check preconditions.
>> App will run preprocess procedures (f.i. Rename by FID).
>> App will wait for recent "latest" dataset.
>> App will retry a failed download for n-times.
>> App will decide what to do if a dataset is missing or outdated.
>> App will verify each step to be successful (or not).
>> App will install correctly created maps (read log files) into BaseCamp.
>> Installer exe files and log files will be renamed by date (history,
>> back-up and review options).
>> App will run by Windows Scheduled Tasks.
>>
>> Recreating a specific map (because of style change) is as simple as
>> clicking checkboxes.
>>
>> Time investment for mkgmap and Mapillary: one weekend development.
>> Payback period in 2021 is already passed. Not by lead time (although
>> Rename is faster than Merge) but most impressive by needed attention time.
>> Don't shut down PC and tomorrow my new maps have been created somewhere
>> between 06:00 and 09:00 AM or they failed.
>>
>> A one time problem is resolved by a new procedure for next occurrence.
>> One time problems don't exist 😝
>>
>> AnkEric (Eric)
>>
>> On Tue, Aug 31, 2021, 10:15 Felix Hartmann <extremecarver at gmail.com>
>> wrote:
>>
>>> yes - exactly what Ticker wrote is my intention. It seems fine using o5m
>>> and merging with osmconvert however for right now.... If it's only a single
>>> tile no problems, but if its 2-X files - then you can only use
>>> keep-complete=false - or instead of using osm.pbf output in splitter o5m,
>>> and then merge the failed o5m tiles into a new input2.o5m and process that
>>> one again with splitter and lower max-nodes value. Instead if several input
>>> files are given and keep-complete=true (or default which is true) - then
>>> use keep-complete for each input file only.
>>>
>>> I do run into another problem there on the huge tiles, will have to
>>> check on that later. Seems to be general about maps with very large tiles
>>> and not related to splitter.
>>>
>>> On Tue, 31 Aug 2021 at 09:52, Ticker Berkin <rwb-mkgmap at jagit.co.uk>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> My understanding of Felix's problem and a possible enhancement to solve
>>>> it:
>>>>
>>>> Splitting a country... into a number of .osm.pfb files. Almost all of
>>>> these are processable by mkgmap to produce tiles, but one or two exceed
>>>> some .img limits.
>>>>
>>>> Rather that resplitting the whole country with a lower --max-nodes and
>>>> trying again:
>>>>
>>>> Run the splitter again using the --split-file=area.list from the
>>>> previous run (maybe with the good tiles deleted) and a new option that
>>>> forces generation of 2 .osm.pbf files for each area. Probably just
>>>> split in half, with some appropriate mapname augmentation. mkgmap
>>>> should now be able to produce pairs of tiles to fill in the holes where
>>>> the first run failed.
>>>>
>>>> Ticker
>>>>
>>>>
>>>> _______________________________________________
>>>> mkgmap-dev mailing list
>>>> mkgmap-dev at lists.mkgmap.org.uk
>>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>>>
>>>
>>>
>>> --
>>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>>
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> mkgmap-dev at lists.mkgmap.org.uk
>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> mkgmap-dev at lists.mkgmap.org.uk
>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210902/051838f2/attachment-0001.html>