[mkgmap-dev] Error when running splitter with several input files, could there be keep-complete=single mode?

Mon Aug 30 20:41:53 BST 2021

Sorry - the last error was a bug in my script. I actually chose the mapid
for splitter too low and while mkgmap did overwrite the .img file without
warning, it did not overwrite the gmapi files without warning (due to a bug
in finding out the highest number of tiles created). So with o5m input this
all works fine. Seems I have to beef up my server by another NVME disk to
accommodate for the larger o5m vs osm.pbf files.
I need to store all splitter created files - because I run mkgmap several
times... Just convenience.

On Mon, 30 Aug 2021 at 12:50, Felix Hartmann <extremecarver at gmail.com>
wrote:

> Well I'm not fully sure this works as intended - though at first glance I
> cannot find a problem - but running mkgmap on the split files gives an:
>
> C:\openmtbmap\maps>start /belownormal /b /wait java -jar
> -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xms5000M -Xmx43000M
> C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area
> --code-page=1252 "--style-file=C:\openmtbmap\buildings_style"
>  --levels="0:24, 1:23, 2:22, 3:21" --ignore-osm-bounds
> --ignore-turn-restrictions --merge-lines --allow-reverse-merge
> --transparent --draw-priority=28 --add-pois-to-areas
> --simplify-polygons=23:4,22:7,21:8
> --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt"
> --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt"
> --add-boundary-nodes-at-admin-boundaries=2
> --poi-excl-index=0x6405,0x4316,0x2f00 --ignore-fixme-values
> --polygon-size-limits="24:16, 23:14, 22:12, 21:11"
> --ignore-turn-restrictions --description=buildings_fr --country-abbr=fr
> --country-name=france --mapname=83910046 --family-id=8391 --product-id=1
> --series-name="buildings_france_30.08.2021" --keep-going
> --family-name="buildings_fr_30.08.2021" --tdbfile --gmapi --gmapsupp
> --overview-mapname=mapsetb --area-name="france_30.08.2021_buildings" -c
> D:\openmtbmap\maps\template.franceb1 8391*.img buildfr.typ  1>NUL
> Mkgmap version 4806M
> Time started: Mon Aug 30 12:34:48 CEST 2021
> Number of MapFailedExceptions: 0
> WARNING (global): Could not copy 83910046.RGN
> uk.me.parabola.imgfmt.FileExistsException: File 83910046.RGN already exists
> WARNING (global): Could not copy 83910046.TRE
> uk.me.parabola.imgfmt.FileExistsException: File 83910046.TRE already exists
> WARNING (global): Could not copy 83910046.LBL
> uk.me.parabola.imgfmt.FileExistsException: File 83910046.LBL already exists
> Number of ExitExceptions: 0
> Time finished: Mon Aug 30 12:35:28 CEST 2021
> Total time taken: 39 seconds
>
>
> Now of course there has been no file at all called 83910046* in the
> directory except 83910046.o5m (referenced via
> D:\openmtbmap\maps\template.franceb1 input file option). Google spits out
> no mention at all concerning this error. I would guess there is an error in
> the --gmapi file?
>
> On Mon, 30 Aug 2021 at 12:38, Felix Hartmann <extremecarver at gmail.com>
> wrote:
>
>> okay - well the only working solution to this problem is to use o5m
>> output format with mkgmap, and then use osmconvert to join the files before
>> splitting them again.
>> Would it be possible for mkgmap to have a mode to split the files one by
>> one without the detour via osmconvert?
>>
>> o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster
>> with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m
>> files vs osm.pbf - but small input files make not much time difference)...
>>
>> On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver at gmail.com>
>> wrote:
>>
>>> I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start
>>> /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m
>>> C:\openmtbmap\splitter.jar --max-nodes=9600000  --max-threads=12
>>> --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024
>>> --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip
>>> --description=france --mapid=83910000 france.osm  1>NUL
>>>
>>> Then I compiled the map using mgkamp.jar - and want to resplit all
>>> remaining files that did not compile with smaller max-nodes value. However
>>> splitter.jar chokes if I pass several files... Is there any way to have
>>> splitter just working with several files without trying to merge them? I
>>> just need the files split to the new max-nodes value and incrementing the
>>> map id. If I pass only one file at a time, it is a PITA to script this (is
>>> already hard enough figuring out the new map-id and creating a list of the
>>> tiles).
>>>
>>> D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java
>>> -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar
>>> --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf
>>> --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip
>>> --description=france --mapid=83910046  "83910015.osm.pbf"
>>> "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf"    1>NUL
>>> Warning: --keep-complete is only used for the first input file. Further
>>> files must use higher ids.
>>> Error: Node ids are not sorted. Use e.g. osmosis to sort the input data.
>>> This is not supported with keep-complete=true or --problem-list
>>> uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted
>>>         at
>>> uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497)
>>>         at
>>> uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126)
>>>         at
>>> uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82)
>>>         at
>>> uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157)
>>>         at
>>> uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255)
>>>         at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503)
>>>         at uk.me.parabola.splitter.Main.start(Main.java:127)
>>>         at uk.me.parabola.splitter.Main.main(Main.java:81)
>>>
>>>
>>> The option to first join the files with osmconvert - is a PITA - as you
>>> cannot parse several osm.pbf files to osmconvert (it only supports joining
>>> osm or o5m files).
>>> The option to supply splitter.jar with each file on it's own - is also a
>>> PITA that would require horrendous for loops because after each split you
>>> need to check which new map-id to use for the next file!
>>>
>>> So it would be great to have a keep-complete option that for each input
>>> file, but not for all input files, uses keep-complete. Actually I think
>>> this should be the default mode when supplying several input files. Even if
>>> mkgmap actually cannot run this in parallel / multithreading it would be
>>> way better than parsing each file on it's own...
>>> Or am I dumbstruck in finding a way to resplit too big input files?
>>>
>>> --
>>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>>
>>>
>>
>> --
>> Felix Hartman - Openmtbmap.org & VeloMap.org
>>
>>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210830/d769715e/attachment.html>