[mkgmap-dev] Error when running splitter with several input files, could there be keep-complete=single mode?

Tue Aug 31 02:41:47 BST 2021

Hi Felix,
I did not yet try to unterstand what your goal is but it sounds wrong to merge files produced by spltter. This can only work well when the merged tiles form a rectangular area without gaps.
Gerd

---- Felix Hartmann schrieb ----

Sorry - the last error was a bug in my script. I actually chose the mapid for splitter too low and while mkgmap did overwrite the .img file without warning, it did not overwrite the gmapi files without warning (due to a bug in finding out the highest number of tiles created). So with o5m input this all works fine. Seems I have to beef up my server by another NVME disk to accommodate for the larger o5m vs osm.pbf files.
I need to store all splitter created files - because I run mkgmap several times... Just convenience.

On Mon, 30 Aug 2021 at 12:50, Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>> wrote:
Well I'm not fully sure this works as intended - though at first glance I cannot find a problem - but running mkgmap on the split files gives an:

C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xms5000M -Xmx43000M C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area --code-page=1252 "--style-file=C:\openmtbmap\buildings_style"   --levels="0:24, 1:23, 2:22, 3:21" --ignore-osm-bounds --ignore-turn-restrictions --merge-lines --allow-reverse-merge --transparent --draw-priority=28 --add-pois-to-areas --simplify-polygons=23:4,22:7,21:8 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --ignore-fixme-values --polygon-size-limits="24:16, 23:14, 22:12, 21:11" --ignore-turn-restrictions --description=buildings_fr --country-abbr=fr --country-name=france --mapname=83910046 --family-id=8391 --product-id=1 --series-name="buildings_france_30.08.2021" --keep-going --family-name="buildings_fr_30.08.2021" --tdbfile --gmapi --gmapsupp --overview-mapname=mapsetb --area-name="france_30.08.2021_buildings" -c D:\openmtbmap\maps\template.franceb1 8391*.img buildfr.typ  1>NUL
Mkgmap version 4806M
Time started: Mon Aug 30 12:34:48 CEST 2021
Number of MapFailedExceptions: 0
WARNING (global): Could not copy 83910046.RGN uk.me.parabola.imgfmt.FileExistsException: File 83910046.RGN already exists
WARNING (global): Could not copy 83910046.TRE uk.me.parabola.imgfmt.FileExistsException: File 83910046.TRE already exists
WARNING (global): Could not copy 83910046.LBL uk.me.parabola.imgfmt.FileExistsException: File 83910046.LBL already exists
Number of ExitExceptions: 0
Time finished: Mon Aug 30 12:35:28 CEST 2021
Total time taken: 39 seconds

Now of course there has been no file at all called 83910046* in the directory except 83910046.o5m (referenced via D:\openmtbmap\maps\template.franceb1 input file option). Google spits out no mention at all concerning this error. I would guess there is an error in the --gmapi file?

On Mon, 30 Aug 2021 at 12:38, Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>> wrote:
okay - well the only working solution to this problem is to use o5m output format with mkgmap, and then use osmconvert to join the files before splitting them again.
Would it be possible for mkgmap to have a mode to split the files one by one without the detour via osmconvert?

o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m files vs osm.pbf - but small input files make not much time difference)...

On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>> wrote:
I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m C:\openmtbmap\splitter.jar --max-nodes=9600000  --max-threads=12 --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024  --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910000 france.osm  1>NUL

Then I compiled the map using mgkamp.jar - and want to resplit all remaining files that did not compile with smaller max-nodes value. However splitter.jar chokes if I pass several files... Is there any way to have splitter just working with several files without trying to merge them? I just need the files split to the new max-nodes value and incrementing the map id. If I pass only one file at a time, it is a PITA to script this (is already hard enough figuring out the new map-id and creating a list of the tiles).

D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf  --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip  --description=france --mapid=83910046  "83910015.osm.pbf" "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf"    1>NUL
Warning: --keep-complete is only used for the first input file. Further files must use higher ids.
Error: Node ids are not sorted. Use e.g. osmosis to sort the input data.
This is not supported with keep-complete=true or --problem-list
uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted
        at uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497)
        at uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126)
        at uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82)
        at uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157)
        at uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255)
        at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503)
        at uk.me.parabola.splitter.Main.start(Main.java:127)
        at uk.me.parabola.splitter.Main.main(Main.java:81)

The option to first join the files with osmconvert - is a PITA - as you cannot parse several osm.pbf files to osmconvert (it only supports joining osm or o5m files).
The option to supply splitter.jar with each file on it's own - is also a PITA that would require horrendous for loops because after each split you need to check which new map-id to use for the next file!

So it would be great to have a keep-complete option that for each input file, but not for all input files, uses keep-complete. Actually I think this should be the default mode when supplying several input files. Even if mkgmap actually cannot run this in parallel / multithreading it would be way better than parsing each file on it's own...
Or am I dumbstruck in finding a way to resplit too big input files?

--
Felix Hartman - Openmtbmap.org & VeloMap.org

--
Felix Hartman - Openmtbmap.org & VeloMap.org

--
Felix Hartman - Openmtbmap.org & VeloMap.org

--
Felix Hartman - Openmtbmap.org & VeloMap.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20210831/9fdf75fd/attachment-0001.html>