[mkgmap-dev] Osmium or Splitter/mkgmap issue?

Fri Jun 12 05:56:50 BST 2020

Hi Enrico,

I  assume your three input files have different bounds. A program which merges them has to decide what to do.
It may
- select one of the existing boundaries
- calculate the smallest bounds covering all three bounds
- ignore the bounds and write no bounds
- calculate the bounds based on the node coordinates found in the data

The best method to avoid this is to tell the merge program the bounds that it should use. It seems osmium doesn't (yet) support this?
Or may you must use osmium extract later.
Reg. your log file it seems that the splitter doesn't find any bounds  in the merged file and thus calculates them based on all data.
You can use the --polygon-file option in splitter to specify the wanted area.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Enrico Liboni <eliboni at gmail.com>
Gesendet: Donnerstag, 11. Juni 2020 23:31
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Osmium or Splitter/mkgmap issue?

Gerd thanks for the script. I'll give a try. Anyway, when using osmium merge on   areas bigger than Malta (Italy + countries around Alps) it works fine and the final img file is the expected one.

On the other side, you wrote:
> maybe the two files 70000001.osm.pbf have different bounds?

and you hit it!

# Using Osmosis
enrico at gling:/opt/osm/tmp$ ../osmosis/bin/osmosis --rbf ./malta-latest.osm.pbf --rbf Contours-Malta.pbf --merge --wb all.pbf
enrico at gling:/opt/osm/tmp$ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
enrico at gling:/opt/osm/tmp$ cat areas.list
# List of areas
# Generated Thu Jun 11 23:19:43 CEST 2020
#
70000001: 1658880,649216 to 1693696,694272
#       : 35.595703,13.930664 to 36.342773,14.897461

#Using osmium sort
enrico at gling:/opt/osm/tmp$ osmium sort *.pbf -o all.pbf
[======================================================================] 100%
enrico at gling:/opt/osm/tmp$ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
enrico at gling:/opt/osm/tmp$ cat areas.list
# List of areas
# Generated Thu Jun 11 22:59:26 CEST 2020
#
70000001: 1658880,649216 to 1693696,694272
#       : 35.595703,13.930664 to 36.342773,14.897461

# While when using "osmium merge"...
enrico at gling:/opt/osm/tmp$ osmium merge *.pbf -o all.pbf
[======================================================================] 100%
enrico at gling:/opt/osm/tmp$ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
enrico at gling:/opt/osm/tmp$ cat areas.list
# List of areas
# Generated Thu Jun 11 23:17:46 CEST 2020
#
70000001: 1454080,247808 to 2002944,1394688
#       : 31.201172,5.317383 to 42.978516,29.926758

and thus the latter produces a much bigger img!

Thus question is...is osmium merge that creates a much bigger area? Or is it splitter that gets confused by something?

Attached the logs, around line 40 you can see splitters mentions "Bounding box 13.815377000000002 35.519850000000005 14.856099 36.332958000000005" only when using osmosis or osmium sort.

If anyone is interested to debug this further I can provide the pbf files produced by osmium&osmosis.

Enrico

On Thu, Jun 11, 2020 at 6:50 AM Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>> wrote:
Hi Enrico,

I use osmconvert like this in a script:
set input=f:\osm\niedersachsen-latest.osm.pbf
...
rmdir /s/q split
if not exist part.o5m f:\osm\osmconvert --drop-author -B=map.poly %input% -o=part.o5m
if not exist srtm.o5m f:\osm\osmconvert --drop-author -B=map.poly f:\osm\Hoehendaten_Freizeitkarte_EUROPE.osm.pbf -o=srtm.o5m
rem merge
if not exist work.o5m f:\osm\osmconvert part.o5m srtm.o5m -o=work.o5m
java -Xmx4G -jar d:\splitter\dist\splitter.jar --output-dir=split --max-nodes=1000000 --mapid=%mapid%0001 --write-kml=splitter.kml work.o5m  > splitter_%dt%-%t%.log

I create the file map.poly with JOSM (and the poly plugin). Instead of niedersachsen-latest.osm.pbf I can also use a larger file like europe-latest.osm.pbf if needed.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Enrico Liboni <eliboni at gmail.com<mailto:eliboni at gmail.com>>
Gesendet: Mittwoch, 10. Juni 2020 23:52
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Osmium or Splitter/mkgmap issue?

Acth...

* osmium sort "...will  take  roughly  10 times as much memory as the files take on disk in  osm.pbf format" - thus I can't pursue this  option when dealing with several countries :(
* osmconvert seems to need o5m, not pbf... I have all pbfs.

On Wed, Jun 10, 2020 at 10:56 PM Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com><mailto:gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>> wrote:
Hi Enrico,

I use osmconvert. I am not familar with osmosis and I cannot use osmium on Windows.
sort instead of merge sounds good, but I would expect an error message from splitter when data is not sorted correctly.
You should also check the output from splitter reg. the bounds.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Enrico Liboni <eliboni at gmail.com<mailto:eliboni at gmail.com><mailto:eliboni at gmail.com<mailto:eliboni at gmail.com>>>
Gesendet: Mittwoch, 10. Juni 2020 22:49
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Osmium or Splitter/mkgmap issue?

Gerd -  however I see no reason why bounds should be different, I use the very same ones. By the way, as per my other email, it seems that usong osmium sort instead of osmium merge does the trick. Perhaps objects are not sorted as osmium merge expects...

What do you usually use to merge pbfs?

Thanks!
Enrico

On Wed, Jun 10, 2020 at 10:27 PM Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com><mailto:gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>><mailto:gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com><mailto:gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>>> wrote:
Hi Enrico,

maybe the two files 70000001.osm.pbf have different bounds? If one is much larger the difference could be the additional data for sea and background polygons.
In one command you list the input files, in the other you use *.pbf.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>>>> im Auftrag von Enrico Liboni <eliboni at gmail.com<mailto:eliboni at gmail.com><mailto:eliboni at gmail.com<mailto:eliboni at gmail.com>><mailto:eliboni at gmail.com<mailto:eliboni at gmail.com><mailto:eliboni at gmail.com<mailto:eliboni at gmail.com>>>>
Gesendet: Mittwoch, 10. Juni 2020 22:19
An: Development list for mkgmap
Betreff: [mkgmap-dev] Osmium or Splitter/mkgmap issue?

I'm getting a weird behaviour: I merge <5MB pbfs,  when using osmium I get a 8MB img file while with osmosis it is less than 4MB! The latter seems fine since the initial pbfs are less than 5MB. I'd like to use osmium in my scripts since it performs better.

Am I doing something wrong? Thanks to anyone that could shed some light on this!

# input pbfs
-r--r-----  1 enrico enrico 4673440 Jun 10 21:50 malta-latest.osm.pbf
-r--r-----  1 enrico enrico   15376 Jun 10 21:50 Malta_lon14.03_14.74lat35.65_36.00_view3.osm.pbf
-r--r-----  1 enrico enrico    8300 Jun 10 21:50 Malta_lon14.03_14.74lat36.00_36.18_view3.osm.pbf
# using osmium
$ osmium merge *.pbf -o all.pbf
$ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
$ java -jar ../mkgmap/mkgmap.jar  --family-id=10030 --product-id=1  --route --remove-short-arcs  --bounds=../bounds.zip \
 --precomp-sea=../sea.zip  --location-autofill=is_in,nearest --draw-priority=20 --gmapsupp  --index --housenumbers 7000*pbf

-rw-rw-r-- 1 enrico enrico 4695873 Jun 10 21:54 all.pbf
-rw-rw-r-- 1 enrico enrico 4288669 Jun 10 21:54 70000001.osm.pbf
-rw-rw-r-- 1 enrico enrico 7925760 Jun 10 21:55 70000001.img
-rw-rw-r-- 1 enrico enrico 8171520 Jun 10 21:55 gmapsupp.img

# using osmosis
$ ../osmosis/bin/osmosis --rbf ./malta-latest.osm.pbf \
 --rbf ./Malta_lon14.03_14.74lat35.65_36.00_view3.osm.pbf \
 --rbf ./Malta_lon14.03_14.74lat36.00_36.18_view3.osm.pbf \
 --merge --merge --wb all.pbf
$ java -jar ../splitter/splitter.jar --mapid=70000001 all.pbf
$ java -jar ../mkgmap/mkgmap.jar  --family-id=10030 --product-id=1  --route --remove-short-arcs  --bounds=../bounds.zip \
 --precomp-sea=../sea.zip  --location-autofill=is_in,nearest --draw-priority=20 --gmapsupp  --index --housenumbers 7000*pbf

-rw-rw-r--  1 enrico enrico 4680030 Jun 10 22:03 all.pbf
-rw-rw-r--  1 enrico enrico 4288668 Jun 10 22:04 70000001.osm.pbf
-rw-rw-r--  1 enrico enrico 3547136 Jun 10 22:04 70000001.img
-rw-rw-r--  1 enrico enrico 3788800 Jun 10 22:04 gmapsupp.img

Using very latest splitter, mkgmap, and osmium (tried with 1.10 and 1.12 compiled from source...).
Note both gmapsupp.img seems to work just fine on the garmin device.
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>>>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev