[mkgmap-dev] Splitter doesn't process pbf data created with osmosis 0.39

Sun Sep 25 00:51:22 BST 2011

On Tue, 20 Sep 2011 22:14:36 +0200, Michael Prinzing wrote:

>On Mon, 19 Sep 2011 17:53:03 +0200, Michael Prinzing wrote:
>
>>there seems to be a problem with the latest splitter and data in pbf
>>format if it was created with osmosis 0.39.
>
>After some further testing, it now looks like a problem with osmosis 
>only. If I am converting the pbf data created with osmosis 0.39 back to
>xml then there also is an empty bounding box that did not exist in the
>original data and that does not arise if the pbf file was built with
>osmosis 0.38.
>
>I will report this on the mailing list for osmosis.

Well, there really was a problem with osmosis and it is fixed in the
latest osmosis-snapshot. This version now can read pbf data created
with osmosis 0.39 or newer without seeing an empty bounding box where
no one is present.

However, the bug was in the pbf reader, not with the creation of pbf
files. This means the created pbf data did not change, and therefore
the problem with the splitter still persists.

Igor Podolskiy who fixed the problem in osmosis also looked into the
splitter code. He assumes there could be something similar wrong as 
it was in osmosis. Please see the thread beginning with

http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-September/001125.html

for details.

Here again how to reproduce the problem with the splitter:

I am saving the following xml data (not containing a bounding box!) to
data.osm:

--------------------------------------------------------------------
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="Osmosis 0.35">
  <node id="0000004406" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6325833" lon="6.2575833"/>
  <node id="0000004407" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6334167" lon="6.2584167"/>
  <node id="0000004408" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.63425" lon="6.25925"/>
  <node id="0000004409" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6334167" lon="6.2600833"/>
  <node id="0000004410" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6325833" lon="6.2600833"/>
  <node id="0000004411" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.63175" lon="6.25925"/>
  <node id="0000004412" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.63175" lon="6.2584167"/>
  <way id="1" version="1" timestamp="2011-09-11T03:07:14Z">
    <nd ref="0000004406"/>
    <nd ref="0000004407"/>
    <nd ref="0000004408"/>
    <nd ref="0000004409"/>
    <nd ref="0000004410"/>
    <nd ref="0000004411"/>
    <nd ref="0000004412"/>
    <nd ref="0000004406"/>
    <tag k="ele" v="-220"/>
    <tag k="contour" v="elevation"/>
    <tag k="contour_ext" v="elevation_medium"/>
  </way>
</osm>
--------------------------------------------------------------------

Then, I am converting this xml data to pbf with:

osmosis --read-xml file="data.osm" --write-pbf omitmetadata=true file="data_nnn.osm.pbf"

Depending on the osmosis version I am using for this, I am getting the
following:

With osmosis 0.39:
--------------------------------------------------------------------
00000000: 0000 000d 0a09 4f53 4d48 6561 6465 7218  ......OSMHeader.
00000010: 2f10 231a 2b78 9c53 e2f3 2fce 0d4e ce48  /.#.+x.S../..N.H
00000020: cd4d d40d 33d0 3353 e272 49cd 2b4e f5cb  .M..3.3S.rI.+N..
00000030: 4f49 2d6e 6264 31d0 33b6 0400 bcf0 0a2d  OI-nbd1.3......-
00000040: 0000 000c 0a07 4f53 4d44 6174 6118 8c01  ......OSMData...
00000050: 109f 011a 8601 789c e372 e062 e012 48cd  ......x..r.b..H.
00000060: 492d 4b2c c9cc cf8b cf4d 4dc9 2ccd e562  I-K,.....MM.,..b
00000070: 068a 70b1 27e7 e795 e497 1671 7143 19f1  ..p.'......qqC..
00000080: a915 255c 2cba 4646 065c 9c70 2d42 9642  ..%\,.FF.\.p-B.B
00000090: e65c 1c6f 5c98 c0c0 4974 52fb 9637 cc73  .\.o\...ItR..7.s
000000a0: 9a18 6735 31ce 6c62 9c0d 2619 bc44 365d  ..g51.lb..&..D6]
000000b0: be66 0d11 0522 8699 6061 2149 2971 0e46  .f..."..`a!I)q.F
000000c0: 2166 2666 1629 6656 3646 274e 9831 dc1d  !f&f.)fV6F'N.1..
000000d0: 8c29 1318 5fb0 0300 fd3e 2ca3            .).._....>,.
--------------------------------------------------------------------

With osmosis-snapshot:
--------------------------------------------------------------------
00000000: 0000 000d 0a09 4f53 4d48 6561 6465 7218  ......OSMHeader.
00000010: 3b10 2f1a 3778 9c53 e2f3 2fce 0d4e ce48  ;./.7x.S../..N.H
00000020: cd4d d40d 33d0 3353 e272 49cd 2b4e f5cb  .M..3.3S.rI.+N..
00000030: 4f49 2d6e 6214 08f6 730c 08f6 f00f d12d  OI-nb...s......-
00000040: cdcb cecb 2fcf 0300 5893 0f1c 0000 000c  ..../...X.......
00000050: 0a07 4f53 4d44 6174 6118 8c01 109f 011a  ..OSMData.......
00000060: 8601 789c e372 e062 e012 48cd 492d 4b2c  ..x..r.b..H.I-K,
00000070: c9cc cf8b cf4d 4dc9 2ccd e562 068a 70b1  .....MM.,..b..p.
00000080: 27e7 e795 e497 1671 7143 19f1 a915 255c  '......qqC....%\
00000090: 2cba 4646 065c 9c70 2d42 9642 e65c 1c6f  ,.FF.\.p-B.B.\.o
000000a0: 5c98 c0c0 4974 52fb 9637 cc73 9a18 6735  \...ItR..7.s..g5
000000b0: 31ce 6c62 9c0d 2619 bc44 365d be66 0d11  1.lb..&..D6].f..
000000c0: 0522 8699 6061 2149 2971 0e46 2166 2666  ."..`a!I)q.F!f&f
000000d0: 1629 6656 3646 274e 9831 dc1d 8c29 1318  .)fV6F'N.1...)..
000000e0: 5fb0 0300 fd3e 2ca3                      _....>,.
--------------------------------------------------------------------

Both files can be converted back to xml only with the latest osmosis
from snapshot. osmosis 0.39 or older are reading an empty bounding box
from the pbf data. This was the bug that was fixed in osmosis now.

When I am trying to split one of these files, the splitter sees an
empty bounding box and of course does not write any data to the output:

--------------------------------------------------------------------
cache=
description=
geonames-file=
legacy-mode=false
mapid=63240001
max-areas=255
max-nodes=1600000
max-threads=1 (auto)
mixed=false
no-trim=false
output=unset
output-dir=
overlap=2000
resolution=13
split-file=
status-freq=120
write-kml=
Elapsed time: 0s   Memory: Current 15MB (1MB used, 14MB free) Max 247MB
Time started: Fri Sep 23 23:27:55 CEST 2011
Map is being split for resolution 13:
 - area boundaries are aligned to 0x800 map units
 - areas are multiples of 0x1000 map units wide and high
Processing data_039.osm.pbf
Bounding box 0.0 0.0 0.0 0.0
in 1 file
Time: Fri Sep 23 23:27:56 CEST 2011
Exact map coverage is (0.0,0.0) to
(2.1457672119140625E-5,2.1457672119140625E-5)
Trimmed and rounded map coverage is (0.0,0.0) to
(2.1457672119140625E-5,2.1457672119140625E-5)
Splitting nodes into areas containing a maximum of 1.600.000 nodes each...
0 areas:
Writing out split osm files Fri Sep 23 23:27:56 CEST 2011
Processing 0 areas in a single pass
Time finished: Fri Sep 23 23:27:56 CEST 2011
Total time taken: 0s
--------------------------------------------------------------------

If however I am creating the pbf file with osmosis 0.38, then the pbf
data looks like the following:

--------------------------------------------------------------------
00000000: 0000 000c 0a07 4f53 4d44 6174 6118 8c01  ......OSMData...
00000010: 109f 011a 8601 789c e372 e062 e012 48cd  ......x..r.b..H.
00000020: 492d 4b2c c9cc cf8b cf4d 4dc9 2ccd e562  I-K,.....MM.,..b
00000030: 068a 70b1 27e7 e795 e497 1671 7143 19f1  ..p.'......qqC..
00000040: a915 255c 2cba 4646 065c 9c70 2d42 9642  ..%\,.FF.\.p-B.B
00000050: e65c 1c6f 5c98 c0c0 4974 52fb 9637 cc73  .\.o\...ItR..7.s
00000060: 9a18 6735 31ce 6c62 9c0d 2619 bc44 365d  ..g51.lb..&..D6]
00000070: be66 0d11 0522 8699 6061 2149 2971 0e46  .f..."..`a!I)q.F
00000080: 2166 2666 1629 6656 3646 274e 9831 dc1d  !f&f.)fV6F'N.1..
00000090: 8c29 1318 5fb0 0300 fd3e 2ca3            .).._....>,.
--------------------------------------------------------------------

The obvious difference is that there is no "OSMHeader" included. This
file can be processed by the splitter without an empty bounding box
showing up. Because of this the problem with the splitter does not
occur with pbf files created with osmosis 0.38 or older.

Could please someone look into the splitter code and check if the
corrections that were done to osmosis have to by applied to the
splitter, too?

Thank you,

Michael