[mkgmap-dev] Splitter only writes points, not ways on huge file?

Sat May 29 06:47:47 BST 2021

Hi Felix,
the log shows
"way Map is empty"
so splitter doesn't find any ways in asia10m_.osm.pbf.

So, either the file doesn't contain ways or splitter fails to find them. Maybe use
osmconvert --out-statistics asia10m_.osm.pbf
to find out more.

Reg. performance when splitting huge files with --keep-complete
use o5m format and --max-areas=4096 (or higher)

I'll have to check why splitter needs so much time in the "Problem-list-generator" passes. Without ways or relations there is no need to even execute that.

The asia.poly contains two rings. I think splitter can copy with that but maybe phyghtmap cannot? Try with two separate *.poly files

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver at gmail.com>
Gesendet: Freitag, 28. Mai 2021 19:29
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Splitter only writes points, not ways on huge file?

Attached the full splitter.log - looks fine. Is it most likely that the error is on phyghtmap?

On Fri, 28 May 2021 at 13:12, Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>> wrote:
running now again with resolution 13, I had tried if changing resolution to 12 would solve the problem - but is does not.

On Fri, 28 May 2021 at 13:11, Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>> wrote:
Asia.poly is attached - I did not save the splitter.log - doing this right now (but will take some hours - splitting this huge file takes a lot of time)

The log starts with:
Splitter version 597 compiled 2020-01-30T14:10:47+0000
boundary-tags=use-exclude-list
cache=
description=
geonames-file=
handle-element-version=remove
ignore-osm-bounds=false
keep-complete=true
mapid=75400000
max-areas=2048
max-nodes=12000000
max-threads=7
mixed=false
no-trim=true
num-tiles=
output=pbf
output-dir=
overlap=auto
polygon-desc-file=
polygon-file=
precomp-sea=
problem-file=
problem-report=
resolution=13
route-rel-values=
search-limit=200000
split-file=
status-freq=120
stop-after=dist
wanted-admin-level=5
write-kml=
Elapsed time: 0s   Memory: Current 1012MB (7MB used, 1005MB free) Max 25000MB
Time started: Fri May 28 12:06:59 CEST 2021
Map is being split for resolution 13:
 - area boundaries are aligned to 0x800 map units (0.0439453125 degrees)
 - areas are multiples of 0x800 map units wide and high
Processing asia10m_.osm.pbf
Bounding box -179.99990000000003 -13.011650000000001 179.99990000000003 84.52666
10,000,000 nodes parsed... id=19999999
20,000,000 nodes parsed... id=29999999
30,000,000 nodes parsed... id=39999999
40,000,000 nodes parsed... id=49999999
50,000,000 nodes parsed... id=59999999
60,000,000 nodes parsed... id=69999999
70,000,000 nodes parsed... id=79999999
80,000,000 nodes parsed... id=89999999
90,000,000 nodes parsed... id=99999999
100,000,000 nodes parsed... id=109999999

On Fri, 28 May 2021 at 12:25, Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>> wrote:
Hi Felix,

the pbf file really contains only nodes.
Can you attach the splitter log file and the asia.poly, please?

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com>>
Gesendet: Freitag, 28. Mai 2021 11:18
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Splitter only writes points, not ways on huge file?

Hi Gerd,
sorry - fixed. I attached the file. I could also put the whole 60GB file for analysis. But do not know if you could download such a huge file or analyse it better.
https://openmtbmap.org/75400407.osm.pbf

Felix

On Fri, 28 May 2021 at 09:03, Gerd Petermann <gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com><mailto:gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>> wrote:
Hi Felix,

the link to the pbf file seems to be wrong?  I get an 404

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann <extremecarver at gmail.com<mailto:extremecarver at gmail.com><mailto:extremecarver at gmail.com<mailto:extremecarver at gmail.com>>>
Gesendet: Freitag, 28. Mai 2021 07:56
An: Development list for mkgmap
Betreff: [mkgmap-dev] Splitter only writes points, not ways on huge file?

I used phyghtmap to create a huge file (60GB) of 10m equidistance osm.pbf contourlines data.

phyghtmap --jobs=2 --polygon=/home/contourlines/bounds/asia.poly --step=10  --no-zero-contour --void-range-max=-420 --output-prefix="Asia10m" --line-cat=500,100--start-node-id=10000000 --start-way-id=10000000 --source=view1,srtm1,view3,srtm3 --max-nodes-per-way=230 --max-nodes-per-tile=0 --pbf

I am not fully sure that this pbf file is correct, but I created it exactly the same way as smaller countries. Maybe its also phyghtmap going wrong. But I doubt it with phyghtmaps incremental approach of just constantly writing out data.

When splitting it with splitter - and opening up one of the osm.pbf files in JOSM - I noticed it contains only the contourline nodes, no lines.
I think the Asia continent file spans 360 degrees - so maybe that causes problems with splitter?

I have uploaded a splitted file here: https://openmtbmap.org/75400407.osm.pbf
If you load it in Josm, you will notice it is alll the points but no lines...

--
Felix Hartman - Openmtbmap.org & VeloMap.org

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--
Felix Hartman - Openmtbmap.org & VeloMap.org

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--
Felix Hartman - Openmtbmap.org & VeloMap.org

--
Felix Hartman - Openmtbmap.org & VeloMap.org

--
Felix Hartman - Openmtbmap.org & VeloMap.org