[mkgmap-dev] Exception in thread "main" java.lang.IllegalArgumentException . on Europe extract

Wed Dec 1 16:44:51 GMT 2021

yes it is some sort of overflow - if I do the map in two runs (1500 tiles
approx)
with 65500*
and 65501*
it compiles fine.
That's why I use a pretty long exclusion list - but this is not enough. The
index without split-name-index is about half the size. I think the first
list of options only matters on the actual mkgmap creation run - and  does
not change anything anymore on the second step of creating the address
index. Maybe I missed a couple - but most should be excluded.

--mdr7-del=pri;sec;ter;cw;min;unsf,uncl;living;pdstrn;trk;pth;ft;fp;brdlw;rd;byw;ser;cw;mrt;crt;Bklane;Bktrk;opp;Opptrk;g0;g1;g2;g3;g4;g5;g6;t1;t2;t3;t4;t5;t6;xbk;cr;mr;hr;mr0;mr1;mr2;mr3;mr4;mr5;mr0.;mr1.;mr2.;mr3.;mr4.;mr5.;mr6.;mr00;mr01;mr02;mr03;mr04;mr05;mr06;mr11;mr12;mr13;mr14;mr15;mr16;mr20;mr21;mr22;mr23;mr24;mr25;mr26;mr30;mr31;mr32;mr33;mr34;mr35;mr35;mr40;mr41;mr41;mr43;mr44;mr45;mr46;mr50;mr51;mr52;mr53;mr54;mr55;mr56;mr60;mr61;mr62;mr63;mr64;mr65;mr66;m0;m1;m2;m3;m4;m5;m0.;m1.;m2.;m3.;m4.;m5.;m6.;m00;m01;m02;m03;m04;m05;m06;m11;m12;m13;m14;m15;m16;m20;m21;m22;m23;m24;m25;m26;m30;m31;m32;m33;m34;m35;m35;m40;m41;m41;m43;m44;m45;m46;m50;m51;m52;m53;m54;m55;m56;m60;m61;m62;m63;m64;m65;m66;mr.0;mr.1;mr.2;mr.3;mr.4;mr.5;mr.6;m.0;m.1;m.2;m.3;m.4;m.5;m.6;m1.;m2.;m3.;m4.;m5.;m6.;lmr;mrx0;mrx1;mrx2;mrx3;mrx4;mrx5;mrx6;mx0;mx1;mx2;mx3;mx4;mx5;mx6;bridge;swing;aqueduct;viaduct;"ntnl
bndry";admin_lv=4;"ntnl
park";weir;dam;waterfall;rapids;stream;river;drain;ditch;"huge
river";"hugest river";"medium river";"small river";"wide river";"hugest
canal";"huge canal";"medium canal";"small canal";"big
canal";canal;snowpark;Skitour-descent;Skipiste;Skislope;illuminated;Nordic-Skiing;skitour;platter;ropetow;tbar;chairlift;goods_lift;runway;taxiway;apron;VFS0;VFS1;VFS2;VFS3;VFS4;VFS5;VFS6;VFS7;usf;cstrn;steps;parking_aisle;res;cob;cobstn;public_footway;public_footpath
--mdr7-excl=pri;sec;ter;cw;min;unsf,uncl;pdstrn;trk;pth;ft;fp;brdlw;rd;byw;ser;cw;mrt;crt;Bklane;Bktrk;opp;Opptrk;g0;g1;g2;g3;g4;g5;g6;t1;t2;t3;t4;t5;t6;xbk;cr;mr;hr;mr0;mr1;mr2;mr3;mr4;mr5;mr0.;mr1.;mr2.;mr3.;mr4.;mr5.;mr6.;mr00;mr01;mr02;mr03;mr04;mr05;mr06;mr11;mr12;mr13;mr14;mr15;mr16;mr20;mr21;mr22;mr23;mr24;mr25;mr26;mr30;mr31;mr32;mr33;mr34;mr35;mr35;mr40;mr41;mr41;mr43;mr44;mr45;mr46;mr50;mr51;mr52;mr53;mr54;mr55;mr56;mr60;mr61;mr62;mr63;mr64;mr65;mr66;m0;m1;m2;m3;m4;m5;m0.;m1.;m2.;m3.;m4.;m5.;m6.;m00;m01;m02;m03;m04;m05;m06;m11;m12;m13;m14;m15;m16;m20;m21;m22;m23;m24;m25;m26;m30;m31;m32;m33;m34;m35;m35;m40;m41;m41;m43;m44;m45;m46;m50;m51;m52;m53;m54;m55;m56;m60;m61;m62;m63;m64;m65;m66;mr.0;mr.1;mr.2;mr.3;mr.4;mr.5;mr.6;m.0;m.1;m.2;m.3;m.4;m.5;m.6;m1.;m2.;m3.;m4.;m5.;m6.;lmr;mrx0;mrx1;mrx2;mrx3;mrx4;mrx5;mrx6;mx0;mx1;mx2;mx3;mx4;mx5;mx6;"ntnl
bndry"

On Wed, 1 Dec 2021 at 17:39, Gerd Petermann <gpetermann_muenchen at hotmail.com>
wrote:

> Hi Felix,
>
> your index is probably simply too large (> 2GB).
> The attached patch fixes another possible overflow, but I am not sure if
> such a file size is allowed for the index.
>
> Maybe try to remove specific items from the index (see --mdr7-excl,
> --mdr7-del)
> I think the reason for the huge size is your style which combines a lot of
> information in the road name, e.g. "102 Thw-Göttinger Weg T2 Pth
> Zentralalpenweg 02 Hr"
> or "137 Mindener Jubiläumsweg T2 Pth Mindener Jubiläumsweg Hr"
> (found these in omtb_austria_13.05.2021)
> All this information is stored in the PC index (a string table), and with
> --split-name-index lots of entries are produced for strings like "Pth" or
> "T2" or "Hr"
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von
> Felix Hartmann <extremecarver at gmail.com>
> Gesendet: Mittwoch, 1. Dezember 2021 17:16
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Exception in thread "main"
> java.lang.IllegalArgumentException . on Europe extract
>
> I found out that if I remove --split-name-index then it compiles fine. I'm
> just trying a minimal setup option wise and could look if it's a special
> .img tile or not.
> I could upload all map tiles somewhere for further analysis - or try some
> more things before doing that (12GB data or so).
>
> On Wed, 1 Dec 2021 at 13:08, Felix Hartmann <extremecarver at gmail.com
> <mailto:extremecarver at gmail.com>> wrote:
> Hi Gerd,
>
> here is the log - hope you can find out something with it - but I think it
> is missing the problem - there are only 3 lines before the crash - all else
> is many minutes before.
>
> 2021/12/01 12:55:08 INFO (ImgFS): bs=512, whole size=-1789952000,
> hb=-14501, fb=-3481499, blocks=-3496000
>
> 2021/12/01 12:55:08 INFO (ImgFS): bs=1024, whole size=-1786235904,
> hb=-3623, fb=-1740748, blocks=-1744371
>
> 2021/12/01 12:55:08 INFO (ImgFS): Best block size: 512
> sizeInBlocks=-3496000, reserved=-14501
>
>
> On Wed, 1 Dec 2021 at 08:57, Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com>>
> wrote:
> Hi Felix,
>
> hm, in that case one of the two factors is already negative.
> You can enable logging and run the combiner step again (no need to compile
> the tiles again)
> Add this to the log config:
> uk.me.parabola.imgfmt.sys.ImgFS.level=FINE
>
> BTW: The index contains not only address search data, also POI.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <
> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>
> Gesendet: Mittwoch, 1. Dezember 2021 02:27
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Exception in thread "main"
> java.lang.IllegalArgumentException . on Europe extract
>
> Hi Gerd,
>
> thanks for the quick response. Any other way I can try to get more debug
> information?
>
> I tried with the patch but I still get:
> gmapi-minimal: Skipping file
> E:\OpenMTBMap\contourlines20\europe\75500448.img
> Exception in thread "main" java.lang.IllegalArgumentException
>         at java.base/sun.nio.ch<http://sun.nio.ch
> >.FileChannelImpl.position(FileChannelImpl.java:355)
>         at uk.me.parabola.imgfmt.sys.ImgFS.sync(ImgFS.java:249)
>         at uk.me.parabola.imgfmt.sys.ImgFS.close(ImgFS.java:325)
>         at
> uk.me.parabola.mkgmap.combiners.MdrBuilder.onFinish(MdrBuilder.java:334)
>         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:690)
>         at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
>         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:147)
>         at uk.me.parabola.mkgmap.main.Main.main(Main.java:118)
>
> On Tue, 30 Nov 2021 at 14:48, Gerd Petermann <
> gpetermann_muenchen at hotmail.com<mailto:gpetermann_muenchen at hotmail.com
> ><mailto:gpetermann_muenchen at hotmail.com<mailto:
> gpetermann_muenchen at hotmail.com>>> wrote:
> Hi Felix,
>
> might be an overflow. The code line is
> file.position(param.getReservedDirectoryBlocks() * param.getBlockSize());
>
> so two ints are multiplied.
>
> Please try if the attached patch changes something.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk><mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk<mailto:
> mkgmap-dev-bounces at lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann <
> extremecarver at gmail.com<mailto:extremecarver at gmail.com><mailto:
> extremecarver at gmail.com<mailto:extremecarver at gmail.com>>>
> Gesendet: Dienstag, 30. November 2021 13:23
> An: Development list for mkgmap
> Betreff: [mkgmap-dev] Exception in thread "main"
> java.lang.IllegalArgumentException . on Europe extract
>
> Using mkgmap version 4808 - when compiling Europe. It went through fine on
> all geofabrik country extracts.
> I then upgraded to mkgmap 4819 and yesterday downloaded a new europe
> extract (4 days newer) from geofabrik to see if I can reproduce it again -
> and the error message is identical.
> I'm going to run with default style now again - to see if it makes a
> difference.
>
> C:\openmtbmap\mkgmap.jar --max-jobs=11 --order-by-decreasing-area
> "--generate-sea" --code-page=65001
> "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip"
> "--style-file=C:\openmtbmap\openmtbmap_style"
> --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction
> --improve-overview --drive-on=detect --allow-reverse-merge --lower-case
> --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18"
> --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13, 12:12"
> --add-pois-to-areas
> --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance
> --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6
> --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4
> --add-boundary-nodes-at-admin-boundaries=2 --cycle-map
> --ignore-fixme-values --housenumbers
> --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt
> --split-name-index --link-pois-to-ways --ignore-turn-restrictions
> --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20
>  , 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_eu
> --show-profiles=1  --location-autofill=bounds,is_in,nearest
> --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route
> --country-abbr=eu --country-name=europe --mapname=65500000 --family-id=6550
> --product-id=1 --series-name=omtb_europe_29.11.2021_UC_LOCAL
> --family-name=mtb_eu_29.11.2021_UC_LOCAL --tdbfile --x-gmapi-minimal
> --overview-mapname=mapsetc --keep-going
> --area-name="europe_29.11.2021_UC_LOCAL_omtb" -c
> D:\openmtbmap\maps\template.europe
> E:\OpenMTBMap\contourlines20\europe\7*.img typeu.TYP  1>NUL
>
>
> Number of MapFailedExceptions: 0
> gmapi-minimal: Writing freshly compiled file
> C:\openmtbmap\maps\.\65500000.img
> gmapi-minimal: Writing freshly compiled file
> C:\openmtbmap\maps\.\65500001.img
> gmapi-minimal: Writing freshly compiled file
> C:\openmtbmap\maps\.\65500002.img
> gmapi-minimal: Writing freshly compiled file
> C:\openmtbmap\maps\.\65500003.img
> gmapi-minimal: Writing freshly compiled file
> C:\openmtbmap\maps\.\65500004.img
> ................
> skipped
> .....................
> long list of contourline files:
> .........
> gmapi-minimal: Skipping file
> E:\OpenMTBMap\contourlines20\europe\75500447.img
> WARNING (global): Input file
> E:\OpenMTBMap\contourlines20\europe\75500448.img has different code page
> 1252
> WARNING (global): Input file
> E:\OpenMTBMap\contourlines20\europe\75500448.img has different charset type
> 10
> WARNING (global): Input files have different code pages
> gmapi-minimal: Skipping file
> E:\OpenMTBMap\contourlines20\europe\75500448.img
> .... 448 is the highest number contourlines file
>
> ............And here comes the error
> it happens on writing the address index mdr file - mapset.img and
> mapset.tdb have already been created.
>
> Exception in thread "main" java.lang.IllegalArgumentException
>         at java.base/sun.nio.ch<http://sun.nio.ch><http://sun.nio.ch
> >.FileChannelImpl.position(FileChannelImpl.java:355)
>         at uk.me.parabola.imgfmt.sys.ImgFS.sync(ImgFS.java:249)
>         at uk.me.parabola.imgfmt.sys.ImgFS.close(ImgFS.java:325)
>         at
> uk.me.parabola.mkgmap.combiners.MdrBuilder.onFinish(MdrBuilder.java:334)
>         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:690)
>         at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
>         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:147)
>         at uk.me.parabola.mkgmap.main.Main.main(Main.java:118)
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk
> ><mailto:mkgmap-dev at lists.mkgmap.org.uk<mailto:
> mkgmap-dev at lists.mkgmap.org.uk>>
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk<mailto:mkgmap-dev at lists.mkgmap.org.uk>
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
>
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mkgmap.org.uk/pipermail/mkgmap-dev/attachments/20211201/3c2ed78b/attachment-0001.html>