[mkgmap-dev] Find Address - No cities

Thu Aug 3 10:11:28 BST 2017

Hi Gerd

I introduced them as methods to be used where unsigned bytes/words need
to be written and the range verified with an assertion, rather than
(byte)/(char) casts in the main code that simply discard the unwanted
bytes.

Almost all put()/putChar() in imgfmt/* should use these new methods,
but I didn't want to make excess changes while solving the city...
problem.

Maybe the names of these, which I chose to match existing put3() and
the names in ImageFileReader - where different methods are really
required for signed and unsigned - should be made consistent.

Regards
Ticker  

On Thu, 2017-08-03 at 08:38 +0000, Gerd Petermann wrote:
> Hi  Ticker,
> 
> thanks. Please review:
> You introduced new methods put1 and put2 in Interface ImgFileWriter
> and SectionWriter implements them but I don't find where they are
> used.
> 
> Gerd
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Ticker Berkin <ticker at jagIT.co.uk>
> Gesendet: Donnerstag, 3. August 2017 10:07:30
> An: mkgmap-dev at lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Find Address - No cities
> 
> Hi Gerd
> 
> Sorry - I did do "ant test", but this doesn't detect the missing
> methods unless attempt made to call then.
> 
> New patch attached - I noticed that some of the new method comments
> where not quite correct so I've improved these.
> 
> Regards
> Ticker
> 
> On Thu, 2017-08-03 at 05:15 +0000, Gerd Petermann wrote:
> > Hi Ticker,
> > 
> > sounds good, but I cannot compile:
> > ant clean test
> > gives
> > build-test:
> >     [mkdir] Created dir: D:\mkgmap\build\test
> >     [javac] Compiling 73 source files to D:\mkgmap\build\test
> >     [javac] D:\mkgmap\test\func\lib\ArrayImgWriter.java:26: error:
> > ArrayImgWriter is not abstract and does not override abstract
> > method
> > putN(int,int) in ImgFileWriter
> >     [javac] public class ArrayImgWriter implements ImgFileWriter {
> >     [javac]        ^
> >     [javac] 1 error
> > 
> > ciao,
> > Gerd
> > ________________________________________
> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > Gesendet: Dienstag, 1. August 2017 14:31:57
> > An: mkgmap-dev at lists.mkgmap.org.uk
> > Betreff: Re: [mkgmap-dev] Find Address - No cities
> > 
> > Hi Gerd
> > 
> > I decided to both fix the code and put out a warning message if
> > there
> > are structures that switch to 3 byte pointers (zips, highways and
> > exit
> > facilities as well as cities)
> > 
> > Address search now works when there are > 65535 cities.
> > 
> > The code is a lot tidier - I've done it properly and it gets rid of
> > lots of messy bits of in-line testing against sizes > 255 etc.
> > 
> > Ticker
> > 
> > 
> > On Mon, 2017-07-31 at 13:12 +0000, Gerd Petermann wrote:
> > > Hi Tcker,
> > > 
> > > yes, good enough. I also thought about this alternative.
> > > 
> > > Gerd
> > > 
> > > 
> > > ________________________________________
> > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
> > > Auftrag
> > > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > > Gesendet: Montag, 31. Juli 2017 13:09:49
> > > An: mkgmap-dev at lists.mkgmap.org.uk
> > > Betreff: Re: [mkgmap-dev] Find Address - No cities
> > > 
> > > Hi Gerd
> > > 
> > > I started fixing all the places where there is an assumption
> > > about
> > > the
> > > number of cities (and zips) being < 65535 and I'm only part way
> > > through
> > > my list of suspect files and have make a lot of changes so far.
> > > 
> > > I'm thinking it is not worth while because:
> > > 
> > > The overall MDR size jumps because all city entries change to 3
> > > bytes.
> > > I get a smaller GMAPSUPP by increasing the splitting so that it
> > > stays
> > > in the 2 byte limit.
> > > 
> > > I might miss some subtle coding of the header flags and/or the
> > > input/output.
> > > 
> > > It might not work anyway after I've made all the changes.
> > > 
> > > The changes are a bit messy because I've just hacked each problem
> > > area
> > > with a fix. Really there should be some utility functions,
> > > enhancements
> > > to the readers/writers and pointerSize variables in the relevant
> > > classes.
> > > 
> > > I propose just to signal an error in individual maps when the
> > > exceed
> > > the limit
> > > 
> > > What do you think?
> > > 
> > > Regards
> > > Ticker
> > > 
> > > On Sun, 2017-07-30 at 10:16 +0100, Ticker Berkin wrote:
> > > > Hi Gerd
> > > > 
> > > > I'll just done a bit more investigation and see what seems to
> > > > be
> > > > the
> > > > the same assumption in imgfmt/app/net/RoadDef.java around line
> > > > 296
> > > >                                         if(numCities > 255)
> > > > 
> > > >  writer.putChar(cityIndex);
> > > >                                         else
> > > > 
> > > >  writer.put((byte)cityIndex);
> > > > 
> > > > I'll fix and test tomorrow and if there are still have problems
> > > > I'll
> > > > assemble the problem map, style etc for you
> > > > 
> > > > Thanks
> > > > Ticker
> > > > 
> > > > On Sun, 2017-07-30 at 06:33 +0000, Gerd Petermann wrote:
> > > > > Hi Ticker,
> > > > > 
> > > > > hmm, I think the patch looks good and I have no idea why the
> > > > > search
> > > > > doesn't work
> > > > > for you. Yesterday I thought the patch handles the overall
> > > > > number
> > > > > of
> > > > > cities in the gmapsupp,
> > > > > but it is about the number of cities in a single map tile.
> > > > > Maybe there is another problem caused by the high number of
> > > > > cities.
> > > > > Maybe you can post a link to the tile with
> > > > > that high number?
> > > > > 
> > > > > Gerd
> > > > > 
> > > > > ________________________________________
> > > > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
> > > > > Auftrag
> > > > > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > > > > Gesendet: Samstag, 29. Juli 2017 21:12:27
> > > > > An: mkgmap-dev at lists.mkgmap.org.uk
> > > > > Betreff: Re: [mkgmap-dev] Find Address - No cities
> > > > > 
> > > > > Hi Gerd
> > > > > 
> > > > > Quite possibly, or maybe there are limits in the device.
> > > > > 
> > > > > More testing reveals that although my lists of cities and
> > > > > streets
> > > > > within them appear correctly in Find>Address, it doesn't find
> > > > > anything.
> > > > > 
> > > > > Ticker
> > > > > 
> > > > > On Sat, 2017-07-29 at 17:22 +0000, Gerd Petermann wrote:
> > > > > > Hi Tcker,
> > > > > > 
> > > > > > thanks for the patch. I seem to remember that the number of
> > > > > > entries
> > > > > > in mdr5 is also used
> > > > > > elsewhere. Will have a closer look tomorrow.
> > > > > > 
> > > > > > Gerd
> > > > > > ________________________________________
> > > > > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
> > > > > > Auftrag
> > > > > > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > > > > > Gesendet: Samstag, 29. Juli 2017 19:14:02
> > > > > > An: mkgmap-dev at lists.mkgmap.org.uk
> > > > > > Betreff: Re: [mkgmap-dev] Find Address - No cities
> > > > > > 
> > > > > > Hi Gerd
> > > > > > 
> > > > > > It looks as if can simply write 1/2/3 bytes as have
> > > > > > indicated
> > > > > > in
> > > > > > the
> > > > > > magic field in the MDR header and I've changed it to do
> > > > > > this.
> > > > > > All
> > > > > > seems
> > > > > > OK.
> > > > > > 
> > > > > > Yes - I do misuse a City POI. But now, after having worked
> > > > > > out
> > > > > > how
> > > > > > the
> > > > > > MDR20 index works, I might be able to do what I want in a
> > > > > > better
> > > > > > way.
> > > > > > 
> > > > > > Attached is patch to write the correct number of bytes.
> > > > > > 
> > > > > > Regards
> > > > > > Ticker
> > > > > > 
> > > > > > On Sat, 2017-07-29 at 16:37 +0000, Gerd Petermann wrote:
> > > > > > > Hi Ticker,
> > > > > > > 
> > > > > > > yes, sounds plausible. The comment seems to say that we
> > > > > > > don't
> > > > > > > know
> > > > > > > how to handle so many cities.
> > > > > > > Why do you have so many cities. Do you "abuse" the
> > > > > > > corresponding
> > > > > > > POI
> > > > > > > for something?
> > > > > > > 
> > > > > > > Gerd
> > > > > > > ________________________________________
> > > > > > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk>
> > > > > > > im
> > > > > > > Auftrag
> > > > > > > von Ticker Berkin <ticker at jagIT.co.uk>
> > > > > > > Gesendet: Samstag, 29. Juli 2017 18:13:20
> > > > > > > An: mkgmap-dev at lists.mkgmap.org.uk
> > > > > > > Betreff: Re: [mkgmap-dev] Find Address - No cities
> > > > > > > 
> > > > > > > Hi Gerd
> > > > > > > 
> > > > > > > I think I've found the problem:
> > > > > > > 
> > > > > > > /src/uk/me/parabola/imgfmt/app/mdr/Mdr5.java line 227
> > > > > > > 
> > > > > > >         private void putLocalCityIndex(ImgFileWriter
> > > > > > > writer,
> > > > > > > int
> > > > > > > cityIndex) {
> > > > > > >                 if (localCitySize == 2)
> > > > > > >                 // 3 probably not possible in actual
> > > > > > > maps.
> > > > > > >                         writer.putChar((char) cityIndex);
> > > > > > >                 else
> > > > > > >                         writer.put((byte) cityIndex);
> > > > > > >         }
> > > > > > > 
> > > > > > > There were more than 63335 cities in the failing map.
> > > > > > > 
> > > > > > > Ticker
> > > > > > > 
> > > > > > > 
> > > > > > > On Sat, 2017-07-29 at 16:09 +0100, Ticker Berkin wrote:
> > > > > > > > Hi Gerd
> > > > > > > > 
> > > > > > > > I've done a bit of investigation and debugging. It
> > > > > > > > seems
> > > > > > > > to
> > > > > > > > be
> > > > > > > > building
> > > > > > > > the indexes with enough cities and streets. MDR20 is
> > > > > > > > the
> > > > > > > > suspect
> > > > > > > > index.
> > > > > > > > 
> > > > > > > > I'm now experimenting with mkgmap / Display
> > > > > > > > 
> > > > > > > > With just the suspect map built in gmapsupp:
> > > > > > > > 
> > > > > > > > $ ... test.ExtractFile gmapsupp.img
> > > > > > > > filter[]
> > > > > > > > Copying .               65024
> > > > > > > > Copying MAKEGMAP.MPS    78
> > > > > > > > Copying 74440044.RGN    4154927
> > > > > > > > Copying 74440044.TRE    18772
> > > > > > > > Copying 74440044.LBL    2174485
> > > > > > > > Copying 74440044.NET    1626350
> > > > > > > > Copying 74440044.NOD    3566829
> > > > > > > > Copying 00007444.MDR    2979295
> > > > > > > > Copying 00007444.SRT    912
> > > > > > > > 
> > > > > > > > $ ... test.display.MdrSummary 00007444.MDR
> > > > > > > > initial values    7    2    e
> > > > > > > > MDR 1  NR=1     (000001) RS=4  magic=0x0
> > > > > > > > MDR 4  NR=101   (000065) RS=3  magic=0x0
> > > > > > > > MDR 5  NR=54456 (00d4b8) RS=9  magic=0x152
> > > > > > > > MDR 6  NR=4039  (000fc7) RS=3  magic=0x1
> > > > > > > > MDR 7  NR=17674 (00450a) RS=6  magic=0x66
> > > > > > > > MDR 9  NR=11    (00000b) RS=4  magic=0x0
> > > > > > > > MDR 10 DataSize=427616   (00068660) magic=0x0
> > > > > > > > MDR 11 NR=106904(01a198) RS=10 magic=0x95
> > > > > > > > MDR 12 NR=11    (00000b) RS=7  magic=0x40a
> > > > > > > > MDR 17 DataSize=111036   (0001b1bc) magic=0x0
> > > > > > > > MDR 18 NR=102   (000066) RS=5  magic=0x6
> > > > > > > > MDR 19 NR=106904(01a198) RS=3  magic=0x2
> > > > > > > > MDR 20 NR=20389 (004fa5) RS=6  magic=0xe
> > > > > > > > MDR 22 NR=18117 (0046c5) RS=6  magic=0xc000e
> > > > > > > > MDR 24 NR=2     (000002) RS=6  magic=0x0
> > > > > > > > MDR 25 NR=70016 (011180) RS=3  magic=0x0
> > > > > > > > MDR 29 NR=2     (000002) RS=8  magic=0x26
> > > > > > > > 
> > > > > > > > seems fine but
> > > > > > > > 
> > > > > > > > $ ... test.display.MdrCheck 00007444.MDR
> > > > > > > > ---------- 00007444.MDR --------------------
> > > > > > > > EXTRA
> > > > > > > > remaining {}
> > > > > > > > # reading 74440044
> > > > > > > > mdr5 check
> > > > > > > > Exception in thread "main"
> > > > > > > > java.lang.IndexOutOfBoundsException:
> > > > > > > > Index:
> > > > > > > > 308736, Size: 70018
> > > > > > > >         at
> > > > > > > > java.util.ArrayList.rangeCheck(ArrayList.java:653)
> > > > > > > >         at java.util.ArrayList.get(ArrayList.java:429)
> > > > > > > >         at
> > > > > > > > test.display.check.MapDetails.getCity(MapDetails.java:1
> > > > > > > > 88
> > > > > > > > )
> > > > > > > >         at
> > > > > > > > test.display.MdrCheck.check5(MdrCheck.java:282)
> > > > > > > >         at
> > > > > > > > test.display.MdrCheck.print(MdrCheck.java:87)
> > > > > > > >         at
> > > > > > > > test.display.CommonDisplay.display(CommonDisplay.java:1
> > > > > > > > 71
> > > > > > > > )
> > > > > > > >         at
> > > > > > > > test.display.CommonDisplay.display(CommonDisplay.java:1
> > > > > > > > 96
> > > > > > > > )
> > > > > > > >         at
> > > > > > > > test.check.CommonCheck.runMain(CommonCheck.java:145)
> > > > > > > >         at
> > > > > > > > test.display.MdrCheck.main(MdrCheck.java:1789)
> > > > > > > > + exit
> > > > > > > > 
> > > > > > > > On another map from the split this runs OK, producing
> > > > > > > > masses
> > > > > > > > of
> > > > > > > > output:
> > > > > > > > 
> > > > > > > > $ ... test.display.MdrCheck 00007429.MDR
> > > > > > > > ---------- 00007429.MDR --------------------
> > > > > > > > EXTRA
> > > > > > > > remaining {}
> > > > > > > > # reading 74290047
> > > > > > > > ERROR: map 1: city 1: no name; index 1
> > > > > > > > mdr5 check
> > > > > > > > 1 map1; ABBEY WOOD mapCity=2 reg=0 (LONDON) country=0
> > > > > > > > (UNITED
> > > > > > > > KINGDOM^]GBR) ind20=0 rep=false
> > > > > > > > 2 map1; ABRIDGE mapCity=3 reg=0 (ESSEX) country=0
> > > > > > > > (UNITED
> > > > > > > > KINGDOM^]GBR)
> > > > > > > > ind20=0 rep=false
> > > > > > > > ... 87000 more lines ...
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Do you think mkgmap:Display is finding a real problem
> > > > > > > > and
> > > > > > > > I
> > > > > > > > should
> > > > > > > > pursue this line, or could it be unreliable and this
> > > > > > > > exception
> > > > > > > > is
> > > > > > > > due
> > > > > > > > to it rather than the format of the index?
> > > > > > > > 
> > > > > > > > Thanks
> > > > > > > > Ticker
> > > > > > > > 
> > > > > > > > On Wed, 2017-07-26 at 23:42 -0700, Gerd Petermann
> > > > > > > > wrote:
> > > > > > > > > Ticker Berkin wrote
> > > > > > > > > > With a particular split of 'british-isles', one map
> > > > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > split
> > > > > > > > > > causes the list of cities in the Find>Address
> > > > > > > > > > function
> > > > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > empty.
> > > > > > > > > 
> > > > > > > > > Maybe you have the same problem here which I
> > > > > > > > > described
> > > > > > > > > for
> > > > > > > > > road
> > > > > > > > > name search when road names contain special
> > > > > > > > > characters.
> > > > > > > > > In
> > > > > > > > > your
> > > > > > > > > case
> > > > > > > > > this would mean special characters in the citiy
> > > > > > > > > names.
> > > > > > > > > I suggest to add debug code in
> > > > > > > > > uk.me.parabola.imgfmt.app.mdr.Mdr5.writeSectData(ImgF
> > > > > > > > > il
> > > > > > > > > eW
> > > > > > > > > ri
> > > > > > > > > te
> > > > > > > > > r
> > > > > > > > > writer)
> > > > > > > > > to print all city names which are written to the
> > > > > > > > > global
> > > > > > > > > index.
> > > > > > > > > 
> > > > > > > > > Gerd
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > View this message in context:
> > > > > > > > > http://gis.19327.n8.nabble.com/Find-Add
> > > > > > > > > ress-No-cities-tp5899775p5899880.html
> > > > > > > > > Sent from the Mkgmap Development mailing list archive
> > > > > > > > > at
> > > > > > > > > Nabble.com.
> > > > > > > > > _______________________________________________
> > > > > > > > > mkgmap-dev mailing list
> > > > > > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > > > > _______________________________________________
> > > > > > > > mkgmap-dev mailing list
> > > > > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > > > _______________________________________________
> > > > > > > mkgmap-dev mailing list
> > > > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > > > _______________________________________________
> > > > > > > mkgmap-dev mailing list
> > > > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > > _______________________________________________
> > > > > > mkgmap-dev mailing list
> > > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > _______________________________________________
> > > > > mkgmap-dev mailing list
> > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > _______________________________________________
> > > > > mkgmap-dev mailing list
> > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > _______________________________________________
> > > > mkgmap-dev mailing list
> > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > _______________________________________________
> > > mkgmap-dev mailing list
> > > mkgmap-dev at lists.mkgmap.org.uk
> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > _______________________________________________
> > > mkgmap-dev mailing list
> > > mkgmap-dev at lists.mkgmap.org.uk
> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev