[mkgmap-dev] type/subtype of points and cities

Thu Nov 7 13:15:09 GMT 2019

Hi Gerd

Yes, I'll change allElements to not try subtypes for isCityType, and,
elsewhere, assert that indPoints don't have a subtype. 

My Etrex only shows points in the range you mention in the results for
nearby cities, but doing a spell search it finds whatever is in the
appropriate MDR - presume group 0 in MDR 10.

I find it a useful feature (for UK postcodes) - having a point that can
be searched for by name but doesn't swamp the 'nearby' display. 
City search seemed to be the only place I could do this and the various
old city range definitions allowed it anyway.

Ticker

On Thu, 2019-11-07 at 12:51 +0000, Gerd Petermann wrote:
> Hi Ticker,
> 
> maybe allElements should only produce POIs for types acepted by
> GTYpe.checkType()?
> When I do that I see no crashes in MapSource but the MdrCheck still
> complains about some mdr11 entries.
> 
> Reg. MapPoint.isCityType():
> According to MapSource only type 0x0100 .. 0x0d00 are cities (and
> subtype must be zero)
> When I use "Find Nearest  Places" in Mapsource it doesn't show the
> other types when I limit the search to cities.
> 
> Gerd
> 
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Gerd Petermann <gpetermann_muenchen at hotmail.com>
> Gesendet: Donnerstag, 7. November 2019 13:18
> An: Ticker Berkin; mkgmap development
> Betreff: Re: [mkgmap-dev] type/subtype of points and cities
> 
> Hi Ticker,
> 
> the problems with AllElements are not new. I see them also with much
> older versions of mkgmap.
> 
> Gerd
> 
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> Gesendet: Donnerstag, 7. November 2019 12:56
> An: mkgmap development
> Betreff: Re: [mkgmap-dev] type/subtype of points and cities
> 
> Hi Gerd
> 
> MdrDisplay/cityPtrSize:
> 
> I didn't change any significant behaviour. Because it seemed
> inconsistent, I put in a diagnostic. I also made the switch stmt in
> printSec11_city like the similar instances that do have to handle
> 1/2/3
> byte objects rather than switch ... case 3: ... default: ...
> 
> I should also have added handling cityPtrSize of 4. I fixed mkgmap to
> work correctly for this. However, when I get a tile that needs 4 byte
> pointers, the gmapsupp.img size jumps up, so I re-split the map with
> a
> different number of nodes until the city count reduces again.
> 
> My preference here would be rename one of the cityPtrSize variables
> and
> keep/extend the switch statement. Happy to get rid of the
> test/system.out.printf
> 
> MdrCheck/group/toType:
> 
> The original code in check10() tried to re-constitute a type/subtype
> as
> per mkgmap from group/someBits data and it didn't do it fully or
> correctly.
> 
> I agree that this tool should just display/validate the understanding
> of the Garmin IMG structure. However, in this case, because the
> fullType generated above is then validated against data from mdr11,
> it
> has to do this validation based on the grouping chosen by mkgmap.
> Similarly, if these mappings are changed in mkgmap, equivalent
> changes
> need to be made to Display (I put in a few comments to this effect in
> the relevant places)
> 
> AllElements:
> 
> On my Etrex device, it shows some points not in the correct place
> according to sequence with strange types or names. Some of these
> points
> are outside the map area! I can't quite remember the details, it has
> been a while since I last used it.
> 
> So yes, I'm sure it is generating something it shouldn't and maybe
> the
> lower levels of mkgmap code should be checking/objecting as well. It
> might be related to indPoints sometimes having subtypes; I think
> there
> are assumptions that these are a fixed size items.
> 
> I don't know if this will cause the problems you describe, but I'll
> have a look at it.
> 
> Ticker
> 
> 
> On Thu, 2019-11-07 at 09:15 +0000, Gerd Petermann wrote:
> > Hi Ticker,
> > 
> > I've not yet understood the changes regarding cityPtrSize. Do you
> > think that Garmin supports one-byte pointers when there are less
> > than
> > 128 cities?
> > The transalpin demo map contains only a few cities or regions but
> > uses two  bytes.
> > I've attached a patch with my proposed changes.
> > 
> > One more hint:
> > I don't like the description "Understands the same MDR groups as
> > generated by mkgmap so it can recreate the correct full type."
> > My understanding of display tool is this:
> > - The main purpose is to find out how the Garmin algos encode
> > things
> > in IMG format, so we should use maps produced by Garmin and
> > Mapsource/Basecamp to verify. The code in mkgmap should be the
> > results of those findings, not vice versa.
> > - I also use it to verify that changes made in mkgmap don't do
> > something unexpected.
> > 
> > I've just learned that Mapsource doesn't like the map produced with
> > java -jar mkgmap.jar --index --gmapi test-map:all-elements
> > It crashes when you search for all POI and also when you search for
> > cities (both without giving a name)
> > 
> > Gerd
> > 
> > ________________________________________
> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > Gesendet: Dienstag, 5. November 2019 12:44
> > An: mkgmap development
> > Betreff: Re: [mkgmap-dev] type/subtype of points and cities
> > 
> > Hi Gerd
> > 
> > Here is patch for Display.
> > 
> > Changes are:
> >   Couple of extra diagnostics.
> >   Handles 1 byte cities.
> >   Moves a call of getSection(11).get... out of a loop.
> >   Consistent handling/display of point type/subtype.
> >   Understands the same MDR groups as generated by mkgmap so
> >     it can recreate the correct full type.
> > 
> > Ticker
> > 
> > On Tue, 2019-11-05 at 09:43 +0000, Gerd Petermann wrote:
> > > Hi Ticker,
> > > 
> > > okay, looks good to me. In an earlier post you mentioned that
> > > display
> > > tool needs changes, too. Please post that patch as well.
> > > 
> > > Gerd
> > > 
> > > ________________________________________
> > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
> > > Auftrag
> > > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > > Gesendet: Montag, 4. November 2019 18:12
> > > An: Development list for mkgmap
> > > Betreff: Re: [mkgmap-dev] type/subtype of points and cities
> > > 
> > > Hi Gerd
> > > 
> > > To stay compatible with "Openfietsmap full" style, I've updated
> > > the
> > > patch to change the range of city to be:
> > >   type >= 0x0100 && type < 0x1100;
> > > Release mkgmap had this as:
> > >   type >= 0x0100 && type <= 0x1100;
> > > 
> > > In release mkgmap, a point with value 0x1100 was added to the RGN
> > > as
> > > an
> > > indPoint, but not indexed as a city by MDR. Also 0x1100 is not
> > > findable
> > > as a nearby city, at least on Garmin eTrex. I couldn't detect any
> > > useful behaviour for this combination.
> > > 
> > > Ticker
> > > 
> > > On Tue, 2019-10-22 at 15:03 +0000, Gerd Petermann wrote:
> > > > Hi Minko,
> > > > 
> > > > "(like you say)" should have been ... "(like Ticker wrote)"
> > > > ... (a small dot with the label) ...
> > > > sorry, I forgot to use the typ file, so 0x1101 is displayed
> > > > with
> > > > an
> > > > icon showing a bus and without the label.
> > > > But why do you use a type that is not indexed?
> > > > 
> > > > Gerd
> > > > 
> > > > ________________________________________
> > > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
> > > > Auftrag
> > > > von Gerd Petermann <gpetermann_muenchen at hotmail.com>
> > > > Gesendet: Dienstag, 22. Oktober 2019 16:52
> > > > An: Development list for mkgmap
> > > > Betreff: Re: [mkgmap-dev] type/subtype of points and cities
> > > > 
> > > > Hi Ticker,
> > > > 
> > > > thanks.
> > > > 
> > > >  @Minko:
> > > > I've created a small map with "Openfietsmap full" style and
> > > > trunk
> > > > r4315 , OSM data contains an amenity=bus_station.
> > > > The result is that this POI appears in the map (a small dot
> > > > with
> > > > the
> > > > label), but it is not in the mdr (like you say).
> > > > I found no older versions of mkgmap where this would have
> > > > worked
> > > > different, so it seems useless to me because normally I'd want
> > > > to
> > > > be
> > > > able to find the bus station under transport services.
> > > > 
> > > > Why do you use 0x1101, 0x1102, or 0x1108 for POI?
> > > > 
> > > > Gerd
> > > > 
> > > > ________________________________________
> > > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
> > > > Auftrag
> > > > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > > > Gesendet: Dienstag, 22. Oktober 2019 16:29
> > > > An: Development list for mkgmap
> > > > Betreff: Re: [mkgmap-dev] type/subtype of points and cities
> > > > 
> > > > Hi Gerd
> > > > 
> > > > The old reader/osm/Gtype:checkType() didn't diagnose cities
> > > > (however
> > > > they might be defined) with non-zero subtypes - this is where
> > > > the
> > > > new
> > > > message comes from.
> > > > 
> > > > Old general/MapPoint:isCity[Type] defined city as:
> > > > 
> > > >         return type >= 0x0100 && type <= 0x1100
> > > > 
> > > > ie 0x1100 is city, 0x1101 isn't, which I thought wasn't a good
> > > > definition, so I changed it to:
> > > > 
> > > >         return type >= 0x0100 && type < 0x1200;
> > > > 
> > > > I chose all of 0x11XX to be considered cities so that 0x1100
> > > > processing
> > > > as a city didn't change (this and 0x1000 were put into RGN
> > > > indPoints,
> > > > but were not indexed as cities by MDR)
> > > > 
> > > > Changing it to < 0x1100 would be reasonable, because the device
> > > > range
> > > > seems to be < 0x0e00 and MDR5 city indexing was for points <
> > > > 0x1000
> > > > 
> > > > Ticker
> > > > 
> > > > On Tue, 2019-10-22 at 12:32 +0000, Gerd Petermann wrote:
> > > > > Hi Ticker,
> > > > > 
> > > > > I see some new warnings for Minkos popular Openfietsmap
> > > > > styles
> > > > > with
> > > > > this patch:
> > > > > checking style: Openfietsmap full
> > > > > Warning: invalid type 0x1101 for POINT in style file points,
> > > > > line
> > > > > 97
> > > > > Warning: invalid type 0x1102 for POINT in style file points,
> > > > > line
> > > > > 263
> > > > > Warning: invalid type 0x1102 for POINT in style file points,
> > > > > line
> > > > > 264
> > > > > Warning: invalid type 0x1108 for POINT in style file points,
> > > > > line
> > > > > 336
> > > > > Warning: invalid type 0x1108 for POINT in style file points,
> > > > > line
> > > > > 348
> > > > > Warning: invalid type 0x1108 for POINT in style file points,
> > > > > line
> > > > > 349
> > > > > Warning: invalid type 0x1108 for POINT in style file points,
> > > > > line
> > > > > 350
> > > > > 
> > > > > The corresponding lines in the points file
> > > > > amenity=bus_station [0x1101 resolution 24 continue]
> > > > > railway=halt [0x1102 resolution 22]
> > > > > railway=station [0x1102 resolution 20-23 continue]
> > > > > (barrier=bollard | barrier=bus_trap | barrier=block) [0x1108
> > > > > resolution 23]
> > > > > access:conditional=* & access:conditional ~
> > > > > '!(sunset|sunrise|destination)' & bicycle!=no &
> > > > > bicycle!=permissive
> > > > > &
> > > > > vehicle!=no { name 'access=${access:conditional}' } [0x1108
> > > > > resolution 24]
> > > > > bicycle:conditional=* & bicycle:conditional ~
> > > > > '!(sunset|sunrise|destination)' { name
> > > > > 'bicycle=${bicycle:conditional}' } [0x1108 resolution 24]
> > > > > vehicle:conditional=* & vehicle:conditional ~
> > > > > '!(sunset|sunrise|destination)' { name
> > > > > 'vehicle=${vehicle:conditional}' } [0x1108 resolution 24]
> > > > > 
> > > > > Can you explain what happened with those POI without the
> > > > > patch?
> > > > > 
> > > > > Gerd
> > > > > 
> > > > > ________________________________________
> > > > > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im
> > > > > Auftrag
> > > > > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > > > > Gesendet: Dienstag, 22. Oktober 2019 13:07
> > > > > An: Development list for mkgmap
> > > > > Betreff: Re: [mkgmap-dev] type/subtype of points and cities
> > > > > 
> > > > > Hi Gerd
> > > > > 
> > > > > Here is the patch based on the current trunk
> > > > > 
> > > > > Ticker
> > > > > On Tue, 2019-04-23 at 11:51 +0100, Ticker Berkin wrote:
> > > > > > Hi
> > > > > > 
> > > > > > I think the mkgmap internal handling of types/subtypes of
> > > > > > points
> > > > > > is
> > > > > > obscure.
> > > > > > 
> > > > > > In the points style file, the type code is always full, ie
> > > > > > type
> > > > > > <<
> > > > > > 8
> > > > > > > 
> > > > > > subtype, but when the points are read back from the RGN
> > > > > > file
> > > > > > for
> > > > > > the
> > > > > > MDR processing, the representation is the same provided the
> > > > > > subtype
> > > > > > is
> > > > > > not zero, otherwise it is just type! Logic then decides
> > > > > > that
> > > > > > if
> > > > > > the
> > > > > > value is <= 0xff it is because the subtype is zero.
> > > > > > 
> > > > > > It is simpler and much clearer to preserve the original
> > > > > > representation.
> > > > > > This also allows unambiguous use of type 0 with subtypes,
> > > > > > and,
> > > > > > possibly, city with subtypes.
> > > > > > 
> > > > > > It also allows the same function to be used to test a type
> > > > > > for
> > > > > > being
> > > > > > in
> > > > > > the 'city' range, regardless of the context. mkgmap wasn't
> > > > > > consistent
> > > > > > the ranges for the isCity test.
> > > > > > 
> > > > > > City POI are written in a different manner to the RGN file.
> > > > > > The
> > > > > > old
> > > > > > range for this was:
> > > > > >   type >= 0x0100 && type <= 0x1100
> > > > > > which I've kept, except changing to: ...& type < 0x1200. A
> > > > > > similar
> > > > > > range was used for MDR5.
> > > > > > 
> > > > > > However, city POI are held in group 1 of MDR 9/10. This is
> > > > > > used
> > > > > > in
> > > > > > Find
> > > > > > > City 'By Name'. The old range was: type <= 0xf, where, if
> > > > > > > the
> > > > > > > subtype
> > > > > > was zero, type is right shifted by 8 (see above), non-zero
> > > > > > subtypes
> > > > > > messed up the testing. Now it uses the same range as above.
> > > > > > 
> > > > > > Actually neither of these ranges are correct for my Garmin
> > > > > > devices.
> > > > > > Find > City show nearby POI with types in the range
> > > > > > 0x0100..0x0d1f,
> > > > > > regardless of the img RGN or MDR. The eTrex Legend labels
> > > > > > POIs
> > > > > > in
> > > > > > this
> > > > > > range as {Large/Medium/Small} City/Town and points in the
> > > > > > range
> > > > > > 0x0e00..0x111f as "*".
> > > > > > 
> > > > > > Find > City 'by-name' returns POI if in group 1 of MDR
> > > > > > 9/10.
> > > > > > I
> > > > > > find
> > > > > > it
> > > > > > a useful feature to have a POI that can be searched for but
> > > > > > doesn't
> > > > > > flood the list of nearby points.
> > > > > > 
> > > > > > Generally, non-zero subtypes for cities cause problems in
> > > > > > the
> > > > > > RGN
> > > > > > structure and I've added a test for this to the style
> > > > > > validation.
> > > > > > 
> > > > > > There is also a fix to --make-poi-index, but I can't detect
> > > > > > any
> > > > > > effect
> > > > > > of this option.
> > > > > > 
> > > > > > This patch won't make any difference to the img output
> > > > > > unless
> > > > > > the
> > > > > > there
> > > > > > are points 0x1000, 0x1100.
> > > > > > 
> > > > > > If this patch is accepted, I'll do the equivalent to the
> > > > > > img
> > > > > > display
> > > > > > system.
> > > > > > 
> > > > > > Regards
> > > > > > Ticker
> > > > > > _______________________________________________
> > > > > > mkgmap-dev mailing list
> > > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > > _______________________________________________
> > > > > mkgmap-dev mailing list
> > > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > _______________________________________________
> > > > mkgmap-dev mailing list
> > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > _______________________________________________
> > > > mkgmap-dev mailing list
> > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > > _______________________________________________
> > > > mkgmap-dev mailing list
> > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev