[mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

Wed Oct 20 09:24:24 BST 2021

Hi Ticker & Steve,

I don't understand the mixed use of collator.compare() and String.equals() in the Mdr classes.
When we use the collator to sort the data we probably also have to use it to compare for equality while grouping?

I also see differences between the code in MdrCheck and the classes in mkgmap.

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen at hotmail.com>
Gesendet: Mittwoch, 20. Oktober 2021 09:59
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

Hi Ticker,

please double check Mdr25:
I just wonder why we compare the region name when we sort by the country name.

Looks wrong (also in the unpatched code)

Gerd

________________________________________
Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
Gesendet: Dienstag, 19. Oktober 2021 12:10
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

Hi Gerd

Here it is

Ticker

On Tue, 2021-10-19 at 09:22 +0000, Gerd Petermann wrote:
> Hi Ticker,
>
> yes, please remove all unrelated optimizations.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> Gesendet: Dienstag, 19. Oktober 2021 11:03
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building
> index from unicode tiles
>
> Hi Gerd
>
> I'd removed the change relating to clearing the reference to the Sort
> object to allow garbage garbage collection; as you said, this won't
> happen because Sort is shared. I do notice, however, that on a
> typical
> mkgmap run, Sort is created/read 3 times - it isn't shared as fully
> as
> possible.
>
> The other changes (LargeListSorter) are slight improvements to memory
> usage and/or processing time - I can remove them if you want.
>
> Ticker
>
>
> On Tue, 2021-10-19 at 08:13 +0000, Gerd Petermann wrote:
> > Hi Ticker,
> >
> > please remove the unrelated changes. I think we discussed them with
> > patch mdrSort.patch in May, subject "MDR building out-of-memory".
> >
> > Gerd
> >
> > ________________________________________
> > Von: mkgmap-dev <mkgmap-dev-bounces at lists.mkgmap.org.uk> im Auftrag
> > von Ticker Berkin <rwb-mkgmap at jagit.co.uk>
> > Gesendet: Montag, 18. Oktober 2021 16:36
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building
> > index from unicode tiles
> >
> > Hi Gerd
> >
> > Here is first version of the changes to improve MDR unicode and
> > stop
> > the crash.
> >
> > It always provides a PRIMARY strength sort value, both in the key
> > for
> > sorting and direct comparison when using the collator. Previously
> > neither of these would have anything for a unicode character not
> > mentioned in the sort/cp65001.txt file
> >
> > In an attempt to stop ordering clashes between the specified sort
> > and
> > the ones fudged from the actual unicode value, it orders anything
> > unknown after the known values. Unfortunately these can then become
> > larger than 2 bytes - and, as this is all the space available
> > without
> > re-structuring, they have to wrap onto the known sort region. I
> > only
> > found 1 character that did this and I don't know if it conflicted
> > with
> > an existing sort.
> >
> > Regardless of the character set used, in all the places where
> > sorting
> > is used for de-dupe, I've used the SECONDARY strength collator to
> > detect similar record instead of name.equals(lastName)
> >
> > I also noticed that my source base included optimisation for
> > LargeListSorter, its use of a key cache and some tidy-up of this in
> > mdr7 & mdr11 so these are here as well.
> >
> > Ticker
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > mkgmap-dev at lists.mkgmap.org.uk
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
mkgmap-dev at lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev