[mkgmap-dev] Commit: r3319: Give a warning for probable cases of files with wrong encoding.

Thu Jul 31 17:47:44 BST 2014

Version mkgmap-r3319 was committed by steve on Thu, 31 Jul 2014

Give a warning for probable cases of files with wrong encoding.

The default reading routines in Java cause bad characters in unicode input to be
replaced with the unicode replacement character 0xfffd.  We detect this in the
TokenScanner, and print a message with the file and line number.  This will
apply to all files that are read using TokenScanner, so probably everything
other than the maps themselves.

It is possible for a file that is in the wrong encoding to still be valid utf-8
although this would be rare, and you can always add a comment with some accented
characters that is likely to fail when in the incorrect encoding.

It does mean that you can't genuinely have a replacement character in your input.
I don't think that this is likely to be a real problem, but we did have a few
ourselves already, which have been removed by this patch.