[mkgmap-dev] RE Commit r3801: merge split-shape branch

Thu Feb 16 01:23:31 GMT 2017

HI Ticker, I've tried the patch without the --order-by-decreasing-area option and it does now run without crashing. However, it suffers from the same problem as I mentioned earlier when trying --order-by-decreasing-area without the patch, in that the coastline is corrupted. I haven't yet turned on detailed logging - I can do that if it would help.

I note that in MapSplitter.java there is a new line:

log.info("Single item larger that WANTED_MAX_AREA_SIZE", area.getBounds().getCenter().toOSMURL());

I think this should say 'than' rather than 'that' and probably WANTED_MAX_AREA_SIZE should be outside the string so we see an actual number.

Regards,
Mike

-----Original Message-----
From: Ticker Berkin [mailto:rwb-mkgmap at jagit.co.uk] 
Sent: 15 February 2017 16:14
To: mkgmap-dev at lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] RE Commit r3801: merge split-shape branch

Hi Gerd / Mike

I've just committed a fix for this to the split-shape branch

Ticker

On Wed, 2017-02-15 at 08:53 +0000, Ticker Berkin wrote:
> Hi Mike
> 
> OK - I think I understand the problem.
> 
> Can you try with --order-by-decreasing-area and let me know if that
> stops the unbound recursion, and, in the meantime, I'll work on it.
> 
> Thanks
> Ticker
> 
> 
> On Wed, 2017-02-15 at 00:48 +0000, Mike Baggaley wrote:
> > HI Ticker, having run this under the debugger now, it appears that
> > it
> > is not the postcode data after all - I was getting confused between
> > the too small to split messages and too many POIs messages. I was
> > previously getting too small to split messages in splitter, but not
> > in mkgmap.
> > 
> > I temporarily changed the code to:
> > 
> > 				} else if (mustSplit) { // can't reduce
> > size, so force more subdivisions
> > 					if (depth < 1000) {
> > 						log.debug("splitting
> > area by contents", area);
> > 						MapArea[] sublist =
> > area.split(1, 1, bounds, true);
> > 						addAreasToList(sublist,
> > alist, depth + 1);
> > 						continue;
> > 					} else {
> > 						log.error("Area too
> > small to split at " + area.getBounds().getCenter().toOSMURL() + "
> > (reduce the density of points, length of lines, etc.)");		
> > 				
> > 					}
> > 				}
> > 
> > This ran successfully and gave a single error:
> >  Area too small to split at 
> > http://www.openstreetmap.org/?mlat=51.797190&mlon=0.918646&zoom=17 
> > (r
> > educe the density of points, length of lines, etc.
> > 
> > Stopping in the debugger, I can see that sizes[MapArea.SHAPE_KIND]
> > is
> > 68076 and from the coordinate it is processing the coastline. The
> > call stack has the same value for the sizes[MapArea.SHAPE_KIND] in
> > every instance of addAreasToList, so it looks like the call to
> > area.split is not actually splitting. Looking at the resulting file
> > in BaseCamp, I can see that the coastline is corrupt.
> > 
> > Hope that helps,
> > Mike
> > 
> > -----Original Message-----
> > From: Ticker Berkin [mailto:rwb-mkgmap at jagit.co.uk] 
> > Sent: 14 February 2017 23:21
> > To: Mike Baggaley <mike at tvage.co.uk>
> > Subject: Re: [mkgmap-dev] RE Commit r3801: merge split-shape branch
> > 
> > Hi Mike
> > 
> > I have the raw postcode data (codePointOpen.zip) which I processes
> > just
> > to take the first few at the same point but I can change it do all
> > of
> > them individually. Do you feed it with a map through the splitter
> > or
> > just make a postcode only overlay? What is the bit of style you use
> > to
> > represent each postcode? I can probably knock up something similar.
> > 
> > The code should cope - I can't see anything obviously wrong so need
> > to
> > try and reproduce the problem
> > 
> > Ticker 
> > 
> > On Tue, 2017-02-14 at 18:17 +0000, Mike Baggaley wrote:
> > > Hi Ticker the data is postcode centres (points). As well as
> > > geographic postcodes, the Royal Mail postcode data includes a
> > > number
> > > of postcodes that are centred on sorting offices (presumably for
> > > box
> > > numbers). There are quite a few at large sorting offices, but if
> > > each
> > > split should be able to accept 255 points, and it is exceeding
> > > 2000
> > > splits before crashing, it looks like the splitting is not
> > > working 
> > > -
> > > there won't be more than 2000 * 255 postcodes at the same place.
> > > My
> > > OSM file is rather large. I can see whether I can extract a map
> > > centred on a sorting office.
> > > 
> > > Regards,
> > > Mike
> > > 
> > > -----Original Message-----
> > > From: Ticker Berkin [mailto:rwb-mkgmap at jagit.co.uk] 
> > > Sent: 14 February 2017 17:52
> > > To: mkgmap-dev at lists.mkgmap.org.uk
> > > Subject: Re: [mkgmap-dev] RE Commit r3801: merge split-shape
> > > branch
> > > 
> > > Hi Mike
> > > 
> > > This is an area of code I've changed and it should be able to
> > > cope
> > > with
> > > many items at the same location without recursing.
> > > 
> > > Do you have some sample data? what are the items
> > > (point/line/shape
> > > extended?)
> > > 
> > > Ticker
> > > 
> > > On Tue, 2017-02-14 at 17:40 +0000, Mike Baggaley wrote:
> > > > Hi Gerd, since this change I am getting a
> > > > java.lang.StackOverflowError crash
> > > > caused by the code recursively attempting to split something
> > > > which
> > > > is
> > > > unsplittable (assuming the split is based on location), as I
> > > > have
> > > > a
> > > > large
> > > > number of points at exactly the same location (from external
> > > > data
> > > > I
> > > > add to
> > > > the OSM data).
> > > > 
> > > > The offending line is at
> > > > uk.me.parabola.mkgmap.build.MapSplitter.addAreasToList(MapSplit
> > > > te
> > > > r.
> > > > ja
> > > > va:187)
> > > > . It is failing on my system with a depth of 2342. I suggest
> > > > there
> > > > needs to
> > > > be a maximum depth after which it should give up trying to
> > > > split.
> > > > 
> > > > 				} else if (mustSplit) { //
> > > > can't reduce
> > > > size, so force more subdivisions
> > > > 					log.debug("splitting
> > > > area by
> > > > contents", area);
> > > > 					MapArea[] sublist =
> > > > area.split(1, 1,
> > > > bounds, true);
> > > > 					addAreasToList(sublist,
> > > > alist,
> > > > depth
> > > > + 1);
> > > > 					continue;
> > > > 				}
> > > > 
> > > > I am unsure whether you would want a fixed depth limit, would
> > > > want
> > > > it
> > > > to be
> > > > set by parameter, would prefer to catch the error or think it
> > > > would
> > > > be
> > > > possible to see whether the proposed area split has made any
> > > > improvement,
> > > > and give up if it hasn't.
> > > > 
> > > > Regards,
> > > > Mike
> > > > 
> > > > 
> > > > _______________________________________________
> > > > mkgmap-dev mailing list
> > > > mkgmap-dev at lists.mkgmap.org.uk
> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > > 
> > > 
> > 
> _______________________________________________
> mkgmap-dev mailing list
> mkgmap-dev at lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev