Pixels from the Edge » Geocoding

Geocoding Part 2 – Import your own Zip Code Spatial Database

Richie — Fri, 24 Apr 2009 10:48:00 +0000

In my post yesterday I talked about calculating geographical distances. During my research I’d come across this http://www.scribd.com/doc/2569355/Geo-Distance-Search-with-MySQL. It was a super interesting read and helped me get a much better understanding of what it is I am even doing. Also it really helps you gets super optimized with the whole subject, but as the locations table I am working with has less than 3000 entries my queries were already coming back super fast in less than 1000th of a second, so it really didn’t need to be optimized more. I didn’t really have time to mess around and investigate it all, and seeing as I still didn’t really fully understand the whole subject too well it was something I’d come back to at another time if need be. However it did point me to geonames.org that has free databases with geographical data to download. This is incredibly useful and whenever I’ve needed this data in the past I’ve had my client buy it (the disk is like $30 so no biggie, but it seems this might even be the free source they are pulling it all from). They have individual country zip codes files available to export, the US file contains exactly what I needed – all the US zip codes with their respective latitude and longitude coordinates. This site has a whole range of useful features on the subject – they appear to even have their own webservice, and a list client libraries in various languages, which pointed me towards a PHP library that appears to be doing a lot of what I need (but also way more than what I need), but by this point I already have everything I needed and had already spent too much time on the subject (not really but I didn’t have as much time allocated to the subject as it really needs – geo-location is a subject unto itself but only a small piece of the website I am building).

So geonames.org offers a free webservice, but I was already using the google API which is also free so I didn’t see any benefit to switch. But they did offer for free all the zip codes with the geographical coordinates so I decided to import this into my database and cut out the 3rd party google Geocode API. It would mean quicker response time and not having to worry about if the service is unavailable or returning errors, I was actually planning to cache the results locally anyway. And perhaps the zip code table might have to be updated on an annual basis to maintain its accuracy (like do zip codes really change that much anyways?), but the web service API is subject to change and it would be less maintainable to monitor it and react to the changes.

So I downloaded the tab delimited US Zip Codes file and imported it into a table in my database called “tbl_Zipcodes_Spatial”, with the following fields (from the included readme):

Country_code : iso country code, 2 characters
Postal_code : varchar(10)
Place_name : varchar(180)
Admin_name1 : 1. order subdivision (state) varchar(100)
Admin_code1 : 1. order subdivision (state) varchar(20)
Admin_name2 : 2. order subdivision (county/province) varchar(100)
Admin_code2 : 2. order subdivision (county/province) varchar(20)
Admin_name3 : 3. order subdivision (community) varchar(100)
Latitude : estimated latitude (wgs84) float(10,6)
Longitude : estimated longitude (wgs84) float(10,6)
Accuracy : accuracy of lat/lng from 1=estimated to 6=centroid

I created an index on the postal_code field and updated my GeoPos:: prepareGetByDistanceSQL() method to get the zip code from the new table. I actually created a new method so that the class still has the ability to pass in the coordinates to the old method when necessary:

And it worked! Barely increased the query execution time, and I’ve completely cut out the google API middle man so the overall execution time of the page is greatly increased. Now I feel like I’ve put together something pretty damn robust (the final touch would be to create a stored procedure in the database, and this is something I’ll come back to at a future point). Behold the final script:

Response->Placemark->Point->coordinates;
		$coord=explode(",",$coord);
		$r->longitude=$coord[0];
		$r->latitude=$coord[1];
		return $r;
	}

	/*
	returns the radius
	*/
	public function rad($v){
		return ($v*M_PI/180);
	}

	/*
	degree,minute,seconds to decimal degrees
	note: this could also be done in a single line using ternary
	*/
	public function dms2deg($D,$M,$S,$dir){
		if(strpos(' WsSs', $dir)>0){
			return(-1 * ($D + ($M + $S/60)/60));
		}else{
			return($D + ($M + $S/60)/60);
		}
	}

	/*
	degree,minute,seconds
	*/
	public function dms($rad) {
		$d = abs($rad * 180 / M_PI);
		$d += 1/7200; // add ½ second for rounding
		$deg = floor($d);
		$min = floor(($d-$deg)*60);
		$sec = floor(($d-$deg-$min/60)*3600);
		// add leading zeros if required
		if ($deg< 100) $deg = '0' + $deg;
		if ($deg< 10) $deg = '0' + $deg;
		if ($min< 10) $min = '0' + $min;
		if ($sec< 10) $sec = '0' + $sec;
		return $deg + '\u00B0' + $min + '\u2032' + $sec + '\u2033';
	}

	/*
	gets the measuring unit
	*/
	public function convertMeasuringUnit($units){
		switch ($units){
			case "miles": $r = 3963.191; break;;
			case "nmiles": $r = 3441.596; break;;
			case "kilo": $r = 6378.137; break;;
		}
		return $r;
	}

	/*
	distance between two points using sherical law of cosines
	cos c = cos a cos b + sin a sin b cos C
	*/
	public function distance($lat1, $lon1, $lat2, $lon2, $units = 'miles'){
		$lat1 = self::rad($lat1); $lon1 = self::rad($lon1);
		$lat2 = self::rad($lat2); $lon2 = self::rad($lon2);
		$r=self::convertMeasuringUnit($units);

		return acos(sin($lat1)*sin($lat2) + cos($lat1)*cos($lat2)*cos($lon2-$lon1)) * $r;
	}

	/*
	initial bearing from point 1 to point 2
	*/
	public function bearing($lat1,$lon1, $lat2, $lon2) {
		$y = sin($lon2-$lon1) * cos($lat2);
		$x = cos($lat1)*sin($lat2) - sin($lat1)*cos($lat2)*cos($lon2-$lon1);
		return atan2($y, $x);
	}

	/*
	find out what the midpoint between two points is
	*/
	public function midpoint($lat1, $lon1, $lat2, $lon2) {
		$lat1 = self::rad($lat1); $lon1 = self::rad($lon1);
		$lat2 = self::rad($lat2); $lon2 = self::rad($lon2);
		$dLon = $lon2 - $lon1;
		$Bx = (cos($lat2) * cos($dLon));
		$By = (cos($lat2) * sin($dLon));
		$lat3 = atan2( sin($lat1) + sin($lat2), sqrt( (cos($lat1)+$Bx) * (cos($lat1)+$Bx) + ($By * $By)) );
		$lon3 = $lon1 + atan2($By, cos($lat1) + $Bx);

		if (!$lat3 || !$lon3) return false;
		return array($lat3 * 180 / M_PI, $lon3 * 180 / M_PI);
	}

	/*
	returns sql to query a table for rows by distance (miles,nmiles,kilo)
	table must contain fields named latitude and longitude
	*/
	public function prepareGetByDistanceSQL($long,$lat,$distance,$table_name,$fields,$units="miles"){
		$cUnit=self::convertMeasuringUnit($units);
		$sql="
			SELECT ".implode($fields,",").",
				acos(SIN( PI()* ".$lat." /180 )*SIN( PI()*latitude/180 ))+(cos(PI()* ".$lat." /180)*COS( PI()*latitude/180) *COS(PI()*longitude/180-PI()* ".$long." /180))* ".$cUnit." AS distance
			FROM ".$table_name."
			WHERE active=1
				AND ".$cUnit." * ACOS( (SIN(PI()* ".$lat." /180)*SIN(PI() * latitude/180)) +
				(COS(PI()* ".$lat." /180)*cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* ".$long." /180))
				) <= ".$distance."
			ORDER BY ".$cUnit." * ACOS(
				(SIN(PI()* ".$lat." /180)*SIN(PI()*latitude/180)) +
				(COS(PI()* ".$lat." /180)*cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* ".$long." /180))
			)
		";
		return $sql;
	}

	/*
	returns sql to query a table for rows by distance (miles,nmiles,kilo)
	table must contain fields named latitude and longitude
	requires you have the tbl_zipcodes_spatial table http://download.geonames.org/export/zip/
	*/
	public function prepareGetByDistanceWithZipcodeSQL($zip_code,$distance,$table_name,$fields,$units="miles"){
		$cUnit=self::convertMeasuringUnit($units);
		$sql="
			SELECT ".$table_name.".".implode($fields,",".$table_name.".").",
				tbl_zipcodes_spatial.Latitude AS Zipcode_Latitude,
				tbl_zipcodes_spatial.Longitude AS Zipcode_Longitude,
				ACOS(SIN( PI()* tbl_zipcodes_spatial.Latitude /180 )*SIN( PI()*".$table_name.".Latitude/180 ))+(COS(PI()* tbl_zipcodes_spatial.Latitude /180)*COS( PI()*".$table_name.".Latitude/180) *COS(PI()*".$table_name.".Longitude/180-PI()* tbl_zipcodes_spatial.Longitude /180))* ".$cUnit." AS Distance
			FROM tbl_zipcodes_spatial,".$table_name."
			WHERE
				tbl_zipcodes_spatial.postal_code='".$zip_code."'
				AND ".$cUnit." * ACOS( (SIN(PI()* tbl_zipcodes_spatial.Latitude /180)*SIN(PI() * ".$table_name.".Latitude/180)) +
				(COS(PI()* tbl_zipcodes_spatial.Latitude /180)*COS(PI()*".$table_name.".Latitude/180)*COS(PI() * ".$table_name.".Longitude/180-PI()* tbl_zipcodes_spatial.Longitude /180))
				) <= ".$distance."
			ORDER BY ".$cUnit." * ACOS(
				(SIN(PI()* tbl_zipcodes_spatial.Latitude /180)*SIN(PI()*".$table_name.".Latitude/180)) +
				(COS(PI()* tbl_zipcodes_spatial.Latitude /180)*COS(PI()*".$table_name.".Latitude/180)*COS(PI() * ".$table_name.".Longitude/180-PI()* tbl_zipcodes_spatial.Longitude /180))
			)
		";
		return $sql;
	}
}
?>

Geocoding and Distancing and Latitude and Longitude and umm Stuff

Richie — Thu, 23 Apr 2009 19:01:00 +0000

Okay this is complicated stuff. Working on putting together a new site for a client, I have a table of locations with longitude and latitude fields for every row (I actually had a tab delimited file, but I sucked it into the db). For the functionality on the web front end a user needs to search by distance, e.g. find all locations within 10 miles of a zip code. Um okay, so how the hell do I do this?

Well first of I need to get the latitude and longitude for a zip code. For that I very quickly found the Google Geocode API and threw together the following:

Response->Placemark->Point->coordinates;
$coord=explode(",",$coord);
$longitude=$coord[1];
$latitude=$coord[0];
?>

It took like 10 minutes, and that included signing up for the API key and reading the documentation, shit this geocoding stuff is easy, I should be done by lunch…

Yeah okay so I have two sets of coordinates, great but errr now what? So I find the Haversine formula, but alas not really being any good at math I don’t understand it, but it doesn’t really matter because I know how to convert it to do the math in PHP. Cool so I got the Haversine function in PHP now but how do I query it against the data in my database? I’d read some stuff about geographical libraries for databases and it seemed super complicated, like the GIS and Spatial extensions for MySQL. So I figure I’ll use a two step approach and do a simple query to pull out a square block of locations and then apply the math. i.e. if I am searching for within a 10 mile radius of a location I’ll do a search on -10 miles and +10 miles, but as the distance is circular I would be getting results in the four corners of that square block that are greater than the 10 miles, and I would then apply the Haversine function and be left only with those results that where truly within the 10 mile radius.

By sheer luck I found this post. It was insane, I didn’t realize you could apply the Haversine formula directly in SQL, I ran that query in my db and bam it worked instantly (well actually there’s a typo syntax error ‘< =’ -blank space between the less than and equals character - in the where clause). Obviously I had to switch out a couple of things so it played nice with my database, but it was instant success, I tried out some different latitude and longitude coordinates and it worked every time:

SELECT Name,Address,Address2,City,State,Zip,Phone,URL,Latitude,Longitude,
acos(SIN( PI()* 42.4584036 /180 )*SIN( PI()*latitude/180 ))+(cos(PI()* 42.4584036 /180)*COS( PI()*latitude/180) *COS(PI()*longitude/180-PI()* -83.1380955 /180))* 3963.191 AS distance
FROM tbl_retailers
WHERE active=1
AND 3963.191 * ACOS( (SIN(PI()* 42.4584036 /180)*SIN(PI() * latitude/180)) +
(COS(PI()* 42.4584036 /180)*cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* -83.1380955 /180))
) <= 100 ORDER BY 3963.191 * ACOS( (SIN(PI()* 42.4584036 /180)*SIN(PI()*latitude/180)) + (COS(PI()* 42.4584036 /180)*cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* -83.1380955 /180)) )

Wow so this was it, everything I needed. Oh wait, except that I needed to display the distance in miles. But the guy who blogged this was some kind of superman and he talks about this on another post. Really I only needed the rad() and distance() functions (because distance is dependent on rad), but I decided that they’ll probably all come in useful one day, so I took them wrapped them in a class called GeoLoc. The class doesn’t need to be instantiated into an object so all methods can be called as follows

Then decided I could abstract in the SQL statement by passing in all necessary variables, and I added it to the class:

I then understood that the measuring unit the query is using is the same as in the distance() method so I abstracted it out into its own method. At which point I noticed that the numbers didn’t match exactly – for miles the SQL had 3963.191 whereas the distance function only had 3963.1. I decided that the SQL query had the more granular number and when I reviewed its original blog post it was across the board for all three measuring units so I updated all the numbers. I then figured I might as well abstract in the Google Geocode zip code lookup.

This is what I was left with (functions I’m not using removed); I give all credit to joeldg:

Response->Placemark->Point->coordinates;
  $coord=explode(",",$coord);
  $r->longitude=$coord[0];
  $r->latitude=$coord[1];
  return $r;
}
function rad($v){
  return ($v*M_PI/180);
}
/*
distance between two points using sherical law of cosines
cos c = cos a cos b + sin a sin b cos C
*/
function distance($lat1, $lon1, $lat2, $lon2, $units = 'miles'){
  $lat1 = self::rad($lat1); $lon1 = self::rad($lon1);
  $lat2 = self::rad($lat2); $lon2 = self::rad($lon2);
  $r=self::convertMeasuringUnit($units);
  return acos(sin($lat1)*sin($lat2) + cos($lat1)*cos($lat2)*cos($lon2-$lon1)) * $r;
}
/*
gets the measuring unit
*/
function convertMeasuringUnit($units){
  switch ($units){
    case "miles": $r = 3963.191; break;;
    case "nmiles": $r = 3441.596; break;;
    case "kilo": $r = 6378.137; break;;
  }
  return $r;
}
/*
returns sql to query a table for rows by distance (miles,nmiles,kilo)
table must contain fields named latitude and longitude
*/
function prepareGetByDistanceSQL($long,$lat,$distance,$table_name,$fields,$units="miles"){
  $cUnit=self::convertMeasuringUnit($units);
  $sql="
    SELECT ".implode($fields,",").",
    acos(SIN( PI()* ".$lat." /180 )*SIN( PI()*latitude/180 ))+(cos(PI()* ".$lat." /180)*COS( PI()*latitude/180) *COS(PI()*longitude/180-PI()* ".$long." /180))* ".$cUnit." AS distance
    FROM ".$table_name."
    WHERE active=1
      AND ".$cUnit." * ACOS( (SIN(PI()* ".$lat." /180)*SIN(PI() * latitude/180)) +
    (COS(PI()* ".$lat." /180)*cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* ".$long." /180))
    ) <= ".$distance."
    ORDER BY ".$cUnit." * ACOS(
    (SIN(PI()* ".$lat." /180)*SIN(PI()*latitude/180)) +
    (COS(PI()* ".$lat." /180)*Cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* ".$long." /180))
   )
  ";
  return $sql;
}
}
?>