Showing posts with label Offline Mapping. Show all posts
Showing posts with label Offline Mapping. Show all posts

13.2.12

Online/Offline Mapping: Finding tile urls based on a geographic extent

This is my second post in a series on developing web mapping applications which also function offline.  My last post covered map tile url schemas for various basemap providers.  While ESRI, Open Street Map, Google, Bing, and Yahoo / Nokia structure their urls differently, they all use the same column (x), row (y), and zoom level (z) values for a indicating tile position within a quad tree.  If you haven't skimmed over that post, or this sounds like Klingon, I recommend checking that post out

The here's the next challenge: take a geographic area of interest, and find urls for all the tiles which covering that area (at zoom levels of interest). 

My area of interest will be the fine City of New Orleans

    '''
    The City of New Orleans
    xmin: -90.283741
    ymin: 29.890626
    xmax: -89.912952
    ymax: 30.057766
    '''

Disclaimer:  Persisting map tiles locally breaks the terms of service for many popular basemap providers like Google, Bing, and ArcGIS Online.  I'm not a lawyer, nor do I aspire to be one, but if you start storing tiles offline without permission, then you do so at your own risk

Helper Methods for Finding Tiles

There are a bunch of functions you will need at your disposal to accomplish this task. These functions can be found in Joe Schwartz's Awesome White Paper on the Bing Maps Tiling System.  This is one of the best explanations of map tile systems I've found on the web.  Joe goes over the mercator projection, along with explanations of ground resolution, map scale, and pixel coordinates. Really really great stuff and I urge everybody to really dig into it.

At the end of the article, Joe provides sample code which contains the major functions you will need to dominate map tile systems.  While I've grown to really like C# development, I almost always translate interesting code back into warm and friendly Python.   Here's Joe's TileSystem class translated into Python with a couple extra methods for additional abstraction:

'''
Mostly direct port of awesome article by Joe Schwartz - http://msdn.microsoft.com/en-us/library/bb259689.aspx
'''
from __future__ import division
import math

class TileUtils(object):

    earth_radius = 6378137
    min_lat = -85.05112878
    max_lat = 85.05112878
    min_lng = -180
    max_lng = 180

    def clipValue(self, value, minValue, maxValue):
        '''
        Makes sure that value is within a specific range.
        If not, then the lower or upper bounds is returned
        '''
        return min(max(value, minValue), maxValue)

    def getMapDimensionsByZoomLevel(self, zoomLevel):
        '''
        Returns the width/height in pixels of the entire map
        based on the zoom level.
        '''
        return 256 << zoomLevel

    def getGroundResolution(self, latitude, level):
        '''
        returns the ground resolution for based on latitude and zoom level.
        '''
        latitude = self.clipValue(latitude, self.min_lat, self.max_lat);
        mapSize = self.getMapDimensionsByZoomLevel(level)
        return math.cos(latitude * math.pi / 180) * 2 * math.pi * self.earth_radius / mapSize

    def getMapScale(self, latitude, level, dpi=96):
        '''
        returns the map scale on the dpi of the screen
        '''
        dpm = dpi / 0.0254 #convert to dots per meter
        return self.getGroundResolution(latitude, level) * dpm

    def convertLatLngToPixelXY(self, lat, lng, level):
        '''
        returns the x and y values of the pixel corresponding to a latitude and longitude.
        '''
        mapSize = self.getMapDimensionsByZoomLevel(level)
       
        lat = self.clipValue(lat, self.min_lat, self.max_lat)
        lng = self.clipValue(lng, self.min_lng, self.max_lng)

        x = (lng + 180) / 360
        sinlat = math.sin(lat * math.pi / 180)
        y = 0.5 - math.log((1 + sinlat) / (1 - sinlat)) / (4 * math.pi)

        pixelX = int(self.clipValue(x * mapSize + 0.5, 0, mapSize - 1))
        pixelY = int(self.clipValue(y * mapSize + 0.5, 0, mapSize - 1))
        return (pixelX, pixelY)

    def convertPixelXYToLngLat(self, pixelX, pixelY, level):
        '''
        converts a pixel x, y to a latitude and longitude.
        '''
        mapSize = self.getMapDimensionsByZoomLevel(level)
        x = (self.clipValue(pixelX, 0, mapSize - 1) / mapSize) - 0.5
        y = 0.5 - (self.clipValue(pixelY, 0, mapSize - 1) / mapSize)

        lat = 90 - 360 * math.atan(math.exp(-y * 2 * math.pi)) / math.pi
        lng = 360 * x

        return (lng, lat)

    def convertPixelXYToTileXY(self, pixelX, pixelY):
        '''
        Converts pixel XY coordinates into tile XY coordinates of the tile containing
        '''
        return(int(pixelX / 256), int(pixelY / 256))

    def convertTileXYToPixelXY(self, tileX, tileY):
        '''
        Converts tile XY coordinates into pixel XY coordinates of the upper-left pixel
        '''
        return(tileX * 256, tileY * 256)

    def tileXYZToQuadKey(self, x, y, z):
        '''
        Computes quadKey value based on tile x, y and z values.
        '''
        quadKey = ''
        for i in range(z, 0, -1):
            digit = 0
            mask = 1 << (i - 1)
            if(x & mask) != 0:
                digit += 1
            if(y & mask) != 0:
                digit += 2
            quadKey += str(digit)
        return quadKey

    def quadKeyToTileXYZ(self, quadKey):
        '''
        Computes tile x, y and z values based on quadKey.
        '''
        tileX = 0
        tileY = 0
        tileZ = len(quadKey)

        for i in range(tileZ, 0, -1):
            mask = 1 << (i - 1)
            value = quadKey[tileZ - i]

            if value == '0':
                continue

            elif value == '1':
                tileX |= mask

            elif value == '2':
                tileY |= mask

            elif value == '3':
                tileX |= mask
                tileY |= mask

            else:
                raise Exception('Invalid QuadKey')

        return (tileX, tileY, tileZ)

    def convertLngLatToTileXY(self, lng, lat, level):
        pixelX, pixelY = self.convertLatLngToPixelXY(lat, lng, level)
        return self.convertPixelXYToTileXY(pixelX, pixelY)

    def getTileOrigin(self, tileX, tileY, level):
        '''
        Returns the upper-left hand corner lat/lng for a tile
        '''
        pixelX, pixelY = self.convertTileXYToPixelXY(tileX, tileY)
        lng, lat = self.convertPixelXYToLngLat(pixelX, pixelY, level)
        return (lat, lng)

The function convertLngLatToTileXY really sums up the class for our purposes. This function will help get the tile x/y values for the sides of the bounding box.  Using these functions, we can write an additional helper class to generate all the tile urls within our bounding box based off of a template tile url like the ones mentioned in my last post:

class TileFinder(object):

    def __init__(self, tileUtils, tileTemplate):
        '''
        Tile template is a template url which use {{x}}, {{y}}, {{z}} which will be replaced by
        their respective values.
        '''
        self.tileUtils = tileUtils
        self.tileTemplate = tileTemplate
    
    def getTileUrlsByLatLngExtent(self, xmin, ymin, xmax, ymax, level):
        '''
        Returns a list of tile urls by extent
        '''
        #Upper-Left Tile
        tileXMin, tileYMin = self.tileUtils.convertLngLatToTileXY(xmin, ymax, level)

        #Lower-Right Tile
        tileXMax, tileYMax = self.tileUtils.convertLngLatToTileXY(xmax, ymin, level)

        tileUrls = []

        for y in range(tileYMax, tileYMin - 1, -1):
            for x in range(tileXMin, tileXMax + 1, 1):
                tileUrls.append(self.createTileUrl(x, y, level)) 

        return tileUrls

    def createTileUrl(self, x, y, z):
        '''
        returns new tile url based on template
        '''
        return self.tileTemplate.replace('{{x}}', str(x)).replace('{{y}}', str(y)).replace('{{z}}', str(z))

Check out the constructor here.  It has two required arguments, one of which is a tile template.  This tile template uses {{x}}, {{y}}, and {{z}} values to indicate where substitutions should take place.  These substitutions happen in createTileUrl which is called for each x,y pair generated in getTileUrlsByLatLngExtent.

So now we have our helper classes for generating a list of tile urls based on an extent.  The last value to complete the trifecta is the zoom level.  I chose level 11 since my area of interest is fairly small (city level).  Here is a snippet for running the code:

    '''
    The City of New Orleans
    xmin: -90.283741
    ymin: 29.890626
    xmax: -89.912952
    ymax: 30.057766
    '''

    import urllib2
    
    #constructing the template url for arcgis online
    template = 'http://services.arcgisonline.com/ArcGIS/rest/services/World_Topo_Map/MapServer/tile/{{z}}/{{y}}/{{x}}.png'

    tileUtils = TileUtils()
    tileFinder = TileFinder(tileUtils, template)
    tile_urls = tileFinder.getTileUrlsByLatLngExtent(-90.283741, 29.890626, -89.912952, 30.057766, 11)

    for t in tile_urls: urllib2.urlopen(t).read()

So that's about all there is to it.  Below you can see the six tile urls that were produced in the code snippet above.  A special thanks to Stephen Ansari (@sansari22) who really got me started on learning tile schemas and provided me sample code to get going.

29.1.12

Online/Offline Mapping: Map Tiles and their URL Schemas

I've recently been developing online/offline mapping applications for use in places which simply don't have access to internet connections.  I'm currently preparing a presentation discussing topics related to bring online application to offline users, and in doing so will be blogging on related topics.  The first part of the series will be dedicated to working with tile map services.

One of the main challenges of bringing an online map to an offline environment is maintaining access to map tiles. From imagery to streets, tiled map services provide the necessary geographic context and semantic detail to visualize additional spatial layers.  While hard-drive space is still a major limiting factor in ability to store tiles, a significant number of tiles can be stored locally to allow use of an application for limited geographic area.

Disclaimer:  Persisting map tiles locally breaks the terms of service for many popular basemap providers like Google, Bing, and ArcGIS Online.  I'm not a lawyer, nor do I aspire to be one, but if you start storing tiles offline without permission, then you do so at your own risk.  With that said, let's look at how to get started in storing map tiles for offline use.


Map Tiles and their URL Schemas

If you want to save down map tiles, you need to first know the address of each tile.  Map tile addresses take the form of urls (uniform address locators) which web maps access via HTTP GET requests.  Map tile service providers structure their map tile urls differently, but they all use column (x), row (y), and zoom level (z) values somewhere in the url.  Most providers have settled on using the Web Mercator projection (sad but true), and most have used the same origin for tiling and tile dimensions which means that the zoom level, row and column ids are the same across providers.  Dimensions are almost all 256 by 256 pixels and are in png or jpeg format.

Below I have listed requests for a single tile covering the same geographic area (Pacific Ocean / California) for several basemap tile providers including: Open Street Map, ArcGIS Online, Google Maps, Bing Maps, Yahoo! Maps and Nokia.  Here are the details for the tile which is common to ALL providers and demonstrates how easy it can be to locate a tile for any of the providers below:

/*
Tile Details:
COLUMN (X): 2
ROW (Y): 6
ZOOM LEVEL (Z): 4
DIMENSIONS: 256 x 256
*/


Open Street Map:

O.S.M. uses a simple directory structure for storing its map tiles which we will see repeated in many of the other service providers.  While you may think that "Open" means you can freely scrape as many tiles as you want...remember that OSM is hosted by volunteers with limited server resources.  If you want to scrape a large number of tiles, check out hosting your own OSM tile server.  I used Michal Migurski's TileDrawer to quickly setup my own OSM EC2 instance.  Tile Drawer is great and doesn't even require you to SSH into the machine!  Just start it up and start scraping.



/*
Open Street Map Template:
http://{SERVER}/{ZOOM_LEVEL}/{COLUMN}/{ROW}.png

Open Street Map Example:
http://c.tile.openstreetmap.org/4/2/6.png
*/

ArcGIS Server Map Service:

ArcGIS Server is the most powerful, out-of-the-box mapping server currently available.  AGS uses a similiar tile directory structure as OSM, but reverses the column and row values in the url.  ESRI is very gracious with providing access to tons of free tile services.


/*
ArcGIS Server Tiled Map Service Template:
http://{SERVER}/tile/{ZOOM_LEVEL}/{ROW}/{COLUMN}.png

ArcGIS Server Tiled Map Service Example:
http://services.arcgisonline.com/ArcGIS/rest/services/World_Topo_Map/MapServer/tile/4/6/2.png
*/

Google Maps

Google uses a slightly different structure which involves url parameters.  They also include what appears to be a locale parameter which I'm guessing is used to change the language of label (although I have not yet confirmed).


/*
Google Maps Template:
http://{SERVER}/vt/lyrs={layer_id}&hl={locale}&x={COLUMN}&y={ROW}&z={ZOOM_LEVEL}

Google Maps Example: 
http://mt0.google.com/vt/lyrs=m@169000000&hl=en&x=2&y=6&z=4&s=Ga
*/


Bing Map (Formerly Microsoft Virtual Earth)

Bing has an interesting way to access tile which is using a quadkey.  A "quadkey" is basically the position of the tile within the "quadtree".  The real reason why Web Mercator become so successful is that it turns the earth into a square which can be recursively tiled by spliting tiles into 4 to obtain tiles for the next zoom level.  I will be covering how to generate quadkeys in coming posts, but for now, here is a python snippet which I used to get the tile below:

def tileXYZToQuadKey(x, y, z):
    quadKey = ''
    for i in range(z, 0, -1):
        digit = 0
        mask = 1 << (i - 1)
        if(x & mask) != 0:
            digit += 1
        if(y & mask) != 0:
            digit += 2
        quadKey += str(digit)
    return quadKey
>>> print tileXYZToQuadKey(2, 6, 4)
0230
>>> 

I wrote the function above based on some C# that was provided in the bing maps tiling specification which you can read more about here: Bing Maps tile specification


/*
Bing Maps Tile Template:
http://{SERVER}/tiles/a{QUAD_KEY}.jpeg?token={TOKEN}&mkt={LOCALE}&g={??}

Bing Maps Tile Example:
http://t0.tiles.virtualearth.net/tiles/a0230.jpeg?g=854&mkt=en-US&token=Anz84uRE1RULeLwuJ0qKu5amcu5rugRXy1vKc27wUaKVyIv1SVZrUjqaOfXJJoI0

Awesome Explanation: http://msdn.microsoft.com/en-us/library/bb259689.aspx
*/



Ovi Maps (Yahoo! / Nokia)
Yahoo! and Nokia use an API called Ovi Maps which I don't know much about (but I'm guessing Ovi was acquired by Nokia).  The tile structure is straight forward.


/*
Ovi Maps (Nokia and Yahoo! Maps) Template:
http://{SERVER}/maptiler/v2/maptile/???/{STYLE}/{ZOOM_LEVEL}/{COLUMN}/{ROW}/{SIZE}/{IMG_FORMAT}?token={TOKEN}&lg={LOCALE}

Ovi Maps (Nokia and Yahoo! Maps) Example:
http://4.maptile.lbs.ovi.com/maptiler/v2/maptile/279af375be/normal.day/4/2/6/256/png8?lg=ENG&token=TrLJuXVK62IQk0vuXFzaig%3D%3D&requestid=yahoo.prod&app_id=eAdkWGYRoc4RfxVo0Z4B
*/

Yandex Maps


Yandex has really nice maps and great to see soCal in cyrillic.




/*
Yandex Maps Template:
http://vec04.maps.yandex.net/tiles?l=map&v=2.26.0&x={{X}}&y={{y}}&z={{z}}&lang=ru-RU

Yandex Maps Example:
http://vec04.maps.yandex.net/tiles?l=map&v=2.26.0&x=2&y=6&z=4&lang=ru-RU
*/



Well that about concludes my post on map tile url schemas.  Look for my upcoming blog post where I will cover how to generate a list of tile x,y, z values for a specified zoom level based on a given extent.