I had a great time out in Palm Springs this year. Saw some great presentations and will be definitely going back over the videos. Looks like ESRI has released many of them in their re-vamped video site.
I presented this year on converting an existing Flex application to AIR and running it offline:
Here's a link to my presentation.
Showing posts with label ArcGIS Server. Show all posts
Showing posts with label ArcGIS Server. Show all posts
29.1.12
Online/Offline Mapping: Map Tiles and their URL Schemas
I've recently been developing online/offline mapping applications for use in places which simply don't have access to internet connections. I'm currently preparing a presentation discussing topics related to bring online application to offline users, and in doing so will be blogging on related topics. The first part of the series will be dedicated to working with tile map services.
One of the main challenges of bringing an online map to an offline environment is maintaining access to map tiles. From imagery to streets, tiled map services provide the necessary geographic context and semantic detail to visualize additional spatial layers. While hard-drive space is still a major limiting factor in ability to store tiles, a significant number of tiles can be stored locally to allow use of an application for limited geographic area.
Disclaimer: Persisting map tiles locally breaks the terms of service for many popular basemap providers like Google, Bing, and ArcGIS Online. I'm not a lawyer, nor do I aspire to be one, but if you start storing tiles offline without permission, then you do so at your own risk. With that said, let's look at how to get started in storing map tiles for offline use.
Map Tiles and their URL Schemas
If you want to save down map tiles, you need to first know the address of each tile. Map tile addresses take the form of urls (uniform address locators) which web maps access via HTTP GET requests. Map tile service providers structure their map tile urls differently, but they all use column (x), row (y), and zoom level (z) values somewhere in the url. Most providers have settled on using the Web Mercator projection (sad but true), and most have used the same origin for tiling and tile dimensions which means that the zoom level, row and column ids are the same across providers. Dimensions are almost all 256 by 256 pixels and are in png or jpeg format.
Below I have listed requests for a single tile covering the same geographic area (Pacific Ocean / California) for several basemap tile providers including: Open Street Map, ArcGIS Online, Google Maps, Bing Maps, Yahoo! Maps and Nokia. Here are the details for the tile which is common to ALL providers and demonstrates how easy it can be to locate a tile for any of the providers below:
Open Street Map:
O.S.M. uses a simple directory structure for storing its map tiles which we will see repeated in many of the other service providers. While you may think that "Open" means you can freely scrape as many tiles as you want...remember that OSM is hosted by volunteers with limited server resources. If you want to scrape a large number of tiles, check out hosting your own OSM tile server. I used Michal Migurski's TileDrawer to quickly setup my own OSM EC2 instance. Tile Drawer is great and doesn't even require you to SSH into the machine! Just start it up and start scraping.
ArcGIS Server Map Service:
ArcGIS Server is the most powerful, out-of-the-box mapping server currently available. AGS uses a similiar tile directory structure as OSM, but reverses the column and row values in the url. ESRI is very gracious with providing access to tons of free tile services.
Google Maps
Google uses a slightly different structure which involves url parameters. They also include what appears to be a locale parameter which I'm guessing is used to change the language of label (although I have not yet confirmed).
Bing Map (Formerly Microsoft Virtual Earth)
Bing has an interesting way to access tile which is using a quadkey. A "quadkey" is basically the position of the tile within the "quadtree". The real reason why Web Mercator become so successful is that it turns the earth into a square which can be recursively tiled by spliting tiles into 4 to obtain tiles for the next zoom level. I will be covering how to generate quadkeys in coming posts, but for now, here is a python snippet which I used to get the tile below:
I wrote the function above based on some C# that was provided in the bing maps tiling specification which you can read more about here: Bing Maps tile specification.
Ovi Maps (Yahoo! / Nokia)
Yahoo! and Nokia use an API called Ovi Maps which I don't know much about (but I'm guessing Ovi was acquired by Nokia). The tile structure is straight forward.
Yandex Maps
Yandex has really nice maps and great to see soCal in cyrillic.
Well that about concludes my post on map tile url schemas. Look for my upcoming blog post where I will cover how to generate a list of tile x,y, z values for a specified zoom level based on a given extent.
One of the main challenges of bringing an online map to an offline environment is maintaining access to map tiles. From imagery to streets, tiled map services provide the necessary geographic context and semantic detail to visualize additional spatial layers. While hard-drive space is still a major limiting factor in ability to store tiles, a significant number of tiles can be stored locally to allow use of an application for limited geographic area.
Disclaimer: Persisting map tiles locally breaks the terms of service for many popular basemap providers like Google, Bing, and ArcGIS Online. I'm not a lawyer, nor do I aspire to be one, but if you start storing tiles offline without permission, then you do so at your own risk. With that said, let's look at how to get started in storing map tiles for offline use.
Map Tiles and their URL Schemas
If you want to save down map tiles, you need to first know the address of each tile. Map tile addresses take the form of urls (uniform address locators) which web maps access via HTTP GET requests. Map tile service providers structure their map tile urls differently, but they all use column (x), row (y), and zoom level (z) values somewhere in the url. Most providers have settled on using the Web Mercator projection (sad but true), and most have used the same origin for tiling and tile dimensions which means that the zoom level, row and column ids are the same across providers. Dimensions are almost all 256 by 256 pixels and are in png or jpeg format.
Below I have listed requests for a single tile covering the same geographic area (Pacific Ocean / California) for several basemap tile providers including: Open Street Map, ArcGIS Online, Google Maps, Bing Maps, Yahoo! Maps and Nokia. Here are the details for the tile which is common to ALL providers and demonstrates how easy it can be to locate a tile for any of the providers below:
/*
Tile Details:
COLUMN (X): 2
ROW (Y): 6
ZOOM LEVEL (Z): 4
DIMENSIONS: 256 x 256
*/
Open Street Map:
O.S.M. uses a simple directory structure for storing its map tiles which we will see repeated in many of the other service providers. While you may think that "Open" means you can freely scrape as many tiles as you want...remember that OSM is hosted by volunteers with limited server resources. If you want to scrape a large number of tiles, check out hosting your own OSM tile server. I used Michal Migurski's TileDrawer to quickly setup my own OSM EC2 instance. Tile Drawer is great and doesn't even require you to SSH into the machine! Just start it up and start scraping.
/*
Open Street Map Template:
http://{SERVER}/{ZOOM_LEVEL}/{COLUMN}/{ROW}.png
Open Street Map Example:
http://c.tile.openstreetmap.org/4/2/6.png
*/
ArcGIS Server Map Service:
ArcGIS Server is the most powerful, out-of-the-box mapping server currently available. AGS uses a similiar tile directory structure as OSM, but reverses the column and row values in the url. ESRI is very gracious with providing access to tons of free tile services.
/*
ArcGIS Server Tiled Map Service Template:
http://{SERVER}/tile/{ZOOM_LEVEL}/{ROW}/{COLUMN}.png
ArcGIS Server Tiled Map Service Example:
http://services.arcgisonline.com/ArcGIS/rest/services/World_Topo_Map/MapServer/tile/4/6/2.png
*/
Google Maps
Google uses a slightly different structure which involves url parameters. They also include what appears to be a locale parameter which I'm guessing is used to change the language of label (although I have not yet confirmed).
/*
Google Maps Template:
http://{SERVER}/vt/lyrs={layer_id}&hl={locale}&x={COLUMN}&y={ROW}&z={ZOOM_LEVEL}
Google Maps Example:
http://mt0.google.com/vt/lyrs=m@169000000&hl=en&x=2&y=6&z=4&s=Ga
*/
Bing Map (Formerly Microsoft Virtual Earth)
Bing has an interesting way to access tile which is using a quadkey. A "quadkey" is basically the position of the tile within the "quadtree". The real reason why Web Mercator become so successful is that it turns the earth into a square which can be recursively tiled by spliting tiles into 4 to obtain tiles for the next zoom level. I will be covering how to generate quadkeys in coming posts, but for now, here is a python snippet which I used to get the tile below:
def tileXYZToQuadKey(x, y, z):
quadKey = ''
for i in range(z, 0, -1):
digit = 0
mask = 1 << (i - 1)
if(x & mask) != 0:
digit += 1
if(y & mask) != 0:
digit += 2
quadKey += str(digit)
return quadKey
>>> print tileXYZToQuadKey(2, 6, 4)
0230
>>>
I wrote the function above based on some C# that was provided in the bing maps tiling specification which you can read more about here: Bing Maps tile specification.
/*
Bing Maps Tile Template:
http://{SERVER}/tiles/a{QUAD_KEY}.jpeg?token={TOKEN}&mkt={LOCALE}&g={??}
Bing Maps Tile Example:
http://t0.tiles.virtualearth.net/tiles/a0230.jpeg?g=854&mkt=en-US&token=Anz84uRE1RULeLwuJ0qKu5amcu5rugRXy1vKc27wUaKVyIv1SVZrUjqaOfXJJoI0
Awesome Explanation: http://msdn.microsoft.com/en-us/library/bb259689.aspx
*/
Ovi Maps (Yahoo! / Nokia)
Yahoo! and Nokia use an API called Ovi Maps which I don't know much about (but I'm guessing Ovi was acquired by Nokia). The tile structure is straight forward.
/*
Ovi Maps (Nokia and Yahoo! Maps) Template:
http://{SERVER}/maptiler/v2/maptile/???/{STYLE}/{ZOOM_LEVEL}/{COLUMN}/{ROW}/{SIZE}/{IMG_FORMAT}?token={TOKEN}&lg={LOCALE}
Ovi Maps (Nokia and Yahoo! Maps) Example:
http://4.maptile.lbs.ovi.com/maptiler/v2/maptile/279af375be/normal.day/4/2/6/256/png8?lg=ENG&token=TrLJuXVK62IQk0vuXFzaig%3D%3D&requestid=yahoo.prod&app_id=eAdkWGYRoc4RfxVo0Z4B
*/
Yandex Maps
Yandex has really nice maps and great to see soCal in cyrillic.
/*
Yandex Maps Template:
http://vec04.maps.yandex.net/tiles?l=map&v=2.26.0&x={{X}}&y={{y}}&z={{z}}&lang=ru-RU
Yandex Maps Example:
http://vec04.maps.yandex.net/tiles?l=map&v=2.26.0&x=2&y=6&z=4&lang=ru-RU
*/
Well that about concludes my post on map tile url schemas. Look for my upcoming blog post where I will cover how to generate a list of tile x,y, z values for a specified zoom level based on a given extent.
3.8.11
Cannot Log into REST Admin (AGS 10)
I was dealing with an annoying issue today involving logging into the REST admin area of ArcGIS Server 10. Basically I would log in, nothing would happen and no error message would be displayed.
For anybody having this issue, it turns out it was related to the domain of the user's account. Basically if the user account is not local, but instead on a different domain, they will not be able to log into the ArcGIS Server 10 REST admin interface. This is a known bug #NIM059442.
Upgrade to the latest service pack and it should fix the problem.
For anybody having this issue, it turns out it was related to the domain of the user's account. Basically if the user account is not local, but instead on a different domain, they will not be able to log into the ArcGIS Server 10 REST admin interface. This is a known bug #NIM059442.
Upgrade to the latest service pack and it should fix the problem.
18.7.11
Setting Feature Layer maxAllowableOffset property
Derek Swingley from ESRI recently enlightened me on the use of feature layers for the ArcGIS APIs for Flex, JavaScript, and Silverlight.
Max Allowable Offset basically determines the level of geometry generalization when returned from the mapping server. The larger the offset, the greater the simplification. To calculate the appropriate maxAllowableOffset, I am simply using the ground resolution of the current zoom level of the map (AS3):
If you are using a WGS84 Basemap, you can just divide the result above by the number of meters per degree:
These functions make it so that at the equator there should only be one vertex per pixel. I've been wanting a little more stylized features, so I've been adding about 35 - 40% additional offset:
For more information on feature layers, check out Derek's blog posts
Max Allowable Offset basically determines the level of geometry generalization when returned from the mapping server. The larger the offset, the greater the simplification. To calculate the appropriate maxAllowableOffset, I am simply using the ground resolution of the current zoom level of the map (AS3):
private const EARTH_CIRCUM:Number = 2 * 3.14159265 * 6378137;
private function computeAllowableOffset():void
{
fLayer.maxAllowableOffset = (EARTH_CIRCUM / (256 * Math.pow(2, oMap.level)));
}
If you are using a WGS84 Basemap, you can just divide the result above by the number of meters per degree:
private const EARTH_CIRCUM:Number = 2 * 3.14159265 * 6378137;
private function computeAllowableOffset():void
{
fLayer.maxAllowableOffset = (EARTH_CIRCUM / (256 * Math.pow(2, oMap.level))) / (EARTH_CIRCUM / 360);
}
These functions make it so that at the equator there should only be one vertex per pixel. I've been wanting a little more stylized features, so I've been adding about 35 - 40% additional offset:
private const EARTH_CIRCUM:Number = 2 * 3.14159265 * 6378137;
private function computeAllowableOffset():void
{
fLayer.maxAllowableOffset = (EARTH_CIRCUM / (256 * Math.pow(2, oMap.level))) * 1.35;
}
For more information on feature layers, check out Derek's blog posts
2.7.11
Great REST-based Server Object Extension posts from NicoGIS
NicoGIS is an awesome GIS blog. Over the past year, I've been learning REST-based Server Object Extensions for ArcGIS Server 10. NicoGIS has some one of the best posts on the subject. I ended up reading the post in Italian for fun, but I bet Google can translate it.
26.6.11
Python Geoprocessing Vs. C# Server Object Extensions (AGS 10)
The following deals with development of geoprocessing services specifically for ArcGIS Server 10
Introduction Video I made for Penn State's MGIS Program
Over the past weeks, I have written several server-side tools to support web mapping applications as part of my capstone project for Penn State's MGIS Program. The following is a general overview of what I've found to be the strengths and weaknesses of Geoprocessing Services in Python and Server Object Extensions in C#.
Over the coming weeks, I will be releasing the code and discussions of the individual tools in separate posts.
Geoprocessing Services (Python)
For developers of web-based geoprocessing tools for ArcGIS Server 10, the arcpy Python library is now a popular alternative to the more complex ArcObjects library. With arcpy, you can access a decent percent of the ArcGIS tools with a clear syntax for linking them together. Knowledge of object-oriented programming is not required, and you can integrate GIS with an entire world of Python third-party libraries. To web-enable arcpy, scripts are added to toolboxes which get published to the server.
In spite of their ease of development and clear syntax, aspects of python geoprocessing services limit their power. First is that they execute slower that the equivalent code written in .NET ArcObjects. For desktop tools that may not be a problem, but for the web this can negatively impacts user experience.
Even if speed was not an issue, geoprocessing services are more cumbersome to move from one environment to another (e.g. staging -> production) because access to required source data is more tightly coupled with the tool. Take the example of a server-side point clusterer. To access the point feature class containing un-clustered points, a python script may use either of the following logic:
This first example uses a hard-coded absolute path that will need to be changed if the tool moves to an environment with a different directory structure. A better choice would be the relative path example, but there is still hard-coding of the dataset's geodatabase and name. With either choice, the tool suffers from a lack of true encapsulation which ends up making it harder to move and reuse on other point layers.
An aspect of the ArcGIS workflow for creating scripting tools which hinders a tool's flexibility is the need for explicit registration of input and output parameters in a properties dialog.
Python Cons
C# Server Object Extensions (ArcObjects 10 .NET SDK)
The idea of the Server Object Extension is that instead of publishing a service on top of the server, you extend the server itself. This extended functionality can then be enabled for any service along with other out-of-the-box capabilities (e.g. Export Map, Identify, Find, Query, Generate KML, WMS).
The beautiful thing about Server Object Extensions is that ArcObjects has hooks into the source data within a map service. These hooks give C# access to datasets using map service layer indexes instead of directory paths allowing for looser coupling with the server environment. ESRI Web APIs also use map service layer indexes which makes client-server communication easier.
C# Cons
An example of this would be getting a geometry object, passed in as JSON from the client, into a feature class which can be used by a Spatial Analyst tool. The solution I used took 80 lines of not fun code to write. The image below is meant to give an impression of the complexity.
Another area of trouble is .NET and inability to run outside of a Window's environment. Being that ESRI offer's a Java version of the ArcGIS Server, choosing C# means limiting a tool to Window's and the .NET version of ArcGIS Server. This hasn't been a major issue for me yet, but I want as many options in hosting environments as possible.
Overall I have found the benefits of Server Object Extensions outweigh the ease of development offered by Python when performance and reusablilty are important. Initially, I found the complexity of working in ArcObjects discouraging, but slowly I wrote function to deal this data type conversions and development got easier. I now consider arcpy as a solution only on the desktop.
Introduction Video I made for Penn State's MGIS Program
Over the past weeks, I have written several server-side tools to support web mapping applications as part of my capstone project for Penn State's MGIS Program. The following is a general overview of what I've found to be the strengths and weaknesses of Geoprocessing Services in Python and Server Object Extensions in C#.
Over the coming weeks, I will be releasing the code and discussions of the individual tools in separate posts.
Geoprocessing Services (Python)
For developers of web-based geoprocessing tools for ArcGIS Server 10, the arcpy Python library is now a popular alternative to the more complex ArcObjects library. With arcpy, you can access a decent percent of the ArcGIS tools with a clear syntax for linking them together. Knowledge of object-oriented programming is not required, and you can integrate GIS with an entire world of Python third-party libraries. To web-enable arcpy, scripts are added to toolboxes which get published to the server.
Python Pros
- Accelerated development time
- ArcGIS includes many Python code samples
- Easy to debug business logic
In spite of their ease of development and clear syntax, aspects of python geoprocessing services limit their power. First is that they execute slower that the equivalent code written in .NET ArcObjects. For desktop tools that may not be a problem, but for the web this can negatively impacts user experience.
Even if speed was not an issue, geoprocessing services are more cumbersome to move from one environment to another (e.g. staging -> production) because access to required source data is more tightly coupled with the tool. Take the example of a server-side point clusterer. To access the point feature class containing un-clustered points, a python script may use either of the following logic:
This first example uses a hard-coded absolute path that will need to be changed if the tool moves to an environment with a different directory structure. A better choice would be the relative path example, but there is still hard-coding of the dataset's geodatabase and name. With either choice, the tool suffers from a lack of true encapsulation which ends up making it harder to move and reuse on other point layers.
An aspect of the ArcGIS workflow for creating scripting tools which hinders a tool's flexibility is the need for explicit registration of input and output parameters in a properties dialog.
Python Cons
- Slower execution speed when compared to C#
- Tighter coupling with source data
- Double managing of service input/output parameters
C# Server Object Extensions (ArcObjects 10 .NET SDK)
C# Pros
- Faster code execution
- Loose coupling with tool source data
- Easy to deploy across environments
The idea of the Server Object Extension is that instead of publishing a service on top of the server, you extend the server itself. This extended functionality can then be enabled for any service along with other out-of-the-box capabilities (e.g. Export Map, Identify, Find, Query, Generate KML, WMS).
The beautiful thing about Server Object Extensions is that ArcObjects has hooks into the source data within a map service. These hooks give C# access to datasets using map service layer indexes instead of directory paths allowing for looser coupling with the server environment. ESRI Web APIs also use map service layer indexes which makes client-server communication easier.
C# Cons
- Large and complex library
- Not Cross-Platform (won't run on AGS Java version).
- Steeper learning curve
An example of this would be getting a geometry object, passed in as JSON from the client, into a feature class which can be used by a Spatial Analyst tool. The solution I used took 80 lines of not fun code to write. The image below is meant to give an impression of the complexity.
Another area of trouble is .NET and inability to run outside of a Window's environment. Being that ESRI offer's a Java version of the ArcGIS Server, choosing C# means limiting a tool to Window's and the .NET version of ArcGIS Server. This hasn't been a major issue for me yet, but I want as many options in hosting environments as possible.
Overall I have found the benefits of Server Object Extensions outweigh the ease of development offered by Python when performance and reusablilty are important. Initially, I found the complexity of working in ArcObjects discouraging, but slowly I wrote function to deal this data type conversions and development got easier. I now consider arcpy as a solution only on the desktop.
13.9.10
Find Watershed SOE
The Find Watershed Server Object Extension (SOE) plugs into ArcGIS Server 10 and adds functionality for generating watersheds over REST. The SOE takes two parameters: hydroshed id and a pour point location. Proper flow accumulation and flow direction rasters are retrieved by hydroshed id from a file geodatabase. The location parameter is the pour point. The goal of the tool is to generate a polygon representing the area of water which flows into a point.
I like the motto, "if you can't do in on desktop, you won't be able to do it on server." Keeping this idea in mind, I first wrote the tool as a add-in for ArcMap. This allowed me to debug core business logic for the tool without the complications of the server.
The main difference between a 'desktop tool' and Server Object Extension is the ability to deserialize requests and serialize responses over http. Since I was new to C# and Server Object Extensions, I first published a sample SOE called SimpleRESTSOE from the samples folder of the ArcGIS 10 install directory. This tool was simply an echo service (i.e. pass a string in, and get the string back). Requests declare 'text' and 'f' (output format) parameters as part of a url request:
/echo?text=helloWorld&f=json
and they SOE responds:
{"text":"helloWorld"}
I got the sample SOE registered with Windows using the command:
RegAsm SimpleRESTSOE.dll /codebase
I then registered the SOE with ArcCatalog and the Server Manager using the companion file located in the samples folder (RegisterSimpleRESTSOE.dll).
Once I verified the sample SOE was running, I simply started modifing the rest schema to take my custom parameters. To do this, I modified the CreateRestSchema method of the COM class for the tool. I also added some additional variables to support my business logic:
I wrapped my business logic in a function and called it from the REST operation's handler EchoInput. THe EchoInput method parses the incoming request, calculates the watershed, and serialize the output.
There is a block of 3 lines towards the bottom that comprise the business logic from my ArcMap tool. The first two are helper functions (CreateInMemoryWorkspace / CreateFCFromGeometry), and the third is the actual watershed operation:
I built a small client application to test Find Watershed over the web. In the app, the user first supplies the location by clicking on the map. The click location is passes to the server into a query against the lookup map to determine the hydroshed id then :
building a client
I like the motto, "if you can't do in on desktop, you won't be able to do it on server." Keeping this idea in mind, I first wrote the tool as a add-in for ArcMap. This allowed me to debug core business logic for the tool without the complications of the server.
The main difference between a 'desktop tool' and Server Object Extension is the ability to deserialize requests and serialize responses over http. Since I was new to C# and Server Object Extensions, I first published a sample SOE called SimpleRESTSOE from the samples folder of the ArcGIS 10 install directory. This tool was simply an echo service (i.e. pass a string in, and get the string back). Requests declare 'text' and 'f' (output format) parameters as part of a url request:
/echo?text=helloWorld&f=json
and they SOE responds:
{"text":"helloWorld"}
I got the sample SOE registered with Windows using the command:
RegAsm SimpleRESTSOE.dll /codebase
I then registered the SOE with ArcCatalog and the Server Manager using the companion file located in the samples folder (RegisterSimpleRESTSOE.dll).
Once I verified the sample SOE was running, I simply started modifing the rest schema to take my custom parameters. To do this, I modified the CreateRestSchema method of the COM class for the tool. I also added some additional variables to support my business logic:
After modifying the CreateRestSchema method the tool was wired to take the following request:
/echo?hydroshed_id=sa7&location={"x"=-54,"y"=-24}&f=json
I wrapped my business logic in a function and called it from the REST operation's handler EchoInput. THe EchoInput method parses the incoming request, calculates the watershed, and serialize the output.
Global Hydrosheds Lookup
I built a small client application to test Find Watershed over the web. In the app, the user first supplies the location by clicking on the map. The click location is passes to the server into a query against the lookup map to determine the hydroshed id then :
building a client
Subscribe to:
Posts (Atom)