.NET (1) AIR (1) Algorithms (1) ArcGIS (2) ArcGIS Server (7) ArcMap (2) ArcObjects (6) arcpy (1) Arrays (1) AS3 (1) Bing (1) C# (8) Clean Code (1) Clustering (1) COM (1) Design Patterns (1) Developer Interviews (1) Django (1) ESRI Developer Summit (3) ESRI Flex API (1) Flash (1) Flex (7) fuzzy (1) Geometry (1) Geoprocessing (3) Google (1) Imaging (1) JavaScript (2) Map Tiles (2) Mobile (1) Nokia (1) Offline Mapping (2) PIL (1) Pixel Bender (1) Presentations (1) PSU (2) PyTables (1) Python (18) Rasters (2) Server Object Extensions (4) Sorting (1) Spatial Analysis (1) SQLite (1) Sublime Text 2 (1) VB (2) War (1) Where Camp (1) Yahoo (1)
Showing posts with label Server Object Extensions. Show all posts
Showing posts with label Server Object Extensions. Show all posts

11.12.11

Some random ArcObjects that make writing Server Object Extensions easier...(C#)

Server Object Extensions are one of the most powerful features of ArcGIS Server. To make writing them a bit easier for people, I am posting several of the functions I commonly use to deal with incoming data, execute an operation, and provide a response. As an example, I will take the scenario of doing a zonal statistics operation based on a user digitized polygon. The idea is that the user selects a variable for analysis (elevation, temperature, precipitation, etc.), draws a polygon in Flex/JS/Silverlight, the polygon is sent to the server, and zonal statistics for that polygon/variable are returned.


1. JSON to IGeometry: The server will receive a json object representing the polygon digitized by the user. The first step is to convert the JSONObject to IGeometry. I particularly liked NicoGIS's solution to the issue which is shown below:

public IGeometry ConvertAnyJsonGeometry(JsonObject jsonObjectGeometry)
        {
            object[] objArray;

            if (jsonObjectGeometry.TryGetArray("rings", out objArray))
            {
                return Conversion.ToGeometry(jsonObjectGeometry, esriGeometryType.esriGeometryPolygon);
            }

            if (jsonObjectGeometry.TryGetArray("paths", out objArray))
            {
                return Conversion.ToGeometry(jsonObjectGeometry, esriGeometryType.esriGeometryPolyline);
            }

            if (jsonObjectGeometry.TryGetArray("points", out objArray))
            {
                return Conversion.ToGeometry(jsonObjectGeometry, esriGeometryType.esriGeometryMultipoint);
            }

            try
            {
                return Conversion.ToGeometry(jsonObjectGeometry, esriGeometryType.esriGeometryPoint);
            }
            catch
            {
                try
                {
                    return Conversion.ToGeometry(jsonObjectGeometry, esriGeometryType.esriGeometryEnvelope);
                }
                catch
                {
                    return null;
                }
            }
        }  

2. IGeometry to IFeatureClass:  Once I have IGeometry, many times I want to convert it to an IFeatureClass so I can use it in an operation like zonal statistics. Zonal Statistics actually takes an IGeodataset, but IFeatureClass extends IGeodatatset, so all that is required is a cast. Here is a function to convert IGeometry to IFeatureClass:

public IFeatureClass CreateFeatureClassFromGeometry(IGeometry pGeometry, IFeatureWorkspace pOutFeatWorkspace, int wkid = 4236)
        {
            try
            {
                IFields pFields = new Fields() as IFields;
                {
                    // Set up the shape field for the feature class
                    IFieldsEdit pFieldsEdit = (IFieldsEdit)pFields;
                    IField pField = new Field();
                    IFieldEdit pFieldEdit = (IFieldEdit)pField;
                    pFieldEdit.Name_2 = "Shape";
                    pFieldEdit.Type_2 = esriFieldType.esriFieldTypeGeometry;

                    IGeometryDef pGeometryDef = new GeometryDef();
                    IGeometryDefEdit pGeometryDefEdit = (IGeometryDefEdit)pGeometryDef;
                    pGeometryDefEdit.GeometryType_2 = pGeometry.GeometryType;

                    ISpatialReference pSpatialReference;
                    if (wkid == 4326)
                    {
                        ISpatialReferenceFactory2 pSpaRefFact2 = new SpatialReferenceEnvironment() as ISpatialReferenceFactory2;
                        IGeographicCoordinateSystem pGeoCoordSys = pSpaRefFact2.CreateGeographicCoordinateSystem(wkid);
                        pSpatialReference = (ISpatialReference)pGeoCoordSys;
                    }

                    else if (wkid == 102100 || wkid == 3857)
                    {
                        ISpatialReferenceFactory2 pSpaRefFact2 = new SpatialReferenceEnvironment() as ISpatialReferenceFactory2;
                        IProjectedCoordinateSystem pProjCoordSys = pSpaRefFact2.CreateProjectedCoordinateSystem(wkid);
                        pSpatialReference = (ISpatialReference)pProjCoordSys;
                    }

                    else
                    {
                        throw new ArgumentNullException("Invalid Spatial Reference Well Known Id: Please use 4326 for Geographic or 102100 for Web Mercator");
                    }

                    ISpatialReferenceResolution pSpatialReferenceResolution = (ISpatialReferenceResolution)pSpatialReference;
                    pSpatialReferenceResolution.ConstructFromHorizon();

                    pGeometryDefEdit.SpatialReference_2 = pSpatialReference;
                    pFieldEdit.GeometryDef_2 = pGeometryDef;
                    pFieldsEdit.AddField(pField);

                    // Add other required fields to the feature class

                    IObjectClassDescription pObjectClassDescription = new FeatureClassDescription();

                    for (int i = 0; i < pObjectClassDescription.RequiredFields.FieldCount; i++)
                    {
                        pField = pObjectClassDescription.RequiredFields.get_Field(i);
                        if (pFieldsEdit.FindField(pField.Name) == -1)
                            pFieldsEdit.AddField(pField);
                    }
                }

                // Create the feature class
                string sFeatureClassName = "tmp" + Guid.NewGuid().ToString("N");
                IFeatureClass pFeatureClass = pOutFeatWorkspace.CreateFeatureClass(
                   sFeatureClassName, pFields, null, null, esriFeatureType.esriFTSimple, "Shape", null);

                // Add the input geometry to the feature class and return
                IFeatureCursor pFeatureCursor = pFeatureClass.Insert(true);
                IFeatureBuffer pFeatureBuffer = pFeatureClass.CreateFeatureBuffer();

                pFeatureBuffer.Shape = pGeometry;
                pFeatureCursor.InsertFeature(pFeatureBuffer);

                // Flush the feature cursor and release COM objects
                pFeatureCursor.Flush();
                Marshal.ReleaseComObject(pFeatureBuffer);
                Marshal.ReleaseComObject(pFeatureCursor);
                return pFeatureClass;
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                throw e;
            }
        }

3. Create In-Memory Workspace:  As you notice, the function above requires a workspace.  I like to use in-memory workspaces and to create them I use the following function:

        public IWorkspace CreateInMemoryWorkspace()
        {
            try
            {
                // Create an InMemory workspace factory.
                IWorkspaceFactory workspaceFactory = new InMemoryWorkspaceFactory() as IWorkspaceFactory;

                // Create an InMemory geodatabase.
                IWorkspaceName workspaceName = workspaceFactory.Create("", "MyWorkspace", null, 0);

                // Cast for IName.
                IName name = (IName)workspaceName;

                //Open a reference to the InMemory workspace through the name object.
                IWorkspace workspace = (IWorkspace)name.Open();
                return workspace;
            }

            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                throw e;
            }
        }

4. Get Raster Dataset from Map Service:  Keep in mind that an SOE has access to any of the feature classes in the service which uses the SOE as a capability.  This means that you shouldn't usually need to have any hardcoded data paths in the actual code.  Rather you should try an only use data from the map service itself.  If you send in the index of the value raster the user wants to run analysis on, look how easy it is to open:
public IGeoDataset GetGeoDatasetByMapServiceIndex(IMapServer3 mapServer, int layerID)
        {
            IMapServerDataAccess dataAccess = (IMapServerDataAccess)mapServer;
            return dataAccess.GetDataSource(mapServer.DefaultMapName, layerID) as IGeoDataset;
        }

5. Pre-loading Rasters during Init(): Loading and unloading rasters from memory can be an expensive operation.  Fortunately with SOEs, you can load data into memory when the service starts up, and have it standing by for any request which come in.  This is different than Python geoprocessing services which basically need to load data once the actual request is receive.  One idea I've been using is to create a data dictionary when the service starts up which keeps my value rasters standing by:

public Dictionary<int, IGeoDataset> CreateGeodatasetDictionary(IMapServer3 mapServer)
        {
            Dictionary<int, IGeoDataset> data_dictionary = new Dictionary<int, IGeoDataset>();

            IMapServerInfo msInfo = mapServer.GetServerInfo(mapServer.DefaultMapName);
            IMapLayerInfos layerInfos = msInfo.MapLayerInfos;
            int c = layerInfos.Count;

            for (int i = 0; i < c; i++)
            {
                IGeoDataset data = GetGeoDatasetByMapServiceIndex(mapServer, c);
                data_dictionary.Add(c, data);
            }

            return data_dictionary;
        }

Basically this dictionary is a mapping between map service indexes and their IGeodatasets which I create in the SOE Init() function and store in memory for the life of the SOE.

6. Running Zonal Statistics: Now that we have both the zone dataset and the value raster at our figure tips, the zonal statistics operation is just a couple lines of code.  There are many examples of how to use the zonalops class, but here's the idea:

IZonalOp oZonalOp = new RasterZonalOpClass();
ITable zonalStats = oZonalOp.ZonalStatisticsAsTable(zones, valueRaster, true);
            

7. ITable to IRecordSet: To return the results of the Zonal Stats from the SOE, you must first convert the ITable to an IRecordSet.  Here an straightforward way to do it:

public static IRecordSet ConvertTableToRecordset(ITable table)
        {
            IRecordSetInit recordSetInit = new ESRI.ArcGIS.Geodatabase.RecordSetClass();
            recordSetInit.SetSourceTable(table, new QueryFilterClass());
            IRecordSet recordset = recordSetInit as IRecordSet;
            return recordset;
        }

8. Serialize IRecordset: To convert the IRecordset into return JSON, you can simply use the Conversion.ToJSON method as follows:

IRecordSet return_records = BSharpUtilities.ConvertTableToRecordset(areaTable);
byte[] jsonBytes = Conversion.ToJson(return_records);

return Encoding.UTF8.GetString(jsonBytes, 0, jsonBytes.Length);

That's about it for now.  For information on getting started with SOEs, I would check out ESRI's samples that come with the .NET SDK and also NicoGIS posts 

2.7.11

Great REST-based Server Object Extension posts from NicoGIS

NicoGIS is an awesome GIS blog.  Over the past year, I've been learning REST-based Server Object Extensions for ArcGIS Server 10.  NicoGIS has some one of the best posts on the subject.  I ended up reading the post in Italian for fun, but I bet Google can translate it.

26.6.11

Python Geoprocessing Vs. C# Server Object Extensions (AGS 10)

The following deals with development of geoprocessing services specifically for ArcGIS Server 10

Introduction Video I made for Penn State's MGIS Program

Over the past weeks, I have written several server-side tools to support web mapping applications as part of my capstone project for Penn State's MGIS Program.  The following is a general overview of what I've found to be the strengths and weaknesses of  Geoprocessing Services in Python and Server Object Extensions in C#.

Over the coming weeks, I will be releasing the code and discussions of the individual tools in separate posts. 

Geoprocessing Services (Python)

For developers of web-based geoprocessing tools for ArcGIS Server 10,  the arcpy Python library is now a popular alternative to the more complex ArcObjects library.  With arcpy, you can access a decent percent of the ArcGIS tools with a clear syntax for linking them together. Knowledge of object-oriented programming is not required, and you can integrate GIS with an entire world of Python third-party libraries.  To web-enable arcpy, scripts are added to toolboxes which get published to the server.

Python Pros
  • Accelerated development time
  • ArcGIS includes many Python code samples
  • Easy to debug business logic

In spite of their ease of development and clear syntax, aspects of python geoprocessing services limit their power.  First is that they execute slower that the equivalent code written in .NET ArcObjects. For desktop tools that may not be a problem, but for the web this can negatively impacts user experience. 

Even if speed was not an issue, geoprocessing services are more cumbersome to move from one environment to another (e.g. staging -> production) because access to required source data is more tightly coupled with the tool.  Take the example of a server-side point clusterer.  To access the point feature class containing un-clustered points, a python script may use either of the following logic:



This first example uses a hard-coded absolute path that will need to be changed if the tool moves to an environment with a different directory structure.  A better choice would be the relative path example, but there is still hard-coding of the dataset's geodatabase and name.  With either choice, the tool suffers from a lack of true encapsulation which ends up making it harder to move and reuse on other point layers.

An aspect of the ArcGIS workflow for creating scripting tools which hinders a tool's flexibility is the need for explicit registration of input and output parameters in a properties dialog.





Python Cons
  • Slower execution speed when compared to C#
  • Tighter coupling with source data
  • Double managing of service input/output parameters
When using geoprocessing services, the developer gains development time by sacrificing execution speed, and the fine-grain control offered by ArcObjects.

C# Server Object Extensions (ArcObjects 10 .NET SDK)

C# Pros
  • Faster code execution
  • Loose coupling with tool source data
  • Easy to deploy across environments
I began using .NET ArcObjects because python geoprocessing service felt sluggish.  The allure of greater speed justified learning a more complex language.  What I found was not only increased speed, but also easier deployment and lower maintenance.

The idea of the Server Object Extension is that instead of publishing a service on top of the server, you extend the server itself.  This extended functionality can then be enabled for any service along with other out-of-the-box capabilities (e.g. Export Map, Identify, Find, Query, Generate KML, WMS).



The beautiful thing about Server Object Extensions is that ArcObjects has hooks into the source data within a map service.  These hooks give C# access to datasets using map service layer indexes instead of directory paths allowing for looser coupling with the server environment. ESRI Web APIs also use map service layer indexes which makes client-server communication easier.










C# Cons
  • Large and complex library
  • Not Cross-Platform (won't run on AGS Java version).
  • Steeper learning curve
Writing SOEs requires knowledge of a more complex library of tools accessed through a more complex programming language.  If a project does not have somebody with ArcObjects experience, diving into Server Object Extensions will be intimidating. In ArcObjects, finding the core methods for geoprocessing is easy, but getting data into a class which implements the correct interface for use as input in those methods can be difficult.

An example of this would be getting a geometry object, passed in as JSON from the client, into a feature class which can be used by a Spatial Analyst tool.  The solution I used took 80 lines of not fun code to write.  The image below is meant to give an impression of the complexity.





Another area of trouble is .NET and inability to run outside of a Window's environment.  Being that ESRI offer's a Java version of the ArcGIS Server, choosing C# means limiting a tool to Window's and the .NET version of ArcGIS Server.  This hasn't been a major issue for me yet, but I want as many options in hosting environments as possible.

Overall I have found the benefits of Server Object Extensions outweigh the ease of development offered by  Python when performance and reusablilty are important. Initially, I found the complexity of working in ArcObjects discouraging, but slowly I wrote function to deal this data type conversions and development got easier.  I now consider arcpy as a solution only on the desktop.

13.9.10

Find Watershed SOE

 The Find Watershed Server Object Extension (SOE) plugs into ArcGIS Server 10 and adds functionality for generating watersheds over REST.  The SOE takes two parameters: hydroshed id and a pour point location.  Proper flow accumulation and flow direction rasters are retrieved by hydroshed id from a file geodatabase.  The location parameter is the pour point.  The goal of the tool is to generate a polygon representing the area of water which flows into a point. 

I like the motto, "if you can't do in on desktop, you won't be able to do it on server."  Keeping this idea in mind, I first wrote the tool as a add-in for ArcMap. This allowed me to debug core business logic for the tool without the complications of the server.

The main difference between a 'desktop tool' and Server Object Extension is the ability to deserialize requests and serialize responses over http.  Since I was new to C# and Server Object Extensions, I first published a sample SOE called SimpleRESTSOE from the samples folder of the ArcGIS 10 install directory. This tool was simply an echo service (i.e. pass a string in, and get the string back).   Requests declare 'text' and 'f' (output format) parameters as part of a url request:

 /echo?text=helloWorld&f=json

and they SOE responds:

{"text":"helloWorld"} 


I got the sample SOE registered with Windows using the command:

RegAsm SimpleRESTSOE.dll /codebase

I then registered the SOE with ArcCatalog and the Server Manager using the companion file located in the samples folder (RegisterSimpleRESTSOE.dll).

Once I verified the sample SOE was running, I simply started modifing the rest schema to take my custom parameters.  To do this, I modified the CreateRestSchema method of the COM class for the tool.  I also added some additional variables to support my business logic:

After modifying the CreateRestSchema method the tool was wired to take the following request:

/echo?hydroshed_id=sa7&location={"x"=-54,"y"=-24}&f=json

I wrapped my business logic in a function and called it from the REST operation's handler EchoInput. THe EchoInput method parses the incoming request, calculates the watershed, and serialize the output.

There is a block of 3 lines towards the bottom that comprise the business logic from my ArcMap tool.  The first two are helper functions (CreateInMemoryWorkspace / CreateFCFromGeometry), and the third is the actual watershed operation:


Global Hydrosheds Lookup

I built a small client application to test Find Watershed over the web. In the app, the user first supplies the location by clicking on the map.  The click location is passes to the server into a query against the lookup map to determine the hydroshed id then :










building a client