New Big Data Projects

Some interesting big data related projects.

• Databus - Databus provides a timeline-consistent stream of change capture events for a database. Github link.
• Samza - Apache Samza is a distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.
• Druid - Druid is open source infrastructure for Real²time Exploratory Analytics on Large Datasets. The system uses an always-on, distributed, shared-nothing, architecture designed for real-time querying and data ingestion. It leverages column-orientation and advanced indexing structures to allow for cost effective, arbitrary exploration of multi-billion-row tables with sub-second latencies.
• Grape - Grape is a realtime processing pipeline.

Building Booky SPA - Building a RESTFul API for Booky using ASP.NET Web API

REST stands for Representational State Transfer.

• REST is an architectural style that runs over HTTP. 
• REST is used for exchanging data over HTTP. 
• REST is simple and stateless
• REST is not an official protocol or standard.

The core idea behind the creation of REST based services is that we should treat our distributed services as a resource and we should be able to use simple HTTP verbs to perform various operations on that resource.

In booky we have the following resources:

• users
• bookmarks
• boards - not implemented
• tags - not implemented

Each resource is available from a URI. booky REST URI's is created with the following URI http://domain.tld/rest/. Each booky resource will take on a URI such as:

• users - http://domain.tld/rest/user
• bookmarks - http://domain.tld/rest/bookmark
• boards - http://domain.tld/rest/board
• tags - http://domain.tld/rest/tag

In cases where REST-ful style doesn't fit with the application requirements, a REST-hybrid URI design can be used, such as with server side paging and application design

In booky a REST-hybrid URI http://domain.tld/actions is created so that there is an area for REST only URI's and REST-hybrid URI's. 

• http://domain.tld/rest/ -  booky common REST URI's
• http://domain.tld/actions/ - booky common REST-hybrid URI's

Example - booky common REST-hybrid URI's:

• Server side paged bookmarks - http://domain.tld/actions/bookmarkaction/GetPagedBookmarks/
• Updating a single bookmark property/attribute that indicates whether a bookmark has been read. - http://domain.tld/actions/bookmarkaction/toggleread/

HTTP Verbs

On the server each server side operation (Create, Read, Update, DELETE) maps to an HTTP verb:

• GET  
• POST 
• PUT 
• DELETE 

GET

Gets a listing of resources. When no resource id is specified then all resources are returned. When a resource id is specified then a single resource is returned. 

Example: 

• GET http://domain.tld/rest/bookmark/1 - single resource 
• GET http://domain.tld/rest/bookmark - listing of resources

POST

Creates a new resource. On successful creation, return HTTP status 201, returning a Location header with a link to the newly-created resource with the 201 HTTP status.When the resource can't be found return a 404 not found HTTP Response Code. POST is neither safe nor idempotent. It is therefore recommended for non-idempotent resource requests. Making two identical POST requests will most-likely result in two resources containing the same information.

Example: 

• POST http://domain.tld/rest/bookmark

PUT

Updates an existing resource. On successful update, return HTTP status 200, returning a Location header with a link to the updated resource with the 201 HTTP status.When the resource can't be found return a 404 not found HTTP Response Code.

Example: 

• PUT http://domain.tld/rest/bookmark/1 

DELETE

- Deletes an existing resource.  On successful delete, return HTTP status 200, and in the response body the representation of the deleted item.

Example: 

• DELETE http://domain.tld/rest/bookmark/1

HTTP Response Codes for use in response headers

The listing below outlines the typical HTTP Response Code to use for each server operation/HTTP verb combo:

• GET - 200 OK
• POST -  201 Created 
• PUT - 200 OK
• DELETE - 200 OK

A standard that I follow for PUT/DELETE/GET is when the resource can't be found is to return a 404 not found HTTP Response Code.

A listing HTTP Response Codes for reference:

• 200 OK: Success
• 201 Created - Used on POST request when creating a new resource.
• 304 Not Modified: no new data to return.
• 400 Bad Request: Invalid Request.
• 401 Unauthorized: Authentication.
• 403 Forbidden: Authorization
• 404 Not Found – entity does not exist.
• 406 Not Acceptable – bad params.
• 409 Conflict - For POST / PUT requests if the resource already exists.
• 500 Internal Server Error
• 503 Service Unavailable

Additional resources that provide expanded HTTP Response Codes information:

• http://www.w3.org/Protocols/HTTP/HTRESP.html
• http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

REST was introduced and defined by Roy Fielding in 2000.

• Roy Fielding doctoral dissertation - http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

This completes an overview on REST basics. ASP.NET Web API is up next followed by booky ASP.NET Web API Design.

What is ASP.NET Web API?

ASP.NET Web API is a framework for building and consuming HTTP services that can reach a broad range of clients including browsers, phones and tablets. You can use XML or JSON or something else with your API. JSON is nice for mobile apps with slow connections, for example. You can call an API from jQuery and better utilize the client's machine and browser.

Why use ASP.NET Web API?

ASP.NET Web API provides the following benefits: 

• From a focus standpoint MS Web API doesn't try to be all things; Web API is designed and focuses on HTTP
• Easier to secure than other ASP.NET web service options (WCF, ASMX)
• Has close integration of MS Web API with ASP.NET MVC; Uses the controller (Controller class) model found in ASP.NET MVC which developers may already be familiar with.

booky ASP.NET API Design

booky contains the following ASP.NET Web API controllers for each REST resource.

booky Controller Classes for Common REST URI's:

• Bookmark resource - BookmarkController Class - Get Bookmark 
• User resource - UserController Class - Update User (PUT)

booky Controller Classes for Common REST-hybird URI's:

• User Action Controller - Authenticate user and Register user
• Bookmark Action Controller - Get Paged Bookmarks, Get Bookmark By Url, Toggle Favorite, Toggle Read.

Bonus: CORS

CORS - Cross Origin Resource Sharing

Implementation of CORS via Web.config

<system.webServer>
 <httpProtocol>
  <customHeaders>
    <add name="Access-Control-Allow-Origin" value="*" />
    <add name="Access-Control-Allow-Headers" value="Content-Type" />
    <add name="Access-Control-Allow-Methods" value="GET, POST, PUT, DELETE, OPTIONS" />
  </customHeaders>
 </httpProtocol>
</system.webServer>


<system.webServer>
  <modules runAllManagedModulesForAllRequests="true">
    <remove name="WebDAVModule"/>
  </modules>
  <handlers>
    <remove name="WebDAV" />
  </handlers>
</system.webServer>

You can also implement CORS via code by creating a Message Handler:

In Global.asax add the CorsHandler.

GlobalConfiguration.Configuration.MessageHandlers.Add(new CorsHandler());

 public class CorsHandler : DelegatingHandler
    {
        const string Origin = "Origin";
        const string AccessControlRequestMethod = "Access-Control-Request-Method";
        const string AccessControlRequestHeaders = "Access-Control-Request-Headers";
        const string AccessControlAllowOrigin = "Access-Control-Allow-Origin";
        const string AccessControlAllowMethods = "Access-Control-Allow-Methods";
        const string AccessControlAllowHeaders = "Access-Control-Allow-Headers";

        protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            bool isCorsRequest = request.Headers.Contains(Origin);
            bool isPreflightRequest = request.Method == HttpMethod.Options;
            if (isCorsRequest)
            {
                if (isPreflightRequest)
                {
                    return Task.Factory.StartNew(() =>
                    {
                        HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
                        response.Headers.Add(AccessControlAllowOrigin, request.Headers.GetValues(Origin).First());

                        string accessControlRequestMethod = request.Headers.GetValues(AccessControlRequestMethod).FirstOrDefault();
                        if (accessControlRequestMethod != null)
                        {
                            response.Headers.Add(AccessControlAllowMethods, accessControlRequestMethod);
                        }

                        string requestedHeaders = string.Join(", ", request.Headers.GetValues(AccessControlRequestHeaders));
                        if (!string.IsNullOrEmpty(requestedHeaders))
                        {
                            response.Headers.Add(AccessControlAllowHeaders, requestedHeaders);
                        }

                        return response;
                    }, cancellationToken);
                }
                else
                {
                    return base.SendAsync(request, cancellationToken).ContinueWith(t =>
                    {
                        HttpResponseMessage resp = t.Result;
                        resp.Headers.Add(AccessControlAllowOrigin, request.Headers.GetValues(Origin).First());
                        return resp;
                    });
                }
            }
            else
            {
                return base.SendAsync(request, cancellationToken);
            }
        }
    }

Build 2015 - Service Fabric

One exciting announcement from Build this year was that of Service Fabric.

What can you do with Service Fabric?

Build and operate always-on, hyper-scale services using the same technology powering Microsoft’s Cloud.

Overview

Service Fabric is a distributed systems platform used to build scalable, reliable, and easily-managed applications for the cloud. Service Fabric addresses the significant challenges in developing and managing cloud applications. By using Service Fabric developers and administrators can avoid solving complex infrastructure problems and focus instead on implementing mission critical, demanding workloads knowing that they are scalable, reliable, and manageable. Service Fabric represents the next-generation middleware platform for building and managing these enterprise class, Tier-1 cloud scale services.

Service Fabric enables you to build and manage scalable and reliable applications composed of microservices running at very high density on a shared pool of machines (commonly referred to as a Service Fabric cluster). It provides a sophisticated runtime for building distributed, scalable stateless and stateful microservices and comprehensive application management capabilities for provisioning, deploying, monitoring, upgrading/patching, and deleting deployed applications.

Service Fabric powers many Microsoft services today such as Azure SQL Databases, Azure DocumentDB, Cortana, Power BI, Intune, Azure Event Hubs, many core Azure Services, and Skype for Business to name a few.

Where can you run Service Fabric?

On Azure and on premise

More information

Link - aka.ms/servicefabric

Technical Overview - http://azure.microsoft.com/en-us/documentation/articles/service-fabric-technical-overview/

Windows Build Video - skip to 30:13 in - http://channel9.msdn.com/Events/Build/2015/3-618