23 January 2019

Introduction to Git

Git Introduction

In our organization some of our code is stored in a Git repository and I created this document internally to help people understand Git better. So I thought it could be useful for others to use since I try to put the git system in the context that the commands were created (making them more logical for use and easier to remember). This document will explain the basics of the Git repo and how it is used.
In addition to this document, one can review this excellent guide as well: Atlassian Git Guide

What is Git?

Git is a version control system that was original created by Linus Torvalds (Linux creator) to manage the Linux kernel source code.  Its core principle is that it is a distributed source control system (meaning the code does not have to be stored in a central server).  This allows for each developer contributing code to work entirely isolated and then later mesh those changes with other users in a consistent manner. Each developer has a full copy of the source control for the project on their machine. A program named git was created to perform the actions on the repository locally and the synchronize those changes with others.

Git Basics

The entire git system on the development machine is contained the .git folder in the project directory, which contains two primary objects:
  • Repository
  • Staging Area (or Index)
The repository is a database containing all code and history of changes to a set of files for that project. The repository is not directly modified, nor does it contain files that are generally manipulated by hand - rather all changes are performed in a separate workspace and are linked to the git system through commands using the git program.  Here is a representation of the physical layout of the git system:
Git Physical Diagram
Components are:
  • Project Directory - folder for the entire project
    • Workspace - all the source code directories and files for the project
    • .git Directory - container for the local git system and contains:
      • Staging Area - index within the git repository that contains what changes you have made to the files in the workspace.
      • Local Repository - compressed database containing all the files and their history.
    • .gitattributes File - settings for the git repository for this project (i.e. line endings, default editors, etc.)
    • .gitignore File - information about which files to ignore from source control (i.e. compiled output, user settings, etc.)
  • Remote Repository - a remote copy of the project that others may use as a reference.

Local .Git Directory

You can think of the local .git directory as the entire source control system.  All changes, adds, and deletes are done in the workspace and managed in the local git repository using git commands.
Project Directory
In a traditional source control system, there is a central server containing the source files and users checks out and checks in changes to that server.  In git, that all occurs locally against your local git system (.git directory).  Those changes can then be shared to a central location with which others can synchronize.
Local Changes

Remote Repository

The remote repository is nothing more than a copy of the local git system for others to use as a reference.  It serves as a centralized sharing point for everyone to communicate their changes to the others.
Remote Repository
Developers make changes in their local projects, link those changes to their local git systems, then share those changes to the central reference copy known as the "remote repository".  Others may pull down those changes to their local git system and mesh those changes with their own using git commands.

Using a Shared Project (Cloning)

After a project is initially created and shared to a network location, others may make a copy of the project to perform local development work.  This process involves using git to copy and link the remote repository to their local project location and is done using the clone command:
git clone 

This creates the local git system for the project and copies down the history and working files while being linked to the shared reference copy.
Git Clone
All developers who are linked to the same remote repository can now share their changes with the same centralized set of changes using git commands.
Project Sharing

Working with Remote Repositories

There are several commands to synchronize your local project's git source control system with the shared reference copy. These commands allow for pushing of your changes to the shared reference copy and for merging of changes done by others with your own. Here is the basic diagram for these commands:
Remote Repository Commands Diagram
Common Remote Repository Commands:
  • git push
    - pushes changes make to your local repository (linked to a remote repository) to the remote.
  • git fetch
    - fetches the changes made by others into your local database without making changes to your working files
  • git merge
    - merges changes in your local repository with your working files.  Normally changes are meshed by git seamlessly unless two conflicting changes were made (known as a "merge conflict", which must be dealt with manually).
  • git pull
    - performs a
    git fetch
    git merge

Working with Local Repositories

There are several commands to add your workspace changes to the local git source control system.  At any point your changes can be undone by either overwriting your local repository with the remote reference, or by undoing changes you have committed to your local repository.
Local Repository Commands

Git Branches

In addition to the history of the current files in the project, git can also manage changes to multiple branches of the same project, each with their own history and common ancestor.
Git Branching
The history of all the branches is stored in the same local repository as before.  This allows for distruptive development to occur in parallel with a separate copy of the files.  In addition, it can encapsulate a series of changes in a visually separate stream in git visualization tools.
The git branch is not specific to a single file, but to the entire project.  So there can be changes to any files within the project in the separate branch and they are all stored in the same repository.
Multi-Branch Local Repository
The commands to view branches in your local repository are:
  • git branch
    - lists all branches in your repository (same as git branch --list)
  • git branch <branchname>
    - Creates a branch named <branchname> (but does not switch the workspace to it)
  • git branch -d <branchname>
    - Deletes a branch named <branchname> locally (note that if the branch exists in the reference remote repo, you need to do git push origin --delete <branchname> in order to delete from the shared repo)

Working With Branches

The user may chose to work with any branch in their repository by switching the files in the workspace to the branch of their choosing, or creating a new branch.
Switching Branches
Once you have switched branches, all changes made to the project are isolated to that branch and do not affect files in the other branches.

The commands to switch branches are:
  • git checkout <branchname>
    - Switches the current workspace to the <branchname> branch and sets all edits to be on that branch.
  • git checkout -b <branchname>
    - Creates and switches to <branchname>.  This is the same as running
    git branch <branchname>
    followed by
    git checkout <branchname>

Feature Branch

Generally, unique features are developed in a separate "feature branch" to isolate multiple changes until the feature is ready to be brought back in the the standard branch.  This allows for multiple advantages:
  1. Breaking changes will not disrupt the main branch.
  2. Multiple changes can be made slowly over time and tested independently of the main branches changes.
  3. Multiple feature commits can be compressed into a single commit on the main branch.
To start a feature branch, you simply create a branch and check it out using the command:
git checkout -b coolfeature

Then they would push the feature branch to the shared repo if others are working it too (so that everyone is working on the feature in the same branch):
git push -u origin coolfeature
git push -u origin HEAD

Create a Feature Branch
Next, changes may occur on either the source branch or the feature branch independently.  The feature branch can be shared in the remote repository or it can be isolated to a single users workspace.
Working the Branch Master branch developers would be performing these commands while making changes:

Dev A:
git pull,   git add .,   git commit,   git push

Dev B:
git pull,   git add .,   git commit,   git push


Feature branch developers would be performing these commands:
git checkout coolfeature
to switch their workspace to the feature branch
git pull,   git add .,   git commit,   git push

git pull,   git add .,   git commit,   git push


Once all work has been completed on the feature branch, it is ready to bring back into the master branch.  Each of the branch 'tips' will be merged into a single branch.
Feature Complete
A merge occurs that combines the two branch tips together in a single commit on the branch where the integration occurs.
Using the merge command:
git checkout master
to switch to the master branch (destination of merge)
git merge coolfeature
to merge the changes from feature branch "coolfeature" to master

Feature Branch Merged into Master Branch
This merge represents the simplest flow, but much more complicated branching can occur.

Additional Advanced Topics

Initial Creation of a Git Enabled Project

*WARNING: This section is only done ONCE by the project creator during project creation!*

In order for others to contribute to the same project, the project creator performs two steps:

  1. Create a local git system to handle source control for a project.
  2. Share that local git system with others to synchronize.
The creation is done using the command:

git init

This take all files and folders in the current directory and creates a .git directory with the necessary git database and other components.

The project repository can be shared to a web/network location for others to to use a reference for cloning and synchronizing their changes. The commands to create the shared reference copy of the project are:

// Add changes to a commit
git add .

// Save the commit in the local repository
git commit -m 'initial commit comment'

// Add a remote repo with name "origin" (like clone would do if you were pulling it down)
git remote add origin <url location of shared repo>

// push your repository to the master branch
git push -p origin master

At this point the reference copy of the repository is available for others to clone and to submit changes.

16 May 2012

API Example Using REST

In the last post I talked about the non-RESTfulness of most "REST" api implementations. I will attempt to show a more RESTful implementation and some examples of how this benefits the software. While researching this topic, it became clear (by both lack of examples and poor examples) that this is a tough concept. Before I jump into the example, lets examine why its so hard - because these issues will drive some of the design.

Difficulties with API and REST

An Application Programming Interface (API) is supposed to be:

And when creating a traditional API, you come up with methods and data formats to tell people how to "talk" to the system. But one of the primary REST tenets as mentioned by Roy Fielding is his blog is that you NOT have "out-of-band information driving interaction instead of hypertext." To put more simply, you should not pre-define the format of the data or the methods to be used, they should be discovered. But isn't that what an API is? How does one define the methods and the data formats but not have to know the methods or data formats? And why should I follow this "silly rule" anyway?

Well, the silly rule is the brilliance of REST and also what allows for the incredible flexibility that is the internet as we know today. If we know the format or the calls to make ahead of time, and write software against those, the control is no longer in the data providers hands. The control has shifted back from server to client. To understand the problem with this better, imagine that you wrote a custom browser and hard-coded it to a specific news article page with a fixed format and calling it silly that the news site wants to change the url or page format. (This example is stretched a bit far because an API has the understanding of being slightly more permanent than a generic webpage, but it still applies in spirit to RESTful APIs.)

The solution here is that there was a loophole of sorts, baked into the "Rules of REST" - note the last part in the rule: "out-of-band information driving information instead of hypertext." You can tell the client what it is allowed to do, as long as its part of the data (hypertext) being returned. The "media type" can have the information about what is allowed and what format things should be in! This, combined with the specifics of our chosen protocol (HTTP here), should give us all we need. So how do we do that?

REST Example

In the examples in the last post, the user logged in, searched for a product and created an order in both a traditional website and with a traditional API. So lets now make a REST version:

NOTE: I am going to ignore security at the beginning because it doesn't help in understanding the example, but I will discuss what could be done at the end.

Client starts with the root URL www.example.com/api/ and a discoverable media type apiIndex-v1+xml.

GET / www.example.com/api/ HTTP/1.1
Accept: application/vnd.example.coolapp.apiIndex-v1+xml

HTTP/1.1 200 OK
Content-Type: application/vnd.example.coolapp.apiIndex-v1+xml
    <description>API Help</description>

Here I have defined a set of valid hyperlinks the client can travel to from the root. It is reasonable to assume that the client could get more stuff or less stuff depending on what the business wanted to provide at the root, or based on what the user is authorized to use. This exchange would be the equivalent of a generic "index.html" page for the web. The client could show the descriptions as buttons or a drop down or whatever makes sense for the client (but it doesn't matter at the API level because it is not providing a page to be rendered, just resources to be processed as the client sees fit).

Lets say the client chooses to view a list of products, but does not know the allowed operations - the client could then perform an HTTP OPTIONS call against the resource to get the list (there are few standards for the format of the response of an OPTIONS call, but I went with a custom response body for the API to show a useful example):

OPTIONS /api/products
HTTP/1.1 200 OK
Content-Length: 1032
      <description>Retrieve a list of products</description>

Now seeing the options and the calling parameters, the client then provides a product search entry form in its UI, and then decides to perform the search against the API:

GET /api/products?Keyword=green%20widget

HTTP/1.1 200 OK
Content-Length: 1032
    <description>389349 green widget with bells</description>
    <price currencyCode="USD">3.56</price>
      <description>Product 12343</description>
    <description>2343 green widget copper clad</description>
    <price currencyCode="USD">4.93</price>
      <description>Product 2343</description>

The client then processes the results and allows the user to add items to a cart. This can be done against a cart resource so that the cart object is validated by the server (minimum quantities, availability, etc) and so that the client can save the cart if it doesn't want to commit to an order just yet:

POST /api/carts
Content-Length: 1032

HTTP/1.1 201 Created
Location: /api/carts/323392

Now that the cart is all set, perhaps the client decides to create an order:

POST /api/orders
Content-Type: application/vnd.example.coolapp.order-v1+xml
Content-Length: 2939

HTTP/1.1 201 Created
Location: /api/orders/O293920

That is the basic REST API example.

Security Aside

Let me walk through some of the security handling in this example since it can be confusing the first time you work through it. When you hit the index page for the API, the server would likely be expecting a valid session token in the form of an "Authorization" header so that we can authorize access to resources appropriately. If that header is not present, the server could send back a 302 redirect to a session resource (so that the client could submit a username and password to get a valid session) like so:

GET /api HTTP/1.1
Accept: application/vnd.example.coolapp.apiIndex-v1+xml
HTTP/1.1 302 Moved
Location: http://www.example.com/api/session

At which point the client could do an OPTIONS to learn the valid session options (not shown), then POST credentials to get a new session resource back:

POST /api/session
Content-Type: application/vnd.example.coolapp.sessionCredentials-v1+xml
Content-Length: 100
HTTP/1.1 201 Created
Location: http://www.example.com/api.session/74823
Content-Type: application/vnd.example.coolapp.session-v1+xml
Content-Length: 203

Typically, you could do something similar to all resources, but probably using 401 Not Authorized instead of the redirect since you really want the user to go through the index page instead of "knowing" the URL ahead of time. This is just a quick example of security and there are a lot of security vulnerabilities here that you should be careful to avoid. Plus, the client would have to start over at the index page, using the valid token - which is a little awkward. There are dozens of different ways to handle security, like OAuth, BASIC Authentication, etc, so more research is warranted with this topic, but you get the basic idea.

So now let me talk about some of the nuances with the flow.

Notes about REST APIs

I am going to parallel these comments with my interpretation of the "rules" that Mr. Fielding presented in his REST blog post.

  1. The only thing the client should know ahead of time is the root URL.  It shouldn't know or expect as specific resource to be available. For example, the client can't assume an "order" resource or a "search" resource - those should be provided as choices to the client from the server.

The first thing to note is that although the client only knows the root URL, it still must be coded to handle the link structure that was defined for the root, plus the media types that were presented. It is very difficult to avoid writing something akin to a browser when consuming a REST API. To create a client properly in this example, it should handle the creation or deletion of links on the index page, and gracefully handle any unknown media types that are presented. Otherwise, you are just basically making an RPC call, because you both know the format and nothing can change without both sides changing. That destroys the REST concept and prevents changes from occurring as seamlessly as it should (such as the host deciding to change the URL, or alter the flow through the API.)


The bulk of what can be done or what is available is meant to be part of the object that is returned. You will notice above that I included a special "resultInformation" block in the product search results. This allows the client to present a "1 of X" link in the app, or make multiple requests to cycle over the results from the server (and limit the bandwidth from the server). Also you will notice that the flow through the API does not jump around, for example, already "knowing" the URL of orders. You don't just jump to /api/orders/123, because the server may have moved it, or deleted it, or you may not be authorized to view it. If you want to give the client the ability to view orders, you create an orderView resource that returns a displayable form of the order and lets the client pick from a list. (server is in control what is available).

  1. "A REST API should not contain any changes to the communication protocols aside from filling-out or fixing the details of under specified bits of standard protocols, such as HTTP’s PATCH method or Link header field."

You will notice above that I presented a use of the HTTP OPTIONS method to assist with the consumption of the API, but tried to stay within the bounds of what is "standard" HTTP. In a normal non-REST API, these method signatures are predefined in a document and both sides code to them. If you do that in REST, you couple the client and server so that neither may change without the other changing. You want the server to decide what is presented and make the client smart enough to handle those choices. Again, it is very difficult to write a client that that is not browser-like and is a little unnatural for an API, but gives great power to the server to evolve.

Again, I have been more long-winded with my post that I would like, but hopefully it gives some value to the developers and REST-ees out there. This was a very difficult post for me to come up with because I had to constantly focus on the REST principles and avoid the classic API mindset. I found it very helpful to discuss with colleagues and to compare what a browser does with what we are trying to get out of the API. There are lots of things that I didn't have space to discuss, like versioning, content-negotiation, internationalization, etc that make this a very rich communication platform to build on. I would appreciate any comments or questions about this flow and REST in general. I know that there are things I can do better, and things that likely need fixing (and that everyone is going to do it differently for their purposes since there is no "right way") so be sure and add a comment.

27 April 2012

Putting the 'ST' in REST

Starting to implement REST has led to some interesting discussions and research topics with regards to what it means to be "REST". After reading Roy Fielding's post "REST APIs must be hypertext driven" - I think a walk through of this point might help things (both for my own understanding and for anyone who might be reading).

Unfortunately, Roy's "rant" comes off a little theoretical - and I think that might be one of the reasons why so many people are missing the point he is trying to make. The basic idea is that the API itself gives you all the information for the route your application can go. All the "application" knows is a root address - everything else flows from the choices the user is given and what they choose. In API land, this is a very foreign concept. Normally, when interfacing to an API, the application consuming the API has a structure and pages hardcoded, and those pages have buttons which restrict the flow for the user. So the API is just a wrapper around the data that the application will be using.

Web vs API

The real difference here is how web pages work versus how the typical API consumption works. For example, if you want to place an order on a website, the flow is something like this:

Webpage Application Example:

  • User goes to the root URL (http://www.awesomeproduct.com), and is presented with links to product pages of stuff on sale and a search box of some sort.

  • The user types in "green widget" to the search box and hits the "search" button.

  • The user is redirected to a product search result page with all the green widgets listed.

  • User clicks a product result link and is taken to a product information page with a "add to cart" button.

  • User likes the product, clicks "add to cart" and is taken either to the cart or the general search page

  • For web pages, this all works because there is an understanding that you are using a web browser and are expecting HTML to come back from nearly every resource. Your browser negotiates with the web server and gets an HTML representation of the information to display and displays it as text/images/buttons/forms. The returned page may contain links to other pages or related information and you can choose to browse to that information. The page could conceivably have a form that you fill out to send a message or something.

    Now compare this to an example ordering application consuming a typical "secure" non-REST API:

    Proprietary API Example:

  • User opens a pre-built login form in the application. This page has a username/password pair that get encrypted and sent to a Login method in the API via a socket connection, which returns a session token for subsequent requests.

  • The application then displays a predefined search page to locate products to order. The user enters "green widget" and hits search (passing the token).

  • "green widget" is sent to the SearchProducts API method and it returns a list of products matching "green widget".

  • The application displays the list in a table and the user hits "Add to order" button on the app. This triggers the application to package up a new order object and send it to the API's OrderCreate method.

  • An order is created on the backend and a success message is returned

  • As you can see the flows are very similar - but there are some important differences which will be discussed in a minute.

    Lets go "REST" !!

    This is pretty much how things are done for most applications, but then some ambitious programmer comes along and says "Lets make our API REST so we can use the same API for multiple apps" (or some such reasoning). The sales people chime in and say "Yeah, lets go REST because everyone is clamoring for that buzzword" and so the decision is made to do REST. After the 10 minute "Google Tutorial" on REST, they implement the following BAD implementation:

    Login method:
    GET to www.awesomeproducts.com/api/v1/login?user=x;encryptedpassword=y
    Product search:
    GET to www.awesomeproducts.com/api/v1/products?keyword="green widget"
    Order create:
    POST to www.awesomeproducts.com/api/v1/orders

    The developers of the app then change over their calls from their proprietary calls and responses to the new "REST" versions using XML defined in an XSD and the app is declared REST compliant! The problem here is that it is NOT REST and it is really no better than the previous proprietary non-REST version before. All that was really changed was the formatting of the data going to and from the backend, and the standardization on a communication protocol (HTTP).  Whats worse, is that it gives the appearance of REST, but it is not really REST at all.

    Problems with RE"ST"

    So why is this bad and how is doing it "properly" going to help anyway? Let now add a simple change to the situation to make the problem more clear - lets say management says that they want to display an advertisement for the new and amazing purple widgets after a customer logs in, but before the search. That will surely drive sales and we have to have this new feature yesterday.  So the app designers go to work and change the application to stick in this new advertisement (because changes to the API can't really make an ad show up in the hard-coded flow).

    BUT WAIT - why does the application have to change?!   That is the problem with this implementation of "REST"!  The issue here, is that if it was REST, the application would not have changed to accommodate this new 'page' in the flow.  To put it in perspective, imagine that if Amazon or Google wanted to change their pages, you had to upgrade your browser to see the new pages!!  That would make the web unworkable.  Yet this is what purported 'REST' designers have been doing and the type of thing that prompted Roy Fielding to go off on his well-deserved "rant" post.  With 'proper' REST, you should have been able to insert or change the flow by simply changing the transitions returned from the API server - not changes in the client.  What you have here is what my coworker called "REST without the 'ST'", there are no state transitions being provided by the "REST" server.  Its akin to web pages with hard-coded pages and no way to link between them - you would just have to know all the pages to go to and manually type them in (or have them hard-coded in your browser) - and any changes to the web page would require you to alter the browser.

    The brilliance of REST is not with the format of the data, or the HTTP negotiation - its with the transfer of state control from the client to the server.   You are allowing a third party to not only provide you with information in a format you can understand, but to control the valid paths through the "application" by letting it tell you were you can go - and then letting you choose.  Prior to REST, it was just a program that was hard-coded to known resources on a server and to find new information, you had to re-code the program to access those resources.  After REST, programs let you connect to servers which provide you a list of information(resources) you could go to.  You could then "choose-your-own-adventure" through the resources and, more importantly, the information providers could control your experiences as they saw fit - directing you down whichever back alley your choices lead.  This paradigm shift cannot be understated and is what gives REST its power.

    Aside: If you want to read more about this shift, it goes under the more technical name HATEOAS (Hypermedia As The Engine Of Application State)

    Roy was justifiably upset to see that great idea applied to situations where all that was "REST-a-fied" was the format of the data and some details around the protocol.  Which is really the same as the Proprietary API Example I provided above.  You basically hard code a program to pull resources in a predefined format from fixed resources.  The only thing that was really added was the standardization of format - which does allow others to more easily pick up the API, but its not REST.

    So How do We Do It More RESTful?

    So this is the point where most REST folks (myself included) start getting flak about being "REST purist" and that providing the data in XML via HTTP using a "REST" URL is good enough.  But you miss the entire crux of what makes REST so great - transfer of state control from the client to the server.  So what changes in the above example to make the Proprietary API Example a REST API example?

    Because this post is running long, I will make a new post explaining an example of how to use REST in an API context.

    24 March 2011

    "Standards First" Domain Object Design


    Our development team is starting to build up some domain objects for the various business objects that we will need and there are a variety of approaches that people seem to take as they start this endeavor. Many of the approaches seem hap-hazard and often capricious with respect to consistency, re-usability, and simplicity. So I decided to talk about my thoughts and maybe solidify my own ideas.

    What is Standards First Domain Object Design

    My tack has been to use a "Standards first" domain object design - meaning first look to see if there is a well recognized standard on how something should be represented before I venture out on my own. So for instance, lets say we need to pass a date out of the API, what should the XML for that date look like? I would ask myself, Are there standard ways to represent date in XML? For this example there is a standard XML date representation. I kind of expected everyone to follow this approach when I first started, but it seems this is not the way many developers operate. There are a lot of reasons to look to industry standards first, but there are also some good reasons to burn your own trail.

    Pros and Cons

    So what are the advantages of using the standards when possible? Well, for one, it often saves a bunch of time later (often in situations where you wouldn't anticipate it). For instance, In the last project I worked on, I was responsible for formatting the data coming out of the API and I choose to format times using the XML date time standard. I didn't realize at the time that when I went to parse it in our Android application, that android already had an ISO 3339 time/date parser built into the standard libraries - which works with the XML date time format (the XML format is a subset of ISO 3339). So I got automatic time parsing for free by just choosing the right format. I did not know the parser existing when I choose to use standard XML dates with timezones, but it ended up making all the pieces work together much smoother than it would have if I had chosen a "custom" format. These kinds of "happy accidents" seem to follow when standards are used (and it forms the basis for why computer software expands so rapidly - building on things before using components).

    Another big advantage to standards first domain objects would be when the time comes to share your API with 3rd parties. The adoption of an API occurs far faster when the formats look like things that developers have worked with before. Not having to explain a format is something that everyone should strive for. Also, the use of the API is better when the developers can use your data using their standard tools (which were developed to standards) with components that are already written.

    OK, OK, standards are great blah, blah - why would you choose to NOT use them? The primary disadvantage to them is that the research takes time. It takes a lot of time to look up standards for the data you are trying to use, figure out which one is the "best" or "most common" format. Often there are competing standards, many times the standards bodies want you to pay for the standards documents, sometimes the "standards" are far too complex for the task at hand. I personally like standards that are freely available (like RFCs), because they seem to be more highly vetted and have far more adoption because of the availability. I also like standards that are easy to read - often not needing explanation because they just make sense. So research is necessary and it take a lot of time.

    At what point do I decide that I am spending too much time researching and not enough time coding? It is a tough balance that I struggle with, and every developer situation is unique, so I can't generalize (and there is no right way). But there is a point of diminishing returns when it comes to software architecture research. I tend to strive the answer 3 questions when I design my domain objects/API:
    1. Would I want to use it?
    2. And by this I mean, if I were writing my own software for my own company, and I had 10 APIs to chose from, would this be the one I would go with?
    3. Does it handle my complexity?
    4. Does it handle the nuances that I may not have time to fully flesh out for this project at hand (thus saving me time in the long run). As an example, the time format (ISO 3339) handles time zones and durations and daylight savings time - things that might require me making a few stabs at to get right.
    5. Does it give me room to grow?
    6. Is the structure so strictly defined that I can't build on it, or is it componentized enough that I can use the structure as pieces to my architected objects.

    There are a lot of considerations to take into account when designing formats for domain objects and this post merely scratches the surface of a crazy complex issue, but standing on the shoulders of those who came before is definitely in your best interest. And a "standards first" domain object design is a great way to crank out a sustainable and flexible design.

    24 February 2011

    Versioning and Types in REST/HTTP API Resources

    There are a variety of ways to type and version the data with REST services, many of which are used successfully. Because of the living nature of APIs, changing versions and changing data types can lead to API designer headaches that saturate much of their time. I am going to discuss the ways to avoid some of these problems by configuring things properly up front.

    Dealing with Types

    Lets look at a common way REST calls are made:

    GET /customer/123 HTTP/1.1
    Accept: application/xml
    HTTP/1.1 200 OK
    Content-Type: application/xml
      <name>Neil Armstrong</name>
    Ok, we have an API that returns a customer - looks good. The problem here is that the API does not return a customer - but rather a generic XML document. When the designer made this particular API, they didn't bother to specify the type of document being returned. Sure there is an API document somewhere that defines the customer XML being returned, and as soon as you call it you are going to see that it is a customer - what is the big deal? Well, the problem is that the API is never static. And as far as REST/HTTP is concerned, sticking a product in the response is perfectly valid (but totally wrong in concept). Wouldn't it be nice if the client could actually validate the information coming back? Wouldn't the client be more stable and predictable if we knew the format of the data that was going to come back? Wouldn't it be great if the server knew which format the client wanted and could give the client what they ask for? Sure it would, so lets change the call to make sure we are asking for what we want:

    GET /customer/123 HTTP/1.1
    Accept: application/vnd.company.myapp.customer+xml
    HTTP/1.1 200 OK
    Content-Type: application/vnd.company.myapp.customer+xml
      <name>Neil Armstrong</name>
    Here I have substituted the generic xml request MIME type with a vendor specific MIME type. Awesome, now we ask for a customer XML formated customer, and we get a customer XML back from the call - which can even be validated on the client! That should solve our data format problem once and for all, right?

    Dealing with Versions

    BUT WAIT - there is still an issue! Lets say you decide to change the customer XML because you want to add some really cool new stuff to make that million dollar sale. How does that impact your client programs?
    Here is the new flow:

    GET /customer/123 HTTP/1.1
    Accept: application/vnd.company.myapp.customer+xml
    HTTP/1.1 200 OK
    Content-Type: application/vnd.company.myapp.customer+xml
    This change has just broken all the clients using that resource since they can no longer parse the changed XML properly! Further, there is no way for the client to check the returned version without calling it or asking for a specific version of the API return object - it just always gets the latest format. Sure you could say that your API needs to maintain backward compatibility - but that is not very realistic when you are properly reusing your API across your product line. To demonstrate further, lets say you have 30 applications (and maybe a handful of external companies using the API), all of which are relying on the "customer" REST resource - your choices now are:
    1. Keep it backward compatible (and lose the million dollar sale because you couldn't implement cool feature X)
    2. Change all 30 applications simultaneously to handle the new data (you likely don't have enough resource to do this and deliver on time)
    3. Make the change, breaking the apps you don't have time to upgrade, but get the sale. (of course you will fix the remaining apps in the future, right?)
    Basically, you lose no matter which choice you make. So what can we do now to avoid problems with newer versions within the API?

    Common Solutions to Versioning

    Some common ways of handling this problem are to request a specific version in the call URI:

    Method 1 (Put API version in the URI):

    Method 2 (Add a request parameter in the URI):

    Both of these methods are used to request a specific version of the resource we are looking for. And for many situations, these work fine. But we are munging up the resource identification with the resources representation (breaking an essential REST tenant). We don't want the identifier that we use to find an object to be mingled with the format we are requesting (for reasons discussed in my last post)! But we are smart designers, and this is essentially the same problem we had above, so we immediately recognize that we can solve this with the same solution as above - just make the type requested more specific:

    Method 3 (version the request type):
    GET /customer/123 HTTP/1.1
    Accept: application/vnd.company.myapp.customer-v1+xml
    HTTP/1.1 200 OK
    Content-Type: application/vnd.company.myapp.customer-v1+xml
      <name>Neil Armstrong</name>
    Similarly, the newer clients make a different call since they are aware of the new version:

    GET /customer/123 HTTP/1.1
    Accept: application/vnd.company.myapp.customer-v2+xml
    HTTP/1.1 200 OK
    Content-Type: application/vnd.company.myapp.customer-v2+xml
    Great! Now you have added the changes and made the sale and not broken any of the clients - you are the company hero! As an added bonus, clients and servers can work together to keep your API stable going forward. By requesting a specific type and version in the request header, the server can decide if it is capable of fulfilling the request and inform the client appropriately (i.e. return a HTTP 415 if it can't fulfill the request, or a 301 if it is changing). This is a much more stable and polite way of dealing with API changes for the client.


    We can see that the versioning and the typing of the data are not independent concepts for REST APIs - they are in fact the same, since changing either one results in different data being returned. And since REST is built on a very well tested technology like HTTP, lets take advantage of the capabilities of that technology. It is a simply matter to type the responses being generated and it adheres to 'proper' calling conventions, simplifies the URI and solves our versioning and typing issues cleanly.

    31 January 2011

    URI design in REST

    Background and the Issue

    We just began switching our existing architecture over to a proper SOA design (yeah!) and decided on using REST (yeah!) for the "API". One of the first discussions we got into was what the REST URI's will look like in the new design. We all agree that proper URI design is a core factor in determining the success of REST so the debate is lively and its good to see everyone passionate about the technology.

    One of the sticking points is how to return different formats of the Restful "objects" - say XML and JSON. Our desires come down to 3 possible designs all of which are considered acceptable from what I have read. So let me present you the design possibilities before I go into the pros and cons of each:

    Method 1 - Use the URI to configure the return type

    So to get a customer with ID 123 you would call:
    http://server:port/api/customer/123.json (for JSON)
    http://server:port/api/customer/123.xml (for XML)

    Method 2 - Use a query parameter to set the return format


    Method 3 - Use the HTTP headers to get the objects

    (GET request using an XML or JSON mime-type in the request header)

    One of our architects choose the first for the prototyping we were doing and when I saw it, something didn't sit right about it. So I decided to do a little research -- and it turns out that there are a lot of folks who do it this way, including the Twitter REST API, and the seeming "reference" text on the subject (RESTful Web Services by Leonard Richardson & Sam Ruby).

    Even though I found a lot of great references praising this method, that nagging sense that I was doing something wrong kept creeping in, so I figured it was worth the time to analyze the situation a bit more. So I came up with some reasons for an against each design to help me decide on a proper way to go:

    Analysis of Method 1

    The first solution has several advantages and it is certainly a quick solution to multiple formats.
    1. Easily testable in a browser - you just type in the format you want.
    2. Simple for most to understand - because you explicitly ask for the format you need.
    1. Can lead to ambiguity in the API. For instance, lets say you needed to return data that required multiple input parameters, like some reporting data. You request could be something like this (I put the parameters in []):

      But if you need to specify the format, what does the final URL look like:
      While this is a sort of a minor problem (you could just say "put it at the end"), it does represent something you will have to think about when writing the calls and something you will have to explain whenever a 3rd party uses your API.
    2. Requires a separate handler for each URI type that you add - meaning its more difficult to write a generic format parser and reuse it on all the methods. On the receiving side, most languages (.net and java) have annotation type tags that can be placed above code to handle a specific format. So you client has something like:

      public method URLHandler()
      [annotation to handle "/customer/[ID].xml"]
      [annotation to handle "/customer/[ID].json"]
      [annotation to handle "/customer/[ID].newformat"]

      rather than a generic format identifier:

      public method URLHandler()
      [annotation to handle "/customer/[ID]"]
      format = getFormat(request)

      You can certainly do this generic handler, but you end up parsing the type out of the URL which can be tricky if your URI doesn't contain the format or contains the format keywords in the URI.

    3. Provides some confusion with the return type.
      What are you really requesting, if we watch the wire while the browser makes this request, you see the browser pass an HTTP GET with an accept header of type "HTML" and the function returns you XML or JSON data. This seems wrong to me. From a browser you won't notice this as much, but if you are actually writing a client you will have to understand that you are not asking for the format in the request header - just the URL.

    4. Can't rely on the REST hosting framework to enforce the accept formats and deny requests the server cannot fulfill without manually checking the formats. If you are going to be doing this, you probably aren't going to be writing the whole REST framework from the socket up - you are going to be depending on .NET or a Java library to "help" you out with the coding and you just want to slap some tags on a function to do handle it. If you stick the return type in the URI, you are not letting the framework handle the format negotiations for you. So you are essentially manually overriding what part of the framework was designed to handle.

    5. Put yourself in a position to really complicate your URI. Lets say you have developed your solution using method 1). Now the team comes along and says, we also want to be able to return the customer with spanish tags to expedite the rollout to Mexico (or whatever wacky format users now want). The logical place to put this with this design would be in the URI so that you aren't mixing methods:
      or maybe
      or worse (mix the two designs above)
      This seems to be fighting against something that is already supported in the HTTP header into the URI instead.

    Analysis of Method 2

    The first method has some problems, so lets look at Method 2.


    1. Again, easily testable in a browser - you just type in the format you want.
    2. Again, simple for most to understand - because you explicitly ask for the format you need.
    1. Still have the issue of confusion with the return type.
      The format query parameter serves as an override to whatever HTTP header request type you have passed.
    2. Still have an unnecessarily complicated URI to specify the format you need without relying on the headers:
    This method seems to fix many of the problems mentioned with method 1 and I feel is also preferrable. The cons are relatively minor and it seems workable from a design perspective.

    Analysis of Method 3

    1. Consistent URIs for all objects regardless of format.
    2. Adheres to the expectations of HTTP with regard to requests
    1. Much more difficult to test than method 1) or 2), but still doable with a plugin in most browsers (Firefox). Since you overriding the browsers default return type (HTML), you must use a plugin to alter the request that gets sent.
    2. More training of the staff to understand how HTTP works and why you have to fill out the HTTP request information properly to get the data you want.


    Adhering to the blog title - there is no "right" way to do REST API design. From the analysis, I feel that method 3 is the "best" choice for our architecture and our particular needs. But all 3 have their place and all are used successfully in a variety of architectures. What do you think, what are the pros and cons I didn't think of?


    I named this blog "There is No Right Way" to remind myself of that fact when it comes to programming. Often there is a general consensus about which way to go, but with the constant flux of technologies and capabilities, even the sure way will change over time. I will attempt to stir up some discussion about programming ideas and situations as I encounter them in my day-to-day programming.