Omar Shahine has a blog post entitled Hotmail + Outlook = Sweet where he writes

At long last... experience Hotmail inside of Outlook.

What used to be a subscription only offering is now available to anyone that wants it. While Outlook used to have the ability to talk to Hotmail via DAV it was flaky and 2 years ago we no longer offered it to new users of the service.

Well the new Outlook Connector has a few notable features that you didn't get with the old DAV support:

1. uses DeltaSync, a Microsoft developed HTTP based protocol that sync's data based on change sequence numbers. This means that the server is stateful about the client. Whenever the client connects to Hotmail, the server tells the clients of any changes that happened since the last time the client connected. This is super efficient and allows us to offer the service to many users at substantially lower overhead than stateless protocols. This is the same protocol utilized by Windows Live Mail. It's similar in nature to exchange Cached Mode or AirSync, the mobile sync stack used by Windows Mobile Devices.
2. Sync of Address Book. Your Messenger/Hotmail contacts get stored in Outlook.
3. Sync of Calendar (currently premium only)
4. Sync of allow/block lists for safety/spam

I've been using the Microsoft Office Outlook Connector for a few years now and have always preferred it to the Web interface for accessing my email on Hotmail. It's great that this functionality is now free for anyone who owns a copy of Microsoft Outlook instead of being a subscription service.

PS: Omar mentioning Hotmail and Microsoft Outlook's use WebDAV reminds me that there have been other times in recent memory when using RESTful Web protocols swept Microsoft. Without reading old MSDN articles like Communicating XML Data over the Web with WebDAV when Microsoft Office, Microsoft Exchange and Internet Information Services (IIS) it's easy to forget that Microsoft almost ended up standardizing on WebDAV as the primary protocol for reading and writing Microsoft data sources on the Web. Of course, then SOAP and WS-* happened. :)

 

If you don't read Stevey Yegge's blog, you should. You can consider him to be the new school version of Joel Spolsky especially now that most of Joel's writing is about what's going on at Fog Creek software and random rants about applications he's using. However you should be warned that Stevey writes long posts full of metaphors which often border on allegory.

Consider his most recent post That Old MarshMallow Maze Spell which is an interesting read but full of obfuscation. I actually haven't finished it since it is rather longer than I tend to devote to a single blog post. I've been trying to track down summaries of the post and the best I've gotten so far are some comments about the post on reddit which seem to imply that the allegory is about being burned out due to some death march project at his current employer.

I'm as down with schadenfreude as the next guy but a death march project seems wildly contradictory to Stevey's previous post Good Agile, Bad Agile where he wrote

The basic idea behind project management is that you drive a project to completion. It's an overt process, a shepherding: by dint of leadership, and organization, and sheer force of will, you cause something to happen that wouldn't otherwise have happened on its own.
Project management comes in many flavors, from lightweight to heavyweight, but all flavors share the property that they are external forces acting on an organization.

At Google, projects launch because it's the least-energy state for the system.
...
Anyway, I claimed that launching projects is the natural state that Google's internal ecosystem tends towards, and it's because they pump so much energy into pointing people in that direction. All your needs are taken care of so that you can focus, and as I've described, there are lots of incentives for focusing on things that Google likes.

So launches become an emergent property of the system.

This eliminates the need for a bunch of standard project management ideas and methods: all the ones concerned with dealing with slackers, calling bluffs on estimates, forcing people to come to consensus on shared design issues, and so on. You don't need "war team meetings," and you don't need status reports. You don't need them because people are already incented to do the right things and to work together well.

So, did anyone else get anything else out of Stevey's post besides "even at Google we have death marches that suck the soul out of you"? After all, I just kinda assumed that came with the territory.

 

One of the accusations made by Tim Bray in his post I’ve Seen This Movie is that my recent  posts about the Atom Publishing Protocol are part of some sinister plot by Microsoft to not support it in our blogging clients. That's really ironic considering that Microsoft is probably the only company that has shipped two blogging clients that support APP.

Don't take my word for it. In his blog post entitled Microsoft is not sabotaging APP (probably) Joe Cheng of the Windows Live Writer team writes

1. Microsoft has already shipped a general purpose APP client (Word 2007) and GData implementation (Windows Live Writer). These are the two main blogging tools that Microsoft has to offer, and while I can’t speak for the Word team, the Writer team is serious about supporting Atom going forward.
2. These two clients also already post to most blogs, not just Spaces. In particular, Writer works hard to integrate seamlessly with even clearly buggy blog services. I don’t know anyone who works as hard as we do at this.
3. ...
4. Spaces may not support APP, but it does support MetaWeblog which Microsoft has a lot less influence over than APP (since MW is controlled by Dave Winer, not by an official standards body). Consider that many of its main competitors, including MySpace, Vox, and almost all overseas social networking sites, have poor or nonexistent support for any APIs.

The reasoning behind Windows Live Spaces supporting the MetaWeblog API and not the Atom Publishing Protocol are detailed in my blog posts What Blog Posting APIs are supported by MSN Spaces? and Update on Blog Posting APIs and MSN Spaces which I made over two years ago when we were going through the decision process for what the API story should be for Windows Live Spaces. For those who don't have time to read both posts, it basically came down to choosing a mature de facto standard (i.e. the MetaWeblog API) instead of (i) creating a proprietary protocol which better our needs or (ii) taking a bet on the Atom Publishing Protocol spec which was a moving target in 2004 and is still a moving target today in 2007.

I hope this clears up any questions about Microsoft and APP. I'm pretty much done talking about this particular topic for the next couple of weeks.

PS: You can download Windows Live Writer from here and you can buy Office 2007 wherever great software is sold.

 

I recently posted a blog post entitled Why GData/APP Fails as a General Purpose Editing Protocol for the Web which pointed out some limitations in the Atom Publishing Protocol (APP) and Google's implementation of it in GData with regards to being a general purpose protocol for updating data stores on the Web. There were a lot of good responses to my post from developers knowledgeable about APP including the authors of the specification, Bill de hÓra and Joe Gregorio. Below are links to some of these responses

Joe Gregorio: In which we narrowly save Dare from inventing his own publishing protocol
Bill de hÓra: APP on the Web has failed: miserably, utterly, and completely
David Megginson: REST, the Lost Update Problem, and the Sneakernet Test
Bill de hÓra: Social networks, web publishing and strategy tax
James Snell: Silly

There was also a post by Tim Bray entitled So Lame which questions my motives for writing the post and implies that it is some sinister plot by Microsoft to make sure that we use proprietary technologies to lock users in. I guess I should have given more background in my previous post. The fact is that lots of people have embraced building RESTful Web Services in a big way. My primary concern now is that we don't end up seeing umpteen different RESTful protocols from Microsoft [thus confusing our users and ourselves] and instead standardize on one or two. For example, right now we already have Atom+SSE, Web3S and Project Astoria as three completely different RESTful approaches for updating or retrieving data from a Microsoft data source on the Web. In my mind, that's two too many and that's just the stuff we've made public so there could be more. I'd personally like to see us reduce the set of RESTful protocols coming out of Microsoft to one and even better end up reusing existing Web standards, if possible. Of course, this is an aspiration and it is quite possible that all of these protocols are different for a reason (e.g. we have FTP, SMTP, and HTTP which all can be used to transfer files but have very different use cases) and there is no hope for unification let alone picking some existing standard. My previous post  was intended to point out the limitations I and others had noticed with using the Atom Publishing Protocol (APP) as a general protocol for updating data stores that didn't primarily consist of authored content. The point of the post was to share these learnings with other developers working in this space and get feedback from the general developer community just in case there was something wrong with my conclusions.

Anyway, back to the title of this post. In my previous post I pointed out to the following limitations of APP as a general purpose protocol for editing Web content

1. Mismatch with data models that aren't microcontent
2. Lack of support for granular updates to fields of an item
3. Poor support for hierarchy

I have to admit that a lot of my analysis was done on GData because I assumed incorrectly that it is a superset of the Atom Publishing Protocol. After a closer reading of the fifteenth most recent draft APP specification spurred by the responses to my post by various members of the Atom community it seems clear that the approaches chosen by Google in GData run counter to the recommendations of Atom experts including both authors of the spec.

For problem #1, the consensus from Atom experts was that instead of trying to map a distinct concept such as a Facebook user to an Atom entry complete with a long list of proprietary extensions to the atom:entry element, one should instead create a specific data format for that type then treat it as a distinct media type that is linked from atom:entry. Thus in the Facebook example from my previous post, one would have a distinct user.xml file and a corresponding atom:entry which linked to it for each user of the system. Contrast this with the use of the gd:ContactSection in an atom:entry for representing a user. It also seems that the GData solution to the problem of what to put in the elements such as atom:author and atom:summary which are required by the specification but make no sense outside of content/microcontent editing scenarios is to omit them. It isn't spec compliant but I guess it is easier than putting in nonsensical values to satisfy some notion of a valid feed.

For problem #2, a number of folks pointed out that conditional PUT requests using ETags and the If-Match header are actually in the spec. This was my oversight since I skipped the section since the title "Caching and Entity Tags" didn't imply that it had anything to do with dealing with the lost update problem. I actually haven't found a production implementation of APP that supports conditional PUTs this shouldn't be hard to implement for services that require this functionality. This definitely makes the lost update problem more tractable. However a model where a client can just say "update the user's status message to X" still seems more straightforward than one where the client says "get the entire user element", "update the user's status message to X on the client", "replace the user on the server with my version of the user", and potentially "there is a version mismatch so merge my version of the user with the most recent version of the user from the server and try again". The mechanism GData uses for solving the lost update problem is available in the documentation topic on Optimistic concurrency (versioning). Instead of using ETags and If-Match, GData appends a version number to the URL to which the client publishes the updated atom:entry and then cries foul if the client publishes to a URL with an old version number. I guess you could consider this a different implementation of conditional PUTs from what is recommended in the most recent version of the APP draft spec.

For problem #3, the consensus seemed to be to use a atom:link elements to show hierarchy similar to what has been done in Atom threading extensions. I don't question the value of linking and think this is a fine approach for the most part. However, the fact is that in certain scenarios [especially high traffic ones] it is better for the client to be able to make requests like "give me the email with message ID 6789 and all the replies in that thread" than "give me all the emails and I'll figure out the hierarchy I'm interested in myself by piecing together link relationships". I notice that GData completely punts on representing hierarchy in the MessageKind construct which is intended for use in representing email messages.

Anyway I've learned my lesson and will treat the Atom Publishing Protocol (APP) and GData as separate protocols instead of using them interchangeably in the future.

 

Recently I wrote a blog post entitled Google GData: A Uniform Web API for All Google Services where I pointed out that Google has standardized on GData (i.e. Google's implementation of the Atom 1.0 syndication format and the Atom Publishing Protocol with some extensions) as the data access protocol for Google's services going forward. In a comment to that post Gregor Rothfuss wondered whether I couldn't influence people at Microsoft to also standardize on GData. The fact is that I've actually tried to do this with different teams on multiple occasions and each time the I've tried, certain limitations in the Atom Publishing Protocol become quite obvious when you get outside of blog editing scenarios for which the protocol was originally designed. For this reason, we will likely standardize on a different RESTful protocol which I'll discuss in a later post. However I thought it would be useful to describe the limitations we saw in the Atom Publishing Protocol which made it unsuitable as the data access protocol for a large class of online services. 

Overview of the Atom's Data Model

The Atom data model consists of collections, entry resources and media resources. Entry resources and media resources are member resources of a collection. There is a handy drawings in section 4.2 of the latest APP draft specification that shows the hierarchy in this data model which is reproduced below.

                    
             Member Resources
                    |
             -----------------
            |                 |
      Entry Resources     Media Resources
            |
      Media Link Entry      

A media resource can have representations in any media type. An entry resource corresponds to an atom:entry element which means it must have an id, a title, an updated date, one or more authors and textual content. Below is a minimal atom:entry element taken from the Atom 1.0 specification

<entry>
    <title>Atom-Powered Robots Run Amok</title>
    <link href="http://example.org/2003/12/13/atom03"/>
    <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
    <updated>2003-12-13T18:30:02Z</updated>
    <summary>Some text.</summary>
</entry>

The process of creating and editing resources is covered in section 9 of the current APP draft specification. To add members to a Collection, clients send POST requests to the URI of the Collection. To delete a Member Resource, clients send a DELETE request to its Member URI. While to edit a Member Resource, clients send PUT requests to its Member URI.

Since using PUT to edit a resource is obviously problematic, the specification notes two concerns that developers have to pay attention to when updating resources.

To avoid unintentional loss of data when editing Member Entries or Media Link Entries, Atom Protocol clients SHOULD preserve all metadata that has not been intentionally modified, including unknown foreign markup.
...
Implementers are advised to pay attention to cache controls, and to make use of the mechanisms available in HTTP when editing Resources, in particular entity-tags as outlined in [NOTE-detect-lost-update]. Clients are not assured to receive the most recent representations of Collection Members using GET if the server is authorizing intermediaries to cache them.

The [NOTE-detect-lost-update] points to Editing the Web: Detecting the Lost Update Problem Using Unreserved Checkout which not only talks about ETags but also talks about conflict resolution strategies when faced with multiple edits to a Web document. This information is quite relevant to anyone considering implementing the Atom Publishing Protocol or a similar data manipulation protocol.   

With this foundation, we can now talk about the various problems one faces when trying to use the Atom Publishing Protocol with certain types of Web data stores.

Limitations Caused by the Constraints within the Atom Data Model

The following is a list of problems one faces when trying to utilize the Atom Publishing Protocol in areas outside of content publishing for which it was originally designed.

1. Mismatch with data models that aren't microcontent: The Atom data model fits very well for representing authored content or microcontent on the Web such as blog posts, lists of links, podcasts, online photo albums and calendar events. In each of these cases the requirement that each Atom entry has an an id, a title, an updated date, one or more authors and textual content can be met and actually makes a lot of sense. On the other hand, there are other kinds online data that don't really fit this model.

   Below is an example of the results one could get from invoking the users.getInfo method in the Facebook REST API.

   
   <user>
       <uid>8055</uid>
       <about_me>This field perpetuates the glorification of the ego.  Also, it has a character limit.</about_me>
       <activities>Here: facebook, etc. There: Glee Club, a capella, teaching.</activities>    
       <birthday>November 3</birthday>
       <books>The Brothers K, GEB, Ken Wilber, Zen and the Art, Fitzgerald, The Emporer's New Mind, The Wonderful Story of Henry Sugar</books>
       <current_location>
         <city>Palo Alto</city>
         <state>CA</state>
         <country>United States</country>
         <zip>94303</zip>
       </current_location>    
       <first_name>Dave</first_name>       
        <interests>coffee, computers, the funny, architecture, code breaking,snowboarding, philosophy, soccer, talking to strangers</interests>
        <last_name>Fetterman</last_name>   
        <movies>Tommy Boy, Billy Madison, Fight Club, Dirty Work, Meet the Parents, My Blue Heaven, Office Space </movies>
        <music>New Found Glory, Daft Punk, Weezer, The Crystal Method, Rage, the KLF, Green Day, Live, Coldplay, Panic at the Disco, Family Force 5</music>
        <name>Dave Fetterman</name>  
        <profile_update_time>1170414620</profile_update_time>
        <relationship_status>In a Relationship</relationship_status>
        <religion/>
        <sex>male</sex>
        <significant_other_id xsi:nil="true"/>
        <status>
          <message>Pirates of the Carribean was an awful movie!!!</message>
        </status>    
      </user>
   
   

   How exactly would one map this to an Atom entry? Most of the elements that constitute an Atom entry don't make much sense when representing a Facebook user. Secondly, one would have to create a large number of proprietary extension elements to anotate the atom:entry element to hold all the Facebook specific fields for the user. It's like trying to fit a square peg in a round hole. If you force it hard enough, you can make it fit but it will look damned ugly.

   Even after doing that, it is extremely unlikely that an unmodified Atom feed reader or editing client such as would be able to do anything useful with this Frankenstein atom:entry element. If you are going to roll your own libraries and clients to deal with this Frankenstein element, then it it begs the question of what benefit you are getting from mis using a standardized protocol in this manner?

   I guess we could keep the existing XML format used by the Facebook REST API and treat the user documents as media resources. But in that case, we aren't really using the Atom Publishing Protocol, instead we've reinvented WebDAV. Poorly.

2. Lack of support for granular updates to fields of an item: As mentioned in the previous section editing an entry requires replacing the old entry with a new one. The expected client interaction with the server is described in section 5.4 of the current APP draft and is excerpted below.

   Retrieving a Resource

   Client                                     Server
     |                                           |
     |  1.) GET to Member URI                    |
     |------------------------------------------>|
     |                                           |
     |  2.) 200 Ok                               |
     |      Member Representation                |
     |<------------------------------------------|
     |                                           |
   

   1. The client sends a GET request to the URI of a Member Resource to retrieve its representation.
   2. The server responds with the representation of the Member Resource.

   Editing a Resource

   Client                                     Server
     |                                           |
     |  1.) PUT to Member URI                    |
     |      Member Representation                |
     |------------------------------------------>|
     |                                           |
     |  2.) 200 OK                               |
     |<------------------------------------------|
   

   1. The client sends a PUT request to store a representation of a Member Resource.
   2. If the request is successful, the server responds with a status code of 200.

   Can anyone spot what's wrong with this interaction? The first problem is a minor one that may prove problematic in certain cases. The problem is pointed out in the note in the documentation on Updating posts on Google Blogger via GData which states

   IMPORTANT! To ensure forward compatibility, be sure that when you POST an updated entry you preserve all the XML that was present when you retrieved the entry from Blogger. Otherwise, when we implement new stuff and include <new-awesome-feature> elements in the feed, your client won't return them and your users will miss out! The Google data API client libraries all handle this correctly, so if you're using one of the libraries you're all set.

   Thus each client is responsible for ensuring that it doesn't lose any XML that was in the original atom:entry element it downloaded. The second problem is more serious and should be of concern to anyone who's read Editing the Web: Detecting the Lost Update Problem Using Unreserved Checkout. The problem is that there is data loss if the entry has changed between the time the client downloaded it and when it tries to PUT its changes.

   Even if the client does a HEAD request and compares ETags just before PUTing its changes, there's always the possibility of a race condition where an update occurs after the HEAD request. After a certain point, it is probably reasonable to just go with "most recent update wins" which is the simplest conflict resolution algorithm in existence. Unfortunately, this approach fails because the Atom Publishing Protocol makes client applications responsible for all the content within the atom:entry even if they are only interested in one field.

   Let's go back to the Facebook example above. Having an API now makes it quite likely that users will have multiple applications editing their data at once and sometimes these aplications will change their data without direct user intervention. For example, imagine Dave Fetterman has just moved to New York city and is updating his data across various services. So he updates his status message in his favorite IM client to "I've moved" then goes to Facebook to update his current location. However, he's installed a plugin that synchronizes his IM status message with his Facebook status message. So the IM plugin downloads the atom:entry that represents Dave Fetterman, Dave then updates his address on Facebook and right afterwards the IM plugin uploads his profile information with the old location and his new status message. The IM plugin is now responsible for data loss in a field it doesn't even operate on directly.

3. Poor support for hierarchy: The Atom data model is that it doesn't directly support nesting or hierarchies. You can have a collection of media resources or entry resources but the entry resources cannot themselves contain entry resources. This means if you want to represent an item that has children they must be referenced via a link instead of included inline. This makes sense when you consider the blog syndication and blog editing background of Atom since it isn't a good idea to include all comments to a post directly children of an item in the feed or when editing the post. On the other hand, when you have a direct parent<->child hierarchical relationship, where the child is an addressable resource in its own right, it is cumbersome for clients to always have to make two or more calls to get all the data they need.

UPDATE: Bill de hÓra responds to these issues in his post APP on the Web has failed: miserably, utterly, and completely and points out to two more problems that developers may encounter while implementing GData/APP.

 

A couple of months ago, I was asked to be part of a recruiting video for Microsoft Online Business Group (i.e. Windows Live and MSN). The site with the video is now up. It's http://www.whywillyouworkhere.com. As I expected, I sound like a dork. And as usual I was repping G-Unit. I probably should have worn something geeky like what I have on today, a a Super Mario Bros. 1up T-shirt. :)

 

Via Todd Bishop I found the following spoof of Back to the Future starring Christopher Lloyd (from the actual movie) and Bob Muglia, Microsoft's senior vice president of the Server and Tools Business. The spoof takes pot shots at various failed Microsoft "big visions" like WinFS and Hailstorm in a humorous way. It's good to see our execs being able to make light of our mistakes in this manner. The full video of the keynote is here. Embedded below is the first five minutes of the Back to the Future spoof.

Video: Microsoft Back to the Future Parody

 

I've been thinking a little bit about Google Gears recently and after reading the documentation things I've realized that making a Web-based application that works well offline poses an interesting set of challenges. First of all, let's go over what constitutes the platform that is Google Gears. It consists of three components

• LocalServer: Allows you to cache and serve application resources such as HTML pages, scripts, stylesheets and images from a local web server. 

• Database: A relational database where the application can store data locally. The database supports both full-text and SQL queries.

• WorkerPool: Allows applications to perform I/O expensive tasks in the background and thus not lock up the browser. A necessary evil. 

At first, this seemed like a lot to functionality being offered by Google Gears until I started trying to design how I'd take some of my favorite Web applications offline. Let's start with a straightforward case such as Google Reader. The first thing you have to do is decide what data needs to be stored locally when the user decides to go offline. Well, a desktop RSS reader has all my unread items even when I go offline so a user may expect that if they go offline in Google Reader this means all their unread items are offline. This could potentially be a lot of data to transfer in the split instant between when the user selects "go offline" in the Google Reader interface and she actually loses her 'net connection by closing her laptop. There are ways to work around this such as limiting how many feeds are available offline (e.g. Robert Scoble with a thousand feeds in his subscription list won't get to take all of them offline) or by progressively downloading all the unread content while the user is viewing the content in online mode. Let's ignore that problem for now because it isn't that interesting.

The next problem is to decide which state changes while the app is offline need to be reported back when the user gets back online. These seem to be quite straightforward,

• Feed changed
• Feed added
• Feed deleted
• Feed renamed
• Feed moved
• News item changed
• Item marked read/unread
• Item flagged/starred
• Item tag updated
  

The application code can store these changes as a sequential list of modifications which are then executed whenever the user gets back online. Sounds easy enough. Or is it?

What happens if I'm on my laptop and I go offline in Google Reader and mark a bunch of stuff as read then unsubscribe from a few feeds I no longer find interesting. The next day when I get to work, I go online on my desktop, read some new items and subscribe to some new feeds. Later that day, I go online with my laptop. Now the state on my laptop is inconsistent from that on the Web server. How do we reconcile these differences?

The developers at Google have anticipated these questions and have answered them in Google Gears documentation topic titled Choosing an Offline Application Architecture which states

No matter which connection and modality strategy you use, the data in the local database will get out of sync with the server data. For example, local data and server data get out of sync when:

• The user makes changes while offline
• Data is shared and can be changed by external parties
• Data comes from an external source, such as a feed

Resolving these differences so that the two stores are the same is called "synchronization". There are many approaches to synchronization and none are perfect for all situations. The solution you ultimately choose will likely be highly customized to your particular application.

Below are some general synchronization strategies.

Manual Sync

The simplest solution to synchronization is what we call "manual sync". It's manual because the user decides when to synchronize. It can be implemented simply by uploading all the old local data to the server, and then downloading a fresh copy from the server before going offline.
...

Background Sync

In a "background sync", the application continuously synchronizes the data between the local data store and the server. This can be implemented by pinging the server every once in a while or better yet, letting the server push or stream data to the client (this is called Comet in the Ajax lingo).

I don't consider myself some sort of expert on data synchronization protocols but it seems to me that there is a lot more to figuring out a data synchronization strategy than whether it should be done based on user action or automatically in the background without user intervention. It seems that there would be all sorts of decisions around consistency models and single vs. multi-master designs that developers would have to make as well. And that's just for a fairly straightforward application like Google Reader. Can you imagine what it would be like to use Google Gears to replicate the functionality of Outlook in the offline mode of Gmail or to make Google Docs & Spreadsheets behave properly when presented with conflicting versions of a document or spreadsheet because the user updated it from the Web and in offline mode?  

It seems that without providing data synchronization out of the box, Google Gears leaves the most difficult and cumbersome aspect of building a disconnected Web app up to application developers. This may be OK for Google developers using Google Gears since the average Google coder is a Ph.D but the platform isn't terribly useful to Web application developers who want to use it for anything besides a super-sized HTTP cookie. 

A number of other bloggers such as Roger Jennings and Tim Anderson have also pointed that the lack of data synchronization in Google Gears is a significant oversight. If Google intends for Google Gears to become a platform that will be generally useful to the average Web developer then the company will have to fix this oversight. Otherwise, they haven't done as much for the Web development world as the initial hype led us to believe. 

 

Ian McKellar has a blog post entitled Insecurity is Ruby on Rails Best Practice where he points out that by default the Ruby on Rails framework makes sites vulnerable to a certain class of exploits. Specifically, he discusses the vulnerabilities in two Ruby on Rails applications, 37 Signals Highrise and Magnolia, then proposes solutions. He writes

Cross Site Request Forgery
CSRF is the new bad guy in web application security. Everyone has worked out how to protect their SQL database from malicious input, and RoR saves you from ever having to worry about this. Cross site scripting attacks are dying and the web community even managed to nip most JSON data leaks in the bud.

Cross Site Request Forgery is very simple. A malicious site asks the user's browser to carry out an action on a site that the user has an active session on and the victim site carries out that action believing that the user intended that action to occur. In other words the problem arises when a web application relies purely on session cookies to authenticate requests.
...
Solutions
Easy Solutions
There aren't any good easy solutions to this. A first step is to do referrer checking on every request and block GET requests in form actions. Simply checking the domain on the referrer may not be enough security if there's a chance that HTML could be posted somewhere in the domain by an attacker the application would be vulnerable again.

Better Solutions
Ideally we want a shared secret between the HTML that contains the form and the rails code in the action. We don't want this to be accessible to third parties so serving as JavaScript isn't an option. The way other platforms like Drupal achieve this is by inserting a hidden form field into every form that's generated that contains a secret token, either unique to the current user's current session or (for the more paranoid) also unique to the action. The action then has to check that the hidden token is correct before allowing processing to continue.

Incidents of Cross Site Request Forgery have become more popular with the rise of AJAX and this is likely to become as endemic as SQL injection attacks until the majority of Web frameworks take this into account in their out of the box experience. 

At Microsoft, the Web teams at MSN and Windows Live have given the folks in Developer Division the virtue of their experience building Web apps which has helped in making sure our Web frameworks like ASP.NET Ajax (formerly codenamed Atlas) avoid this issue in their default configuration. Scott Guthrie outlines the safeguards against this class of issues in his post JSON Hijacking and How ASP.NET AJAX 1.0 Avoids these Attacks where he writes

Recently some reports have been issued by security researchers describing ways hackers can use the JSON wire format used by most popular AJAX frameworks to try and exploit cross domain scripts within browsers. Specifically, these attacks use HTTP GET requests invoked via an HTML include element to circumvent the "same origin policy" enforced by browsers (which limits JavaScript objects like XmlHttpRequest to only calling URLs on the same domain that the page was loaded from), and then look for ways to exploit the JSON payload content.

ASP.NET AJAX 1.0 includes a number of default settings and built-in features that prevent it from being susceptible to these types of JSON hijacking attacks.
...
ASP.NET AJAX 1.0 by default only allows the HTTP POST verb to be used when invoking web methods using JSON, which means you can't inadvertently allow browsers to invoke methods via HTTP GET.

ASP.NET AJAX 1.0 requires a Content-Type header to be set to "application/json" for both GET and POST invocations to AJAX web services. JSON requests that do not contain this header will be rejected by an ASP.NET server. This means you cannot invoke an ASP.NET AJAX web method via a include because browsers do not allow append custom content-type headers when requesting a JavaScript file like this.

These mitigations would solve the issues that Ian McKellar pointed out in 37 Signals Highrise and Magnolia because HTML forms hosted on a malicious site cannot set the Content-Type header so that exploit is blocked. However neither this approach nor referrer checking to see if the requests come from your domain is enough if the malicious party finds a way to upload HTML or script onto your site. 

To completely mitigate against this attack, the shared secret approach is most secure and is what is used by most large websites. In this approach each page that can submit a request has a canary value (i.e. hidden form key) which must be returned with the request. If the form key is not provided, is invalid or expired then the request fails. This functionality is provided out of the box in ASP.NET by setting the Page.ViewStateUserKey property. Unfortunately, this feature is not on by default. On the positive side, it is a simple one line code change to get this functionality which needs to be rolled by hand on a number of other Web platforms today.

 

The good folks on the Microsoft Experimentation Platform team have published a paper which gives a great introduction to how and why one can go about using controlled experiments (i.e. A/B testing) to improve the usability of a website. The paper is titled Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO and will be published as part of the Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. The paper begins

In the 1700s, a British ship’s captain observed the lack of scurvy among sailors serving on the naval ships of Mediterranean countries, where citrus fruit was part of their rations. He then gave half his crew limes (the Treatment group) while the other half (the Control group) continued with their regular diet. Despite much grumbling among the crew in the Treatment group, the experiment was a success, showing that consuming limes prevented scurvy. While the captain did not realize that scurvy is a consequence of vitamin C deficiency, and that limes are rich in vitamin C, the intervention worked. British sailors eventually were compelled to consume citrus fruit regularly, a practice that gave rise to the still-popular label limeys.

Some 300 years later, Greg Linden at Amazon created a prototype to show personalized recommendations based on items in the shopping cart (2). You add an item, recommendations show up; add another item, different recommendations show up. Linden notes that while the prototype looked promising, ―a marketing senior vice-president was dead set against it, claiming it will distract people from checking out. Greg was ―forbidden to work on this any further. Nonetheless, Greg ran a controlled experiment, and the ―feature won by such a wide margin that not having it live was costing Amazon a noticeable chunk of change. With new urgency, shopping cart recommendations launched. Since then, multiple sites have copied cart recommendations.

The authors of this paper were involved in many experiments at Amazon, Microsoft, Dupont, and NASA. The culture of experimentation at Amazon, where data trumps intuition (3), and a system that made running experiments easy, allowed Amazon to innovate quickly and effectively. At Microsoft, there are multiple systems for running controlled experiments. We describe several architectures in this paper with their advantages and disadvantages. A unifying theme is that controlled experiments have great return-on-investment (ROI) and that building the appropriate infrastructure can accelerate innovation.

I learned quite a bit from reading the paper although I did somewhat skip over some of the parts that involved math. It's pretty interesting when you realize how huge the impact of changing the layout of a page or moving links can be on the bottom line of a Web company. Were talking millions of dollars for the most popular sites. That's pretty crazy.

Anyway, Ronny Kohavi from the team mentioned that they will be giving a talk related to the paper at eBay research labs tomorrow at 11AM. The talk will be in Building 0 (Toys) in room 0158F. The address is 2145 Hamilton Avenue, San Jose, CA. If you are in the silicon valley area, this might be a nice bring your own lunch event to attend.