People who've been following me on Twitter know that I've been spending my weekends turning RSS Bandit into a desktop client for Google Reader and NewsGator Online. Although I was familiar with NewsGator's SOAP API, I didn't have the patience to figure out the differences in using SOAP services that had been made in Visual Studio 2008 given the last time I tried it I was using Visual Studio 2003 and there seemed to be fairly significant differences. In addition, the chance to program against a fully featured RESTful API also piqued my curiosity. For these reasons I decided to use the NewsGator REST API when working on the features for synchronizing with NewsGator Online.

After I completed work on the synchronization with NewsGator Online, it was quite clear to me that the developer(s) who designed the NewsGator REST API didn't really understand the principles behind building RESTful services. For the most part, it looks like a big chunk of the work done to create a REST API was to strip the SOAP envelopes from their existing SOAP services and then switch to URL parameters instead of using SOAP messages for requests. This isn't REST, it is POX/HTTP (Plain Old XML over HTTP). Although this approach gets you out of the interoperability and complexity tax which you get from using XSD/SOAP/WS-*, it doesn't give you all of the benefits of conforming to the Web's natural architecture.

So I thought it would be useful to take a look at some aspects of the NewsGator REST API and see what is gained/lost by making them more RESTful.

What Should a Feed Reader's REST API Look Like?

Before the NewsGator REST API can be critiqued, it is a good idea to have a mental model of what the API would look like if it was RESTful. To do this, we can follow the steps in Joe Gregorio's How to Create a REST Protocol which covers the four decisions you must make as part of the design process for your service

• What are the URIs? [Ed note - This is actually  "What are the resources?" ]
• What's the format?
• What methods are supported at each URI?
• What status codes could be returned?

There are two main resources or data types in a feed reader; feeds and subscription lists. When building RESTful protocols, it pays to reuse standard XML formats for your message payloads since it increases the likelihood that your clients will have libraries and tools for dealing with your payloads. Standard formats for feeds are RSS and Atom (either is fine although I'd prefer to go with Atom) and a standard format for feed lists is OPML. In addition, there are bits of state you want to associate with feeds (e.g. what items have been read/unread/flagged/etc) and with subscription lists (e.g. username/password of authenticated feeds, etc). Since the aforementioned XML formats are extensible, this is not a problem to accommodate.

What methods to support also seems straightforward. You will want full Create, Retrieve, Update and Delete operations on the subscription list. So you will need the full complement of supporting POST, GET, PUT and DELETE on subscription lists. For feeds, you will only need to fetch them and update the user-specific associated with each feed such as if an item has been read or flagged. So you'd want to support GET and PUT/POST.

The question of what error codes to return would probably be at least 200 OK (success), 304 Not Modified (feeds haven't changed since last fetch), 400 Bad Request (invalid or missing request parameters) ,401 Unauthorized (invalid or no authentication credentials provided) and 404 Not Found (self explanatory).

This seems like a textbook example of a situation that calls for using the  Atom Publishing Protocol (AtomPub). The only wrinkle would be that AtomPub requires uploading the entire document with changes when updating the state of an item or media resource. This is pretty wasteful when updating the state of an item in an RSS feed (e.g. indicating that the user has read the item) since you'd end up re-uploading all the content of the item just to change the equivalent of a flag on the item. There have been a number of proposals and workarounds proposed for this limitation of AtomPub such as the HTTP PATCH Internet Draft and Astoria's introduction of a MERGE HTTP method. In thinking about this problem, I'm going to assume that the ideal service supports partial updates of feed and subscription documents using PATCH/MERGE or something similar.

Now we have enough of a background to thoughtfully critique some of the API design choices in the NewsGator API.

REST API Sin #1: URIs Identify Methods Instead of Resources (The Original Sin)

When a lot of developers think of REST they do not think of the REpresentational State Transfer (REST) architectural style as described in chapter five of Roy Fieldings' Ph.D thesis. Instead they think of exposing remote procedure calls (RPC) over the Web using plain old XML payloads, without schemas and without having to use an interface definition language (i.e XML-based RPC without the encumbrance of SOAP, XSD or WSDL).  

The problem with this thinking is that it often violates the key idea behind REST. In a RESTful Web service, the URIs identify the nouns (aka resources) that occur in the service and the verbs are the HTTP methods that operate on those resources. In an RPC-style service, the URLs identify verbs (aka methods) and the nouns are the parameters to these methods. The problem with the RPC style design is that it increases the complexity of the clients that interact with the system. Instead of a client simply knowing how to interact with a single resource (e.g. an RSS feed) and then signifying changes in state by the addition/removal of data in the document (e.g. adding an <is-read>true</is-read> element as an extension to indicate an item is read), it has to specifically target each of the different behaviors a system supports for that item explicitly (e.g. client needs to code against markread(), flagitem(), shareitem(), rateitem(), etc). The reduced surface area of the interface is not only a benefit to the client but to the service as well.  

Below are a few examples which contrast the approach taken by the NewsGator API with a more RESTful method based on AtomPub.

1a) Modifying the name of one of the locations a user synchronizes data from using the Newsgator RESTful API

Service URL	http://services.newsgator.com/ngws/svc/Location.aspx/<locationId>/changelocationnamebylocationid
Method	POST
Request Format	application/x-www-form-urlencoded
Payload	"locationname=<new location name>”
Response	No response

 

1b.) Modifying the name of one of the locations a user synchronizes data from using a more RESTful approach 

Service URL	http://services.newsgator.com/ngws/svc/Location.aspx/<locationId>/
Method	POST
Request Format	application/xml
Payload	<opml xmlns:ng="http://newsgator.com/schema/opml"> <head /> <body> <outline text="<location name>" ng:isPublic="<True|False>" ng:autoAddSubs="<True|False>" /> </body> </opml>
Response	No response

2a.) Deleting a folder using the NewsGator REST API

Service URL	http://services.newsgator.com/ngws/svc/Folder.aspx/delete/
Method	DELETE
Request Format	application/x-www-form-urlencoded
Payload	"fld=<NewsGator folder id>"
Response	No response

2b.) Deleting a folder using a more RESTful approach

Service URL	http://services.newsgator.com/ngws/svc/Folder.aspx/<folder-id>
Method	DELETE
Request Format	Not applicable
Payload	Not applicable
Response	No response

3a.) Retrieve a folder or create it if it doesn't exist using the NewsGator REST API

Service URL	http://services.newsgator.com/ngws/svc/Folder.aspx/getorcreate
Method	POST
Request Format	application/x-www-form-urlencoded
Payload	"parentid=<NewsGator folder id>&name=<folder name>&root=<MYF|MYC>"
Response	<opml xmlns:ng="http://newsgator.com/schema/opml"> <head> <title>getorcreate</title> </head> <body> <outline text="<folder name>" ng:id="<NewsGator folder id>" /> </body> </opml>

3b.) Retrieve a folder or create it if it doesn't exist using a more RESTful approach (the approach below supports creating multiple and/or nested folders in a single call)

Service URL	http://services.newsgator.com/ngws/svc/Folder.aspx/<root>/<folder-id>
Method	POST
Request Format	application/xml
Payload	<opml xmlns:ng="http://newsgator.com/schema/opml"> <head /> <body> <outline text="<folder name>" /> </body> </opml>
Response	<opml xmlns:ng="http://newsgator.com/schema/opml"> <head> <title>getorcreate</title> </head> <body> <outline text="<folder name>" ng:id="<NewsGator folder id>" /> </body> </opml>

REST API Sin #2: Not Providing a Consistent Interface  (A Venial Sin)

In REST, all resources are interacted with using the same generic and uniform interface (i.e. via the HTTP methods - GET, PUT, DELETE, POST, etc). However it defeats the purpose of exposing a uniform interface if resources in the same system respond very differently when interacting with them using the same HTTP method. Consider the following examples taken from the NewsGator REST API.

1.) Deleting a folder using the NewsGator REST API

Service URL	http://services.newsgator.com/ngws/svc/Folder.aspx/delete
Method	DELETE
Request Format	application/x-www-form-urlencoded
Payload	"fld=<NewsGator folder id>"
Response	No response

2.) Deleting a feed using the NewsGator REST API

Service URL	http://services.newsgator.com/ngws/svc/Subscription.aspx
Method	DELETE
Request Format	application/xml
Payload	<opml xmlns:ng="http://newsgator.com/schema/opml"> <head> <title>delete</title> </head> <body> <outline text=”subscription title” ng:id=”<NewsGator subscription id>” ng:statusCode=”<0|1>” /> </body> </opml>
Response	No response

From the end user or application developer's perspective the above actions aren't fundamentally different. In both cases, the user is removing part of their subscription list which in turn deletes some subset of the <outline> elements in the user's subscriptions OPML file.

However the NewsGator REST API exposes this fundamentally identical task in radically different ways for deleting folders versus subscriptions. The service URL is different. The request format is different. Even the response is different. There's no reason why all three of these can't be the same for both folders and subscriptions in a user's OPML feed list. Although it may seem like I'm singling out the NewsGator REST API, I've seen lots of REST APIs that have similarly missed the point when it comes to using REST to expose a uniform interface to their service.

Conclusion

These aren't the only mistakes developers make when designing a REST API, they are just the most common ones. They often are a sign that the developers simply ported some old or legacy API without actually trying to make it RESTful. This is the clearly case with the NewsGator REST API which is obviously a thin veneer over the NewsGator SOAP API.

If you are going to build a RESTful API, do it right. Your developers will thank you for it.

Now Playing: Montell Jordan - This is How We Do It

Disclaimer: This post does not reflect the opinions, thoughts, strategies or future intentions of my employer. These are solely my personal opinions. If you are seeking official position statements from Microsoft, please go here.

About a month ago Joel Spolsky wrote a rant about Microsoft's Live Mesh project which contained some interesting criticism about the project and showed that Joel has a personal beef with the Googles & Microsofts of the world for making it hard for him to hire talented people to work for his company. Unsurprisingly, lots of responses focused on the latter since it was an interesting lapse in judgement for Joel inject his personal frustrations into what was meant to be a technical critique of a software project. However there were some ideas worthy of discussion in Joel's rant that I've been pondering over the past month. The relevant parts of Joel's article are excerpted below

It was seven years ago today when everybody was getting excited about Microsoft's bombastic announcement of Hailstorm, promising that "Hailstorm makes the technology in your life work together on your behalf and under your control."

What was it, really? The idea that the future operating system was on the net, on Microsoft's cloud, and you would log onto everything with Windows Passport and all your stuff would be up there. It turns out: nobody needed this place for all their stuff. And nobody trusted Microsoft with all their stuff. And Hailstorm went away.
...
What's Microsoft Live Mesh?

Hmm, let's see.

"Imagine all your devices—PCs, and soon Macs and mobile phones—working together to give you anywhere access to the information you care about."

Wait a minute. Something smells fishy here. Isn't that exactly what Hailstorm was supposed to be? I smell an architecture astronaut.

And what is this Windows Live Mesh?

It's a way to synchronize files.

Jeez, we've had that forever. When did the first sync web sites start coming out? 1999? There were a million versions. xdrive, mydrive, idrive, youdrive, wealldrive for ice cream. Nobody cared then and nobody cares now, because synchronizing files is just not a killer application. I'm sorry. It seems like it should be. But it's not.

But Windows Live Mesh is not just a way to synchronize files. That's just the sample app. It's a whole goddamned architecture, with an API and developer tools and in insane diagram showing all the nifty layers of acronyms, and it seems like the chief astronauts at Microsoft literally expect this to be their gigantic platform in the sky which will take over when Windows becomes irrelevant on the desktop. And synchronizing files is supposed to be, like, the equivalent of Microsoft Write on Windows 1.0.

As I read the above rant I wondered what world Joel has been living in the past half decade. Hailstorm has actually proven to have been a very visionary and accurate picture of how the world ended up. A lot of the information that used to sit in my desktop in 2001 is now in the cloud. My address book is on Facebook, my photos are on Windows Live Spaces and Flickr, my email is in Hotmail and Yahoo! Mail, while a lot of my documents are now on SkyDrive and Google Docs. Almost all of these services provide XML-based APIs for accessing my data and quite frankly I find it hard to distinguish the ideas behind a unified set of user-centric cloud APIs that was .NET My Services from Google GData. A consistent set of APIs for accessing a user's contact lists, calendar, documents, inbox and profile all stored on the servers of a single company? Sounds like we're in a time warp doesn't it? Even more interesting is that outlandish sounding scenarios at the time such as customers using a delegated authentication model to grant applications and Web sites temporary or permanent access to their data stored in the cloud are now commonplace. Today we have OAuth, Yahoo! BBAuth, Google AuthSub, Windows Live DelAuth and even just the plain old please give us your email password.

In hindsight the major problem with Hailstorm seems to have been that it was a few years ahead of its time and people didn't trust Microsoft. Funny enough, a lot of the key people who were at Microsoft during that era like Vic Gundotra and Mark Lucovsky are now at Google, a company and a brand which the Internet community trusts a lot more than Microsoft, working on Web API strategy. 

All of this is a long winded way of saying I think Joel's comparison of Live Mesh with Hailstorm is quite apt but just not the way Joel meant it. I believe that like Hailstorm, Live Mesh is a visionary project that in many ways tries to tackle problems that people will have or don't actually realize they have. And just like with Hailstorm where things get muddy is separating the vision from the first implementation or "platform experience" of that vision.

I completely agree with Joel that synchronizing files is not a killer application. It just isn't sexy and never will be. The notion of having a ring or mesh of devices where all my files synchronize across each device in my home or office is cool to contemplate from a technical perspective. However it's not something I find exciting or feel that I need even though I'm a Microsoft geek with a Windows Mobile phone, an XBox 360, two Vista laptops and a Windows server in my home. It seems I'm not the only one that feels that way according to a post by a member of the Live Mesh team entitled Behind Live Mesh: What is MOE? which states

Software + Services

When you were first introduced to Live Mesh, you probably played with the Live Desktop.  It’s pretty snazzy.  Maybe you even uploaded a few files too.  Hey, it’s a cool service!  You can store stuff in a cloud somewhere and access it anywhere using a webpage.  Great!

As I look at the statistics on the service though, I notice that a significant portion of our users have stopped here.  This pains me, as there’s a whole lot more you can do with Live Mesh.  Didn’t you hear all the hoopla about Software + Services?  Ever wonder, “Where’s the software?”

You might have noticed that on the device ring there’s a big orange button with a white ‘+’ sign.  The magic happens when you click that big orange button and opt to “add a device” to your mesh. 

So what excites me as a user and a developer about Live Mesh? I believe seamless synchronization of data as a platform feature is really interesting. Today I use OutSync to add people's Facebook photos to their contact information on my Windows Mobile phone. I've written my own RSS reader which synchronizes the state of my RSS subscriptions with Google Reader. Doug Purdy wrote FFSync so he can share his photos, music taste and other data on his Mac with his friends on FriendFeed. It may soon be possible to synchronize my social graph across multiple sites via Google Friend Connect. Google is working on using Google Gears to give me offline access to my documents in Google Docs by synchronizing the state between my desktop and their cloud. Earlier this week Apple announced mobile.me which enables users to synchronize their contacts, emails, calendar and photos across the Web and all their devices.

Everywhere I look data synchronization is becoming more and more important and also more commonplace. I expect this trend to continue over time given the inevitable march of the Web. Being able to synchronize my data and my actions from my desktop to the Web or across Web sites I frequent is extremely enabling. Thus having a consistent  set of standards-based protocols for enabling these scenarios as well as libraries for the key platforms that make this approachable to developers will be very beneficial to users and to the growth of the Web. 

At the rate things are going, I personally believe that this vision of the Web will come to pass with or without Microsoft in the same way that Hailstorm's vision of the Web actually came to pass even though Microsoft canned the project. Whether Microsoft is an observer or a participant in this new world order depends on whether Live Mesh as a product and as a vision fully embraces the Web and collaboration with Web companies (as Google has ably done with GData/OpenSocial/FriendConnect/Gears/etc) or not. Only time will tell what happens in the end.

Now Playing: Usher - In This Club (remix) (feat. Beyonce & Lil Wayne)

Early this week, Microsoft announced a project code named Velocity. Velocity is a distributed in-memory object caching system in the vein of memcached (aka a Distributed Hash Table).  If you read any modern stories of the architectures of popular Web sites today such as the recently published overview of the LinkedIn social networking site's Web architecture, you will notice a heavy usage of in-memory caching to improve performance. Popular web sites built on the LAMP platform such as Slashdot, Facebook, Digg, Flickr and Wikipedia have all been using memcached to take advantage of in-memory storage to improve performance for years. It is good to see Microsoft step up with a similar technology for Web sites built on the WISC platform.

Like memcached, you can think of Velocity as a giant hash table that can run on multiple servers which automatically handles maintaining the balance of objects hashed to each server and transparently fetches/removes objects from over the network if they aren't on the same machine that is accessing an object in the hash table. In addition, you can add and remove servers from the cluster and the cache automatically rebalances itself.

The Velocity Logical Model

The following diagram taken from the Velocity documentation is helpful in discussing its logical model in detail

In the above diagram, each cache host is a server that participates in the cache cluster. Your application can have multiple named caches (e.g. "Inventory", "Product Catalog", etc) each of which can be configured separately. In addition, each named cache can have one or more named region. For example, the Sports region of your Product Catalog or the Arts region of your product catalog. Below is some sample code that shows putting and getting objects in and out of a named cache.

CacheFactory CacheCluster1 = new CacheFactory();
Cache inventoryCache = CacheCluster1.GetCache("Inventory");

Sneaker s = (Sneaker)inventoryCache.Get("NikeAirForce1");
s.price = s.price * 0.8; //20% discount
inventoryCache.Put("NikeAirForce1", s); 

Velocity ships with the ability to search for objects by tag but it is limited to objects within a specific region. So you can fetch all objects tagged "Nike" or "sneakers" from the sports region of your product catalog. As shown in the above diagram, a limitation of regions is that all items in a region must be on the same physical server. Below is an example of what the code for interacting with regions looks like

CacheFactory CacheCluster1 = new CacheFactory();
Cache catalog= CacheCluster1.GetCache("Catalog");
List <KeyValuePair<string, object>>  sneakers = catalog.GetByTag("Sports", "sneakers");

foreach (var kvp in sneakers)
{
   Sneaker s = kvp.Value as Sneaker;
   /* do stuff with Sneaker object */ 
}

The above sample searches for all items tagged "sneakers" from the Sports region of the Catalog cache.

The notion of regions and tagging is one place Velocity diverges from the simpler model of technologies like memcached and provides more functionality.

Eviction Policy and Explicit Object Expiration

Since memory is limited on a server, there has to be an eviction policy that ensures that the cache doesn't end up growing to big thus forcing the operating system to get all Virtual Memory on your ass by writing pages to disk. Once that happens you're in latency hell since fetching objects from the cache will involve going to disk to fetch them. Velocity gives you a couple of knobs that can be dialed up or down as needed to control how eviction or expiration of objects from the cache works. There is a file called ClusterConfig.xml which is used for configuring the eviction and expiration policy of each named cache instance. Below is an excerpt of the configuration file showing the policies for some named cache instances

<!-- Named cache list -->
 <caches>
   <cache name="default" type="partitioned">
     <policy>
       <eviction type="lru" />
       <expiration isExpirable="false" />
     </policy>
   </cache>
   <cache name="Inventory" type="partitioned">
     <policy>
       <eviction type="lru" />
       <expiration isExpirable="true" defaultTTL="50" />
     </policy>
   </cache>
 </caches>

The above excerpt indicates that the default and Inventory caches utilize a Least Recently Used algorithm for determining which objects are evicted from the cache. In addition, it specifies the default interval after which an object can be considered to be stale in the Inventory cache.

The default expiration interval can actually be overridden when putting an object in the cache by specifying a TTL parameter when calling the Put() method.

Concurrency Models: None, Optimistic, or Pessimistic

One of the first things you learn about distributed computing in the real world is that locks are bad mojo. In the way locks traditionally work, an object can be locked by a caller meaning everyone else interested in the object has to wait their turn. Although this prevents errors in your system occurring due to multiple callers interacting with the object at once, it also means there are built-in bottle necks in your system. So lesson #1 of scaling your service is often to get rid of as many locks in your code as possible. Eventually this leads to systems like eBay which doesn't use database transactions and Amazon's Dynamo which doesn't guarantee data consistency.

So what does this have to do with Velocity? Systems designed to scale massively like memcached don't support concurrency. This leads to developers asking questions like this one taken from the memcached mailing list

Consider this situation:-

• A list with numeric values: 1,2,3
• Server1: Gets the list from memcache.
• Server2: Also gets the list from memcache.
• Server1: Removes '1' from the list.
• Server2: Removes '3' from the list.
• Server1: Puts back a list with 2,3 in list in memcache.
• Server2: Puts back a list with 1,2 in list in memcache.

Note:Since, both servers have their instance of list objs.
This is not what we need to do. Becoz, both servers are putting an incorrect list in memcache.Ideally what shud have happened was that in the end a list with only '1' shud be put back in memcache. This problem occurs under load and happens in case of concurrent threads.
What I want is that memcache shud restrict Server2 and a consistent list shud be there in memcache. How do I handle such problem in memcache environment?? I know we can handle at application server end by doing all these operations through a centralized place(gets and puts), but how do I handle it in Memcache????
  Any help wud be appreciated?

Unfortunately for the author of the question above, memcached doesn't provide APIs for concurrent access and enforcing data consistency (except for numeric counters). So far, the code samples I've shown for Velocity also do not support concurrency.

However there are APIs for fetching or putting objects that support optimistic and pessimistic concurrency models. In the optimistic concurrency model, instead of taking a lock, the objects are given a version number and the caller is expected to specify the version number of the object they have modified when putting it back in the cache. If the object has been modified since the time it was retrieved then there is a version mismatch error. At this point, the caller is expected to re-fetch the object and make their changes to the newly retrieved object before putting it back in the cache. Below is a code sample taken from the Velocity documentation that illustrates what this looks like in code

        /* At time T0, cacheClientA and cacheClientB fetch the same object from the cache */ 

         //-- cacheClientA pulls the FM radio inventory from cache
         CacheFactory clientACacheFactory = new CacheFactory();
         Cache cacheClientA = clientBCacheFactory.GetCache("catalog");
         CacheItem radioInventory = cacheClientA.GetCacheItem("electronics", "RadioInventory");


         //-- cacheClientB pulls the same FM radio inventory from cache
         CacheFactory clientBCacheFactory = new CacheFactory();
         Cache cacheClientB = clientBCacheFactory.GetCache("catalog");
         CacheItem radioInventory = cacheClientA.GetCacheItem("electronics", "RadioInventory");


         //-- At time T1, cacheClientA updates the FM radio inventory
         int newRadioInventory = 155;
         cacheClientA.Put("electronics", "RadioInventory", newRadioInventory,
                          radioInventory.Tags, radioInventory.Version);

         //-- Later, at time T2, cacheClientB tries to update the FM radio inventory
         // AN ERROR OCCURS HERE 
         int newRadioInventory = 130;
         cacheClientB.Put("electronics", "RadioInventory", newRadioInventory,
                          radioInventory.Tags, radioInventory.Version);

In the pessimistic concurrency model, the caller specifically takes a lock by calling GetAndLock() with a lock time out. The lock is then held until the time out or until the object is put back using PutAndUnlock(). To prevent this from being a performance nightmare, the system does not block requests if a lock is held on an object they want to manipulate. Instead the request is rejected (i.e. it fails).

Update: Some people have commented here and elsewhere that memcached actually does support the optimistic concurrency model using the gets and cas commands. Sorry about that, it wasn't exposed in the memcached libraries I've looked at.

Final Thoughts

From my perspective, this is a welcome addition to the WISC developer's toolkit. I also like that it pushes the expectations of developers on what they should expect from a distributed object cache which I expect will end up being good for the industry overall and not just developers on Microsoft's platforms.

If the above sounds interesting, there is already a technology preview available for download from MSDN here. I've downloaded it but haven't tried to run it yet since I don't have enough machines to test it in the ways I would find interesting. As you can expect there is a Velocity team blog. Subscribed.

Now Playing: 50 Cent - These N*ggas Ain't Hood (feat. Lloyd Banks & Marquis)

Matt Asay of C|Net has an article entitled Facebook adopts the CPAL poison pill where he writes

Instead, by choosing CPAL, Facebook has ensured that it can be open source without anyone actually using its source. Was that the intent?

As OStatic explains, CPAL requires display of an attribution notice on derivative works. This practice, which effectively requires downstream code to carry the original developer(s)' logo, came to be known as "badgeware." It was approved by the OSI but continues to be viewed with suspicion within the open-source community.

I've written before about how most open-source licenses don't apply themselves well to the networked economy. Only the OSL, AGPL, and CPAL contemplate web-based services. It's not surprising that Facebook opted for one of these licenses, but I am surprised it chose the one least likely to lead to developers actually modifying the Facebook platform.

If the point was to protect the Facebook platform from competition (i.e., derivative works), Facebook chose a good license. If it was to encourage development, it chose the wrong license.

But if the purpose was to prevent modifications of the platform, why bother open sourcing it at all?

I've seen more than one person repeat the sentiment in the above article which leaves me completely perplexed. With fbOpen Facebook has allowed anyone who is interested to run Facebook applications and participate in what is currently the most popular & vibrant social network widget ecosystem in the world.

I can think of lots of good reasons for not wanting to adopt fbOpen. Maybe the code is in PHP and you are a Ruby On Rails shop. Or maybe it conflicts with your company's grand strategy of painting Facebook as the devil and you the heroes of openness (*cough* Google *cough*). However I can't see how requiring that you mention somewhere on your site that your social network's widget platform is powered by the Facebook developer platform is some sort of onerous POISON PILL which prevents you from using it. In the old days, companies used to charge you for the right to say your application is compatible with theirs, heck, Apple still does. So it seems pretty wacky for someone to call Facebook out for letting people use their code and encouraging them to use the Facebook brand in describing their product. Shoot!

The premise of the entire article is pretty ridiculous, it's like calling the BSD License a poison pill license because of the advertising clause. This isn't to say there aren't real issues with an advertising clause as pointed out in the GNU foundation's article The BSD License Problem. However as far as I'm aware,  adopters of fbOpen don't have to worry about being obligated to display dozens powered by X messages because every bit of code they depend on requires that it is similarly advertised. So that argument is moot in this case.

Crazy article but I've come to expect that from Matt Asay's writing.

Now Playing: Eminem & D12 - Sh*t On You

After weeks of preparatory work we are now really close to shipping the alpha version of the next release of RSS Bandit codenamed Phoenix. As you can see from the above screen shot, the key new feature is that you can read feeds from Google Reader, NewsGator Online and the Windows Common Feed List from RSS Bandit in a fully synchronized desktop experience.

This has been really fun code to write and I'm pretty sure I have a pending blog post in me about REST API design based on my experiences using the NewsGator REST API. The primary work items we have are around updating a bunch of the GUI code to realize that there are now multiple feed lists loaded and not just one. I estimate we'll have a version ready for our users to try out on the 14th or 15th of this month.

Your feedback will be appreciated.

Now Playing: Ben Folds - The Luckiest

Recently the folks behind Twitter came clean on the architecture behind the service and it is quite clear that the entire service is being held together by chewing gum and baling wire. Only three MySQL database servers for a service that has the I/O requirements of Twitter? Consider how that compares to other Web 2.0 sites that have come clean with their database numbers; Facebook has 1800, Flickr has 166, even Wikipedia has 20. Talk about bringing a knife to a gunfight.

Given the fact that Twitter has had scaling issues for over a year it is surprising that not only has it taken so long for them to figure out that they need a re-architecture but more importantly they decided that having a developer/sys admin manage fail over and traffic spikes by hand was cheaper to the business than buying more hardware and a few weeks of coding. 

A popular social networking that focuses on features instead of performance while upstart competitors are waiting in the wings? Sounds like a familiar song doesn't it? This entire episode reminds me of a story I read in the New York Times a few years ago titled The Wallflower at the Web Party which contains the following familiar sounding excerpts

But the board also lost sight of the task at hand, according to Kent Lindstrom, an early investor in Friendster and one of its first employees. As Friendster became more popular, its overwhelmed Web site became slower. Things would become so bad that a Friendster Web page took as long as 40 seconds to download. Yet, from where Mr. Lindstrom sat, technical difficulties proved too pedestrian for a board of this pedigree. The performance problems would come up, but the board devoted most of its time to talking about potential competitors and new features, such as the possibility of adding Internet phone services, or so-called voice over Internet protocol, or VoIP, to the site.
...
In retrospect, Mr. Lindstrom said, the company needed to devote all of its resources to fixing its technological problems. But such are the appetites of companies fixated on growing into multibillion-dollar behemoths. They seek to run even before they can walk.

“Friendster was so focused on becoming the next Google,” Professor Piskorski said, “that they weren’t focused on fixing the more mundane problems standing in the way of them becoming the next Google.”
...
“We completely failed to execute,” Mr. Doerr said. “Everything boiled down to our inability to improve performance.”

People said about Friendster the same thing they say about Twitter, we travel in tribes - people won't switch to Pownce or Jaiku because all their friends use Twitter. Well Friendster thought the same thing until MySpace showed up and now we have Facebook doing the same to them.

It is a very vulnerable time for Twitter and a savvy competitor could take advantage of that by adding a few features while courting the right set of influential users to jumpstart an exodus. The folks at FriendFeed could be that competitor but I suspect they won't. The Bret & Paul have already boxed their service into being an early adopter's play thing when there's actually interesting mainstream potential for their service. They'd totally put paid to their dreams of being a household brand if they end up simply being a Twitter knock off even if they could end up outplaying Evan and Biz at the game they invented.

Now Playing: Bob Marley - Redemption Song

The Live Search team has a blog post entitled Wikipedia Gets Big which reveals

Check it out:

We realize that often you just need to get a sense of what your query is about. Wikipedia is great for that — you can learn enough from the first paragraph of a Wikipedia article to start you out on the right path.

For Wikipedia results, we now show a good portion of the first paragraph and a few links from the table of contents. You can see more about the topic right there and see what else the article offers.

We hope you learn more, faster with our expanded Wikipedia descriptions. Let us know what you think.

After trying out on a few queries like "rain slick precipice", "wireshark" and "jeremy bentham" I definitely see this as a nice addition to the repertoire of features search engines use to give the right answer directly in the search results page. I've already found this to be an improvement compared to Google's habit of linking to definitions on Answer.com.

The interesting thing to note is just how often Wikipedia actually shows up in the top tier of search results for a diverse set of query terms. If you think this feature has legs why not leave a comment on the Live Search team's blog telling them what you think about it?

Now Playing: Abba - The Winner Takes It All

I've been having problems with hard drive space for years. For some reason, I couldn't get over the feeling that I had less available space on my hard drive than I could account for. I'd run programs like FolderSizes and after doing some back of the envelope calculations it would seem like I should have gigabytes more free space than what was actually listed as available according to my operating system.

Recently I stumbled on a blog post by Darien Nagle which claimed to answer the question Where's my hard disk space gone? with the recommendation that his readers should try WinDirStat. Seeing nothing to lose I gave it a shot and I definitely came away satisfied. After a quick install, it didn't take long for the application to track down where all those gigabytes of storage I couldn't account for had gone. It seems there was a hidden folder named C:\RECYCLER that was taking up 4 GigaBytes of space.

I thought that was kind of weird so I looked up the folder name and found Microsoft KB 229041 - Files Are Not Deleted From Recycler Folder which listed the following symptoms

SYMPTOMS
When you empty the Recycle Bin in Windows, the files may not be deleted from your hard disk.

NOTE: You cannot view these files using Windows Explorer, My Computer, or the Recycle Bin.

I didn't even have to go through the complicated procedure in the KB article to delete the files, I just deleted them directly from the WinDirStat interface.

My only theory as to how this happened is that some data got orphaned when I upgraded my desktop from Windows XP to Windows 2003 since the user accounts that created them were lost. I guess simply deleting the files from Windows Explorer as I did a few years ago wasn't enough.

Good thing I finally found a solution. I definitely recommend WinDirStat, the visualizations aren't half bad either.

Now Playing: Eminem - Never Enough (feat. 50 Cent & Nate Dogg)

A coworker forwarded me a story from a Nigerian newspaper about a cat turning into a woman in Port Harcourt, Nigeria. The story is excerpted below

This woman was reported to have earlier been seen as a cat before she reportedly turned into a woman in Port Harcourt, Rivers State, on Thursday. Photo: Bolaji Ogundele.

WHAT could be described as a fairy tale turned real on Wednesday in Port Harcourt, Rivers State, as a cat allegedly turned into a middle-aged woman after being hit by a commercial motorcycle (Okada) on Aba/Port Harcourt Expressway. Nigerian Tribune learnt that three cats were crossing the busy road when the okada ran over one of them which immediately turned into a woman. This strange occurrence quickly attracted people around who descended on the animals. One of them, it was learnt, was able to escape while the third one was beaten to death, still as a cat though. Another witness, who gave his name as James, said the woman started faking when she saw that many people were gathering around her. “I have never seen anything like this in my life. I saw a woman lying on the road instead of a cat. Blood did not come out of her body at that time. When people gathered and started asking her questions, she pretended that she did not know what had happened," he said.

Reading this reminds me how commonplace it was to read about the kind of mind boggling supernatural stories that you'd expect to see in the Weekly World News in regular newspapers alongside sports, political and stock market news in Nigeria.  Unlike the stories of alien abduction you find in the U.S., the newspaper stories of supernatural events often had witnesses and signed confessions from the alleged perpetrators of supernatural acts. Nobody doubted these stories, everyone knew they were true. Witches who would confess to being behind the run of bad luck of their friends & family or who'd confess that they key to their riches was offering their family members or children as blood sacrifices to ancient gods. It was all stuff I read in the daily papers as a kid as I would be flipping through looking for the comics. 

The current issue of Harper's Bazaar talks about the penis snatching hysteria from my adolescent years. The story is summarized in Slate magazine shown below

Harper's, June 2008
An essay reflects on the widespread reports of "magical penis loss" in Nigeria and Benin, in which sufferers claim their genitals were snatched or shrunken by thieves. Crowds have lynched accused penis thieves in the street. During one 1990 outbreak, "[m]en could be seen in the streets of Lagos holding on to their genitalia either openly or discreetly with their hand in their pockets." Social scientists have yet to identify what causes this mass fear but suspect it is what is referred to as a "culture-bound syndrome," a catchall term for a psychological affliction that affects people within certain ethnic groups.

I remember that time fairly well. I can understand that this sounds like the kind of boogie man stories that fill every culture. In rural America, it is aliens in flying saucers kidnapping people for anal probes and mutilating cows. In Japan, it's the shape changing foxes (Kitsune). In Nigeria, we had witches who snatched penises and could change shape at will.

In the cold light of day it sounds like mass hysteria but I wonder which is easier to believe sometimes. That a bunch of strangers on the street had a mass hallucination that a cat transformed into a woman or that there really are supernatural things beyond modern science's understanding out there? 

Now Playing: Dr. Dre - Natural Born Killaz (feat. Ice Cube)

When Google Gears was first announced, it was praised as the final nail in the coffin for desktop applications as it now made it possible to take Web applications offline. However in the past year that Gears has existed, there hasn't been as much progress or enthusiasm in taking applications offline as was initially thought when Gears was announced. There are various reasons for this and since I've already given my thoughts on taking Web applications offline so I won't repeat myself in this post. What I do find interesting is that many proponents of Google Gears including technology evangelists at Google have been gradually switching to pushing Gears as a way to route around browsers and add features to the Web, as opposed to just being about an 'offline solution'. Below are some posts from the past couple of months showing this gradual switch in positioning.

Alex Russell of the Dojo Toolkit wrote a blog post entitled Progress Is N+1 in March of this year that contained the following excerpt

Every browser that we depend on either needs an open development process or it needs to have a public plan for N+1. The goal is to ensure that the market knows that there is momentum and a vehicle+timeline for progress. When that’s not possible or available, it becomes incumbent on us to support alternate schemes to rev the web faster. Google Gears is our best hope for this right now, and at the same time as we’re encouraging browser venders to do the right thing, we should also be championing the small, brave Open Source team that is bringing us a viable Plan B. Every webdev should be building Gear-specific features into their app today, not because Gears is the only way to get something in an app, but because in lieu of a real roadmap from Microsoft, Gears is our best chance at getting great things into the fabric of the web faster. If the IE team doesn’t produce a roadmap, we’ll be staring down a long flush-out cycle to replace it with other browsers. The genius of Gears is that it can augment existing browsers instead of replacing them wholesale. Gears targets the platform/product split and gives us a platform story even when we’re neglected by the browser vendors.

Gears has an open product development process, an auto-upgrade plan, and a plan for N+1.

At this point in the webs evolution, I’m glad to see browser vendors competing and I still feel like that’s our best long-term hope. But we’ve been left at the altar before, and the IE team isn’t giving us lots of reasons to trust them as platform vendors (not as product vendors). For once, we have an open, viable Plan B.

Gears is real, bankable progress.

This was followed up by a post by Dion Almaer who's a technical evangelist at Google who wrote the following in his post Gears as a bleeding-edge HTML 5 implementation

I do not see HTML 5 as competition for Gears at all. I am sitting a few feet away from Hixie’s desk as I write this, and he and the Gears team have a good communication loop.

There is a lot in common between Gears and HTML 5. Both are moving the Web forward, something that we really need to accelerate. Both have APIs to make the Web do new tricks. However HTML 5 is a specification, and Gears is an implementation.
...
Standards bodies are not a place to innovate, else you end up with EJB and the like.
...
Gears is a battle hardened Web update mechanism, that is open source and ready for anyone to join and take in interesting directions.

and what do Web developers actually think about using Google's technology as a way to "upgrade the Web" instead of relying on Web browsers and standards bodies for the next generation of features for the Web? Here's one answer from Matt Mullenweg, founder of WordPress, taken from his post Infrastructure as Competitive Advantage 

When a website “pops” it probably has very little to do with their underlying server infrastructure and a lot to do with the perceived performance largely driven by how it’s coded at the HTML, CSS, and Javascript level. This, incidentally, is one of the reasons Google Gears is going to change the web as we know it today - LocalServer will obsolete CDNs as we know them. (Look for this in WordPress soonish.)

That's a rather bold claim (pun intended) by Matt. If you're wondering what features Matt is adding to WordPress that will depend on Gears, they were recently discussed in Dion Almaer's post Speed Up! with Wordpress and Gears which is excerpted below

However, Gears is so much more than offline, and it is really exciting to see “Speed Up!” as a link instead of “Go Offline?”

This is just the beginning. As the Gears community fills in the gaps in the Web development model and begins to bring you HTML5 functionality I expect to see less “Go Offline” and more “Speed Up!” and other such phrases. In fact, I will be most excited when I don’t see any such linkage, and the applications are just better.

With an embedded database, local server storage, worker pool execution, desktop APIs, and other exciting modules such as notifications, resumable HTTP being talked about in the community…. I think we can all get excited.

Remember all those rumors back in the day that Google was working on their own browser? Well they've gone one better and are working on the next Flash. Adobe likes pointing out that Flash has more market share than any single browser and we all know that has Flash has gone above and beyond the [X]HTML standards bodies to extend the Web thus powering popular, rich user experiences that weren't possible otherwise (e.g. YouTube). Google is on the road to doing the same thing with Gears. And just like social networks and content sharing sites were a big part in making Flash an integral part of the Web experience for a majority of Web users, Google is learning from history with Gears as can be seen by the the recent announcements from MySpace. I expect we'll soon see Google leverage the popularity of YouTube as another vector to spread Google Gears.  

So far none of the Web sites promoting Google Gears have required it which will limit its uptake. Flash got ahead by being necessary for sites to even work. It will be interesting to see if or when sites move beyond using Gears for nice-to-have features and start requiring it to function. It sounds crazy but I never would have expected to see sites that would be completely broken if Flash wasn't installed five years ago but it isn't surprising today (e.g. YouTube).

PS: For those who are not familiar with the technical details of Google Gears, it currently provides three main bits of functionality; thread pools for asynchronous operations, access to a SQL database running on the user's computer, and access to the user's file system for storing documents, images and other media. There are also beta APIs which provide more access to the user's computer from the browser such as the Desktop API which allows applications to create shortcuts on the user's desktop.  

Now Playing: Nas - It Ain't Hard To Tell