Via Sam Ruby's post Embrace, Extend then Innovate I found a link to Joe Gregorio's post entitled How to do RESTful Partial Updates. Joe's post is a recommendation of how to extend the Atom Publishing Protocol (RFC 5023) to support updating the properties of an entry without having to replace the entire entry. Given that Joe works for Google on GData, I have assumed that Joe's post is Google's attempt to float a trial balloon before extending AtomPub in this way. This is a more community centric approach than the company has previously taken with GData, OpenSocial, etc where these protocols simply appeared out of nowhere with proprietary extensions to AtomPub with an FYI to the community after the fact.

The Problem Statement

In the Atom Publishing Protocol, an atom:entry represents an editable resource. When editing that resource, it is intended that an AtomPub client should download the entire entry, edit the fields it needs to change and then use a conditional PUT request to upload the changed entry.

So what's the problem? Below is an example of the results one could get from invoking the users.getInfo method in the Facebook REST API.

   <user>
    <uid>8055</uid>
    <about_me>This field perpetuates the glorification of the ego.  Also, it has a character limit.</about_me>
    <activities>Here: facebook, etc. There: Glee Club, a capella, teaching.</activities>   
    <birthday>November 3</birthday>
    <books>The Brothers K, GEB, Ken Wilber, Zen and the Art, Fitzgerald, The Emporer's New Mind, The Wonderful Story of Henry Sugar</books>
    <current_location>
      <city>Palo Alto</city>
      <state>CA</state>
      <country>United States</country>
      <zip>94303</zip>
    </current_location>   
    <first_name>Dave</first_name>      
     <interests>coffee, computers, the funny, architecture, code breaking,snowboarding, philosophy, soccer, talking to strangers</interests>
     <last_name>Fetterman</last_name>  
     <movies>Tommy Boy, Billy Madison, Fight Club, Dirty Work, Meet the Parents, My Blue Heaven, Office Space </movies>
     <music>New Found Glory, Daft Punk, Weezer, The Crystal Method, Rage, the KLF, Green Day, Live, Coldplay, Panic at the Disco, Family Force 5</music>
     <name>Dave Fetterman</name> 
     <profile_update_time>1170414620</profile_update_time>
     <relationship_status>In a Relationship</relationship_status>
     <religion/>
     <sex>male</sex>
     <significant_other_id xsi:nil="true"/>
     <status>
       <message>Pirates of the Carribean was an awful movie!!!</message>
     </status>   
   </user>

If this user was represented as an atom:entry then each time an application wants to edit the user's status message it needs to download the entire data for the user with its over two dozen fields, change the status message in an in-memory representation of the XML document and then upload the entire user atom:entry back to the server.  This is a fairly expensive way to change a status message compared to how this is approached in other RESTful protocols (e.g. PROPPATCH in WebDAV).

Previous Discussions on this Topic: When the Shoe is on the Other Foot

A few months ago I brought up this issue as one of the problems encountered when using the Atom Publishing Protocol outside of blog editing contexts in my post Why GData/APP Fails as a General Purpose Editing Protocol for the Web. In that post I wrote

Lack of support for granular updates to fields of an item: As mentioned in the previous section editing an entry requires replacing the old entry with a new one. The expected client interaction with the server is described in section 5.4 of the current APP draft and is excerpted below.

Retrieving a Resource

              Client Server

| |

| 1.) GET to Member URI |

|------------------------------------------>|

| |

| 2.) 200 Ok |

| Member Representation |

|<------------------------------------------|

| |

            

1. The client sends a GET request to the URI of a Member Resource to retrieve its representation.
2. The server responds with the representation of the Member Resource.

Editing a Resource

              Client Server

| |

| 1.) PUT to Member URI |

| Member Representation |

|------------------------------------------>|

| |

| 2.) 200 OK |

|<------------------------------------------|

            

The client sends a PUT request to store a representation of a Member Resource.

If the request is successful, the server responds with a status code of 200.

Can anyone spot what's wrong with this interaction? The first problem is a minor one that may prove problematic in certain cases. The problem is pointed out in the note in the documentation on Updating posts on Google Blogger via GData which states

IMPORTANT! To ensure forward compatibility, be sure that when you POST an updated entry you preserve all the XML that was present when you retrieved the entry from Blogger. Otherwise, when we implement new stuff and include <new-awesome-feature> elements in the feed, your client won't return them and your users will miss out! The Google data API client libraries all handle this correctly, so if you're using one of the libraries you're all set.

Thus each client is responsible for ensuring that it doesn't lose any XML that was in the original atom:entry element it downloaded. The second problem is more serious and should be of concern to anyone who's read Editing the Web: Detecting the Lost Update Problem Using Unreserved Checkout. The problem is that there is data loss if the entry has changed between the time the client downloaded it and when it tries to PUT its changes.

That post was negatively received by many members of the AtomPub community including Joe Gregorio. Joe wrote a scathing response to my post entitled In which we narrowly save Dare from inventing his own publishing protocol  where he addressed that particular issue as follows

The second complaint is one of data loss:

The problem is that there is data loss if the entry has changed between the time the client downloaded it and when it tries to PUT its changes.

Fortunately, the only real problem is that Dare seems to have only skimmed the specification. From Section 9.3 :

To avoid unintentional loss of data when editing Member Entries or Media Link Entries, Atom Protocol clients SHOULD preserve all metadata that has not been intentionally modified, including unknown foreign markup as defined in Section 6 of [RFC4287].

And further, from Section 9.5 :

Implementers are advised to pay attention to cache controls, and to make use of the mechanisms available in HTTP when editing Resources, in particular entity-tags as outlined in [NOTE-detect-lost-update]. Clients are not assured to receive the most recent representations of Collection Members using GET if the server is authorizing intermediaries to cache them.

Hey look, we actually reference the lost update paper that specifies how to solve this problem, right there in the spec! And Section 9.5.1 even shows an example of just such a conditional PUT failing. Who knew? And just to make this crystal clear, you can build a server that is compliant to the APP that accepts only conditional PUTs. I did , and it performed quite well at the last APP Interop .

The bottom line of Joe's response is that he didn't think it was a real problem. My assumption is that his perspective on the problem has broadened now that he has a responsibility to the wide breadth of AtomPub implementations at Google as opposed to when his design decisions were being influenced by a home grown blogging server he wrote in his free time.

The Google Solution: Embrace, Extend then Innovate

Now that Joe thinks supporting granular updates of a resource is a valid scenario, he and the folks at Google have proposed the following solution to the problem. Joe writes

Now if I wanted to update part of this entry, say the title, using the mechanisms in RFC 5023 then I would change the value of the title element and PUT the whole modified entry back to the the URI http://example.org/edit/first-post.atom. Now this document isn't large, but we'll use it to demonstrate the concepts. The first thing we want to do is add a URI Template that allows us to construct a URI to PUT changes back to:

            <?xml version="1.0"?> <entry xmlns="http://www.w3.org/2005/Atom" xmlns:t="http://blah..."><t:link_template
ref="sub" href="http://example.org/edit/first-post/{-listjoin|;|id}"/> <title>Atom-Powered
Robots Run Amok</title> <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
<updated>2003-12-13T18:30:02Z</updated> <author><name>John
Doe</name></author> <content>Some text.</content> <link
rel="edit" href="http://example.org/edit/first-post.atom"/>
</entry>
          

Then we need to add id's to each of the pieces of the document we wish to be able to individually update. For this we'll use the W3C xml:id specification:

            <?xml version="1.0"?> <entry xmlns="http://www.w3.org/2005/Atom"
xmlns:t="http://blah..."> <t:link_template ref="sub" href="http://example.org/edit/first-post/{-listjoin|;|id}"/>
<title xml:id="X1">Atom-Powered Robots Run Amok</title>
<id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id> <updated>2003-12-13T18:30:02Z</updated>
<author xml:id="X2"><name>John Doe</name></author>
<content xml:id="X3">Some text.</content> <link
rel="edit" href="http://example.org/edit/first-post.atom"/>
</entry>
          

So if I wanted to update both the content and the title I would construct the partial update URI using the id's of the elements I want to update:

http://example.org/edit/first-post/X1;X3

And then I would PUT an entry to the URI with only those child elements:

            PUT /edit/first-post/X1;X3 Host: example.org <?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom"> <title xml:id="X1">False
alarm on the Atom-Powered Robots things</title> <content xml:id="X3">Sorry
about that.</content> </entry>
          

The Problems with the Google Solution: Your Shipment of FAIL has Arrived

Ignoring the fact that this spec depends on specifications that are either experimental (URI Templates) or not widely supported (xml:id), there are still significant problems with how this approach (mis)uses the Atom Publishing Protocol. Sam Ruby eloquently points out the problems in his post Embrace, Extend then Innovate where he wrote

With HTTP PUT, the the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server.  Having some servers interpret the removal of elements (such as content) as a modification, and others interpret the requests in such a way that elided elements are to be left alone is hardly uniform or self-descriptive.  In fact, depending on usage, it is positively stateful.

I’m fine with a server choosing to interpret the request anyway it sees fit.  As a black box, it could behave as if it updated the resource as requested and then one nanosecond later — and before it processes any other requests — fill in missing data with defaults, historical data, whatever.  My concern is with clients coding with to the assumption as to how the server works.  That’s called coupling.

The main problem is that it changes the expected semantics of HTTP PUT in a way that not only conflicts with how PUT is typically used in other HTTP-based protocols but also how it is used in AtomPub. It's also weird that the existence of xml:id in an Atom document is now used to imply special semantics (i.e. this field supports direct editing). I especially don't like that after all is said and done, the server controls which fields can be partially updated or not which seems to imply a tight coupling between clients and servers (e.g. some servers will support partial updates on all fields, some may only support partial updates on atom:title + atom:category while others will support partial updates on a different set of fields). So the code for editing a title or category changes depending on which AtomPub service you are talking to.  

From where I stand Joe has pretty much invented yet another diff + patch protocol for XML documents. When I worked on the XML team at Microsoft, there were quite a few floating around the company including Diffgram, UpdateGram, and Patchgrams to name three. So I've been around the block when it comes to diff + patch formats for XML and this one has its share of issues.  The most eye brow raising issue with the diff + patch protocol is that half the semantics of the update are in the XML document (which elements to add/edit) while the other half are in the URL (if an ID exists in the URL but is not in the document then it is a delete). This means the XML isn't very self describing nor can it really be said that the URL is identifying a resource [more like it identifies an operation].

Actual Solution: Read the Spec

In Joe's original response to my post his suggestion was that the solution to the "problem" of lack of support for granular updates of entries in AtomPUb is to read the spec. In retrospect, I agree. If a field is important enough that it needs to be identifiable and editable then it should be its own resource. If you want to make it part of another resource then use atom:link to link both resources.

Case closed. Problem solved.

I just realized that the current released version of RSS Bandit doesn’t have a working code name based on a character from the X-Men comic book. The previous 1.5.0.17 release was codenamed ShadowCat while the next release is codenamed Phoenix. Since the v1.6.0.x releases have been an interim releases on the road to Phoenix, I’ve decided to give them the codename Jean Grey retroactively. Now, on to the updates.

Jean Grey (v1.6.0.x) Update

The last bug fix release of RSS Bandit fixed a few bugs but introduced a couple of even worse bugs [depending on your perspective]. We’ve shipped version 1.6.0.2 that addresses the following issues

• Application crashes with AccessViolationException on startup on Windows XP.  
• Application crashes and red 'X' shows in feed subscriptions window on Windows XP.  
• User's credentials are not used when accessing feeds via a proxy server leading to proxy errors when fetching feeds.  
• Duplicate feed URLs not detected if they differ by trailing slash or "www." in the host name 
• Application crashes when displaying an error dialog when a certificate issue is detected with a secure feed.

The first three issues are regressions that were introduced as part of refactoring the code and making it work better on Windows Vista. Yet another data point that shows that you can never have too many unit tests and that beta testing isn’t a bad idea either.

You can download the new release from http://downloads.sourceforge.net/rssbandit/RssBandit1.6.0.2_Installer.zip

Phoenix (v2.0)  Update

I’m continuing with my plan to make RSS Bandit a desktop client for Web based feed readers like NewsGator Online and Google Reader. I’ve been slightly sidetracked by the realization that it would be pretty poor form for a Microsoft employee to write an application that synchronized with Google’s RSS reader but not any of Microsoft’s, even if it is a side project. My current coding project is to integrate with the Windows RSS platform which would allow one to manipulate the same set of feeds in RSS Bandit, Internet Explorer 7 and Outlook 2007. The good news is that with Outlook 2007 integration, you also get Exchange synchronization for free.

The bad news has been having to use the RSS reading features of Internet Explorer 7 and Outlook 2007 on a regular basis as a way of eating my own dog food with regards to the integration features. It’s pretty stunning to see not one but two RSS reading applications that assume “mark all items as read” or “delete all feeds” are actions that users never have to take. When you have people writing shell scripts to perform basic tasks in your application then it is a clear sign that somewhere along the line, the user experience for that particular set of features got the shaft. 

I’m about half way through the integration after which I’ll continue with integrating with Google Reader and finally NewsGator Online using an Outlook + Exchange style model. While I’m working on this, both Oren and Torsten will be mapping out the rewrite of the graphical user interface using WPF. I’ll probably need to buy a book on XAML or something in the next few months so I can contribute to this effort. The only thing I’ve heard about any of the various books about the subject on the market is that they all seem to have had their forewords written by Don Box. Does anyone have recommendations on which book or website I should use to start learning XAML + WPF?

Now playing: Eminem - Sing For The Moment

It seems I missed the news when this happened last week but according to Greg Reinacker’s post NewsGator’s RSS clients are now free! 

We’ve got a lot of big news today at NewsGator.

First, we’ve got new releases of our most popular applications: FeedDemon 2.6 , NetNewsWire 3.1 , Inbox 3.0 (beta) , and NewsGator Go! for Windows Mobile 2.0 . Each of these is a pretty major release on its own - tons of new features in all of them.

But second, every one of those applications is now free! Free as in beer, that is. And add to the free list NewsGator Go! for BlackBerry as well. And not only are they free, but our online services (including synchronization) are now free as well! Not to mention our iPhone reader , HTML mobile reader , and all of the other applications that are part of our online platform.

According to Greg and Nick Bradbury, the reason they are doing this is because the bulk of their profits/revenues come from selling Enterprise licenses and the desktop readers are now being used as advertising to get enterprise customers.

They also mention that the other reason they are giving away their desktop application is that they see a lot of financial value from collecting information about what feeds their users are reading. My assumption was that this is because the demographic data is being resold to marketers although both Greg and Nick make it seem like the collection of this data is benign and only used for end user facing features. 

Anyway, this is pretty great news for fans of desktop RSS readers. If I didn’t already have RSS Bandit, FeedDemon would be my first choice when it comes to a desktop RSS reader for Windows. I also like the fact that we get a shout out as part of the setup experience for FeedDemon which is shown below

Thanks for the shout out Nick.

Now that these apps are free, it does encourage us to step our game up with RSS Bandit. Right now, my thinking is that official version number for the release currently codenamed Phoenix will be RSS Bandit 2.0. This release will be deserve the monicker “2.0” for two reasons

1. The user interface will be completely rewritten from scratch by Oren and Torsten using WPF.
2. The application will become capable of being a full-blown desktop client to both Google Reader and NewsGator Online.

If I seem to have been blogging less, it is because I’ve been spending more time reading about code, thinking about code and writing code in my free time. I can’t wait to get the first beta out to you guys in a few months.

Now playing: Kid Rock - Welcome To The Party (Ode 2 the Old School)

The top story in my favorite aggregator today is the announcement on Scott Guthrie’s blog of the ASP.NET 3.5 Extensions CTP Preview. Normally, announcements related to ASP.NET would not interest me except this time is an interesting item in the list of technologies being released

ADO.NET Data Services: In parallel with the ASP.NET Extensions release we will also be releasing the ADO.NET Entity Framework.  This provides a modeling framework that enables developers to define a conceptual model of a database schema that closely aligns to a real world view of the information.  We will also be shipping a new set of data services (codename "Astoria") that make it easy to expose REST based API endpoints from within your ASP.NET applications.

Wow. It looks like Astoria has quickly moved from being an experimental project to see what it would like to place RESTful interfaces on top of SQL Server database to being very close to shipping a production version.  I dug around for more posts about Astoria ADO.NET Data Services so I could find out what was in the CTP and came across two posts from Mike Flasko and Andy Conrad respectively.

In his post entitled ADO.NET Data Services ("Project Astoria") CTP is Released on the ADO.NET Data Services team blog Mike Flasko writes

The following features are in this CTP:

• Support to create ADO.NET Data Services backed by:
  • A relational database by leveraging the Entity Framework . 
  • Any data source (file, web service, custom store, application logic layer, etc)
• Serialization Formats:
• Industry standard AtomPub serialization
• JSON serialization
• Simple HTTP interface
  • Any platform with an HTTP stack can easily consume a data service
  • Designed to leverage HTTP semantics and infrastructure already deployed at large
• Client libraries:
  • .NET Framework
  • ASP.NET AJAX
  • Silverlight (coming soon)

This is sick. With Astoria I can expose my relational database or even a local just an XML file using a RESTful interface that utilizes the Atom Publishing Protocol or JSON. I am somewhat amused that one of the options is placing a RESTful interface over a SOAP Web Service. My, how times have changed…

It is pretty cool that Microsoft is the first major database vendor to bring the dream of the Atom store to fruition. I also like that one of the side effects of this is that there is now an AtomPub client library for .NET Framework.

Andy Conrad has a blog post entitled Linq to REST which gives an idea of what happens when you combine the Astoria client library with the Language Integrated Query (LINQ) features of C# 3.0

               [OpenObject("PropBag")] publicclass Product{ private Dictionary<string, object>
propBag = new Dictionary<string, object>();
[Key] publicint ProductID { get;
set; } 
              
                public
                string ProductName
{ get; set; } publicint UnitsInStock
{ get; set; } public IDictionary<string, object>
PropBag { get { return propBag; } } }

            

        staticvoid Main(string[] args){
            WebDataContext context = new WebDataContext("http://localhost:18752/Northwind.svc");
            var query = from p in context.CreateQuery<Product>("Products")
                        where p.UnitsInStock > 100
                        select p;

            foreach (Product p in query){
                Console.WriteLine(p.ProductName + " , UnitsInStock= " + p.UnitsInStock);
            }

        } 

If you hover over the query variable, you will actually see the Astoria URI which the Linq query is translated into by the Astoria client library:

http://localhost:18752/Northwind.svc/Products?$filter=(UnitsInStock)%20gt%20(100)

So, there you go.  Linq to Astoria's RESTFUL API.  In other words, Linq to REST. 

Like I said earlier, this is sick. I need to holla at Andy and see if there is a dependency on the Atom feed containing Microsoft specific extensions or whether this Linq to REST capability can be utilized over any arbitrary Atom feed.

Now playing: Jay-Z - Success (feat. Nas)

Disclaimer: This post does not reflect the opinions, thoughts, strategies or future intentions of my employer. These are solely my personal opinions. If you are seeking official position statements from Microsoft, please go here.

One of the Google folks working on OpenSocial sent me a message via Facebook asking what I thought about the technical details of the recent announcements. Since my day job is working on social networking platforms for Web properties at Microsoft and I'm deeply interested in RESTful protocols, this is something I definitely have some thoughts about. Below is what started off as a private message but ended up being long enough to be it's own blog post.

First Impressions

In reading the OpenSocial API documentation it seems clear that is intended to be the functional equivalent of the Facebook platform. Instead of the Facebook users and friends APIs, we get the OpenSocial People and Friends Data API. Instead of the Facebook feed API, we get the OpenSocial Activities API. Instead of the Facebook Data Store API, we get the OpenSocial Persistence Data API. Instead of FQL as a friendly alternative to the various REST APIs we get a JavaScript object model.  

In general, I personally prefer the Facebook platform to OpenSocial. This is due to three reasons

• There is no alternative to the deep integration into the Web site's user experience that is facilitated with FBML.  
• I prefer idiomatic XML to tunnelling data through Atom feeds in ways that [in my opinion] add unnecessary cruft.
• The Facebook APIs encourage developers to build social and item relationship graphs within their application while the OpenSocial seems only concerned with developers stuffing data in key/value pairs.

The Javascript API

At first I assumed the OpenSocial JavaScript API would provide similar functionality to FBML given the large number of sound bites quoting Google employees stating that instead of "proprietary markup" you could use "standard JavaScript" to build OpenSocial applications. However it seems the JavaScript API is simply a wrapper on top of the various REST APIs. I'm sure there's some comment one could make questioning if REST APIs are so simple why do developers feel the need to hide them behind object models?

Given the varying features and user interface choices in social networking sites, it is unsurprising that there is no rich mechanism specified for adding entry points to the application into the container sites user interface. However it is surprising that no user interface hooks are specified at all. This is surprising given that there are some common metaphors in social networking sites (e.g. a profile page, a friends list, etc) which can be interacted with in a standard way.  It is also shocking that Google attacked Facebook's use of "proprietary markup" only to not even ship an equivalent feature.

The People and Friends Data API 

The People and Friends Data API is used to retrieve information about a user or the user's friends as an Atom feed. Each user is represented as an atom:entry which is a PersonKind (which should not be confused with an Atom person construct). It is expected that the URL structure for accessing people and friends feeds will be of the form  http://<domain>/feeds/people/<userid> and http://<domain>/feeds/people/<userid>/friends respectively.

Compare the following response to a request for a user's information using OpenSocial with the equivalent Facebook API call response.

GET http://orkut.com/feeds/people/14358878523263729569

<entry xmlns='http://www.w3.org/2005/Atom' xmlns:georss='http://www.georss.org/georss' xmlns:gd='http://schemas.google.com/g/2005'>

<id>http://sandbox.orkut.com:80/feeds/people/14358878523263729569</id>

<updated>2007-10-28T14:01:29.948-07:00</updated>

<title>Elizabeth Bennet</title>

<link rel='thumbnail' type='image/*' href='http://img1.orkut.com/images/small/1193601584/115566312.jpg'/>

<link rel='alternate' type='text/html' href='http://orkut.com/Profile.aspx?uid=17583631990196664929'/>

<link rel='self' type='application/atom+xml' href='http://sandbox.orkut.com:80/feeds/people/14358878523263729569'/>

<georss:where>

<gml:Point xmlns:gml='http://www.opengis.net/gml'>

<gml:pos>51.668674 -0.066235</gml:pos>

</gml:Point>

</georss:where>

<gd:extendedProperty name='lang' value='en-US'/>

<gd:postalAddress/>

</entry>

Below is the what the above information would look like if returned by Facebook's users.getInfo method

GET

<users_getInfo_response xmlns="http://api.facebook.com/1.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://api.facebook.com/1.0/ http://api.facebook.com/1.0/facebook.xsd" list="true">

<user>

<uid>14358878523263729569</uid>

<current_location>

<city>Palo Alto</city>

<state>CA</state>

<country>United States</country>

<zip>94303</zip>

</current_location>

<first_name>Elizabeth</first_name>

<is_app_user>1</is_app_user>

<has_added_app>1</has_added_app>

<pic>http://photos-055.facebook.com/ip007/profile3/1271/65/s8055_39735.jpg</pic>

</user>

</users_getInfo_response>

I've already mentioned that I prefer idiomatic XML to tunnelling data through Atom feeds. Comparing the readability of both examples should explain why.

The Activities Data API 

A number of social networking sites now provide a feature which enables users to see the recent activities of members of their social network in an activity stream. The Facebook news feed, Orkut's updates from your friends, and the Windows Live Spaces what's new page are all examples of this feature. The OpenSocial Activities Data API provides a mechanism for OpenSocial applications to access and update this activity stream as an Atom feed. All of the users activities or all activities from a specific application can be accessed using URIs of the form  http://<domain>/activities/feeds/activities/user/<userID> and http://<domain>/activities/feeds/activities/user/<userID>/source/<sourceID> respectively.  

Currently there is no reference documentation on this API. My assumption is that since Orkut is the only OpenSocial site that supports this feature, it is difficult to produce a spec that will work for other services without it being a verbatim description of Orkut's implementation.

There are some notes on how Orkut attempts to prevents applications from spamming a user's activity stream. For one, applications are only allowed to update the activity stream for their source directly instead of the activity stream for the user. I assume that Google applies some filter to the union of all the source specific activity streams before generating the user's activity feed to eliminate spam. Secondly, applications are monitored to see if they post too many messages to the activity stream or if they post promotional messages instead of the user's activities to the stream. All of this makes it seem difficult to see how one could specify the behavior of this API and feature set reliably for a diverse set of social networking sites.

The Persistence Data API 

The OpenSocial Persistence API allows applications to store and retrieve key<->value pairs that are either user-specific or are global to the application. An example of the former is a listing of company name and stock ticker pairs while an example of the latter is a user's stock portfolio. The feed of global key<->value pairs for an application can be accessed at a URL of the form http://<domain>/feeds/apps/<appID>/persistence/global for the entire feed and http://<domain>/feeds/apps/<appID>/persistence/global/<key> if seeking a particular key<->value pair. User-specific key<->value pairs are available at the URL of the form http://<domain>/feeds/apps/<appID>/persistence/<userID>/instance/<instanceID>.

This is probably the least interesting aspect of the API. A simple persistence API like this is useful for applications with simple storage needs that need to store user preferences or simple textual data that is needed by the application. However you aren't going to use this as the data storage platform for applications like iLike, Flixster or Scrabulous.

However I will add that an Atom feed seems like a horrible representation for a list of key<->value pairs. It's so bad that the documentation doesn't provide an example of such a feed.

Hosting OpenSocial Applications

The documentation on hosting OpenSocial applications implies that any site that can host Google gadgets can also host OpenSocial applications. In practice, it means that any site that you can place a <script> element on can point to a gadget and thus render it. Whether the application will actually work will depend on whether the hosting service has actually implemented the OpenSocial Service Provider Interface (SPI).

Unfortunately, the documentation on implementing the OpenSocial SPI is missing in action. From the Google site

To host OpenSocial apps, your website must support the SPI side of the OpenSocial APIs. Usually your SPI will connect to your own social network, so that an OpenSocial app added to your website automatically uses your site's data. However, it is possible to use data from another social network as well, should you prefer. Soon, we will provide a development kit with documentation and code to better support OpenSocial websites, along with a sample sandbox which implements the OpenSocial SPI using in-memory storage. The SPI implements:

• Adding and removing friends
• Adding and removing apps
• Storing activities
• Retrieving activity streams for self and friends
• Storing and retrieving per-app and per-app-per-user data

The OpenSocial website development kit will include full SPI documentation. It will provide open source reference implementations for both client and server components.

I assume that the meat of the OpenSocial SPI is documentation is just more detailed rules about how to implement the REST APIs described above. The interesting bits will likely be the reference implementations of the API which will likely become the de facto standard implementations instead of encouraging dozens of buggy incompatible versions of the OpenSocial API to bloom.   

Conclusion

In general I believe that any effort to standardize the widget/gadget APIs exposed by various social networking sites and AJAX homepages (e.g. iGoogle, Netvibes, Live.com, etc) is a good thing. Niall Kennedy has an excellent series of articles on Web Widget formats and Web Widget update technologies that shows how diverse and disparate the technologies that developers have to learn and utilize when they want to build widgets for various sites. Given that Web widgets are now a known quantity, the time is ripe for some standardization.

That said, there are a number of things that give me cause to pause with regards to OpenSocial

1. A common practice in the software industry today is to prefix "Open" to the name of your technology which automatically gives it an aura of goodness while attempting to paint competing technologies as being evil and "closed". Examples include OpenDocument, OpenID, OpenXML, OAuth, etc. In this case, OpenSocial is being positioned as an "open" alternative to the Facebook platform.  However as bloggers like Shelley Powers, Danny Ayers and Russell Beattie have pointed out, there isn't much "open" about OpenSocial. Russell Beattie asks in his post Where the hell is the Container API?

   Would people be jumping on this bandwagon so readily if it was Microsoft unilaterally coming up with an API, holding secret meetings geared towards undercutting the market leader, and then making sure that only those anointed partners get a head start on launch day by making sure a key part of the API isn't released - even in alpha. (It obviously exists already, all the partners have that spec and even sample code, I'm sure. The rest of us don't get access yet, until the GOOG says otherwise).

   Let's say we ignore that the process for creating the technology was not "open" nor have key aspects of the technology even been unveiled [which makes this more of a FUD announcement to take the wind out of Facebook's sails than an actual technology announcement], is the technology itself open? Shelley Powers points out her post Terms that

   Perhaps the world will read the terms of use of the API, and realize this is not an open API; this is a free API, owned and controlled by one company only: Google. Hopefully, the world will remember another time when Google offered a free API and then pulled it. Maybe the world will also take a deeper look and realize that the functionality is dependent on Google hosted technology, which has its own terms of service (including adding ads at the discretion of Google), and that building an OpenSocial application ties Google into your application, and Google into every social networking site that buys into the Dream.

   Google has announced a technology platform that is every bit as proprietary as Facebook's. The only difference is that they've cut deals with some companies to utilize their proprietary platform while Facebook's platform is only for use on the Facebook site. If Zuckerburg announces next week that the Facebook platform is freely implementable by any 3rd party Web site, where does that leave OpenSocial? After all, the Facebook platform is actually a proven, working system with complete documentation instead of the incomplete rush job that OpenSocial clearly is right now.

   There are all sorts of forums for proposing and discussing open Web technologies including the IETF, W3C, OASIS and even ECMA. Until all of the underlying technologies in OpenSocial have been handed over to one or more of these standards bodies, this is a case of the proprietary pot calling the proprietary kettle black.

2. One of the things that comes along with OpenSocial is that Google has now proposed GData as the standard protocol for interacting with social graphs on the Web. This is something that I've been worried about for a while and I've written a couple of blog posts to address this topic because it is not clear that the Atom Publishing Protocol upon which GData is based works well outside it's original purpise of editing blog posts and the like. I'm not the only one that feels this way.

   Danny Ayers wrote in his post Open? Social?

   However the People Data API is cruel and unusual. It first stretches Atom until it creaks with "each entry in the People or Friends feed is a PersonKind"; then gives a further tug  (a person's name is represented using atom:title) then extends it even more (a person's email is gd:email) and finally mops up all the blood, sweat and dribble:

   Key value parameters - gd:extendedProperty - "As different social networks and other sources of People data have many different named fields, this provides a way for them to be passed on generally. Agreeing on common naming conventions is to be decided in future."

   Got to admire the attempt, but (to mix the metaphorical namespaces) silk purses don't make very good sow's ears either.

   In addition, AtomPub geek extraordinairre, Tim Bray wrote in his blog post entitled Web3S

   If you decide you totally can’t model your world as collections of entries populated with hyperlinks to express relationships, well then I guess APP’s not for you. And at the level of engineering intuition, I have to say that a monster online address book does feel different at a deep level from most online “publications” (I thought that was why we had LDAP... but I repeat myself).

   Now that we have AtomPub/GData as a de facto standard protocol for accessing various kinds of non-microcontent data on the Web as a reality, I'm done debating its suitability for the task since the horse has already left the barn. However I will continue to ask when will GData be RFC 5023 compliant?

3. At the end of the day, the most disappointing thing about OpenSocial is that it doesn't really further the conversation about actual interoperability across social networking sites. If I use Orkut, I still need a MySpace account to interact with my friends on that site. Some people have claimed that OpenSocial will enable routing around such lock-in via applications like iLike and Flixster which have their own social networks and thus could build cross-site social networking services since they will be hosted on multiple social networking sites. However the tough part of this problem is how a hosted application knows that carnage4life@windowslivespaces is the same user as DareObasanjo@Facebook? It seems OpenSocial completely punts on satisfying this scenario even though it wouldn't be hard to add this as a requirement of the system. I guess the various applications can create their own user account systems and then do the cross-site social network bridging that way, which sucks because it will be a lot of duplicative work and will require users to create even more accounts with various services.

   Given that the big widget vendors like iLike, Slide and RockYou already have their users creating accounts on their sites that can be tied back to which social networking site the user utilizes their widgets on, this might be a moot point. Wouldn't it be mad cool if the Top Friends Facebook application could also show your top friends from MySpace or Orkut? I suspect the valuation of various widget companies will be revised upwards in the coming months.

4. There is no mention of a user-centric application authorization model. Specifically, there is no discussion of how users grant and revoke permission to access their personal data to various OpenSocial applications. Regular readers of my blog are familiar with my mantra of putting the user in control which is why I've been so enthusiastic about OAuth. Although there is some mention of Google's Authentication for Web Application in the documentation, this seems specific to Google's implementation of OpenSocial hosting and it is unclear to me that we should expect that this is the same model that will be utilized by MySpace, Bebo, TypePad or any of the other social networking sites that have promised to implement OpenSocial. On the other hand, Facebook has a well thought out applications permission model and I would have thought it would be quite easy to simply reverse engineer that and add it to the OpenSocial spec than to simply punt on this problem.

Despite these misgivings, I think this is a step in the right direction. Web widget and social graph APIs need to be standardized across the Web.

PS: I've subscribed to the Google OpenSocial blog. So far there have only been posts by clueless marketing types but I'm sure interesting technical information that addresses some of the points above will be forthcoming.

James Snell has a blog post entitled Batch! which talks about the Batch processing model in GData APIs. He provides a sample of a GData batch request and points out the following

If the mere sight of this doesn’t give you shivers and shakes, let me give you a few reasons why it should:

1. It’s not valid Atom. Note the first entry in the feed for instance. An Atom entry has an id, a title, some content, an author, some links, maybe some categories, etc. If the type of objects you want to represent does not also have those things, Atom is not the right format to use.
2. It only works with Atom. What about binary resources like Jpeg’s? I guess we could base64 encode the binary data and stuff that into our invalid Atom entries but doing so would suck.
3. We can’t use Etag’s and conditional requests
4. I’m sure there are more reasons but these should be enough to convince you that a better approach is needed.

In a previous post I entitled One Protocol to Rule Them All and in the Darkness Bind Them I pointed out that since the Atom Publishing Protocol is not a good fit for interacting with data types that aren’t microcontent, it would need to be embraced and extended to satisfy those needs. In addition, this leads to problems because different vendors will embrace and extend it in different ways which fragments interoperability.

An alternative approach would be for vendors to utilize protocols that are better suited for the job instead of creating incompatible versions of a standard protocol. However the response I’ve seen from various people is that it is better if we have multiple slightly incompatible implementations of a single standard than multiple completely incompatible proprietary technologies. I’ve taken to calling this “the ODF vs. OOXML lesson”. This also explains why there was so much heat in the RSS vs. Atom debates but not so much when it came to debates over Yahoo’s podcasting extensions vs. Apple’s podcasting extensions to RSS.

Let’s say we now take it as a given that there will be multiple proprietary extensions to a standard protocol and this is preferable to the alternative, what should we have as the ground rules to ensure interoperability isn’t completely thrown out the windows? A fundamental ground rule should be that vendors should actually provide standards compliant implementations of the protocol before deciding to embrace and extend it. That way clients and services that conform to the standard can interoperate with them. In this regard, GData falls down horribly as James Snell points out.

Given that Joe Gregorio now works at Google and is a co-author of RFC 5023, I assume it is just a matter of time before Google fixes this brokenness. The only question is when?

PS: Defining support for batch operations in a standard way is going to be rather difficult primarily because of how to deal with failure modes. The fact that there is always the struggle between consistency and availability in distributed systems means that some folks will want a failure in any of the batched operations to result in the equivalent of a rollback while there are others that don’t care if one or two out of a batch of fifty operations fails. Then there are some folks in the middle for whom “it depends on the context” for which kind of failure mode they want.  

Now playing: Method Man - Say (feat. Lauryn Hill)

With the hubbub now settling down down I decided to go back and try out Yahoo! Pipes. For a while, I've wanted a feed for articles by Chris Kelly over on Huffington Post so I decided to build that.  After a couple of false starts I created the feed which currently doesn't have any items because there aren't any posts by Chris Kelly in the Huffington Post feed.

Now that I've actually used the service I'm pretty surprised that anyone thinks that this is a service that non-geeks will use. Programming with flowcharts to process RSS feeds seems even geekier than having a Star Trek wedding which was my previous bar for geekiest thing ever.

While everyone else was raving about the fact that Feedburner can now count RSS subscribers coming from Google reader I've been noticing that there was another discrepancy in the Feedburner data that didn't seem to be accounted for. Below is a screenshot of number of hits from Web browsers on my RSS feed

It seems pretty unlikely that people have clicked on my RSS feed over 5000 times today. At first I thought Feedburner was miscounting feeds that had been subscribed from IE 7 but a quick look in Fiddler shows that IE 7 requests feeds using Windows-RSS-Platform as the User-Agent and is correctly counted by Feedburner.

So I sent some mail to Eric Lunt who's a co-founder and the  CTO of Feedburner to see if he knew what was wrong. He let me know that the problem is that Outlook 2007 doesn't identify itself in the User-Agent string and instead pretends to be Internet Explorer 7. This means there is no way to separate out accesses of your feed from Outlook 2007 from people clicking on your feed in IE 7.

This seems like a fairly rookie mistake to ship in a bigtime product like Outlook. I don't have the latest version installed so I can't confirm that this is truly the case but if it is I hope they plan to fix this soon. It's really lame to not identify your product correctly in the User-Agent string.
...
Oops. I should have done a search before sending out mail. It looks like this was already covered in a blog post entitled Outlook, RSS, & the user-agent string by Michael Affronti who was the PM for RSS in Outlook 2007. He wrote

For Outlook 2007 we will unfortunately not be able to report any custom user agent string for our RSS aggregation.  Due to the way we integrate with IE across many parts of the application (the WININET stack is the underlying infrastructure for all of Outlook’s internet communication), we cannot easily and safely change the way we broadcast ourselves when connecting to external servers.  To do so would require a fundamental change in the way the WININET stack is called from Outlook and could affect all of the Office applications.  The scope of this fix is unfortunately outside of what we can provide this release.

I guess this won't be fixed anytime soon, if ever. Anyway, I hope this post helps out other users of Feedburner who've also been curious about their weird number of hits supposedly from IE 7. 

A couple of blogs I'm subscribed to are pimping the brand new Yahoo! Pipes which I unfortunately can't seem to access right now. You can read some of the hype in blog posts like Jeremy Zawodny's Yahoo! Pipes: Unlocking the Data Web and Tim O'Reilly's Pipes and Filters for the Internet where it is described as "milestone in the history of the internet". I'd have loved to try out the service giving my interest in mashups and feed syndication but the site seems to be down or is just really, really slow.

As Dave Winer writes in his post Pipes Investigation

I see that Yahoo has a new web app, called Pipes, that looks to me like a feed construction kit. It takes RSS inputs, processes them in ways that are specified by the user, and produces feeds as its output.
...
From a quick persual of the functionality last night and the fact that the server isn't responding right now (5:45AM Pacific), it seems this app uses lots of CPU on the server. 

As someone who works on large scale online services for a living, Yahoo! Pipes seems like a scary proposition. It combines providing a service that is known for causing scale issues due to heavy I/O requirements (i.e. serving RSS feeds) with one that is known for scaling issues due to heavy CPU and I/O requirements (i.e. user-defined queries over rapidly changing data). I suspect that this combination of features makes Yahoo! Pipes resistant to popular caching techniques especially if the screenshot below is any indication of the amount of flexibility [and thus processing power required] that is given to users in creating queries.

Really interesting idea though. I agree with Dave Winer that this is definitely fodder for geeks and not the average Web user. After all, RSS still hasn't crossed the adoption chasm with average Web users let alone an RSS feed remixing service.

I've noticed that some problems with viewing feeds of sites hosted on TypePad for the past few months in RSS Bandit. The problem was that every other post in a feed would display raw markup instead of correctly rendered HTML. I decided to look into the problem this morning and tracked down the problem. Take a look at http://blog.flickr.com/flickrblog/atom.xml. Here are relevant excerpts from the feed

            
              

              <content type="html" xml:lang="en-ca" xml:base="http://blog.flickr.com/flickrblog/">
              

&lt;div xmlns=&quot;http://www.w3.org/1999/xhtml&quot;&gt;&lt;p&gt;&&nbsp;&lt;/p&gt;

&lt;div style=&quot;text-align: center;&quot;&gt;&lt;a href=&quot;http://www.flickr.com/gift/&quot;&gt;&lt;img
border=&quot;0&quot; src=&quot;http://us.i1.yimg.com/us.yimg.com/i/ww/news/2006/12/12/gtfof.gif&quot;
style=&quot;padding-bottom: 6px;&quot; /&gt;&lt;/a&gt;&lt;/div&gt;

&lt;p&gt;It&#39;s now easier than ever to spread joy this holiday season
by giving the &lt;a href=&quot;http://www.flickr.com/gift/&quot;&gt;&lt;strong&gt;Gift
of Flickr&lt;/strong&gt;&lt;/a&gt;. You can purchase a special activation
code that you can give to anyone, whether or not they have an existing Flickr account.
We&#39;ve even created a special Gift Certificate card that you can print out
yourself, fold up and stuff in a stocking, under a tree or hidden away for after the
candles are lit (of course, you can also send the gift code in an email).&lt;/p&gt;


&lt;p&gt;And it&#39;s even better to give the gift of Flickr since now
your recipients will get &lt;a href=&quot;http://www.flickr.com/help/limits/#28&quot;&gt;&lt;strong&gt;unlimited
uploads&lt;/strong&gt;&lt;/a&gt; — the two gigabyte monthly limit
is no more (&lt;em&gt;yep, pro users have no limits on how many photos they
can upload&lt;/em&gt;)! At the same time, we&#39;ve upped the limit for
free account members as well, from &lt;a href=&quot;http://www.flickr.com/help/limits/#28&quot;&gt;&lt;strong&gt;20MB
per month up to 100MB&lt;/strong&gt;&lt;/a&gt; (yep, five times more)!&lt;/p&gt;


&lt;p&gt;The Flickr team also wants to take this opportunity to thank you
for a wonderful year and wish you and yours all the best of the season. Yay!&lt;/p&gt;&lt;/div&gt;
</content>

...
<content type="xhtml" xml:lang="en-ca" xml:base="http://blog.flickr.com/flickrblog/">

<div xmlns="http://www.w3.org/1999/xhtml"><p><a href="http://www.flickr.com/photos/eye_spied/313572883/"
title="Photo Sharing"><img width="500" height="357" border="0" src="http://static.flickr.com/117/313572883_8af0cddbc7.jpg"
alt="Dec 2 2006 208 copy" /></a></p>


<p><a title="Photo Sharing" href="http://www.flickr.com/photos/mrtwism/71294604/"><img
width="500" height="375" border="0" alt="riding" src="http://static.flickr.com/34/71294604_b887c01815.jpg"
/></a></p>


<p>See more photos in the <a href="http://www.flickr.com/photos/tags/biggame/clusters/cal-berkeley-stanford/">"Berkeley,"
"Stanford," "big game" cluster</a>.</p>


<p>Photos from <a href="http://www.flickr.com/photos/eye_spied/" title="Link
to caryniam's photos">caryniam</a> and <a title="Link to mrtwism's photos"
href="http://www.flickr.com/photos/mrtwism/">mrtwism</a>.</p></div>
</content>

          

So the first mystery is solved. The reason some posts look OK and some don't is that for some reason TypePad seems to alternate between escaped HTML and well-formed XHTML as the content of an entry in the feed. When the feed uses well-formed XHTML the item looks fine but when it uses escaped HTML it looks like crap. The next question is why the items aren't rendered correctly when escaped HTML is used.

So I referred to section 3.1 of the Atom 0.3 specification and saw the following

3.1.2  "mode" Attribute

Content constructs MAY have a "mode" attribute, whose value indicates the method used to encode the content. When present, this attribute's value MUST be listed below. If not present, its value MUST be considered to be "xml".

"xml":

A mode attribute with the value "xml" indicates that the element's content is inline xml (for example, namespace-qualified XHTML).

"escaped":

A mode attribute with the value "escaped" indicates that the element's content is an escaped string. Processors MUST unescape the element's content before considering it as content of the indicated media type.

"base64":

A mode attribute with the value "base64" indicates that the element's content is base64-encoded [RFC2045]. Processors MUST decode the element's content before considering it as content of the the indicated media type.

To prevent aggregators from having to use their psychic powers to determine when an item contains plain text or escaped HTML, the Atom folks introduced a mode attribute that indicated whether the content should be treated as is or should be unescaped. As you can see the default value for this is not "escaped". Since the TypePad Atom feeds do not state that the HTML content is escaped then the aggregator is not expected to unescape the content before rendering it. Second mystery solved. Buggy feeds are the culprit. 

Even though these feeds are broken it is probably faster for me to special case feeds fromTypePad than trying to track down and convince the folks at SixApart that this is a bug worth fixing. This issue will be fixed in the next beta of the Jubilee release of RSS Bandit.

Shelley Powers has a good pair of posts critical of TechMeme, a technology meme-tracker. In her post entitled Techmeme tells us to Feed the Daddy she writes

Techmeme heard the recent discussion about sites not appearing, and responds with a post (at http://blog.memeorandum.com/061115/how-to-show-up) on how to show up on techmeme. The money shot:

Early on I noticed my system occasionally missed good posts from blogs that link back to my sites. So recently I extended my system to take referrals into account. Now if your blog or news article sends a moderate level of traffic to one of my sites, it will be evaluated for inclusion. Linking certainly doesn't guarantee you'll appear, since all posts are run though the usual tests for newsworthiness. In fact, extra steps to avoid spam are now in effect since faked referrals and splogs are already commonplace. So in summary, sending memeorandum (or Techmeme or…) visitors is another way to "enable discovery of your post".

In other words, if you puff up Gabe Riviera's empire, giving it lots of Google rank, as well as do all the marketing for him (such as techmeme's primary gatekeeper, Scoble, for whom Riviera sends special love and kisses), you might be able to 'buy' your way into being listed.

In her followup post entitled Feed your Daddy Follow-up, Shelley adds

I wouldn't 'fix' Techmeme. What I would like to see is a growth in sites that provide topic aggregations, each using its own metrics and filtering algorithms. The more of these there are, the more likely we'll see a more fair distribution of attention, as well as a greater variety of stories, and more timely ones at that. In history, a way to discover an unbiased view of a fact or an event is to seek out at least three separate sources of information. The same can be said of topic aggregators. More than three; I'd actually like to see at least five.

One of the biggest problems with Techmeme is that it is asserted to be the 'ultimate authority' on what are the top stories in technology (or politics for Memeorandum). Yet according to it, 30% of us spend all of our time talking about Google, 10% of us discuss new startup funding, 10% talk about Microsoft, how it is, or is not clued; probably about 15% of us talk about some variation of gadget, typically iPod and now Zune; the rest talk about Techcrunch, Scoble, Second Life, or Techcrunch and Scoble in a Second Life. I could go on, but the point is that Techmeme is based more or less on seeded terms and seeded webloggers, and it can't shake that influence. As such, it provides an incredibly skewed look at the tech area of weblogging–completely ignoring most of what is truly technology.

Techmeme serves a purpose for those who are into Google and VC and San Francisco and startups and money, and Michael Arrington and Calacanis, and Scoble and the scene there, and that's fine. But that doesn't make it an authority on what's important, interesting, or even timely.

As usual Shelley hits the nail on the head. TechMeme is good at what it does, gathering the popular or interesting links among the Silicon Valley blogger crowd. However this is just one particular view into the technology industry and specifically the technology blogosphere. Most of the content isn't particularly relevant if you aren't a regular reader of sites like TechCrunch and Robert Scoble's blog.

Me, I personally would prefer a meme tracker that was heavy on bloggers like Sam Ruby, Tim Bray and Jon Udell instead of the large number of PR hacks and VCs that populate TechMeme. Where I disagree with Shelley is that I don't think the answer is more meme trackers each with their own bias yet which will likely overlap significantly. We already have that today, if you read sites like TechMeme, TailRank and Megite. I think the future is in personalization and not more news aggregator whose bias you can't control.

I had some DNS issues last week because I forgot to renew my domain name registration. According to my FeedBurner statistics page, one of the consequences of this lapse on my part is that I've lost about 1,000 subscribers to my feed on Rojo. It looks like they may have decommissioned my feed in their service once the domain name stopped resolving after a couple of hours. This seems like a reasonable thing to do.

I've sent some mail to the Rojo folks through their feedback email address but don't hold out much for ever getting a response. It may be that there is nothing for them to fix since the relationship between my feed and the subscribers may have been deleted from their database once my domain was considered to be gone.

Anyway, if you are a Rojo user wondering why my feed disappeared from you subscription list, now you know. 

Tim Bray has a blog post entitled On Comments where he writes

I’ve had comments running for a few days here now (I prefer to say “contributions”, but whatever). People are irritated at me because an ongoing fragment shows up as unread in their feed-reader whenever a new comment comes in. I’m not sure what the right thing to do is. This piece outlines a few options and asks the community for discussion.

This is one of the reasons I've given about disliking the atom:updated element in blog posts like Indicating Updated Items in RSS Bandit. It should be up to the user to decide what count as 'significant' updates that warrant marking the item as changed or new in the user interface, not the publisher. Tim thinks that new comments in a blog post should lead to the reader being notified by their aggregator, I think this should only be the case when the user has explicitly opted in for notifications on changed new comments. This doesn't extend to updates because the definition of what counts as a 'significant' update is going to vary from publisher to publisher and from user to user.

My advice to Tim is to  use the Atom threading extensions which provides explicit mechanisms for indicating changes to the number of comments and provides a way to link to comment feeds as opposed to hacks like changing the value of atom:updated or putting the comments into the atom:content of the entry. Those both sound like recipes for a negative user experience when reading his blog in many aggregators.

The title of this blog post is probably harsher than I intend. I think it is useful to have a last modified date in the form of atom:updated on items in a feed. What I disagree with is impacting the user experience based on changes to that element.

Niall Kennedy has a blog post entitled Authenticated and private feeds where he writes

Examples of private feeds intended for 1:1 communication include bank balances, e-mail notifications, project status, and the latest bids on that big contract. Data in the wrong hands could be dangerous, and many companies will stay away from the feed syndication space until they feel their users' personal data is secure.

A private feed's data could be exposed in a variety of ways. A desktop aggregator's feed content might be available to other users on the same computer, either through directory access or desktop search. An online aggregator might expose a feed and its content in search results or a preview mode.
...
A feed publisher could whitelist the user-agents it knows comply with its access policies. SSL encryption might not be a bad idea either as shared aggregation spaces might not store content requested over HTTPS. It would place extra load on the server as each request requires extra processing, but if the alternative is placing your customer's data in the Yahoo! search index then that's not such a bad thing.

I believe large publishers such as Salesforce.com or eBay would produce more feed content if they knew their customers' data was kept private and secure. There's a definite demand for more content transmitted over feed syndication formats but it will take the cooperation and collaboration of security formats and consistent aggregation practices to really move the needle in the right direction.

How to properly support private and authenticated feeds is a big problem which Niall highlights but fails to go into much detail on why it is hard. The main problem is that the sites providing the feed have to be sure that the application consuming the feed is secure. At the end of the day, can Bank of America trust that RSS Bandit or Bloglines is doing a good job of adequately protecting the feed from spyware or malicious hackers?

More importantly, even if they certify these applications in some way how can they verify that the applications are the ones accessing the feed? Niall mentions white listing user agents but those are trivial to spoof. With Web-based readers, one can whitelist their IP range but there isn't a good way to verify that the desktop application accessing your web server is really who the user agent string says it is.

This seems to be yet another example of where Web-based software trumps desktop software.

Gabe Rivera, author of Techmeme, has a blog post entitled Why I don't offer a personal filter where he writes

I'm facing another round of inquiries on personal filtering, mostly from Techmeme fans who've read Ross Mayfield's or Dare Obasanjo'sJeff Clavier and Ted Leung nearly a year ago!) recent thoughts on the matter. (Just for the record, the first round included requests from

Why don't I offer a personal filter service aka "meMeme" aka "my.memeorandum"? Briefly, filters based on the editorial approach used for Techmeme/memeorandum don't work well outside of a few topic domains (like politics and tech), because cross linking is typically too sparse to produce a compelling mix of news. Sam Ruby unintentionally confirmed this yesterday should you pause to consider what sort of daily news selection could be derived from his Venus output. While it's true that cross linking is dense in some blogospheres, these are largely the same domains already covered by my existing sites.

Why not try editorial approaches based on new kinds of semantic analyses? My belief is that the requisite technology is harder than anything powering Google News, Topix, or my current sites. Attempts based on current technologies come up woefully short, with the resulting "Daily Me" consisting of a seemingly random mix of content missing most or all "must have" articles and posts. And having the "must haves" is essential for winning the earlier adopter types that would dominate the userbase of such a filter in the first place.

I reread the output from Sam's blogroll and it reminded me that there is a difference between the scenarios that sites like Techmeme and Tailrank are interested in and the goals of a personalized meme tracker. Here are a copuple of questions to get you started on understanding the differences in implementation choices one might make between implementing a personalized meme tracker vs. a topic specific memetracker,

1. Q: How do you deal with "noise" links such as http://del.icio.us/tag/rest or http://www.technorati.com/tag/AJAX which may be common in the feeds the user is interested in?

   A: In both cases, it would seem the first step is to hard code the application to understand certain kinds of links as "noise". The interesting question is how to deal with the introduction of new types of "noise" links to the ecosystem. A web-based application may be easily updated as new "noisy" links enter the system but things are a bit more difficult for a desktop application. Perhaps allowing users to nominate certain classes of links as noise?

2. Q: What 'class' of news items or blog posts should be used in evaluating what is [currently] popular?

   A: It is quite obvious that simply using the entirety of the posts from a particular feed to calculate a links popularity is flawed. Using that metric, I suspect that links such as http://adaptivepath.com/publications/essays/archives/000385.php or http://scobleizer.wordpress.com/2006/06/10/correcting-the-record-about-microsoft/ would always be the most popular links from my blogroll. Using a specific date or time range (e.g. over the past 24-48 hours) seems to be what sites such as Techmeme and Tailrank seem to do. An aggregator such as RSS Bandit or FeedDemon may use other techniques such as only using 'unread' items to calculate currently popular topics.

3. Q: How do you deal with link blogs?

   A: A number of people in my blogroll have blog posts that are basically a repost of al the links they have posted to del.icio.us that day (e.g. Stephen O'Grady and Mark Baker). Sites like Techmeme and Tailrank filter these posts because no one wants to see a bunch of headlines that are all of the form 'links for 2006-09-05' with no real content. On the other hand, if a large number of folks in my blogroll are linking to a particular news item then it is likely to be interesting to me regardless of whether there 'meaty' blog posts behind their links or just linkblog style postings.

These are a couple of the queestions that I've been pondering since I started thinking about this feature a couple of months ago. At the end of the day I think that although Gabe's perspective is useful since he did build the site that inspired this thinking, the scenarios are different enough to change some of the implementation choices in ways that may seem surprising to some.

PS: It seems Sam has already turned Gabe's feedback into code based on reading his blog post MeMeme 2.0. There are definitely interesting times ahead.

About six months ago, I wrote a blog post entitled Jubilee Thoughts: Tracking Hot Topics where I talked about adding meme tracking functionality similar to the features of Memeorandum and TailRank to RSS Bandit. Since I wrote that blog post I haven't written a lick of code that actually does this but I've thought and talked about it a lot. While all I've done is talk I can't help but notice that a few others have been writing code while I've been pontificating in my blog.

In his blog post entitled Spyder Spots a Memetracker Nick Bradbury writes

Andy "Spyder" Herron writes about the "personal memetracker" that's hidden in FeedDemon 2.0.0.25.

I had hoped to complete this feature by now, but as Andy points out, it still needs some work (which is why I hid it and gave it a "beta" label). If you'd like to try it out, select "Popular Topics" from the Browse menu (or just add the "Popular Topics" toolbutton to the toolbar above FeedDemon's browser).

I should add that this feature will probably be useful only to people who subscribe to a lot of feeds since it relies on common links to determine popularity. So if you're not subscribed to feeds which link to the same articles, chances are it won't show you any results.

In another blog post entitled MeMeme Sam Ruby writes

Ross Mayfield: Cue up not what is popular, or what the people I subscribed to produced.  Cue up what my social network has found interesting.

Herewith, a simple demonstration of what aggressive canonicalization can produce.  Venus may be in Python, but suppose I’m in a Ruby mood.  The cache is simply files in Atom 1.0 format, with all textual content normalized to XHTML.

Lets make a few simplifying assumptions: all posts are created equal, each post can only vote once for any given link (this also takes care of things like summaries which partially repeat content), posts implicitly vote (once!) for themselves, and the weight of a vote degrades as the square of the distance between when the post was made and now.

Here’s the code, and here’s a snapshot of the output.  The output took 6.239 elapsed seconds to produce on my laptop.  I still have more work to do to eliminate some of the self-referential links (in fact, I a priori removed Bob Sutor’s blog from the analysis as it otherwise he would dominate the results).  But I am confident that this is solvable, in fact, I am working on expanding what filters can do.  I’ll post more on that shortly.

With both Sam and Nick on the case, I'm quite sure that within the next few months it will be taken for granted that one of the features of news aggregator is to provide personalized meme tracking. Although I'm sure that we'll all use the same set of basic rules for providing this feature, I suspect that the problems that we are trying to solve will end up being different which will influence how we'll implement the feature.

For example, the main reason I want this feature isn't to track what the popular topics are across multiple blogs but instead to find what the popular topics are across aggregated blog feeds such as blogs.msdn.com and the numerous planet sites. In reading Sam's blog it seems he'd consider the same feed linking to the same news items as spam to filter out while I consider it be the only part of the feature I'd use. This issue illustrates the main problems I've had with designing the feature in my head. What "knobs" or options should we give users to control how the meme tracker decides what is interesting or not vs. what should be ignored when generating the list of 'hot topics' (e.g. the various meme trackers have said they filter out link blogs since they tend to dominate the results as well)?

Since I've decided to be more focused with regards to RSS Bandit development, I won't touch this feature until podcasting support is done. However I'd like to hear thoughts from our users in the meantime.

I just read the rather brief Feed Access Control RSS and ATOM specification from the Bloglines team. It defines the access:control element as

<access:restriction> element
Sub element of <rss> or <feed>. Used to indicate the re-distribution restrictions for a feed. The 'relationship' attribute is used to indicate whether a feed will 'allow' or 'deny' access.

To 'allow' access means a feed may be redistributed to other public sources, including search. To allow access, for example:

             <access:restriction relationship="allow" />
          

To 'deny' access means a feed should not be redistributed to other public sources, including search. To deny access, for example:

             <access:restriction relationship="deny" />
          

The default relationship is to allow access. However, if a feed is currently set to 'deny', the relationship must be explicitly set back to 'allow' for it to be registered (Simply ommiting it from the feed is not sufficient to turn access back on).

The problem with this 'specification' is that it says nothing about its goals, scenarios or expected use cases. Without these it is hard to tell whether this is a good idea or a bad idea. Danny Ayers points out that this mimics the behavior of the Robots META tag that can be placed in HTML pages. I guess this means it prevents search engines from indexing your page and showing it in search results which makes sense in certain limited scenarios. For example, it makes sense to exclude a search engine from indexing the search results page of another search engine or the RSS feed of some search results. Hints like the Robots META tag and robots.txt are good ways to prevent this from happening for HTML pages. I guess this proposal does the same for RSS and Atom feeds.

On the other hand, it is definitely not an access control mechanism. You wouldn't want your bank to tell you that the way that they prevent anyone from viewing your bank account details is via robots.txt would you?

Yesterday, I spent way too much time trying to figure out how to import an OPML feed list into Bloglines from the UI before giving up and performing a Web search to find out the how to do it. Below is a screenshot of the key choices one has for managing ones feeds in Bloglines.

And this is what the Bloglines FAQ has in response to the question How Can I Import An Existing List of Subscriptions?

Once you have registered with Bloglines and replied to the confirmation email, click on the My Feeds tab at the top of the screen. Then, click on the Edit link. At the bottom of the left panel will be a link to import subscriptions. The subscription list must be in OPML format.

Nathan Torkington has a blog post entitled A Week in the Valley: GData on the O'Reilly Radar blog that talks about the growth of the usage of GData & the Atom Publishing Protocol within Google as well as Marc Lukovsky's take on how this compared to his time at Microsoft working on Hailstorm. Nat writes

They're building APIs to your Google-stored data via GData, and it's all very reminiscent of HailStorm. Mark, of course, was the architect of that. So why's he coming up with more strategies to the same ends? I figure he's hoping Google won't screw it up by being greedy, the way Microsoft did...The reaction to the GData APIs for Calendar have been very positive. This is in contrast to HailStorm, of course, which was distrusted and eventually morphed its way through different product names into oblivion. Noting that Mark's trying again with the idea of open APIs to your personal data, I joked that GData should really be "GStorm". Mark deadpanned, " I wanted to call it ShitStorm but it didn't fly with marketing".

Providing APIs to access and manipulate data owned by your users is a good thing. It extends the utility of the data outside that of the Web applications that may be the primary consumer of the data and it creates an ecosystem of applications that harness the data. This is beneficial to customers as can be seen by looking around today at the success of APIs such as the MetaWeblog API, Flickr API or del.icio.us API.

Five years ago, while interning at Microsoft, I saw a demo about Hailstorm in which a user visiting an online CD retailer was showed an ad for a concert they'd be interested in based on their music preferences in Hailstorm. The thinking here was that it would be win-win because (i) all the user's data is entered and stored in one place which is convenient for the user (ii) the CD retailer can access the user's preferences from Hailstorm and cut a deal with the concert ticket provider to show their ads based on user preferences and (iii) the concert ticket provider gets their ads shown in a very relevant context.

The big problem with Hailstorm is that it assumed that potential Hailstorm partners such as retailers and other businesses would give up their customer data to Microsoft. As expected most of them told Microsoft to take a long walk of a short pier. 

Unfortunately Microsoft didn't take the step of opening up these APIs to its online services such as Hotmail and MSN Messenger but instead quietly canned the project. Fast forward a few years later and the company is now playing catchup to ideas it helped foster. Amusingly, people like Mark Lucovsky and Vic Gundotra who were influential during the Hailstorm days at Microsoft are now at Google rebuilding the same thing.

I've taken a look at GData and have begun to question the wisdom of using Atom/RSS as the baseline for information interchange on the Web. Specifically, I have the same issues as Steven Ickman raised in a comment on DeWitt Clinton's blog where he wrote

From a search perspective I’d argue that the use of either format, RSS or Atom, is pretty much a hack. I think OpenSearch is awesome and I understand the motivators driving the format choices but it still feels like a hack to me.

Just like you I want to see rich structured results returned for queries but both formats basically limit you to results of a single type and contain a few known fields (i.e. link, title, subject, author, date, & enclosure) that are expected to be common across all items.

Where do we put the 100+ Outlook defined contact fields and how do we know that a result is a contact and not an appointment or auction? Vista has almost 1000 properties defined in its schema so how do we convey that much metadata in a loseless way? Embedded Microformats are a great sugestion for how to deal with richer content but it sort of feels like a hack on top of a hack to me? What’s the Microformat for an auction? Do I have to wait a year for some committee to arrive at joint aggreement on what attributes define an auction before I can return structured auction results?

When you have a hammer, everything looks like a nail. It seems Steven Ickman and I reviewed OpenSearch/GData/Atom with the same critical lens and came away with the same list of issues. The only thing I'd change in his criticism is the claim that both formats (RSS & Atom) limit you to results of a single type, that isn't the case. Nothing stops a feed from containing data of wildly varying types. For example, a typical MSN Spaces RSS feed contains items that represent blog posts, photo albums, music lists, and book lists which are all very different types.

The inability to represent hierarchical data in a natural manner is a big failing of both formats. I've seen the Atom Threading Extensions but that seems to be a very un-XML way for an XML format to represent hierarchy. Especially given how complicated message threading algorithms can be for clients to implement.

It'll be interesting to see how Google tackles these issues in GData.

About two weeks ago, Greg Reinacker wrote about NewsGator's past, present and future in two blog posts entitled NewsGator platform roadmap - Part I (a look back) and NewsGator platform roadmap - Part II (a look forward). The blog posts are a good look at the achievements of a company that has gone from a one-man shop building an RSS reading plugin for Outlook into being the dominant syndication platform company on almost any platform from Windows & Mac to the Web & mobile phones. If you are interested in XML syndication, then Greg's posts are bookmark-worthy since they describe the future plans of a company that probably has the best minds building RSS/Atom applications working there today. Below are some excerpts from his posts in my areas of interest

NewsGator Online

As I said 16 months ago, the proposed feature list is long and distinguished - and it still is.  There is so much to do here...some of the short-term planned additions range from more interactive feed discovery mechanisms (based on the larger community of users and their subscriptions), to completely different user interface paradigms (where a user could potentially select from different options, each catering to a different kind of user).

A larger initiative is around the whole paradigm. Techies aside, users don't want to think about feeds, and subscriptions, and searching for content...Given all that, we're really rethinking the way we present information to the user, and the way users discover new information.  We're designing ways for people to participate in a larger community if they wish, and get more value out of the content they consume, at the point they discover it.  While we all have our own set of feeds, and we all participate to some extent in the larger ecosystem, there is a lot of potential in linking people with similar interests to each other.  Some users will continue to use our system as they always have - and others will use it in completely different ways.  We're testing a couple of approaches on this right now - I think it's truly a game-changer.

NewsGator Inbox, FeedDemon, NetNewsWire

As I mentioned before, the enthusiasm around these products has continued to grow - people obviously see the value in a rich, synchronized, offline-capable user experience for consuming content.  Moving forward, online integration will get tighter, and more complete - ranging from the low hanging fruit like FeedDemon "News Bins" becoming Clippings (and thus synchronize with the entire platform), to more involved features like analytics-related features (recommendations, interest-based surfacing, etc.) and community-related features.
...
NewsGator core platform

This is the heart of our entire product line (with the exception of NewsGator Enterprise Server).  Moving forward, we're investing a lot in the platform.  We're building out more support for deep analytics (which we can use to deliver different kinds of user experience), and building out a much deeper metadata engine (which means if a client retrieves content from our system, they'll get much richer data than they otherwise would).  We'll have other ways to "slice" our data to get what you need, without having to subscribe to hundreds of feeds.

The API has been very successful, and we process millions of API calls per day from client applications, web services, and private label clients.  This traffic actually makes up a large percentage of our overall system traffic - which I think is a testament to the popularity and utility of the API.  Moving forward here, we're obviously very committed to the API story, and we'll continue to enhance it as we add platform capabilities.

There's lots of good stuff here. The first thing that pops out at me is that while a bunch of startups these days tend to proclaim the death of desktop software, NewsGator is actually seeing the best of both worlds and improving the quality of the desktop experience by harnessing a Web-based platform. It's not Web-based software replacing desktop software, it's desktop software becoming better by working in tandem with APIs and applications on the Web. When Ray Ozzie talks about "live software", NewsGator is the company that leaps most readily to my mind.

I like the idea of making discovery of new content more of a social experience. It'd be interesting to see what would happen if NewsGator Online had a del.icio.us-inspired interface for browsing and subscribing to people's feeds. I notice that Gordon Weakliem who works on the NewsGator API recently wrote a post entitled Needles in Haystacks where he talks about serendipitous discovery of new websites by browsing bookmarks of people with similar interests to him in del.icio.us. I'm sure it's just a matter of time before NewsGator adds these features to their platform.

I also like the idea of exposing richer metadata in the NewsGator API especially if it relates to the social features that they plan to unveil in the next couple of months. Unfortunately, I've never been able to get the NewsGator API to work quite right with RSS Bandit but I'll be revisiting that code later in the summer.

Mike Arrington of TechCrunch fame has a blog post entitled where he lays out the demographics of the various RSS readers used to subscribe to his feed. Below is an excerpt of his post and a partial screenshot of his FeedBurner statistics showing the top fourteen feed readers used to access the TechCrunch feed

Firefox (including Flock) accounts for 20% of feed readers. Bloglines is in second place with 13%, followed by NewsGator at 12%, Rojo at 8%, FeedReader at 7%, and Netvibes at 7%. Other notables include Pageflakes, Pluck and Attensa. If you add NetNewsWire to the core NewsGator stats, NewsGator is actually bigger than bloglines.
...

The feed reader statistics are surprising to me both for the feed readers that show up in the list and for those that don't. For example, I'm surprised to see FeedReader at #5 yet not see FeedDemon in the top 14. Similarly, the popularity of AJAX home pages like Pageflakes and Netvibes over those from the big 3 (Google/Yahoo/Microsoft) is also unexpected. Of course, these statistics might be skewed because TechCrunch is one of the default feeds in Netvibes. A final surprise is that NewsGator Online is almost as popular as Bloglines among readers of TechCrunch.This seems to mean that the latter is finally getting a lot of cred among the early adopter crowd especially since the former has been slow to update in the past year.

For a completely different set of demographics, here are the top 14 feed readers used to access my RSS feed according to FeedBurner.

I wonder what conclusion you draw from how different the distribution of feed readers is in the above screenshots. For example, I think the fact that a bunch of Microsoft employees and developers on Microsoft's platfoms read my blog explains why there are multiple instances of feed readers based on the .NET Framework in the above list. In addition, I suspect this also explains why there is an entry for the Windows RSS platform in the top 10 applications hitting my feed. 

On the flip side, I have no explanation for why it seems that NewsGator Online is half as popular as Bloglines among the readers of my blog.

Seeing Jon Udell's post about having difficulty with the Google PR team with regards to discussing the Google GData API reminded me that I needed to write down some of my thoughts on extending RSS and Atom based on looking at GData. There are basically three approaches one can take when deciding to extend an XML syndication format such as RSS or Atom

1. Add extension elements in a different namespace: This is the traditional approach to extending RSS and it involves adding new elements as children of the item or atom:entry element which carry application/context specific data beyond that provided by the RSS/Atom elements. Microsoft's Simple Sharing Extensions, Apple's iTunes RSS extensions, Yahoo's Media RSS extensions and Google's GData common elements all follow this model.
   

2. Provide links to alternate documents/formats as payload: This approach involves providing links to additional data or metadata from an item in the feed. Podcasting is the canonical example of this technique. One argument for this approach is that instead of coming up with extension elements that replicate existing file formats, one should simply embed links to files in the appropriate formats. This argument has been used in various discussions on syndicating calendar information (i.e. iCalendar payloads) and contact lists (i.e. vCard payloads). See James Snell's post Notes: Atom and the Google Data API for more on this topic.
   

3. Embed microformats in [X]HTML content: A microformat is structured data embedded within another markup language (typically HTML/XHTML). This allows one to represent both human-readable data and machine-readable data in a single document. The Structured Blogging initiative is an example of this technique.
   

All three approaches have their pros and cons. Option #1 is problematic because it encourages a proliferation of duplicative extensions and may lead to fragmenting the embedded data into multiple unrelated elements instead of a single document/format. Option #2 requires RSS/Atom clients to either build parsers for non-syndication formats or rely on external libraries for consuming information in the feed. The problem with Option #3 above is that it introduces a dependency on an HTML/XHTML parser for extracting the embedded data from the content of the feed.

From my experience with RSS Bandit, I have a preference for Option #1 although there is a certain architectural purity with Option #2 which appeals to me. What do the XML syndication geeks in the audience think about this?

Last week there was an outage on NewsGator Online. This outage didn't just affect people who use the NewsGator Online but also users of their desktop readers such as FeedDemon which synchronize the users feed state between the desktop and the web-based reader.

In his post Dealing with Connectivity Issues in Desktop Applications Nick Bradbury writes

One of the more frustrating challenges when designing a desktop application that connects to the Internet is figuring out how to deal with connectivity issues caused by firewalls, proxy servers and server outages.
...
And as we discovered last week, when your application relies on a server-side API, it has to be able to deal with the server being unavailable without significantly impacting the customer. This was something FeedDemon 2.0 failed to do, and I have to take the blame for this. Because of my poor design, synchronized feeds couldn't be updated while our server was down, and to make matters worse, FeedDemon kept displaying a "synchronization service unavailable" message every time it tried to connect - so not only could you not get new content, but you were also bombarded with error messages you could do nothing about.

A couple of months ago I wrote a blog post entitled The Newsgator API Continues to Frustrate Me where I complained about the fact that Newsgator Online assumes that clients that synchronize with it actually just fetch all their data from Newsgator Online including feed content. This is a bad design decision because it means that they expected all desktop clients who synchronize with the web-based reader to have a single point of failure. As someone who's day job is working on the platforms that power a number of Windows Live services, I know from experience that service outages are a fact of life. In addition, I also know that you don't want clients making requests to your service unless they absolutely have to. This is not a big deal at first but once you get enough clients you start wanting them to do as much data retrieval and processing as they can without hitting your service. Having a desktop feed reader rely on a web service for fetching feeds instead of having it fetch feeds itself needlessly increases the costs of running your online service and doesn't buy your customers a significantly improved user experience. 

I've bumped into Greg Reinacker since I complained about the Newsgator API and he's been adamant about the correctness of their design decisions. I hope the fallout from the recent outage makes them rethink some of the design of Newsgator's RSS platform.

Via Shelley Powers I found out that Mark Pilgrim has restarted his blog with a new post entitled After the Bath. Ironically, I didn't find this out from my favorite RSS reader because it correctly supports the HTTP 410(GONE) status code which Mark's feed has been returning for over a year.

Mark Pilgrim's feed being resurrected from the dead is another example of why simply implementing support for Web specifications as written sometimes bites you on the butt. :)

I stopped paying attention to the syndication wars a couple of months ago. I barely have time to stay on top of all the stuff I have to worry about as part of my day job let alone keeping track of the pointlessness that is the Atom vs. RSS debate. Unfortunately, every once in a while something happens that forces me to pay attention because I'm also the project lead for RSS Bandit.

One cool thing about XML syndication formats like RSS and Atom is that they are easily extensible. This means that anyone can come up with a new extension to the RSS/Atom formats which adds a new feature but is ignored by feed readers that don't understand the extension. Some of my favorite extensions are slash:comments which provides a count of the number of comments in a feed and wfw:commentRss which provides the URL to the feed for the comments to a blog post in the feed. One of my work items for the next version of RSS Bandit is to make it easier for people to 'watch' the comments for a blog post they are interested in if it supports these extensions. That way I can get get a notification everytime some comment thread I am interested in gets new posts directly from my aggregator instead of using other tools like CoComment.

A few days ago Sam Ruby posted an entry entitled Rogers Switches! where he mentioned that he now redirects all requests for his RSS feeds to his Atom feed. This meant that in RSS Bandit I now no longer see the comment count for the blog posts in Sam's feed nor can I view the comments to his posts directly within my aggregator. Sam and I had the following exchange in his blog comments when I discovered the ramifications of his change

I was trying to figure out if I’d introduced a bug in RSS Bandit to make your comment count and inline comments disappear. Instead, it seems you have made your feed less useful as part of the fallout of yet another iteration of the eternal pissing match which is the XML syndication wars.

sigh

Posted by Dare Obasanjo at 20:25

---

Dare, is this is your way of saying that you don’t intend to support the Feed Thread Extension?  I’d think that you would be on the watch for it.

Posted by Sam Ruby at 22:08

I looked at the draft of the Feed Thread Extension specification Sam linked to and it seems like a reinvention of the functionality provided by the slash:comments, wfw:commentRss and annotate:reference extensions. Great, not only do we now have to deal with the an increase in the number of competing XML syndication formats due to the Atom process ...By the way, have you seen the Atom 0.3 vs. Atom 1.0 debate? I told you so... we now also have to deal with duplicates of all the popular RSS extensions as well? Give me a break!

That said, you can expect support for these new extensions in the next version of RSS Bandit. At the end of the day, what matters is building useful software for our users regardless of how many petty annoyances are thrown in our way on the road there.

Thanks to numerous reports from RSS Bandit users it has come to my attention that the Atom feeds provided by Google's Blogger are invalid and in many cases aren't even well-formed XML. Please fix this. I'm tired of dealing with threads like Blogspot feeds - XML Failure in our support forums.

If you'd like an example of what is wrong with your feeds. Click on http://feedvalidator.org/check?url=http://nothing-more.blogspot.com/atom.xml which shows the results of validating the feed for Derek Denny-Brown's blog. Below is the list of errors returned

This feed does not validate.

• line 4, column 0: This feed uses an obsolete namespace [help]

                    <feed xmlns="http://purl.org/atom/ns#" version="0.3" xml:lang="en-US">
  
                  

• line 4, column 0: Unexpected version attribute on feed element [help]

                    <feed xmlns="http://purl.org/atom/ns#" version="0.3" xml:lang="en-US">
  
                  

• line 7, column 0: type attribute must be "text", "html", or "xhtml" [help]

                    <title mode="escaped" type="text/html">only this, and nothing
  more</title>
  
                  

• line 7, column 0: Unexpected mode attribute on title element (7 occurrences) [help]

                    <title mode="escaped" type="text/html">only this, and nothing
  more</title>
  
                  

• line 8, column 0: Undefined feed element: tagline [help]

                    <tagline mode="escaped" type="text/html">irregular eccentic
  eclecticisms, di ...
  
                  

• line 11, column 0: Undefined feed element: modified [help]

                    <modified>2006-03-27T00:01:47Z</modified>
  
                  

• line 12, column 0: Unexpected url attribute on generator element [help]

                    <generator url="http://www.blogger.com/" version="5.15">Blogger</generator>
  
                  

• line 13, column 0: Undefined feed element: info [help]

                    <info mode="xml" type="text/html">
  
                  

• line 4, column 0: Missing feed element: updated [help]

                    <feed xmlns="http://purl.org/atom/ns#" version="0.3" xml:lang="en-US">
  
                  

• line 22, column 0: Undefined entry element: issued (6 occurrences) [help]

                    <issued>2006-03-26T15:25:00-08:00</issued>
  
                  

• line 23, column 0: Undefined entry element: modified (6 occurrences) [help]

                    <modified>2006-03-27T00:01:47Z</modified>
  
                  

• line 24, column 0: Undefined entry element: created (6 occurrences) [help]

                    <created>2006-03-27T00:01:47Z</created>
  
                  

• line 27, column 0: type attribute must be "text", "html", or "xhtml" (6 occurrences) [help]

                    <title mode="escaped" type="text/html">You call that Democracy?</title>
  
                  

• line 36, column 0: Missing entry element: updated (5 occurrences) [help]

                    </entry>
  
                  

• line 153, column 156: XML parsing error: <unknown>:153:156: unbound prefix [help]

                    
                      ... S-X's niceties. If I knew people on
  the <st1:place st="on">Vista</st1:pl ...
                      
  
                                               ^
                  

In addition, this feed has issues that may cause problems for some users. We recommend fixing these issues.

• line 5, column 134: service.post is not a registered link relationship (2 occurrences) [help]

                    ... hing more" type="application/atom+xml"/>
  
                                               ^
                  

• line 7, column 66: text/html type used for a document fragment [help]

                    <title mode="escaped" type="text/html">only this, and nothing
  more</title>
  
                                                                    ^
                  

• line 4, column 0: Missing atom:link with rel="self" [help]

                    <feed xmlns="http://purl.org/atom/ns#" version="0.3" xml:lang="en-US">
  
                  

• line 18, column 150: service.edit is not a registered link relationship (6 occurrences) [help]

                    ... emocracy?" type="application/atom+xml"/>
  
                                               ^
                  

• line 27, column 63: text/html type used for a document fragment (6 occurrences) [help]

                    <title mode="escaped" type="text/html">You call that Democracy?</title>
  
                                                                 ^
                  

• line 29, column 0: application/xhtml+xml type used for a document fragment (6 occurrences) [help]

                    <div xmlns="http://www.w3.org/1999/xhtml">
                    
  
                  

Thanks for listening.

Recently there was a question asked on the RSS Bandit forums from a user who was Unable to Import RSSBandit-Exported OPML into IE7. The question goes

I exported my feeds from RSSBandit 1.3.0.42 to an OPML file in hopes of trying the feed support in IE7. IE7 seems to try to import, but ultimately tells me no feeds were imported. The exported file must have over a 100 feeds, so it's not that. Has anyone else been able to import feeds from RSSBandit into IE7?

I got an answer for why this is the case from the Internet Explorer's RSS team.  The reason is provided in the the RSS Bandit bug report, Support type="rss" for export of feeds, where I found out that somewhere along the line someone came up with the convention of adding a type="rss" attribute to indicate which were the RSS feeds in an OPML file. The Internet Explorer RSS team has decided to enforce this convention for indicating RSS feeds in an OPML file and will ignore entries that don't have this annotation.

Since RSS Bandit supports both RSS/Atom feeds and USENET newsgroups, I can see the need to be able to differentiate which are the feeds in an OPML file without having applications probe each URL. However I do think that type="rss" is a misnomer since it should also apply to Atom feeds. Perhaps type="feed" instead?

Work has been quite hectic the past few weeks so I've gotten behind on checking out the sexy new startups coming out of Silicon Valley. The startup that has recently caught my interest is Edgeio, the brain child of Mike Arrington of TechCrunch.

The most succint post I've found on the company is Edgeio - Mike’s Little eBay Killer by Pete Cashmore where he writes

Essentially, Edgeio is an aggregator for classified listings. You can write a classified ad on your blog, tag it with "listing" and let Edgeio pick it up from your feed. Add a few more tags to describe your ad and Edgeio will grab those too. The service will pick up anything tagged with "listing" and obviously that raises the question of spam. But after speaking to Mike, I’m pretty sure he’s on top of it. For instance, you can claim your blog on Edgeio, just like on Technorati. Claiming your blog means that you are now a "member" and your listings are considered more trustworthy. There are also automated ways to remove the worst of the spam. And then there are the user-powered methods - "report spam" buttons and the like.
...
Last of all: the business model. Unlike about 90% of the stuff that gets labelled (tagged?) Web 2.0, Edgeio actually has one. Actually it has a few, but the main monetization method appears to be sponsored listings - pay 25 cents a day to get your listing bumped up to the top. I would have been tempted to pursue a transaction-based model (ie. you take a cut from every sale), but I can see why Edgeio isn’t taking that path for now - handling transactions is a huge job and requires a reputation system, among other things. (And if Edgeio did build a reputation system, I’m pretty sure it would be portable).

Calling Edgeio an eBay killer is probably a bit hyperbolic, but I do think it points the way to how decentralization will undermine the centralized business models of old. Your little walled garden will never be as large, rich and varied as the content that exists out on the open web.

The more interesting thing to me is that the folks at Edgeio are implying that there is a market for a 'Make this blog post a classified listing' checkbox in traditional blog posting tools. From my perspective as someone who works closely with the MSN Spaces and Windows Live Expo teams this sounds very interesting. There is already some integration planned between both services but I'm not sure this is one of the options that was considered. I wonder how much user validation of this beilief Mike Arrington and company did before going ahead with launching their startup?

As far as business models go, I find it hard to imagine why anyone would consider this an eBay killer. I'm not going to claim that people posting things for sale on their blog and then having that picked up by classified listing services is inferior to eBay's model. However, I wonder why anyone thinks that services like eBay, Craig's List and Windows Live Expo wouldn't jump into this market if it turned out to be profitable. Since there doesn't seem to be any barrier to entry, ability to write a Web crawler and minor HTML parsing is all that is required, I wouldn't start eulogizing  eBay just yet. 

In his blog post The RSS Experience in IE7 Joshua Allen wrote

Dare says as much; IE7 was not intended to replace tools like RSS Bandit, NewsGator, or Outlook 12. It's not a matter of trying to keep small ISVs in business, as much as a decision to put the RSS-Bandit style reading experience in the products where it belongs; namely Outlook and OE. IE7 doesn't read NNTP feeds either; that's what OE is for.

In his blog post Email is Abused Omar Shahine wrote

I firmly believe that email is a fantastic tool, and that it’s also heavily abused in the work place. More often than not, what you hear when you send an email is deafening silence or a flurry of incomprehensible replies breaking threading and screwing up the conversation flow.

It is my firm belief that many folks don’t have any system for dealing with their email. They get overwhelmed by the amount of mail that they have, and as a result are unpredictable in getting back to you (if they do).

What this means is that not only do you have to manage your inbox, but you have to manage their inbox. I’ve started to write things down that I want to talk to people about, and every so often, walk into their offices and talk about the issues. It’s weird as this is what I used to do long before email got crazy.

On the one hand, Joshua Allen argues that consuming RSS feeds should be the purvey of traditional mail readers. On the other, Omar Shahine points out that traditional mail readers do a poor job of enabling people to manage information overload in environments with high rates of information flow. I agree 100% with the implications of Omar's post. Traditional 3-pane mail readers do a very poor job of enabling people keep on top of the information they consume. Thus, I think it's a bad idea to add yet another fire hose of information into the mix (i.e. making a traditional mail reader like Outlook my primary RSS reader).

I've not always been of this opinion. A few years ago I wrote a blog post entitled RSS, WinFS and Building a Universal Information Client where I discussed the concept of a universal information aggregator and argued that Outlook was the closest application to what I envisioned. Since then I've become familiar with the term digital lifestyle aggregator (DLA) which is similar to and better defined than my idea of a universal information aggregator. I believe that the DLA concept gives a clear idea of what information aggregators such as personal information managers and RSS readers should evolve into.

Why did I change my mind about Outlook being the ideal DLA? Well, the longer I worked on RSS Bandit, the more I felt that mimicing Outlook in its entirety wasn't the right approach for approaching building an RSS reader. I mentioned some of the problems I have with the Outlook model in my post The Problem With RSS Readers Inspired By Outlook where I wrote

The major problem is that the Outlook mail reading paradigm has a fundamental assumption which turns out to be flawed. It assumes you want to read every item you get in your inbox. This flawed assumption leads to the kind of information overload that hampers the productivity of lots of people I know at work. I've met several people who seem to always have hundreds unread items in their email inbox. For this reason I always have to learn who's easier to reach via IM or swinging by their office in person than sending them mail.

Most people I know get four classes of messages in their information aggregators (I am lumping reading email, reading news and reading RSS/Atom feeds into a single category). These are

1. notifications (checkin mails, comments to my blog, etc)
2. headlines (email newsletters, feeds from news sites, etc)
3. messages sent directly to me or that is similarly relevant
4. messages sent to an interest group I am a part of (XML-DEV mailing list, comp.text.xml newsgroup, etc)

The problem is that the typical Outlook inspired information aggregator treats all of the above as being of equal relevance. Even though Outlook does provide mechanisms for managing assigning relevance to incoming messages, they are either hard to find or cumbersome to use.

This is definitely one of the areas that needs to be improved in the world of information aggregators in general and RSS/Atom readers in particular.

The bottom line is that I think that traditional mail readers do a poor job of enabling people to manage the amount of information they consume today. With RSS, we've had the opportunity to experiment with different models of presenting information to users from "river of news" style aggregators to personalized portal pages instead of sticking to the traditional 2  or 3 pane readers which dominate email and news readers.

Unfortunately, the major browser vendors haven't gotten in on the act. Instead of using RSS as an opportunity to explore new ways of presenting information to users we've seen rather lame attempts at RSS integration into the browser such as Firefox's Live Bookmarks feature and the upcoming integration of RSS into IE 7 which is just slightly better.

So where are we? The major browsers have punted on solving the information overload problem caused by RSS while integrating it into their products. Similarly, mail readers already suck at dealing with email information overload let alone when RSS feeds are added to the mix. As it stands, I'm not sure where we're going to from here. In the meantime, I'm going to start exploring alternative Web browsers like Flock. Perhaps they'll be bolder in re-imagining how to improve the overall experience of people using the World Wide Web today.

Nick Bradbury has a post entitled Feedback on IE7 Beta 2 from the Developer of FeedDemon where he gives a lot of good feedback on the recently released IE 7 beta from his perspective as the developer of an RSS reader. Although I've given some feedback on the RSS reading functionality of  the IE 7 beta, I realize it would be more valuable to give my thoughts on the Winows RSS platform since this is supposed to make the job of people like me who've built RSS readers better. Below is a smattering of feedback divided into pros and cons of using the Windows RSS platform versus using the version we've built for RSS Bandit. Note that as Nick says in his post given that I've already written a lot of the ugly code needed to handle feed downloading, caching, parsing, etc. actually switching to use the Windows RSS platform is a load of unnecessary work for me. My feedback is based on the kind of support I'd need from the platform to implement the scenarios currently supported by RSS Bandit .

PRO

• COM API  was very straightforward to interact with from  .NET applications
  
• Built-in support for downloading enclosures in the background is nice
• Good support for asynchronously downloading feeds. This means application developers don't need to write a bunch of multithreaded/asynchronous code themselves. That is definitely a godsend.
• One can serialize feed objects to  XML
  

CON

• No support for application specific feeds. The Common Feed List assumes that user needs to use the same list of feeds in the various applications used for subscribing to feeds. I think this assumption is fundamentally flawed. I might use one application for downloading podcasts (e.g. iTunes), another for reading blogs (e.g. RSS Bandit), and yet another for browsing photo feeds. Since it doesn't make sense for my blogs to show up in iTunes, it would be cool if I could identify either the type of feed (podcast, text-based, etc) or the favored application for reading the feed via the API.
  
• No support for password protected feeds. The number of password protected feeds on the Web continues to grow, Web sites such as GMail and LiveJournal provide authenticated feeds for users today. As the usage of syndication technologies like RSS continues to grow, the need to support authentication by feed readers will also grow as well. I can imagine a day when I can subscribe to a password protected feed from my bank or credit card company. Not having support for this today is a non-starter.
  
• Support for obtaining XML elements which aren't supported by the API. It would be nice if there was a property for obtaining extension elements in a feed that didn't involve having to convert the feed object to XML then using XPath. Being able to perform a call like  Feed.GetItem("http://wellformedweb.org/CommentAPI/ ", "commentRss") to get an element which isn't mapped to a property in the Feed object is a lot more desirable than writing DOM or XPath code to extract that element from the results of calling Feed.Xml
  
• No ability to append application specific metadata to feeds. RSS Bandit supports notions like flagging items and we'd need some way to indicate that items are flagged if we are using the API.
  

Most of this is just based on reading the Using the Microsoft Feeds API document on MSDN. I'm sure I'd have more feedback if I took a pass at replacing all the feed processing code in RSS Bandit with the Windows RSS platform. However I don't think I'll have time to do that anytime soon.

A few days ago, in my blog post entitled Some Thoughts on the IE 7 Beta 2 Preview release I described the RSS features of IE 7 as unsatisfactory and disappointing. It seems I'm not the only one who dislikes what the Internet Explorer team has done with RSS.

In his post RSS Is a Glorified "Favorites" Feature Scott Karp writes

RSS is in Internet Explorer 7!!! The blogosphere is shouting from the rooftops. Yawn. I tried RSS in IE7, and it highlights the true shortcoming of current RSS applications — it’s really not much of an improvement over “favorites” or “bookmarks.”

IE7 goes so far as to put the RSS reader in same menu as favorites (or as TDavid puts it “A separate “Feed Center” exists inside the Favorites area.”), which appears in a left-hand navigation column.

So what’s the real innovation over Favorites/Bookmarks in terms of user experience? That it “automatically updates”? That I can get everything all in one place? That it highlights what’s new?

In his post RSS Really Sucks Paul Kedrosky writes

A while back I wrote that RSS sucks, and now that I've had some more time to think about it I've come to a deeper and more nuanced conclusion: RSS Really Sucks. The point was driven home recently as I read articles by people arguing that IE7 from Microsoft does RSS well enough to kill off a few standalone aggregators. I suppose, although that's a little like saying that buggy whips drive milk-wagons so well that people will soon stop using willow branches to goad horses.

Why? Because, as Scott Karp points out, the IE7 RSS implementation is as glorified "favorites" -- bookmarks, in other words. And they are particularly irritating bookmarks, ones that continually change and needle you as more "information" (I use that advisedly) comes beeping and streaming into your computer.

The main reason I am so irritated by IE 7's lackluster user experience around RSS is that you only get one chance to make a first impression. Using IE 7 will be the first time millions of people will be introduced to RSS and it would be unfortunate if they come away from thinking that is potentially transformative and liberating technology is simply a kind of "bookmarks that nag you all the time" feature.

I've heard some people say that if Microsoft integrates a high quality RSS reader into the browser then it would kill the desktop aggregator market which is the kind of thing Microsoft gets in legal trouble for all the time. My response? That is Death by Risk Aversion. What matters is making end users happy, not worrying about making features that suck just enough that people have to go out and buy software that does the job well so we don't get in legal trouble. 

In his post Thanks Bloglines Mike Torres writes

Over the course of the last few days, we noticed a problem in the way Bloglines was displaying feeds from MSN Spaces.  This problem was due to our recent URL change and the way we're redirecting visitors from http://spaces.msn.com/members/mike (as an example) to http://spaces.msn.com/mike.  Instead of providing the absolute URL to the RSS feed when Bloglines and others requested the feed, we're only returning the relative URL (i.e. "/mike").

 

Because of this, Bloglines had to turn around a fix to support relative URL redirects in record time.  Within just a couple of hours of contacting them, they had diagnosed the issue, fixed up all the Spaces feeds in their entire system, and patched the redirect logic to make sure it wouldn't happen again.  During this time, the subscriber lists/counts associated with a feed weren't updated for a little while (my 362 subscribers showed as 9, but my ego wasn't bruised) and they even did the extra work to merge "new" feeds with "old" feeds (because when the feed broke, and someone subscribed to the correct feed, Bloglines then had two records for the 'same' feed).

 

In short, this was truly great work by Mark Fletcher and the Bloglines staff.  Sorry guys for keeping you up so late on a Tuesday night!  We'll be making a change to the way we redirect shortly just to make sure this won't be a problem for anyone else in the future.  And for you Bloglines users, you should be back to normal for any MSN Spaces feeds in your list!

Mike and I exchanged mail with Mark Fletcher about this issue on Tuesday, and as he writes we were both grateful and impressed at how quickly the Bloglines folks made changes to fix the consequences of a bug in how we were sending HTTP redirects. Mad props to Mark Fletcher and the folks at Bloglines. You guys definitely rock.

In a recent post on the RSS Bandit forums entitled Microsoft Feeds API, one of our users asked when we plan to take addvantage of the Windows RSS Platform. Specifically the question asked was

I downloaded IE7 beta 2 preview and one of the new features is feed support. There really is no comparison between RSS Bandit and the minimilist support IE7 provideds however it is my understanding that there is more to the feed support then the basic UI IE provides. Apparently the Feed API is highly integrated into Vista and has been backported to XP as part of IE7. I would like to suggest (and I have no idea if this is even possible) that RSS Bandit embrace this new API. It's in it's early stages and I'd like to see it mature with the help of RSS Bandit into a usable common feed store for any number of apps (RSS Bandit, IE, plus anything else that decides to use it like the sample screensaver app in the API documentation.) Perhaps that's asking too much but I figured I should at least put the idea out there. It would be really great if, while surfing the net in IE I could subscribe to rss feeds using the IE mechanism, maybe take a quick look through them in IE, then later go back in RSS Bandit and be able to use the much more powerful features (such as stored searches, and folder aggregation) without having to have two copies of my actual feed lists to maintain.

Here's a link to the only info I could find on it so far:
http://msdn.microsoft.com/library/en-us/FeedsAPI/rss/rss_entry.asp

To the various people who have asked this question [including my friends on the IE team], the answer is YES we will support the Windows RSS Platform. As Walter, wrote in his post on Windows RSS Platform there are 3 main components of the platform; the Common Feed List, Feed Synchronization Engine, and Feed Store. Ideally I'd like to use all 3 in RSS Bandit but suspect that it'll be difficult to switch to using the Feed Synchronization Engine or the Feed Store provided by the Windows RSS platform. For example, our feed synchronization engine supports subscribing to USENET newsgroups which I doubt the Feed Syncrhonization Engine in the Windows RSS platform supports. On the other hand, it should be straightforward to satisfy the scenario requested in the quoted post where items subscribed in IE 7 are reflected in RSS Bandit and vice versa.

I need to work out the user interface with Torsten but it should be easy for us to support the Common Feed List. My current thinking is that we'll have a special folder for "My Internet Explorer Subscriptions" as opposed to mirroring the entire feed list in RSS Bandit within IE 7 and vice versa. Thoughts?

Niall Kennedy has a blog post entitled Exclusive: Google to offer feed API where he reveals

Google plans to offer a feed reader API to allow third-party developers to build new views of feed data on top of Google's backend. The new APIs will include synchronization, feed-level and item-level tagging, per-item read and unread status, as well as rich media enclosure and metadata handling. Google Reader PM Jason Shellen and engineer Chris Wetherell both confirmed Google's plans after I posted my reverse-engineering analysis of the Google Reader backend.

The new APIs will allow aggregator developers to build new views and interactions on top of Google's data. Google currently has at least two additional Google Reader views running on current development builds.

Google may offer public access to the feed API as early as next month. Shellen said the team wants to nail a few more bugs before publicly making the service available to the world.
...
Google's new offering is direct competition to NewsGator's synchronization APIs but are easier to code against (no SOAP required). Google currently does not have the same reach across devices as NewsGator but an easy-to-use API from the guys who brought you the Blogger API and "Blog This!" might really shake up the feed aggregator ecosystem.

As someone who's been thinking about synchronization between RSS readers for a few years I definitely see this as a welcome development. The Bloglines sync API is too limited in its functionality to be useful while the NewsGator API is both complex and designed with too many assumptions to be widely usable. However, unlike Niall, I blame the complexity of the NewsGator API on the data model and expected data flow than on whether it uses SOAP versus Plain Old XML (POX) as the wire format.

Once the Google Reader API ships, I'll definitely investigate the feasibility of adding support for it to the Jubilee release of RSS Bandit.

We were planning to ship a bugfix release of RSS Bandit before Christmas which fixed all the major issues reported in the recently Nightcrawler edition of RSS Bandit.

Unfortunately, it seems that either due to complexity or buginess I simply can't get the NewsGator API to perform the straightforward task of marking something as read in NewsGator Online which was viewed in RSS Bandit. I spent all of yesterday afternoon plus a couple of hours this morning working on it and I've finally given up. This feature simply won't work in the bugfix release shipping later this week. Maybe I'll have better luck when we ship the Jubilee release.

To make myself feel better, I'll work on fixing some of the Atom parsing bugs reported by Phil Ringnalda and the issues with password protected newsgroups. Nothing like having your self-worth defined by how many bugs you close in a database on SourceForge.

Update: So not only have I already fixed the newsgroup issues and the problems with parsing Atom feeds pointed out by Phil Ringnalda but I just got pinged by Gordon Weakliem who is the developer of the NewsGator API. Perhaps my Christmas can be salvaged after all.

Alan Kleymeyer has a post entitled NewsGator Online where he writes

I've switched my online news aggregator from Bloglines to Newsgator.  First, I wanted to try it out and compare it to Bloglines.  I like the interface better, especially in how you mark things as read.  I've swithched for good.  I mainly switched so that I can continue using RSS Bandit and get the benefit of syncing between it and an online news aggregator (supported in latest RSS Bandit 1.3.0.38 release)

Alan's post describes exactly why creating APIs for your online service and treating it as a Web platform and not just a web site is important. What would you rather use, a web-based aggregator which provides limited integration with a few desktop aggregators (i.e.Bloglines) OR a web-based aggregator which provides full integration with a variety of free and payware aggregators including RSS Bandit, NetNewsWire and FeedDemon? Building out a Web platform is about giving users choice which is what the NewsGator guys have done by providing the NewsGator API.

The canonical example of the power of Web APIs and Web platforms is RSS. Providing an RSS feed liberates your readers from the limitations of using one application (the Web browser) and one user interface (your HTML website) to view your content. They can consume it on their own terms using the applications that best fit their needs. Blogging wouldn't be as popular as it is today if not for this most fundamental of web services.

The topic of my ThinkWeek paper was turning Web sites into Web platforms and I was hoping to get to give a presentation about it at next year's O'Reilly Emerging Technology Conference but it got rejected. I guess I'll just have to keep shopping it around. Perhaps I can get it into Gnomedex or Mix '06. :)

The number one problem that faces developers of feed readers is how to identify posts. How does a feed reader tell a new post from an old one whose title or permalink changed? In general how you do this is to pick a unique identifier from the metadata of the feed item to use to tell it apart from others. If you are using the Atom 0.3 & 1.0 syndication formats the identifier is the <atom:id> element, for RSS 1.0 it is the rdf:about attribute and for RSS 0.9x & RSS 2.0 it is the <guid> element.

The problem is that many RSS 0.9x & 2.0 feeds do not have a <guid> element which usually means a feed reader has to come up with its own custom mechanism for identifying items. In many cases, using the <link> element is enough because most items in a feed map to a single web resource with a permalink URL. In some pathological cases, a feed may not have <guid> or <link> OR even worse may use the same value in the <link> element for each item in the feed. In such cases, feed readers usually resort to heuristics which are guaranteed to be wrong at least some of the time.

So what does this have to do with the Newsgator API? Users of recent versions of RSS Bandit can synchronize the state of their RSS feeds with Newsgator Online using the Newsgator API. Where things get tricky is that this means that both the RSS Bandit and Newsgator Online either need to use the same techniques for identifying posts OR have a common way to map between their identification mechanisms. When I first used the API, I noticed that Newsgator has it's own notion of a "Newsgator ID" which it expects clients to use. In fact, it's worse than that. Newsgator Online assumes that clients that synchronize with it actually just fetch all their data from Newsgator Online including feed content. This is a pretty big assumption to make but I'm sure it made it easier to solve a bunch of tricky development problems for their various products. Instead of worrying about keeping data and algorithms on the clients in sync with the server, they just replace all the data on the client with the server data as part of the 'synchronization' process.

Now that I've built an application that deviates from this fundamental assumption I've been having all sorts of interesting problems. The most recent being that some users complained that read/unread state wasn't being synced via the Newsgator API. When I investigated it turned out that this is because I use <guid> elements to identify posts in RSS Bandit while the Newsgator API uses the "Newsgator ID". Even worse is that they don't even expose the original <guid> element in the returned feed items. So now it looks like fixing the read/unread not being synced bug involves bigger and more fundamental changes than I expected. More than likely I'll have to switch to using <link> elements as unique identifiers since it looks like the Newsgator API doesn't throw those away.

Frustrating.

The following is an excerpt from Michael Arrington's post on the announcement

Yahoo gathered a small group of bloggers, press and others at Sauce in San Francisco tonight to announce the launch of two new RSS products. They have integrated an RSS reader directly into Yahoo Mail Beta, and are expanding Alerts to include RSS feeds.

These are significant new products, aimed squarely at new and mainstream RSS users. The service is not live as of the time I am posting this. I’ve added a screen shot picture from the live demo.

Mail

Yahoo has deeply integrated RSS into the Yahoo Mail beta experience. Directly below the email folders are “RSS folders”. Clicking on the top folder show all posts in a “river of news” format, meaning all posts for all subscribed feeds are listed in the order they have appeared in feeds.

Each feed also has its own folder, allowing the user to read feeds individually (more like bloglines).

A post from any feed is treated exactly like an email - any post can be forwarded as an email or dragged into a folder and saved. All of the great AJAX functionality already working in Yahoo’s Mail beta works with the new RSS functionality as well.

Adding feeds is straightforward - include the feed URL or choose from a number of popular feeds.

In his post Really Simple Sharing, Ray Ozzie announced Simple Sharing Extensions for RSS and OPML. He writes

As an industry, we have simply not designed our calendaring and directory software and services for this “mesh” model. The websites, services and servers we build seem to all want to be the “owner” and “publisher”; it’s really inconsistent with the model that made email so successful, and the loosely-coupled nature of the web.

Shortly after I started at Microsoft, I had the opportunity to meet with the people behind Exchange, Outlook, MSN, Windows Mobile, Messenger, Communicator, and more. We brainstormed about this “meshed world” and how we might best serve it - a world where each of these products and others’ products could both manage these objects and synchronize each others’ changes. We thought about how we might prototype such a thing as rapidly as possible – to get the underpinnings of data synchronization working so that we could spend time working on the user experience aspects of the problem – a much better place to spend time than doing plumbing.

There are many great item synchronization mechanisms out there (and at Microsoft), but we decided we’d never get short term network effects among products if we selected something complicated – even if it were powerful. What we really longed for was "the RSS of synchronization" ... something simple that would catch on very quickly.

Using RSS itself as-is for synchronization wasn't really an option. That is, RSS is primarily about syndication - unidirectional publishing - while in order to accomplish the “mesh” sharing scenarios, we'd need bi-directional (actually, multi-directional) synchronization of items. But RSS is compelling because of the power inherent in its simplicity.
...
And so we created an RSS extension that we refer to as Simple Sharing Extensions or SSE. In just a few weeks time, several Microsoft product groups and my own 'concept development group' built prototypes and demos, and found that it works and interoperates quite nicely.

We’re pretty excited about the extension - well beyond the uses that catalyzed its creation. It’s designed in such a way that the minimum implementation is incredibly easy, and so that higher-level capabilities such as conflict handling can be implemented in those applications that want to do such things.

The model behind SSE is pretty straightforward; to sychronize data across multiple sources, each end point provides a feed and the subscribes to the feeds provided by the other end point(s).  I hate to sound like a fanboy but SSE is an example of how Ray Ozzie showed up at Microsoft and just started kicking butt. I've been on the periphery of some of the discussions of SSE and reviewed early drafts of the spec. It's been impressive seeing how much quick progress Ray made internally on getting this idea polished and evangelized.

The spec looks good modulo the issues that tend to dog Microsoft when it ships specs like this. For example,  is a lack of detail around data types (e.g. nowhere is the date format used by the spec documented although you can assume it's RFC 822 dates based on the examples) and there is also the lack of any test sites that have feeds which use this format so enterprising hackers can quickly write some code to prototype implementations and try out ideas.

Sam Ruby has posted a blog entry critical of Microsoft's practices when it publishes RSS extension specifications in his post This is Sharing? where he writes

The first attribute that the the Simple Sharing Extensions for RSS and OPML is to “treat the item list as an ordered set”.  This sounds like something from the Simple List Extensions Specification that was also hatched in private and then unleashed with great fanfare about five months ago. Sure a wiki was set up, but any questions posted there were promptly ignored.  The cone of silence has been so impenetrable that even invoking the name Scoble turns out to be ineffective. 

Now the Simple List Extensions Specification URI redirects to an ad for vaporware.  Some things never change.

Should we wait for version 3.0?

I agree with all of Sam's feedback. Hopefully Microsoft will do better this time around.

Nick Bradbury has a post entitled An Attention Namespace for OPML where he writes

In a recent post I said that OPML would be a great format for sharing attention data, but I wasn't sure whether this would be possible due to uncertainty over OPML's support for namespaces.
...
As I mentioned previously, FeedDemon already stores attention data in OPML, but it uses a proprietary fd: namespace which relies on attributes that make little sense outside of FeedDemon. What I propose is that aggregator users and developers have an open discussion about what specific attention data could (and should) be collected by aggregators.

Although there's a lot of attention data that could be stored in OPML, my recommendation is that we keep it simple - otherwise, we risk seeing each aggregator support a different subset of attention data. So rather than come up with a huge list of attributes, I'll start by recommending a single piece of attention data: rank.

We need a way to rank feeds that makes sense across aggregators, so that when you export OPML from one aggregator, the aggregator you import into would know which feeds you're paying the most attention to. This could be used for any number of things - recommending related feeds, giving higher ranked feeds higher priority in feed listings, etc.

Although user interface and workflow differences require each aggregator to have its own algorithm for ranking feeds, we should be able to define a ranking attribute that makes sense to every aggregator. In FeedDemon's case, a simple scale (say, 0-100) would work: feeds you rarely read would get be ranked closer to zero, while feeds you read all the time would be ranked closer to 100. Whether this makes sense outside of FeedDemon remains to be seen, so I'd love to hear from developers of other aggregators about this.

I used be the program manager responsible for a number of XML technologies in the .NET Framework while I was on the XML team at Microsoft. The technology I spent the most time working with was the XML Schema Definition Language (XSD). After working with XSD for about three years, I came to the conclusion that XSD has held back the proliferation and advancement of XML technologies by about two or three years. The lack of adoption of web services technologies like SOAP and WSDL on the world wide web is primarily due to the complexity of XSD. The fact that XQuery has spent over 5 years in standards committees and has evolved to become a technology too complex for the average XML developer is also primarily the fault of XSD.  This is because XSD is extremely complex and yet is rather inflexible with minimal functionality. This state of affairs is primarily due to its nature as a one size fits all technology with too many contradictory design objectives. In my opinion, the W3C XML Schema Definition language is a victim of premature standardization. The XML world needed experiment more with various XML schema languages like XDR and RELAX NG before we decided to settle down and come up with a standard.

So what does this have to do with attention data and XML? Lots. We are a long way from standardization. We aren't even well into the experimentation stage yet. How many feed readers do a good job of giving you an idea of which among the various new items in your RSS inbox are worth reading? How many of them do a good job suggesting new feeds for you to read based on your reading habits? Until we get to a point where such features are common place in feed readers, it seems like putting the cart way before the horse to start talking about standardizing the XML representation of these features.

Let's look at the one field Nick talks about standardizing; rank. He wants all readers to track 'rank' using a numeric scale of 1-100. This seems pretty arbitrary. In RSS Bandit, users can flag posts as Follow Up, Review, Read, Reply or Forward. How does that map to a numeric scale? It doesn't. If I allowed users to prioritize feeds, it wouldn't be in a way that would map cleanly to a numeric scale. 

My advice to Nick and others who are entertaining ideas around standardizing attention data in OPML; go build some features first and see which ones work for end users and which ones don't. Once we've figured that out amongst multiple readers with diverse user bases, then we can start thinking about standardization.