When it comes to scaling Web applications, every experienced Web architect eventually realizes that Disk is the New Tape. Getting data from off of the hard drive disk is slow compared to getting it from memory or from over the network. So an obvious way to improve the performance of your system is to reduce the amount of disk I/O your systems have to do which leads to the adoption of in-memory caching. In addition, there is often more cacheable data on disk than there is space in memory since memory to disk ratios are often worse than 1:100 (Rackspace's default server config has 1GB of RAM and 250 GB of hard disk ). Which has led to the growing popularity of distributed, in-memory, object caching systems like memcached and Microsoft's soon to be released Velocity. 

memcached can be thought of as a distributed hash table and its programming model is fairly straightforward from the application developer's perspective. Specifically, There is a special hash table class used by your application which is in actuality a distributed hashtable whose contents are actually being stored on a cluster of machines instead of just in the memory of your local machine.

With that background I can now introduce Terracotta, a product that is billed as "Network Attached Memory" for Java applications. Like distributed hash tables such as memcached, Terracotta springs from the observation that accessing data from a cluster of in-memory cache servers is often more optimal than getting it directly from your database or file store.

Where Terracotta differs from memcached and other distributed hash tables is that it is completely transparent to the application developer. Whereas memcached and systems like it require developers to instantiate some sort of "cache" class and then use that as the hash table of objects that should be stored, Terracotta attempts to be transparent to the application developer by hooking directly into the memory allocation operations of the JVM.

The following is an excerpt from the Terracotta documentation on How Terracotta Works

Terracotta uses ASM to manipulate application classes as those classes load into the JVM. Developers can pick Sun Hotspot or IBM's runtime, and any of several supported application servers
…
The Terracotta configuration file dictates which classes become clustered and which do not. Terracotta then examines classes for fields it needs to cluster, and threading semantics that need to be shared. For example, if to share customer objects throughout an application cluster, the developer need only tell Terracotta to cluster customers and to synchronize customers cluster-wide.

Terracotta looks for bytecode instructions like the following (not an exhaustive list):

• GETFIELD
• PUTFIELD
• AASTORE
• AALOAD
• MONITORENTRY
• MONITOREXIT

On each of those, Terracotta does the work of Network Attached Memory. Specifically:

BYTECODE	Injected Behavior
GETFIELD	Read from the Network for certain objects. Terracotta also has a heap-level cache that contains pure Java objects. So GETFIELD reads from RAM if-present and faults in from NAM if a cache miss occurs.
PUTFIELD	Write to the Network for certain objects. When writing field data through the assignment operator "=" or through similar mechanisms, Terracotta writes the changed bytes to NAM as well as allowing those to flow to the JVM's heap.
AASTORE	Same as PUTFIELD but for arrays
AALOAD	Sames as GETFIELD but for arrays
MONITORENTRY	Get a lock inside the JVM on the specified object AND get a lock in NAM in case a thread on another JVM is trying to edit this object at the same time
MONITOREXIT	Flush changes to the JVM's heap cache back to NAM in case another JVM is using the same objects as this JVM

The instrumented-classes section of the Terracotta config file is where application developers specify which objects types should be stored in the distributed cache and it is even possible to say that all memory allocations in your application should go through the distributed cache.

In general, the approach taken by Terracotta seems more complicated, more intrusive and more error prone than using a distributed hash table like Velocity or memcached. I always worry about systems that attempt to hide or abstract away the fact that network operations are occurring. This often leads to developers writing badly performing or unsafe code because it wasn't obvious that network operations are involved (e.g. a simple lock statement in your Terracotta-powered application may actually be acquiring distributed locks without it being explicit in the code that this is occuring).

Now Playing: Dream - I Luv Your Girl (Remix) (feat. Young Jeezy)

 

In the past year both Google and Facebook have released the remote procedure call (RPC) technologies that are used for communication between servers within their data centers as Open Source projects. 

Facebook Thrift allows you to define data types and service interfaces in a simple definition file. Taking that file as input, the compiler generates code to be used to easily build RPC clients and servers that communicate seamlessly across programming languages. It supports the following programming languages; C++, Java, Python, PHP and Ruby.

Google Protocol Buffers allows you to define data types and service interfaces in a simple definition file. Taking that file as input, the compiler generates code to be used to easily build RPC clients and servers that communicate seamlessly across programming languages. It supports the following programming languages; C++, Java and Python.

That’s interesting. Didn’t Steve Vinoski recently claim that RPC and it's descendants are "fundamentally flawed"? If so, why are Google and Facebook not only using RPC but proud enough of their usage of yet another distributed object RPC technology based on binary protocols that they are Open Sourcing them? Didn’t they get the memo that everyone is now on the REST + JSON/XML bandwagon (preferrably AtomPub)?

In truth, Google is on the REST + XML band wagon. Google has the Google Data APIs (GData) which is a consistent set of RESTful APIs for accessing data from Google's services based on the Atom Publishing Protocol aka RFC 5023. And even Facebook has a set of plain old XML over HTTP APIs (POX/HTTP) which they incorrectly refer to as the Facebook REST API.

So what is the story here?

It is all about coupling and how much control you have over the distributed end points. On the Web where you have little to no control over who talks to your servers or what technology they use, you want to utilize flexible technologies that make no assumptions about either end of the communication. This is where RESTful XML-based Web services shine. However when you have tight control over the service end points (e.g. if they are all your servers running in your data center) then you can use more optimized communications technologies that add a layer of tight coupling to your system. An example of the kind of tight coupling you have to live with is that  Facebook Thrift requires specific versions of g++ and Java if you plan to talk to it using code written in either language and you can’t talk to it from a service written in C#.

In general, the Web is about openness and loose coupling. Binary protocols that require specific programming languages and runtimes are the exact opposite of this. However inside your Web service where you control both ends of the pipe, you can optimize the interaction between your services and simplify development by going with a binary RPC based technology. More than likely different parts of your system are already doing this anyway (e.g. memcached uses a binary protocol to talk between cache instances, SQL Server uses TDS as the communications protocol between the database and it's clients, etc).

Always remember to use the right tool for the job. One size doesn’t fit all when it comes to technology decisions.

FURTHER READING

• Exposing a WCF Service With Multiple Bindings and Endpoints – Keith Elder describes how Windows Communication Foundation (WCF) supports multiple bindings that enable developers to expose their services in a variety of ways.  A developer can create a service once and then expose it to support net.tcp:// or http:// and various versions of http:// (Soap1.1, Soap1.2, WS*, JSON, etc).  This can be useful if a service crosses boundaries between the intranet and the Internet.

Now Playing: Pink - Family Portrait

 

A year ago Loren Feldman produced a controversial video called "TechNigga" which seems to still be causing him problems today. Matthew Ingram captures the latest fallout from that controversy in his post Protests over Verizon deal with 1938media where he writes

Several civil-rights groups and media watchdogs are protesting a decision by telecom giant Verizon to add 1938media’s video clips to its mobile Vcast service, saying Loren’s "TechNigga" clip is demeaning to black people. Project Islamic Hope, for example, has issued a statement demanding that Verizon drop its distribution arrangement with 1938media, which was just announced about a week ago, and other groups including the National Action Network and LA Humanity Foundation are also apparently calling for people to email Verizon and protest.

The video that has Islamic Hope and other groups so upset is one called "TechNigga," which Loren put together last August. After wondering aloud why there are no black tech bloggers, Loren reappears with a skullcap and some gawdy jewelry, and claims to be the host of a show called TechNigga. He then swigs from a bottle of booze, does a lot of tongue-kissing and face-licking with his girlfriend Michelle Oshen, and then introduces a new Web app called "Ho-Trackr," which is a mashup with Google Maps that allows prospective johns to locate prostitutes. In a statement, Islamic Hope says that the video "sends a horrible message that Verizon seeks to partner with racists."

I remember encountering the video last year and thinking it was incredibly unfunny. It wasn’t a clever juxtaposition of hip hop culture and tech geekery. It wasn’t satire since that involves lampooning someone or something you disapprove off in a humorous way (see The Colbert Report).  Of course, I thought the responses to the video were even dumber; like Robert Scoble responding to the video with the comment “Dare Obasanjo is black”.

Since posting the video Loren Feldman has lost a bunch of video distribution deals with the current Verizon deal being the latest. I’ve been amused to read all of the comments on TechCrunch about how this violates Loren’s freedom of speech.

People often confuse the fact that it is not a crime to speak your mind in America with the belief that you should be able to speak your mind without consequence. The two things are not the same. If I call you an idiot, I may not go to jail but I shouldn’t expect you to be nice to me afterwards. The things you say can come back and bite you on butt is something everyone should have learned growing up. So it is always surprising for me to see people petulantly complain that “this violates my freedom of speech” when they have to deal with the consequences of their actions.

BONUS VIDEO: A juxtaposition of hip hop culture and Web geekery by a black tech blogger.

Now Playing: NWA – N*ggaz 4 Life

 

Gnip is a newly launched startup that pitches itself as a service that aims to “make data portability suck less”. Mike Arrington describes the service in his post Gnip Launches To Ease The Strain On Web Services which is excerpted below

A close analogy is a blog ping server (see our overview here). Ping servers tell blog search engines like Technorati and Google Blog Search when a blog has been updated, so the search engines don’t have to constantly re-index sites just to see if new content has been posted. Instead, the blog tells the ping server when it updates, which tells the search engines to drop by and re-index. The creation of the first ping server, Weblogs.com, by Dave Winer resulted in orders of magnitude better efficiency for blog search engines.

The same thinking basically applies to Gnip. The idea is to gather simple information from social networks - just a username and the fact that they created new content (like writing a Twitter message, for example). Gnip then distributes that data to whoever wants it, and those downstream services can then access the core service’s API, with proper user authentication, and access the actual data (in our example, the actual Twitter message).

From a user’s perspective, the result is faster data updates across services and less downtime for services since their APIs won’t be hit as hard.

From my perspective, Gnip also shares some similarity to services like FeedBurner as well as blog ping servers. The original purpose of blog ping servers was to make it cheaper for services like Technorati and Feedster to index the blogosphere without having to invest in a Google-sized server farm and crawl the entire Web every couple of minutes. In addition, since blogs often have tiny readerships and are thus infrequently linked to, crawling alone was not enough to ensure that they find their way into the search index. It wasn’t about taking load off of the sites that were doing the pinging.

On the other hand, FeedBurner hosts a site’s RSS feed as a way to take load off of their servers and then provides analytics data so the site doesn’t miss out from losing the direct connection to its subscribers. This is more in line with the expectation that Gnip will take load off of a service’s API servers. However unlike FeedBurner, Gnip doesn’t actually store the user data from the social networking site. It simply stores a record that indicates that “user X on site Y made an update of type Z at time T”.  The thinking is that web sites will publish a notification to Gnip whenever their users perform an update. Below is a sample interaction between Digg and Gnip where Digg notifies Gnip that the users amy and john.doe have dugg two stories.

===>
  POST /publishers/digg/activity.xml
  Accept: application/xml
  Content-Type: application/xml
  
  <activities>
    <activity at="2008-06-08T10:12:42Z" uid="amy" type="dugg" guid="http://digg.com/odd_stuff/a_story"/>
    <activity at="2008-06-09T09:14:07Z" uid="john.doe" type="dugg" guid="http://digg.com/odd_stuff/really_weird"/>
  </activities>

<---
  200 OK
  Content-Type: application/xml
  <result>Success</result> 

There are two modes in which "subscribers" can choose to interact with the data published to Gnip. The first is in a mode similar to how blog search engines interact with the changes.xml file on Weblogs.com and other blog ping servers. For example, services like Summize or TweetScan can ask Gnip for the last hour of changes on Twitter instead of whatever mechanism they are using today to crawl the site. Below is what a sample interaction to retrieve the most recent updates on Twitter from Gnip would look like

===>
  GET /publishers/twitter/activity/current.xml
  Accept: application/xml

<---
  200 OK
  Content-Type: application/xml
  
  <activities>
    <activity at="2008-06-08T10:12:07Z" uid="john.doe" type="tweet" guid="http://twitter.com/john.doe/statuses/42"/>
    <activity at="2008-06-08T10:12:42Z" uid="amy" type="tweet" guid="http://twitter.com/amy/statuses/52"/>
  </activities>   

The main problem with this approach is the same one that affects blog ping servers. If the rate of updates is more than the ping server can handle then it may begin to fall behind or lose updates completely. Services that don’t want to risk their content not being crawled are best off providing their own update stream that applications can poll periodically. That’s why the folks at Six Apart came up with the Six Apart Update Stream for LiveJournal, TypePad and Vox weblogs.

The second mode is one that has gotten Twitter fans like Dave Winer raving about Gnip being the solution to Twitter’s scaling problems. In this mode, an application creates a collection of one or more usernames they are interested in. Below is what a collection document created by the Twadget application to indicate that it is interested in my Twitter updates might look like.

<collection name="twadget-carnage4life">
     <uid name="carnage4life" publisher.name="twitter"/>
</collection>

Then instead of polling Twitter every 5 minutes for updates it polls Gnip every 5 minutes for updates and only talks to Twitter’s servers when Gnip indicates that I’ve made an update since the last time the application polled Gnip. The interaction between Twadget and Gnip would then be as follows

===>
  GET /collections/twadget-carnage4life/activity/current.xml
  Accept: application/xml
<---
  200 OK
  Content-Type: application/xml
  
  <activities>
    <activity at="2008-06-08T10:12:07Z" uid="carnage4life" type="tweet" guid="http://twitter.com/Carnage4Life/statuses/850726804"/>    
  </activities> 

Of course, this makes me wonder why one would think that it is feasible for Gnip to build a system that can handle the API polling traffic of every microblogging and social networking site out there but it is infeasible for Twitter to figure out how to handle the polling traffic for their own service. Talk about lowered expectations.

So what do I think of Gnip? I think the ping server mode may be of some interest for services that think it is cheaper to have code that pings Gnip after every user update instead building out an update stream service. However since a lot of sites already have some equivalent of the public timeline it isn’t clear that there is a huge need for a ping service. Crawlers can just hit the public timeline which I assume is what services like Summize and TweetScan do to keep their indexes of tweets up to date.

As for using Gnip as a mechanism for reducing the load API clients put on a microblogging or similar service? Gnip is totally useless for that in it’s current incarnation. API clients aren’t interested in updates made by single user. They are interested in all the updates made by all the people the user is following. So for Twadget to use Gnip to lighten the load it causes on Twitter’s servers on my behalf, it has to build a collection of all the people I am following in Gnip and then keep that list of users in sync with whatever that list is on Twitter. But if it has to constantly poll Twitter for my friend list, isn’t it still putting the same amount of load on Twitter? I guess this could be fixed by having Twitter publish follower/following lists to Gnip but that introduces all sorts of interesting technical and privacy issues. But that doesn’t matter since the folks at Gnip brag about only keeping 60 minutes of worth of updates as the “secret sauce” to their scalability. This means if I shut my Twitter client hasn’t polled Gnip in a 60 minute window (maybe my laptop is closed) then it doesn’t matter anyway and it has to poll Twitter.  I suspect someone didn’t finish doing their homework before rushing to “launch” Gnip.

PS: One thing that is confusing to me is why all communication between applications and Gnip needs to be over SSL. The only thing I can see it adding is making it more expensive for Gnip run their service. I can’t think of any reason why the interactions described above need to be over a secure channel.

Now Playing: Lil Wayne - Hustler Musik

 

Every once in a while I encounter an online service or Web site that is so irritating that it seems like the people behind the service are just in it to frustrate Web users. And I don’t mean the obvious candidates like email spammers and purveyors of popup ads since they’ve been around for so long I’ve either learned how to ignore and avoid them.

There is a new generation of irritants and many of them are part of the new lunacy we call “Web 2.0”

1. Flash Widgets with Embedded PDF Documents: Somewhere along the line a bunch of startups decided that they needed to put a “Web 2.0” spin on the simple concept of hosting people’s office documents online. You see, lots of people would like to share documents in PDF or Microsoft Office® formats that aren’t particularly Web friendly. So how have sites like Scribd and Docstoc fixed this problem? By creating a Flash widgets containing the embedded PDF/Office documents like the one shown here. So not only are the documents still in a Web unfriendly format but now I can’t even download them and use the tools on my desktop to read them. It’s like let’s combine the FAIL of putting non-Web documents on the Web with the fail of a Web-unfriendly format like Flash. FAIL++. By the way, it’s pretty ironic that a Microsoft enterprise product gets this right where so many “Web 2.0” startups get it wrong.

2. Hovering Over Links Produces Flash Widgets as Pop Over Windows: The company that takes the cake for spreading this major irritant across the blogosphere is Snap Technologies and their Snap Shots™ product. There’s nothing quite as irritating as hovering over a link on your way to click another link and leaving a wake of pop over windows with previews of the Web pages at the end of said links. I seriously wonder if anyone finds this useful?

3. Facebook Advertisers: One of the promises of Facebook is that its users will see more relevant advertising because there is all this rich demographic data about the site’s users in their profiles. Somewhere along the line this information is either getting lost or being ignored by Facebook’s advertisers. Even though my profile says I’m married and out of my twenties I keep getting borderline sleazy ads whenever I login to play Scrabulous asking if I want to meet college girls. Then there are the ads which aren’t for dating sites but still use sleazy imagery anyway. It’s mad embarrassing whenever my wife looks over to see what I’m doing on my laptop to have dating site ads blaring in her face. Obviously she knows I’m not on a dating site but still…

4. Forums that Require Registration Showing Up in Search Results : Every once in a while I do a Web search for a programming problem and a couple of links to Experts Exchange end up in the results. What is truly annoying about this site is that the excerpt on the search result page makes  it seem as though the answer to your question is one click away but when you click through you are greeted with “All comments and solutions are available to Premium Service Members only”. I thought search engines had rules about banning sites with that sort of obnoxious behavior?

5. Newspaper Websites with Interstitial Ads and Registration Requirements: Newspapers such as the New York Times often act as if they don’t really want me reading the content on their Web site. If I click on a link to a story on the New York Times site such as this one, one of two things will happen; I’m either taken to a full page animated advertisement with an option to skip the ad in relatively small font or I get a one sentence summary of the story with a notice that I need to register on their Web site before I can read the story. Either way it’s a bunch of bull crap that prevents me from getting to the news.

There are two things that strike me about this list as notable. The first is that there are an increasing number of “Web 2.0” startups out there who are actively using Flash to cause more problems than they claim to be solving. The second is that requiring registration to view content is an amazingly stupid trend that is beyond dumb. It’s not like people need to register on your site to see ads so why reduce the size of your potential audience by including this road block? That’s just stupid.

Now Playing: Pleasure P - Rock Bottom (feat. Lil Wayne)

 

Jeff Atwood recently published two anti-XML rants in his blog entitled XML: The Angle Bracket Tax and Revisiting the XML Angle Bracket Tax. The source of his beef with XML and his recommendations to developers are excerpted below

Everywhere I look, programmers and programming tools seem to have standardized on XML. Configuration files, build scripts, local data storage, code comments, project files, you name it -- if it's stored in a text file and needs to be retrieved and parsed, it's probably XML. I realize that we have to use something to represent reasonably human readable data stored in a text file, but XML sometimes feels an awful lot like using an enormous sledgehammer to drive common household nails.

I'm deeply ambivalent about XML. I'm reminded of this Winston Churchill quote:

It has been said that democracy is the worst form of government except all the others that have been tried.

XML is like democracy. Sometimes it even works. On the other hand, it also means we end up with stuff like this:

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <SOAP-ENV:Body>
    <m:GetLastTradePrice xmlns:m="Some-URI">
      <symbol>DIS</symbol>
    </m:GetLastTradePrice>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

…
You could do worse than XML. It's a reasonable choice, and if you're going to use XML, then at least learn to use it correctly. But consider:

1. Should XML be the default choice?
2. Is XML the simplest possible thing that can work for your intended use?
3. Do you know what the XML alternatives are?
4. Wouldn't it be nice to have easily readable, understandable data and configuration files, without all those sharp, pointy angle brackets jabbing you directly in your ever-lovin' eyeballs?

I don't necessarily think XML sucks, but the mindless, blanket application of XML as a dessert topping and a floor wax certainly does. Like all tools, it's a question of how you use it. Please think twice before subjecting yourself, your fellow programmers, and your users to the XML angle bracket tax. <CleverEndQuote>Again.</CleverEndQuote>

The question of if and when to use XML is one I am intimately familiar with given that I spent the first 2.5 years of my professional career at Microsoft working on the XML team as the “face of XML” on MSDN.

My problem with Jeff’s articles is that they take a very narrow view of how to evaluate a technology. No one should argue that XML is the simplest or most efficient technology to satisfy the uses it has been put to today. It isn’t. The value of XML isn’t in its simplicity or its efficiency. It is in the fact that there is a massive ecosystem of knowledge and tools around working with XML.

If I decide to use XML for my data format, I can be sure that my data will be consumable using a variety off-the-shelf tools on practically every platform in use today. In addition, there are a variety of tools for authoring XML, transforming it to HTML or text, parsing it, converting it to objects, mapping it to database schemas, validating it against a schema, and so on. Want to convert my XML config file into a pretty HTML page? I can use XSLT or CSS. Want to validate my XML against a schema? I have my choice of Schematron, Relax NG and XSD. Want to find stuff in my XML document? XPath and XQuery to the rescue. And so on.

No other data format hits a similar sweet spot when it comes to ease of use, popularity and breadth of tool ecosystem.

So the question you really want to ask yourself before taking on the “Angle Bracket Tax” as Jeff Atwood puts it, is whether the benefits of avoiding XML outweigh the costs of giving up the tool ecosystem of XML and the familiarity that practically every developer out there has with the technology? In some cases this might be true such as when deciding whether to go with JSON over XML in AJAX applications (I’ve given two reasons in the past why JSON is a better choice).  On the other hand, I can’t imagine a good reason to want to roll your own data format for office documents or application configuration files as opposed to using XML.

FURTHER READING

• The XML Litmus Test - Dare Obasanjo provides some simple guidelines for determining when XML is the appropriate technology to use in a software application or architecture design. (6 printed pages)
• Understanding XML - Learn how the Extensible Markup Language (XML) facilitates universal data access. XML is a plain-text, Unicode-based meta-language: a language for defining markup languages. It is not tied to any programming language, operating system, or software vendor. XML provides access to a plethora of technologies for manipulating, structuring, transforming and querying data. (14 printed pages)

Now Playing: Metallica - The God That Failed

 

Late last week, the folks on the Google Data APIs blog announced that Google will now be supporting OAuth as the delegated authentication mechanism for all Google Data APIs. This move is meant to encourage the various online services that provide APIs that access a user’s data in the “cloud” to stop reinventing the wheel when it comes to delegated authentication and standardize on a single approach.

Every well-designed Web API that provides access to a customer’s data in the cloud utilizes a delegated authentication mechanism which allows users to grant 3rd party applications access to their data without having to give the application their username and password. There is a good analogy for this practice in the OAuth: Introduction page which is excerpted below

What is it For?

Many luxury cars today come with a valet key. It is a special key you give the parking attendant and unlike your regular key, will not allow the car to drive more than a mile or two. Some valet keys will not open the trunk, while others will block access to your onboard cell phone address book. Regardless of what restrictions the valet key imposes, the idea is very clever. You give someone limited access to your car with a special key, while using your regular key to unlock everything.

Everyday new website offer services which tie together functionality from other sites. A photo lab printing your online photos, a social network using your address book to look for friends, and APIs to build your own desktop application version of a popular site. These are all great services – what is not so great about some of the implementations available today is their request for your username and password to the other site. When you agree to share your secret credentials, not only you expose your password to someone else (yes, that same password you also use for online banking), you also give them full access to do as they wish. They can do anything they wanted – even change your password and lock you out.

This is what OAuth does, it allows the you the User to grant access to your private resources on one site (which is called the Service Provider), to another site (called Consumer, not to be confused with you, the User). While OpenID is all about using a single identity to sign into many sites, OAuth is about giving access to your stuff without sharing your identity at all (or its secret parts).

So every service provider invented their own protocol to do this, all of which are different but have the same basic components. Today we have Google AuthSub, Yahoo! BBAuth, Windows Live DelAuth, AOL OpenAuth, the Flickr Authentication API, the Facebook Authentication API and others. All different, proprietary solutions to the same problem.

This ends up being problematic for developers because if you want to build an application that talks to multiple services you not only have to deal with the different APIs provided by these services but also the different authorization/authentication models they utilize as well. In a world where “social aggregation” is becoming more commonplace with services like Plaxo Pulse & FriendFeed and more applications are trying to bridge the desktop/cloud divide like OutSync and RSS Bandit, it sucks that these applications have to rewrite the same type of code over and over again to deal with the basic task of getting permission to access a user’s data. Standardizing on OAuth is meant to fix that. A number of startups like Digg & Twitter as well as major players like Yahoo and Google have promised to support it, so this should make the lives of developers easier.

Of course, we still have work to do as an industry when it comes to the constant wheel reinvention in the area of Web APIs. Chris Messina points to another place where every major service provider has invented a different proprietary protocol for doing the same task in his post Inventing contact schemas for fun and profit! (Ugh) where he writes

And then there were three
...
Today, Yahoo! announced the public availability of their own Address Book API.

However, I have to lament yet more needless reinvention of contact schema. Why is this a problem? Well, as I pointed out about Facebook’s approach to developing their own platform methods and formats, having to write and debug against yet another contact schema makes the “tax” of adding support for contact syncing and export increasingly onerous for sites and web services that want to better serve their customers by letting them host and maintain their address book elsewhere.

This isn’t just a problem that I have with Yahoo!. It’s something that I encountered last November with the SREG and proposed Attribute Exchange profile definition. And yet again when Google announced their Contacts API. And then again when Microsoft released theirs! Over and over again we’re seeing better ways of fighting the password anti-pattern flow of inviting friends to new social services, but having to implement support for countless contact schemas. What we need is one common contacts interchange format and I strongly suggest that it inherit from vcard with allowances or extension points for contemporary trends in social networking profile data.

I’ve gone ahead and whipped up a comparison matrix between the primary contact schemas to demonstrate the mess we’re in.

Kudos to the folks at Google for trying to force the issue when it comes to standardizing on a delegated authentication protocol for use on the Web. However there are still lots of places across the industry where we speak different protocols and thus incur a needless burden on developers when a single language might do. It would be nice to see some of this unnecessary redundancy eliminated in the future.

Now Playing: G-Unit - I Like The Way She Do It

 

Recently I’ve been bumping into more and more people who’ve either left Google to come to Microsoft or got offers from both companies and picked Microsoft over Google. I believe this is part of a larger trend especially since I’ve seen lots of people who left the company for “greener pastures” return in the past year (at least 8 people I know personally have rejoined) . However in this blog post I’ll stick to talking about people who’ve chosen Microsoft over Google.

First of all there’s the post by Sergey Solyanik entitled Back to Microsoft where he primarily gripes about the culture and lack of career development at Google, some key excerpts are

Last week I left Google to go back to Microsoft, where I started this Monday (and so not surprisingly, I was too busy to blog about it)
…
So why did I leave?

There are many things about Google that are not great, and merit improvement. There are plenty of silly politics, underperformance, inefficiencies and ineffectiveness, and things that are plain stupid. I will not write about these things here because they are immaterial. I did not leave because of them. No company has achieved the status of the perfect workplace, and no one ever will.

I left because Microsoft turned out to be the right place for me.
…
Google software business is divided between producing the "eye candy" - web properties that are designed to amuse and attract people - and the infrastructure required to support them. Some of the web properties are useful (some extremely useful - search), but most of them primarily help people waste time online (blogger, youtube, orkut, etc)
…
This orientation towards cool, but not necessarilly useful or essential software really affects the way the software engineering is done. Everything is pretty much run by the engineering - PMs and testers are conspicuously absent from the process. While they do exist in theory, there are too few of them to matter.

On one hand, there are beneficial effects - it is easy to ship software quickly…On the other hand, I was using Google software - a lot of it - in the last year, and slick as it is, there's just too much of it that is regularly broken. It seems like every week 10% of all the features are broken in one or the other browser. And it's a different 10% every week - the old bugs are getting fixed, the new ones introduced. This across Blogger, Gmail, Google Docs, Maps, and more
…
The culture part is very important here - you can spend more time fixing bugs, you can introduce processes to improve things, but it is very, very hard to change the culture. And the culture at Google values "coolness" tremendously, and the quality of service not as much. At least in the places where I worked.
…
The second reason I left Google was because I realized that I am not excited by the individual contributor role any more, and I don't want to become a manager at Google.

The Google Manager is a very interesting phenomenon. On one hand, they usually have a LOT of people from different businesses reporting to them, and are perennially very busy.

On the other hand, in my year at Google, I could not figure out what was it they were doing. The better manager that I had collected feedback from my peers and gave it to me. There was no other (observable by me) impact on Google. The worse manager that I had did not do even that, so for me as a manager he was a complete no-op. I asked quite a few other engineers from senior to senior staff levels that had spent far more time at Google than I, and they didn't know either. I am not making this up!

Sergey isn’t the only senior engineer I know who  has contributed significantly to Google projects and then decided Microsoft was a better fit for him. Danny Thorpe who worked on Google Gears is back at Microsoft for his second stint working on developer technologies related to Windows Live.  These aren’t the only folks I’ve seen who’ve decided to make the switch from the big G to the b0rg, these are just the ones who have blogs that I can point at.

Unsurprisingly, the fact that Google isn’t a good place for senior developers is also becoming clearly evident in their interview processes. Take this post from Svetlin Nakov entitled Rejected a Program Manager Position at Microsoft Dublin - My Successful Interview at Microsoft where he concludes

My Experience at Interviews with Microsoft and Google

Few months ago I was interviewed for a software engineer in Google Zurich. If I need to compare Microsoft and Google, I should tell it in short: Google sux! Here are my reasons for this:

1) Google interview were not professional. It was like Olympiad in Informatics. Google asked me only about algorithms and data structures, nothing about software technologies and software engineering. It was obvious that they do not care that I had 12 years software engineering experience. They just ignored this. The only think Google wants to know about their candidates are their algorithms and analytical thinking skills. Nothing about technology, nothing about engineering.

2) Google employ everybody as junior developer, ignoring the existing experience. It is nice to work in Google if it is your first job, really nice, but if you have 12 years of experience with lots of languages, technologies and platforms, at lots of senior positions, you should expect higher position in Google, right?

3) Microsoft have really good interview process. People working in Microsoft are relly very smart and skillful. Their process is far ahead of Google. Their quality of development is far ahead of Google. Their management is ahead of Google and their recruitment is ahead of Google.

Microsoft is Better Place to Work than Google

At my interviews I was asking my interviewers in both Microsoft and Google a lot about the development process, engineering and technologies. I was asking also my colleagues working in these companies. I found for myself that Microsoft is better organized, managed and structured. Microsoft do software development in more professional way than Google. Their engineers are better. Their development process is better. Their products are better. Their technologies are better. Their interviews are better. Google was like a kindergarden - young and not experienced enough people, an office full of fun and entertainment, interviews typical for junior people and lack of traditions in development of high quality software products.

Based on my observations, I have theory that Google’s big problem is that the company hasn’t realized that it isn’t a startup anymore. This disconnect between the company’s status and it’s perception of itself manifests in a number of ways

1. Startups don’t have a career path for their employees. Does anyone at Facebook know what they want to be in five years besides rich? However once riches are no longer guaranteed and the stock isn’t firing on all cylinders (GOOG is underperforming both the NASDAQ and DOW Jones industrial average this year) then you need to have a better career plan for your employees that goes beyond “free lunches and all the foosball you can handle".

2. There is no legacy code at a startup. When your code base is young, it isn’t a big deal to have developers checking in new features after an overnight coding fit powered by caffeine and pizza. For the most part, the code base shouldn’t be large enough or interdependent enough for one change to cause issues. However it is practically a law of software development that the older your code gets the more lines of code it accumulates and the more closely coupled your modules become. This means changing things in one part of the code can have adverse effects in another. 

   As all organizations mature they tend to add PROCESS. These processes exist to insulate the companies from the mistakes that occur after a company gets to a certain size and can no longer trust its employees to always do the right thing. Requiring code reviews, design specifications, black box & whitebox & unit testing, usability studies, threat models, etc are all the kinds of overhead that differentiate a mature software development shop from a “fly by the seat of your pants” startup. However once you’ve been through enough fire drills, some of those processes don’t sound as bad as they once did. This is why senior developers value them while junior developers don’t since the latter haven’t been around the block enough.

3. There is less politics at a startup. In any activity where humans have to come together collaboratively to achieve a goal, there will always be people with different agendas. The more people you add to the mix, the more agendas you have to contend with. Doing things by consensus is OK when you have to get consensus from two or three people who sit in the same hallway as you. It’s a totally different ball game when you need to gain it from lots of people from across a diverse company working on different projects in different regions of the world who have different perspectives on how to solve your problems. At Google, even hiring an undergraduate candidate has to go through several layers of committees which means hiring managers need to possess some political savvy if they want to get their candidates approved.  The founders of Dodgeball quit the Google after their startup was acquired after they realized that they didn’t have the political savvy to get resources allocated to their project.

The fact that Google is having problems retaining employees isn't news, Fortune wrote an article about it just a few months ago. The technology press makes it seem like people are ditching Google for hot startups like FriendFeed and Facebook. However the truth is more nuanced than that. Now that Google is just another big software company, lots of people are comparing it to other big software companies like Microsoft and finding it lacking.

Now Playing: Queen - Under Pressure (feat. David Bowie)

 

Last week TechCrunch UK wrote about a search startup that utilizes AI/Semantic Web techniques named True Knowledge. The post entitled VCs price True Knowledge at £20m pre-money. Is this the UK’s Powerset?  stated

The chatter I’m hearing is that True Knowledge is being talked about in hushed tones, as if it might be the Powerset of the UK. To put that in context, Google has tried to buy the Silicon Valley search startup several times, and they have only launched a showcase product, not even a real one. However, although True Knowledge and Powerset are similar, they are different in significant ways, more of which later.
...
Currently in private beta, True Knowledge says their product is capable of intelligently answering - in plain English - questions posed on any topic. Ask it if Ben Affleck is married and it will come back with "Yes" rather than lots of web pages which may or may not have the answer (don’t ask me!).
...
Here’s why the difference matters. True Knowledge can infer answers that the system hasn’t seen. Inferences are created by combining different bits of data together. So for instance, without knowing the answer it can work out how tall the Eiffel Tower is by inferring that it is shorter that the Empire State Building but higher than St Pauls Cathedral.
...
AI software developer and entrepreneur William Tunstall-Pedoe is the founder of True Knowledge. He previously developed a technology that can solve a commercially published crossword clues but also explain how the clues work in plain English. See the connection?

The scenarios described in the TechCrunch write up should sound familiar to anyone who has spent any time around fans of the Semantic Web. Creating intelligent agents that can interrogate structured data on the Web and infer new knowledge has turned out to  be easier said than done because for the most part content on the Web isn't organized according to the structure of the data. This is primarily due to the fact that HTML is a presentational language. Of course, even if information on the Web was structured data (i.e. idiomatic XML formats) we still need to build machinary to translate between all of these XML formats.

Finally, in the few areas on the Web where structured data in XML formats is commonplace such as Atom/RSS feeds for blog content, not a lot has been done with this data to fulfill the promise of the Semantic Web.

So if the Semantic Web is such an infeasible utopia, why are more and more search startups using that as the angle from which they will attack Google's dominance of Web search? The answer can be found in Bill Slawski's post from a year ago entitled Finding Customers Through Anti-Commercial Queries where he wrote

Most Queries are Noncommercial

The first step might be to recognize that most queries conducted by people at search engines aren't aimed at buying something. A paper from the WWW 2007 held this spring in Banff, Alberta, Canada, Determining the User Intent of Web Search Engine Queries, provided a breakdown of the types of queries that they were able to classify.

Their research uncovered the following numbers: "80% of Web queries are informational in nature, with about 10% each being navigational and transactional." The research points to the vast majority of searches being conducted for information gathering purposes. One of the indications of "information" queries that they looked for were searches which include terms such as: “ways to,” “how to,” “what is.”

Although the bulk of the revenue search engines make is from people performing commercial queries such as searching for "incredible hulk merchandise", "car insurance quotes" or "ipod prices", this is actually a tiny proportion of the kinds of queries people want answered by search engines. The majority of searches are about the five Ws (and one H) namely "who", "what", "where", "when", "why" and "how". Such queries don't really need a list of Web pages as results, they simply require an answer. The search engine that can figure out how to always answer user queries directly on the page without making the user click on half a dozen pages to figure out the answer will definitely have moved the needle when it comes to the Web search user experience.

This explains why scenarios that one usually associates with AI and Semantic Web evangelists are now being touted by the new generation of "Google-killers". The question is whether knowledge inference techniques will prove to be more effective than traditional search engine techniques when it comes to providing the best search results especially since a lot of the traditional search engines are learning new tricks.

Now Playing: Bob Marley - Waiting In Vain

 

 At the end of February of this year, I wrote a post entitled No Contest: FriendFeed vs. The Facebook News Feed where I argued that it would be a two month project for an enterprising developer at Facebook to incorporate all of the relevant features of FriendFeed that certain vocal bloggers had found so enticing. Since then we've had two announcements from Facebook

From A new way to share with friends on April 15th

we've introduced a way for you to import activity from other sites into your Mini-Feed (and into your friends' News Feeds).

The option to import stories from other sites can be found via the small "Import" link at the top of your Mini-Feed. Only a few sites—Flickr, Yelp, Picasa, and del.icio.us—are available for importing at the moment, but we'll be adding Digg and other sites in the near future. These stories will look just like any other Mini-Feed stories, and will hopefully increase your ability to share information with the people you care about.

From on We're Open For Commentary on June 25th (Yesterday)

In the past, you've been able to comment on photos, notes and posted items, but if there was something else on your friend's profile—an interesting status, or a cool new friendship—you'd need to send a message or write a Wall post to talk about it. But starting today, you can comment on your friends' Mini-Feed stories right from their profile.

Now you can easily converse around friends' statuses, application stories, new friendships, videos, and most other stories you see on their profile. Just click on the comment bubble icon to write a comment or see comments other people have written.

It took a little longer than two months but it looks like I was right. For some reason Facebook isn't putting the comment bubbles in the news feed but I assume that is only temporary and they are trying it out in the mini-feed first.

FriendFeed has always seemed to me to be a weird concept for a stand alone application. Why would I want to go to whole new site and create yet another friend list just to share what I'm doing on the Web with my friends? Isn't that what social networking sites are for? It just sounds so inconvenient, like carrying around a pager instead of a mobile phone.

As I said in my original post on the topic, all FriendFeed has going for it is the community that has built around the site. Especially since the functionality it provides can be easily duplicated and actually fits better as a feature of an existing social networking site. The question is whether that community is the kind that will grow into making it a mainstream success or whether it will remain primarily a playground for Web geeks despite all the hype (see del.icio.us as an example of this). So far, the chance of the latter seems strong. For comparison, consider the growth curve of Twitter against that of FriendFeed on Google Trends and Alexa.  Which seems more likely to one day have the brand awareness of a Flickr or a Facebook?

Now Playing: Bob Marley - I Shot The Sheriff