In my previous post, I talked about some of the issues I saw with the idea of doing away with operations teams and merging their responsibilities into the development team's tasks [as practised at companies like Amazon]. Justin Rudd, who is a developer at Amazon, posts his first-hand perspective of this practice in his blog post entitled Expanding on the Pain where he writes

Since I am a current employee of Amazon in the software development area, I probably shouldn’t be saying this, but…
...

First a few clarifications - there is no dedicated operations team for Amazon as a whole that is correct.  But each team is allowed to staff as they see fit.  There are teams within Amazon that have support teams that do handle quite a bit of the day to day load.  And their systems tend to be more “smooth” because this is what that team does - keep the system up and running and automate keeping the system up and running so they can sleep at night.

There are also teams dedicated to networking, box failures, etc.  So don’t think that developers have to figure out networking issues all the time (although there are sometimes where networking doesn’t see a problem but it is affecting a service).

Now for those teams that do not have a support team (and I am on one of them), at 3 in the morning you tend to do the quickest thing possible to get the problem rectified.  Do you get creative?  After being in bed for 3 hours (if you’re lucky) and having a VP yell at you on the phone that this issue is THE most important issue there is or having someone yell at you that they are going to send staff home, how creative do you think you can be?  Let me tell you, not that creative.  You’re going to solve the problem, make the VP happy (or get the factory staff back to work), and go back to bed with a little post it note to look for root cause of the problem in the AM.

Now 1 of 2 things happens.  If you have a support team, you let them know about what happened, you explain the symptoms that you saw, how you fixed it, etc.  They take your seed of an idea, plant it, nurture it, and grow it.

If you don’t have a support team and you are lucky, in the morning there won’t be another THE most important thing to work on and you can look at the problem with some sleep and some creativity.  But the reality is - a lot of teams don’t have that luxury.  So what happens?  You end up cronning your solution which may be to bounce your applications every 6 hours or run a perl script that updates a field at just the right place in the database, etc.

We all have every intention of fixing it, but remember that VP that was screaming about how this issue had to be fixed?  Well now that it isn’t an issue anymore and it’s off his radar screen, he has new features that he wants pushed into your code.  And those new features are much more important than you fixing the issue from the night before because the VP really doesn’t care if you get sleep or not at night.

Justin's account jibes with other accounts I've heard [second hand] from other ex-Amazon developers about what it means to live without an operations team. Although it sounds good on paper to have the developers responsible for writing the code also responsible when there are issues with the code on the live site, it leads to burning the candle at both ends. Remember, division of labor exists for a reason.
 

Categories: Web Development

A few weeks ago, I bookmarked a post from Sam Ruby entitled Collapsing the Stack where he wrote

Werner Vogels: Yep, the best way to completely automate operations is to have to developers be responsible for running the software they develop. It is painful at times, but also means considerable creativity gets applied to a very important aspect of the software stack. It also brings developers into direct contact with customers and a very effective feedback loop starts. There is no separate operations department at Amazon: you build it; you run it.

Sounds like a very good idea.

I don't see how this sounds like a good idea. This reminds me of a conversation I once had with someone at Microsoft who thought it would be a good idea to get rid of their test team and replace them all with developers once they moved to Test Driven Development. I used to be a tester when I first joined Microsoft and this seemed to me to be the kind of statement made by someone who assumed that the only thing testers do is write unit tests. Good test teams don't just write unit tests. They develop and maintain test tools. They perform system integration testing. They manage the team's test beds and test labs. They are the first line of defence when attempting to reproduce customer bug reports face before pulling in developers who may be working on your next release. All of this can be done by the development team but it means that your developers spend less time developing and more time testing. This cost will show up either as an increment in the amount of time it takes to get to market or a reduction in quality if schedules are not adjusted to account for this randomization of the development team. Eventually you'll end up recreating your test team so there are specific people responsible for test-related activities [which is why software companies have test teams in the first place].

The same reasoning applies to the argument for folding the responsibilities of your operations team into the development team's tasks. A good operations team isn't just responsible deployment/setup of applications on your servers and monitoring the health of the Web servers or SQL databases inyour web farm. A good operations team is involved in designing your hardware SKUs and understanding your service's peak capacity so as to optimize purchase decisions. A good operations team makes the decisions around your data centers from picking locations with the best power prices and ensuring that you're making optimal use of all the physical space in your data center to making build . A good operations team is the first line of defence when your service is being hit by a Denial of Service attack. A good operations team insulates the team from worrying about operating system, web server or database patches being made to the live site. A good operations team is involved in the configuration, purchase, deployment and [sometimes] development of load balancing, database partitioning and database replication tools. Again, you can have your development team do this but eventually it would seem to make sense that these tasks be owned by specific individuals instead of splitting them across the developers building one's core product.

PS: I've talked to a bunch of folks who know ex-Amazon developers and they tend to agree with my analysis above. I'd be interested in getting the perspective of ex-Amazon developers like Greg Linden on replacing your operations team with your core development staff.

PPS: Obviously this doesn't apply if you are a small 2 to 5 person startup. Everyone ends up doing everything anyway. :)


 

Categories: Web Development

August 8, 2006
@ 08:30 PM

This morning I got an IM from Niall Kennedy letting me know that he was Leaving Microsoft. He begins his blog post about leaving by writing

I am leaving Microsoft to start my own company. My last day at Microsoft is next Friday, August 18. It's uncertain whether Microsoft will continue the feed platform work I started, but it's some good stuff so I hope they do.

As the person who referred Niall to the company and gave him some advice when he was weighing whether to join Windows Live, I am sad to see him leave so soon. I sympathize with his reasons for leaving although some of what he wrote is inaccurate and based on speculation rather than the actual facts of the matter. That said, I found Niall to be quite knowledgeable, smart and passionate, so I expect him to do well in his endeavors.

Good luck, Niall.


 

Don Demsak has a blog post entitled Open Source Projects As A Form Of Community Service which links to a number of blog posts about the death of the NDoc project. He writes

Open source projects have been the talk of the tech blogs recently with the announcement that NDoc 2 is Offcially Dead, along with the mention that the project's sole develop was a victim of an automated mail-bomb attack because the project wasn't getting a .Net 2.0 version out fast enough for their liking.  Kevin has decided to withdraw from the community, and fears for himself and his family.  The .Net blogging community has had a wide range of reactions:

  • Phil Haack talks about his ideas behind helping/saving the open source community and laid down a challenge. 
  • Eric Wise mentions that he will not work on another FOSS project. 
  • Scott Hanselman laments that Microsoft hasn't put together an Ineta like organization to handle giving grants to open source projects, and also shows how easy it is to submit a patch/fix to a project.
  • Peter Provost worries that bringing money into the equation may spoil the cool part of community developed software, and that leadership is the key to good open source projects.
  • Derek Denny-Brown says that "Microsoft needs to understand that Community is more than just lots of vendors creating commercial components, or MVPs answering questions on newsgroups".

I've been somewhat disappointed by the Microsoft developer division's relationship with Open Source projects based on the .NET Framework and it's attitude towards source code availability in general. Derek Denny-Brown's post entitled Less Rambling Venting about Developing for .Net hit the nail on the head for me. There are a number of issues with the developer community around Visual Studio and the .NET Framework that are raised in Derek's post and the others mentioned above. The first, is what seems like a classic case of Not Invented Here (NIH) in how Microsoft has not only failed to support Open Source projects that fill useful niches in the Visual Studio ecosysem but eventually competes with them (Nant vs. MSBuild, NUnit vs. Visual Studio Team System and now Sandcastle vs. NDoc). My opinion is that this is a consequence of Microsoft's strategy of integrated innovation which encourages Microsoft's product teams to pursue a seamless end-to-end experience where every software application in the business process is a Microsoft product. 

Another issue is Microsoft's seeming ambivalence and sometimes antipathy towards Open Source software. This is related to the fact that the ecosystem around Microsoft's software platforms (i.e. customers, ISVs, etc) is heavily tilted towards commercial software development. Or is that vice versa? Either way, commercial software developers tend to view Open Source as the bane of their existence. This is unfortunate given that practically every major software development platform that the .NET Framework and Visual Studio competes with is either Open Source (e.g. PHP, Perl, Python, Ruby) or at the very least encourages source code availability (e.g. Java). Quite frankly, I personally would love to the .NET Framework class libraries being Open Source or at the very least have the source code available in the same way Sun has done with the JDK. I know that there is the Shared Source Common Language Infrastructure (SSCLI) which I have used on occassion when having issues during RSS Bandit development but it isn't the same.

So we have a world where the developer community around Microsoft's products is primarily interested in building and using commercial software while the company pursues an integration strategy that guarantees that it will compete with projects that add value on its platform. The questions then are whether this is a bad thing and if so, how do we fix it?


 

August 8, 2006
@ 01:29 AM

Like every other company out there, Microsoft likes to encourage employees to refer people they know who might be a good fit to join the company. It seems the HR department sent out a mass mailing last week and guess whose ugly mug was used as part of the campaign?

Front

Back

I assume this was just reusing the stock photo from my page on the Microsoft Careers site as opposed to an exhortation to Microsoft employees to hire more people like me. We only need so many paper pushing PMs. ;)

It's still pretty sweet though.


 

Categories: Life in the B0rg Cube

August 6, 2006
@ 04:47 PM

I've seen Yochai Benkler mentioned twice in the past few weeks in blogs I read semi-regularly. It seems that he recently tangled with Jason Calacanis over his attempt to pay top contributors of social bookmarking sites like Digg to using his service. Jason Calacanis documents their encounter in his post entitled Calacanis vs. Benkler Round One. Yochai Benkler also posted a comment in Nick Carr's post which was then elevated to a post by Carr entitled . Below is an excerpt from Yochai Benkler's comment.

The reason is that the power of the major sites comes from combining large-scale contributions from heterogeneous participants, with heterogeneous motivations. Pointing to the 80/20 rule on contributions misses the dynamic that comes from being part of a large community and a recognized leader or major contributors in it, for those at the top, and misses the importance of framing this as a non-priced social process. Adding money alters the overall relationship. It makes some people "professionals," and renders other participants, "suckers." It is not impossible to mix paid and unpaid participants, as we see in free and open source software and even to a very limited extent in Wikipedia. It is just hard, and requires a cultural form that is definitely not "now at long last we can tell who's worth something and pay them, while everyone else is just worthelss." What Calacanis is doing now with his posts about the top contributors to Digg is trying to alter the cultural interpretation of what they are doing: from leaders in an engaged community, to suckers who are being taken for a ride by Ross.Maybe he will succeed in raining on Digg's parade, though I doubt it, but that does not mean that he will succeed in building an alternative sustained social process of peer production, or in replacing peer production with a purely paid service. Once you frame the people who aren't getting paid as poor sods being taken for a ride, for example, the best you can hope for is that some of the "leaders" elsewhere will come and become your low-paid employees (after all, what is $1,000 a month relative to the millions Calacanis would make if his plan in fact succeeds? At that point, the leaders are no longer leaders of a community, and they turn out to be suckers after all, working for pittance, comparatively speaking.)

I'm quite surprised to see Benkler mention and dismiss the example of Open Source software since what is happening between Calacanis and Digg seems to be history repeating itself. Back in the day, Open Source software like Linux was primarily built by hobbyists who worked on such projects in their free time without intention of getting financially rewarded. Later on, companies showed up that wanted to make money from Open Source software and there was a similar kind of angst to what we are seeing today about social bookmarking. If you want to take a trip down memory lane, go back and read all the comments on the various stories about the Redhat IPO on Slashdot to see the same kind of arguments and hurt feelings that you see in the arguments made by Benkler and from people such as the Backstabbed by Netscape blogger.

The fact is that since we are human, the 80/20 rule still applies when it comes to the value of the contributions by individuals. This means that it is beneficial to the 'community' if those that contribute the most value to the system are given as much incentive as possible to contribute. After all, I doubt that there is anyone who would argue that the fact that Linus Torvalds and Alan Cox are paid to work on Linux or that Miguel De Icaza is paid to work on Mono is harmful to the communities around these projects or that it makes the unpaid contributors to these projects "suckers".

I think where people are getting confused is that they are mixing up giving the most valuable contributors to the system more incentive with trying to incentivize the entire community with financial reward. They are not the same thing. Open Source projects wouldn't be successful if everyone contributing to a project did so with the expectation of being paid. On the flip side, Open Source projects benefit the most when the top contributors to the project can dedicate their 100% of their efforts on the project without having to worry about a day job. That's the difference.

Also it seems that Benkler seems to think that Whuffie (aka respect from the community) is a better incentive than money when it comes to influencing top contributors. I think that's a pipe dream that will only occur when we live in a world where money can't buy you anything such as the one in Cory Doctorow's Down and Out in the Magic Kingdom.


 

Categories: Social Software

August 4, 2006
@ 07:55 PM

Jason Fried over at the Signal vs. Noise blog has an entry entitled Don’t believe BusinessWeek’s bubble-math where he writes

This week’s BusinessWeek cover story features a beaming Kevin Rose from Digg. Across his chest it says “How this kid made $60 million in 18 months.” Wow, now that sounds like a great success story.

Too bad it’s a blatent lie. BusinessWeek knows it. They prove it themselves in the article:

So far, Digg is breaking even on an estimated $3 million annually in revenues. Nonetheless, people in the know say Digg is easily worth $200 million.

$3 million in revenues and they’re breaking even. That means no meaningful profits. That’s the first hint no one has made $60,000,000. Their gross revenues aren’t even anywhere close to that number. And let’s leave out the “people in the know say it’s easily worth” fantasy numbers. And certainly don’t use those numbers to do the math that makes the cover (we’ll get to that in a minute).

I can't believe BusinessWeek ran such a misleading cover story. I guess sensational, fact-less headlines aren't just for the supermarket tabloids these days.


 

Earlier this week I was showing off the new Windows Live Spaces to my girlfriend in an attempt to try and explain what I do at Microsoft all day. When I showed her the Friends list feature she was surprised that the site had morphed from a blogging service into a social networking service and wondered whether our users wouldn't react negatively to the change. Actually that's paraphrasing what she said. What she said was

If I was using your site to post my blog and my pictures for people I know, I'd be really annoyed if I started having to deal with strangers asking me to be their friend. If I wanted to deal with that shit crap I'd have just gone to MySpace.
That's valid criticism and is something the people who worked on the design of this feature (i.e. me, Mike, Neel, Matt, John and a bunch of others) took into account. One of the key themes of Windows Live is that it puts users in control of their Web experience. Getting repeated email requests to be some stranger's "friend" without a way to stop them doesn't meet that requirement. This is why there is a communications preferences feature in Windows Live Spaces which can be reached by clicking on http://[yourspacename].spaces.live.com/Settings/Communication/

Below is a screenshot of the page

Don't like getting friend requests as email? Disable this by unchecking 'Also send invitations and e-mails to my email address'. Don't want to deal with requests from total strangers wanting to be on your friend's list? Then move the setting on 'These people can request to be on your friends list' to something more restrictive like 'Messenger buddies' so that you only get friend requests from people on your IM buddy list who you already know.You hang out a virtual DoNotDisturb sign and we'll honor it.

Making sure that our users have total control over who they communicate and share information with is key to how we approach building social software for Windows Live. Thanks to all the users of Windows Live Spaces who have made it the most popular blogging service in the world. 


 

Categories: Windows Live

I've gotten some reports from people that they've had problems using their blogging tool of choice to post to Windows Live Spaces using the MetaWeblog API. I haven't had any problems posting using W.Bloggar but have had some issues using Blogjet recently. I'd appreciate it if anyone else who's having issues posting to their blog responds with a comment to this blog post with information about which blogging tool you are using and what error message the tool reports.

Thanks.


 

Categories: Windows Live

A couple of days ago, I wrote that based on my experiences with Kuro5hin I wouldn't be surprised by small sets of users dominating the content and focus of Digg since it is practically the same kind of site. Duggtrends has a blog post entitled Digg user statisitcs & trends which confirms these suspicions. It states

From our database, for the period of 6/19/2006 9:31:28 PM to 7/30/2006 4:41:34 PM a total of 6013 stories were promoted to front page of these
  • top 10 users contributed 1792 i.e 29.8%
  • top 100 contributed 3324 stories i.e 55.28% (which is again what exactly SEOMOZ reported)
This clearly shows the shift from the Kevin Rose reported numbers from 26.8% to 55.28%; top users are contributing more & more to digg

As per Jason Martinez (and Calacanis points in his blog) 60% of Digg’s front page is the top 0.03% users and provides this graph.

It looks like the 1% rule is in full effect. This has always been the nature of online communication from mailing lists and newsgroups to Web-based message boards and social bookmarking sites. Human nature will always shine through at the end.

PS: Has anyone else seen that the Digg REST API has now been documented? Thanks to Michael Brundage for the tip.