Up until a few months ago, the term DevOps was simply another buzzword which filled my Twitter feed that evoked a particular idea but wasn’t really concrete to me. Similar to other buzzwords related to software development such as NoSQL and Agile, it is hard to pin down what the definitive definition of the term is just what it wasn’t. If you aren’t familiar with DevOps, a simple definition is that the goal of DevOps is to address this common problem when building online services

The Big Switch

A couple of months ago, my work group took what many would consider a rather extreme step in eliminating this wall between developers and operations. Specifically, Bing Ads transitioned away from the traditional Microsoft engineering model of having software design engineers (aka developers), software design engineers in test (testers) and service operations (ops) and merged all of these roles into a single engineering role. As it states in the Wikipedia entry for DevOps, the adoption of DevOps was driven by the following trends

  1. Use of agile and other development processes and methodologies
  2. Demand for an increased rate of production releases from application and business unit stakeholders
  3. Wide availability of virtualized and cloud infrastructure from internal and external providers
  4. Increased usage of data center automation and configuration management tools

All of these trends already applied to our organization before we made the big switch to merge the three engineering disciplines into a DevOps role. We’d already embraced the Agile development model complete with two to four week sprints, daily scrums, burn-down charts, and senior program managers playing the role of the product owner (although we use the term scenario owner). Given our market position as the underdog to Google in search and advertising, our business leaders always wants to ship more features, more quickly while maintaining high product quality. In addition, there’s a ton of peer pressure for all of us at Microsoft to leverage internal tools Windows Azure and Autopilot for as much of our cloud services needs as possible instead of rolling our own data centers and hardware configurations.

Technically our organization was already committed to DevOps practices before we made the transition that eliminated roles. However the what the organization realized is that a bigger change to the culture was needed for us to get the most value out of these practices. The challenge we faced is that the organizational structure of separate roles for developers, testers and operations tends to create these walls where one role feels their responsibility is for a certain part of the development cycle and then tosses the results of their efforts down stream to the next set of folks in the delivery pipeline. Developers tended to think their job was to write code and quality was the role of testers. Testers felt their role was to create test frameworks and find bugs then deployment was the role of the operations team. The operations team tended to think their role was keeping the live site running without the ability to significantly change how the product was built. No matter how open and collaborative the people are on your team, these strictly defined roles create these walls. My favorite analogy for this situation is like comparing two families who are on a diet trying to lose weight and one of them has fruit, veggies and healthy snacks in the pantry while the other has pop tarts, potato chips, chocolate and ice cream in theirs. No matter how much will power the latter family has, they are more likely to “cheat” on their diet than the first family because they have created an environment that makes it harder for them to do the right thing.

Benefits

The benefits of fully embracing DevOps are fairly self-evident so I won’t spend time on discussing the obvious benefits that have been beaten to death elsewhere. I will talk about the benefits I’ve seen in our specific case of merging the 3 previous engineering roles into a single one. The most significant change is the cultural change towards how we view automation of every step related to deployment and monitoring. It turns out that there is a big difference when approaching a problem from the perspective of taking away people’s jobs (i.e. automating what the operations team does) versus making your team more effective (i.e. reducing the amount of time the engineering team spends on operational tasks that can be automated thus giving us more time to work on features that move the business forward). This has probably the biggest surprise, although obvious in hindsight, as well as the biggest benefit.

We’ve also begun to see faster time to resolve issues from build breaks to features failing in production due to fact that the on-call person (we call them Directly Responsible Individuals or DRIs) is now a full member of the engineering team who is expected to be capable of debugging and fixing issues encountered as part of being on-call. This is an improvement from prior models where the operations team were the primary folks on-call and would tend to pull in the development team as a last resort outside of business hours.  

As a program manager (or product manager if you’re a Silicon Valley company), I find it has made my job easier since I have fewer people to talk to because we’ve consolidated engineering managers. No longer having to talk to an development manager separately from the manager of systems engineers separately from a test manager has made communication far more efficient for me.

Challenges

There are a number of risks with any organization taking the steps that we have at Bing Ads. The biggest risk is definitely attrition especially at a company like Microsoft where these well-defined roles have been a part of the culture for decades and are still part & parcel of how the majority of the company does business. A number of people may feel that this is a bait and switch on their career plans with the new job definitions not aligning with how they saw their roles evolving over time. Others may not mind that as much but may simply feel that their skills may not be as valuable in the new world especially as they now need to learn a set of new skills. I’ve had one simple argument when I’ve met people with this mindset. The first is that DevOps is here to stay. The industry trends that have had more and more companies from Facebook and Amazon to Etsy and Netflix blurring the lines between developers, test engineers and operations staff will not go away. Companies aren’t going to want to start shipping less frequently nor will they want to bring back manual deployment processes instead of automating as much as possible. The skills you learn in a DevOps culture will make you more broadly valuable wherever they find their next role whether it is a traditional specialized engineering structure or in a DevOps based organization.

Other places where we’re still figuring things out are best practices around ownership of testing. We currently try to follow a “you build it, you test it, you deploy it” culture as much as possible although allowing any dev to deploy code has turned out to be bit more challenging than we expected since we had to ensure we do not run afoul of the structures we had in place to stay compliant with various regulations. Testing your own code is one of topics where many in the industry have come out against as being generally a bad idea. I remember arguments from my college classes from software engineering professors about the blind spots developers have about their software requiring the need for dedicated teams to do testing. We do have mitigations in place such as test plan reviews and code reviews to ensure there are alternate pairs of eyes looking at the problem space not just the developer who created the functionality. There is also the school of thought that since the person who wrote the code will likely be the person woken up in the middle of the night if it goes haywire at an inopportune moment, there is a sense of self preservation that will cause more diligence to be applied to the problem than was the case in the previous eras of boxed software which is when most of the anti-developer testing arguments were made.

Further Reading

 

Note Now Playing: EminemRap God Note


 

Earlier this week my Twitter feed was flooded with reactions to the announcement of Amazon Prime Air. A vision which is compelling and sounds like something from a science fiction novel.

 

Flying robots delivering your impulse buys ordered from you smartphone within 30 minutes? Sign me up. 

Amazon's Robotic Vision

A few have been skeptical of Amazon Prime Air and some such as Konstantin Kakaes at Wired have described the announcement as an hour long infomercial hosted by Charlie Rose that is full of hot air. There are definitely a lot of things that need to get better before drone-based package delivery is a reality. On the technology end there are challenges such as improving navigation software and the getting more efficiency out of battery power. On the regulatory end the rules and regulations needed to ensure the safety of the populace in the midst of these flying drones still needs to be figured out.

Unlike a number of the skeptics, I'm quite confident that a number of the technological and regulatory hurdles will be surmounted in the next 5 years. On the other hand, I also believe Amazon oversold the technology. The logistics of actually delivering a package to a customer has been glossed over quite a bit. If you look at the video, this is the optimal situation for delivering a package.

Now think about all the places in urban environments that don't meet this criteria; apartment complexes, condos, office building, etc. Then think about all the places where packages are best left on the doorstep close to the house which would be infeasible for a drone to land close to. In fact it's likely the 30 minute delivery claim is meant to address some logistics challenges by assuming you'll be there to pick up anything the drone drops off on your lawn or driveway before Johnny Sticky Fingers does.

My suspicion is that the true impactful usage of Amazon Prime Air will be 30 minute delivery drone delivery to Amazon Locker locations.

Google's Robotic Vision

This morning my Twitter feed is abuzz with the news that Google's Andy Rubin, creator of Android and longtime robotics enthusiast, has acquired seven robotics companies that are creating technologies to build a mobile dexterous robot. This effort will be part of Google X which is also the home of Google Glass & self-driving cars. Andy Rubin describes the effort as follows

“I have a history of making my hobbies into a career,” Mr. Rubin said in a telephone interview. “This is the world’s greatest job. Being an engineer and a tinkerer, you start thinking about what you would want to build for yourself.”

He used the example of a windshield wiper that has enough “intelligence” to operate when it rains, without human intervention, as a model for the kind of systems he is trying to create. That is consistent with a vision put forward by the Google co-founder Larry Page, who has argued that technology should be deployed wherever possible to free humans from drudgery and repetitive tasks.

To spell it out, Google’s efforts in automation under project X have been about augmenting the human experience in ways that eliminate repetitive tasks which humans are poor at such as driving where human error is a significant cause of the over 30,000 deaths a year in the US due to automobile accidents. Even this somewhat frivolous example of ringing the doorbell at Andy Rubin’s home taken from a 2007 profile of the Android founder is consistent with that vision

If the scanner recognizes you, the door unlocks automatically. (The system makes it easier to deal with former girlfriends, Mr. Rubin likes to joke. No messy scenes retrieving keys — it’s just a simple database update.)

Those forced to use the doorbell are greeted with another technological marvel: a robotic arm inside the glass foyer grips a mallet and then strikes a large gong. Although Mr. Rubin won’t reveal its cost, it may be one of the world’s most expensive doorbells.

At the end of the day, I’m more inspired by a company looking at automating away all of the tediousness of every day life with Star Trek style technology from automated doorways and self-driving cars to fully autonomous robots than a vision of making impulse buying at Walmart 2.0 Amazon more convenient. I for one welcome our new robotic overlords.

Note Now Playing: AdeleRumor Has It Note


 

Categories: Technology

Danny Sullivan wrote an interesting blog post this morning titled Google’s Broken Promises & Who’s Running The Search Engine? whose central thesis is that Google now does a number of things it once described as “evil” when it comes to how search results and ads work in Google Search. Given that I now work in Bing Ads, this is a fairly interesting topic to me and one I now have some degree of industry knowledge about.

Promises Are Like Pie Crust, Easy to Make and Easy to Break

Danny Sullivan categorizes two big broken promises in his article, one in 2012 and one from 2013. The 2012 broken promise is excerpted below

The first broken promise came last year, when Google took the unprecedented step of turning one of its search products, Google Product Search, into a pure ad product called Google Shopping.

Google Shopping is a different creature. No one gets listed unless they pay. It’s as if the Wall Street Journal decided one day that it would only cover news stories if news makers paid for inclusion. No pay; no coverage. It’s not perfect metaphor. Paid inclusion doesn’t guarantee you’ll rank better or get favorable stories. But you don’t even get a chance to appear unless you shell out cold hard cash.

What Was Evil In 2004, Embraced In 2012

Shopping search engines have long had paid inclusion programs, but not Google. Google once felt so strongly that this was a bad practice that when it went public in 2004, it called paid inclusion evil, producing listings that would be of poor relevancy and biased. The company wrote, in part:

Because we do not charge merchants for inclusion in [Google Shopping], our users can browse product categories or conduct product searches with confidence that the results we provide are relevant and unbiased.

There is a similar Google then versus Google now perspective when looking at the second broken promise related to banner ads in the search results page.

“There will be no banner ads on the Google homepage or web search results pages,” Google promised in December 2005, on its main blog, to reassure consumers concerned that its new partnership with AOL would somehow change the service. Eight years later, Google’s testing big banner ads like these:

These excerpts could almost be a cautionary tale to idealistic young companies about making absolute statements and the staking one’s brand on these statements without thinking about the future. However that isn’t the point of this post.

I decided to write this post because Danny Sullivan’s article starts starts out quite strongly by pointing to this misalignment between Google’s past statements and their current behavior but then peters out. The rest of the article is spent studying Google’s org chart trying to figure out which individual to blame for these changes as well as trying to come up with a rationalization for these moves in the context of making search better for consumers. As an industry watcher the rationale for these moves is quite straightforward and has been a natural progression for years.

The Love of Money is the Root of all Evil

Any analysis of business decisions Google makes in the arena of search, is remiss if it fails to mention Google makes the majority of its revenue and profits from ads running on Google sites. As an example, Google made $9.39 billion last quarter from ads running on its sites whereas of the $3.15 billion it made from ads running on other people’s websites it paid those people $2.97 billion. Combining that with the fact that its $12.5 billion acquisition of Motorola has so far produced nothing but financial losses, there is a lot of pressure for Google to make as much money as possible from ads running on its sites specifically in Google Search results pages. 

When it comes to search engine advertising, the money is primarily in queries with “commercial intent”. This is a fancy way of saying that the person who is performing the search is planning to spend money. Advertisers are willing to pay several dollars to a search engine each time a customer clicks on their ad when the search term has commercial intent. In fact, companies are willing to spend up to $40 – $50 each time a user clicks on an ad if the user is searching for a life insurance policy or a home loan.

Over time both search engines and advertisers have figured out exactly where the money is and how to extract the most value from each other. Google has slowly been making changes to their search engine that implies that for queries with commercial intent they always want a cut of the action. This is why if you perform a search today that has commercial intent, there are an order of magnitude (i.e. ten times) as many links to ads as there are unpaid search engine results. For example, take a look at this screenshot of a query for “northface jackets” on Google.

There are two links on this page that are unpaid search results and eighteen links where Google gets paid if you click on them. Given that context, it is no surprise that Google eventually realized it was competing with itself by having a “free” shopping search engine. This explains the broken promise in 2012 related to paid inclusion.

Now if you take a close look at the above screenshot, you’ll notice that The North Face is actually the top advertiser on this page. This means that despite the fact that the user was specifically looking for the North Face brand products, the company has to still compete with other advertisers by paying people for clicks to their website from Google search results. Brand advertisers hate this. A lot. Not only did they spend a lot of money and effort to become a well-known brand but now they still end up paying when this brand recognition pays off and people explicitly are looking for them on Google.

This leads us to the second broken promise, banner ads in search results. What Google is trying to do is to appease brand advertisers by letting them “take over” the search engine results page in cases where the user is quite clearly searching for their brand. Treating this is a giant billboard that reinforces their brand as opposed to scrabbling with other advertisers for a user which they already consider theirs is a more amenable pitch. This explains the broken promise of 2013.

I expect to see more aggressive commercialization of the search results page given Google’s seeming lack of interest and inability to diversify their sources of income. Doing this while preserving the customer experience will be the #1 challenge of their search engine and other similarly advertising focused web products in 2014 and beyond.

Note Now Playing: Jay-ZF*ckwithmeyouknowigotit (featuring Rick Ross) Note


 

It isn't hard to find criticisms of Microsoft's employee appraisal system. Whether it's almost decade old complaints such as Mini-Microsoft’s Microsoft Stack Ranking is not Good Management or more recent forays into blaming it for the company's "decline" such as Kurt Eichenwald's 2012 Vanity Fair opus Microsoft’s Lost Decade or the follow up The Poisonous Employee-Ranking System That Helps Explain Microsoft’s Decline from Slate, there are many who would lay the practice of ranking employees on a vitality curve as the root cause of any problems facing Microsoft today. Kurt Eichenwald's article persuasively makes that argument in the in excerpt below

Every current and former Microsoft employee I interviewed—every one—cited stack ranking as the most destructive process inside of Microsoft, something that drove out untold numbers of employees. The system—also referred to as “the performance model,” “the bell curve,” or just “the employee review”—has, with certain variations over the years, worked like this: every unit was forced to declare a certain percentage of employees as top performers, then good performers, then average, then below average, then poor. …

For that reason, executives said, a lot of Microsoft superstars did everything they could to avoid working alongside other top-notch developers, out of fear that they would be hurt in the rankings. And the reviews had real-world consequences: those at the top received bonuses and promotions; those at the bottom usually received no cash or were shown the door. …

“The behavior this engenders, people do everything they can to stay out of the bottom bucket,” one Microsoft engineer said. “People responsible for features will openly sabotage other people’s efforts. One of the most valuable things I learned was to give the appearance of being courteous while withholding just enough information from colleagues to ensure they didn’t get ahead of me on the rankings.” Worse, because the reviews came every six months, employees and their supervisors—who were also ranked—focused on their short-term performance, rather than on longer efforts to innovate.

This reads as a very damning indictment of the Microsoft performance appraisal system. Although I've now been at Microsoft for 12 years and a manager for the last three of them, the purpose of this blog post isn't to defend or expand on the details of performance reviews at Microsoft. The purpose of this blog post is to point out that one often needs multiple data points before coming to sweeping generalizations when discussing something as complex as the success or failure of technology companies.

 

The Four Horsemen: Facebook, Amazon, Google and Apple

A few years ago, Eric Schmidt described the "gang of four" companies driving innovation and growth in tech as Facebook, Amazon, Google and Apple.  The implication being that these were the four leading companies in the tech industry. For the purpose of this blog post, I’ll take this implication at face value and will consider these 4 companies as leading examples of what other companies should aim to emulate in the technology industry. In fact, I’ll go one further and reference Mobile is eating the world, autumn 2013 edition by Benedict Evans where he cites these four companies as “setting the agenda” whereas when Microsoft is mentioned, it’s only to speak about it’s “growing irrelevance”.

Now that we’ve established that these four companies are worthy of emulation, how exactly do these companies evaluate employee performance anyway?

Unfortunately, I could not find any information either off-the-record from friends or on the Internet about how performance reviews work at Apple. On the other hand, there has been enough written about the other three companies that we can still draw some conclusions about how performance appraisal works at relevant technology companies. 

Let’s start with Facebook. A good overview of Facebook’s performance review system can be found in the answers to the Quora question, What does Facebook's performance review process look like? Below is the answer from Molly Graham who used to work in Facebook HR

Then there is a (roughly) two week period of calibration where managers meet to look at the assessments of everyone on their team and ensure that people are rated correctly relative to their peers. Facebook has seven performance assessments as well as a guideline for what % of employees should be at each level, however it is explicitly not a forced curve, particularly for small teams. The curve exists to ensure that extraordinary performance is rewarded (I believe the distribution is such that only 2% or less of employees are given the highest rating every cycle) and that if hard conversations need to happen, they happen.


Calibration happens at the team level and at the senior management level (Mark, Sheryl, and all of their direct reports look at the numbers for the whole company, lists of the highest performers, lists of the lowest performers, etc). Performance Assessments are final and they are used to determine compensation like raises, bonuses, and additional equity grants. Facebook gives out raises and additional equity once a year but they do promotions and bonuses twice a year. Compensation at Facebook is almost entirely formulaic with multipliers (based on the Performance Assessment) for bonuses, raises, and additional equity grants.

Hmmm, so it looks like Facebook uses a curve but the argument seems to be that it is to ensure that extraordinary performance can be rewarded yet there is a quota on how many extraordinary performers that can exist according to the system.

Let’s see what Amazon does in the area of employee appraisals next. For this topic, I’ll use an excerpt from an article in Business Insider which references a leaked document that has since been pulled and an article by Amazon chronicler, Brad Stone. The Business Insider article is titled Amazon Has A Brutal System For Employees Trying To Get Promoted and is excerpted below

In the second meeting, which takes place in September or October, the leaders talk some more about who's getting a promotion, and talk about who is doing well and who is doing poorly. Amazon's managers group employees into three tiers: The top 20%, who are groomed for promotions, the next 70% who are kept happy, and the bottom 10%, who are either let go, or told to get it together.

This system, which was created by Jeff Bezos, is supposed to cut down on politics and in-fighting. Unfortunately, Stone says it has the opposite effect."Ambitious employees tend to spend months having lunch and coffee with their boss’s peers to ensure a positive outcome once the topic of their proposed promotion is raised in [the meetings]," says Stone.

Stone also notes that promotions are very limited at Amazon, so if you fight for your employee to get a promotion, it means someone else's employee gets snubbed. And anyone in the room can nuke someone else's promotion.

OK, so it seems Amazon also has a curve and it seems more explicit that the bottom 10% are targeted for negative messaging. But it seems there is a new concept we’re being introduced to. A peer review based system for promotion which in theory is in place to reduce cronyism (e.g. working for a boss that’s a friend who then promotes you for simply having a pulse) which in reality turns into politics-driven affair since everyone needs to like you for you to get ahead at the company. Good luck, rocking the boat in such environments.

So far we’re not really seeing much alternative ideas for tech companies that decide Microsoft’s employee appraisal system is one they don’t want to emulate. Let’s see what we can learn from how performance reviews are done at Google to turn the tide. For this I think we’ll look at two perspectives on the Google performance appraisal system. First here is an excerpt from a review on Glassdoor from a Google employee that loves everything about working at the company describing the performance review system.

Promotion and work performances is entirely reliant on peer reviews. In other words, to get ahead at Google and to get a positive performance review, you must get positive reviews from your fellow co-workers. Your manager might love you, but if your co-workers don't like you, you have some work to do. Managers are also required to seek peer review from those they manage. (I have never seen this before in my career.) Senior level employees from other fields are also encouraged to seek peer reviews from people in other departments. For example, engineers need reviews from people other than engineers in order to advance. For this reason, a culture of cooperation is endemic at Google. This is great because the percentage of "cowboys" that seems common at other high tech companies is quite low at Google. It also fosters an awareness of the type of contribution made by people outside your department, since everyone reviews people in other fields, and therefore must learn a bit about what others do outside their sphere.

Peer reviews sound like a great idea. Now we have a performance appraisal technique we can emulate that is different from applying recommended or forced curves as Facebook and Amazon do. Before embracing this whole heartedly, let’s balance our perspective with excerpts from Quora answers to What are the major deficiencies of the performance review process at Google?

Google's original intention in designing the byzantine monstrosity known as "Perf" was noble: to provide multiple avenues toward success. Someone who got mediocre reviews from his manager but excellent peer reviews could move up, or at least laterally. (This prevented the scenario where a manager uses mediocre or even negative reviews in order to prevent transfers, a known problem at Google.) It was an "OR-gate". If you had good managerial reviews or good peer reviews or objectively demonstrable accomplishments, you'd be in good standing and move up.Eventually, it became an AND-gate. To get a promotion or even a transfer, you had to have managerial blessing and good peer reviews and high visibility and the willingness (as Piaw Na alluded) to spend considerable amounts of time and energy marketing yourself. So it became a morale-killing, time-consuming "No Machine" that people spent a considerable amount of time figuring out how to game. The typical corporate manager-as-SPOF dynamic that Perf was invented to extinguish was strengthened by the "objective" soothsaying they call "calibration scores". – Anonymous

The big one is that engineers have to apply for a promotion and put together their own promotion packet. There's no human being who can do that and not end the process thinking, "Oh boy, I did so much work. I really deserve a promotion." Since the process doesn't promote everyone, that creates a number of disgruntled employees. Even if these employees were to eventually get promoted later, they tend to think, "I should have gotten this N quarters/years ago,", not "I'm so glad I got this promotion." The net result is that very few people are pleasantly surprised when they get promoted, while a lot of people get disappointed.Piaw Na

Although there isn’t forced ranking in place, it does looks like Google one-upped Amazon in making it difficult to climb up their corporate ladder by having a promotion process based on pleasing everyone.

 

In the Land of the Blind, the One-Eyed Man is King

To summarize, so far it looks like Amazon and Facebook have fairly similar performance review structures as Microsoft much-lambasted system while Google seems to have a performance that seems to trade one set of problems for a different set. As an honorary mention, I’d like to point to the QPR system recently put in place at Yahoo by Marissa Mayer which is also a vitality curve based system.

This then raises the question of why if forced ranking and other similarly disheartening employee appraisal processes are commonplace in the industry that tech blogs makes it seem the practice is limited to Microsoft and then blame it for the challenges the company has faced in recent years? From what I can tell, the reason is twofold. The first is that Microsoft employees stick around at the company for far longer than peers at other companies. From the Geekwire article Amazon, Google employees ranked as ‘least loyal’ which looks at data from PayScale, we learn

Amazon.com tied for second for the least loyal employees with a median tenure of one year, while Google tied for fourth with just 1.1 years of tenure on average. Apple, meanwhile, tied for 36th at two years…Microsoft, however, was all the way down the list tied for 259th with an average tenure of four years.

Most people at companies like Amazon, Facebook and Google either just got there or left before they would have to deal with the frustration of being disappointed by the performance appraisal system over multiple cycles. On the other hand, the average Microsoft employee has been at the company for far longer and when they do leave have had multiple brushes with the performance appraisal system. Just from a raw numbers perspective given average tenure at the company and number of employees, there are a lot more people who you can find that would complain about the performance review system at Microsoft than say at Amazon which has very similar systems in place.

The second reason I believe there are more people willing to talk to the press about the performance review system at Microsoft than at say Google or Amazon, is that it doesn’t fit the narrative to air those complaints. When you see a chart like below, it is easy to look for simple answers to explain the differences in the stock market’s belief in the success of Microsoft versus the “four horsemen”.

On the other hand, it is also hard for employees to complain when the company they work for is winning in new markets and is being praised by the industry press. Glassdoor is full of complains about poor work/life balance at Apple but no one is going to write a damning expose about the company’s employee morale problems as long as the company doesn’t slip in the marketplace. However once a company starts to falter in the marketplace, everything they do is bad and is the cause of their demise according to the pundits. I’ve been amused by the number of articles blaming Blackberry’s dual CEO model for the company’s failures even though that is effectively the Facebook model if you ask anyone who works there and one could argue at one point Google had a triple CEO model with Eric, Larry & Sergey all running the company in their different ways.

 

Everything Sucks

The bottom line is that performance appraisal systems at large companies always suck for the set of reasons covered extremely well by Steven Sinofsky in his blog post Realities of Performance Appraisal. He does a good job of pointing out some of the realities of businesses and human nature that guarantee that these processes will always come across as soul crushingly awful when applied at large enough scale including 

  • Performance systems conflate performance and compensation with organizational budgets. No matter how you look at it, one person cannot be evaluated and paid in isolation of budgets. The company as a whole has a budget for how much to pay people (salary, bonus, stock, etc.) No matter what an individual’s compensation is part of a system that ultimately has a budget. The vast majority of mechanical or quantitative effort in the system is not about one person’s performance but about determining how to pay everyone within the budget. While it is desirable to distinguish between professional development and compensation, that will almost certainly get lost once a person sees their compensation or once a manager has to assign a rating. Any suggestion as to how to be more fair, allow for more flexibility, provide more absolute ratings, or otherwise separate performance from compensation must still come up with a way to stick to a budget. The presence of a budget drives the existence of a system. There is always a budget and don’t be fooled by “found money” as that’s just a budget trick.

This is the fundamental conceit of performance appraisal systems. For large companies they are primarily about answering the question of “how do we distribute our promotion and bonus budget?” by drawing a fuzzy line between employee work activities and how much money they actually have to spend (e.g. policy that only 2% of Facebook employees can have extraordinary rewards is a function of budgets not a natural law of distribution of extraordinary employees at Facebook or any other company in the world). Companies can’t just say everyone who does an excellent job gets $1,000 bonus because they may not have $1,000 to spend per employee in the budget.  

And with that you have the answer to the question in the title of this blog post.

Note Now Playing: The GameMartians vs Goblins (featuring Tyler the Creator & Lil Wayne) Note


 

Categories: Life in the B0rg Cube

The big news in tech circles is this is the week Google Reader died. There has been a lot of collective shock about this since Google Reader was a staple of the sort of technology news omnivore and early adopter that was once considered to be the core fan base for Google. Despite Google’s protestations it’s quite clear that there was ample usage of the service with sites like Buzzfeed providing evidence that up to its last day it was driving over a million referrals a day. As someone who wrote a feed reader that synced with Google Reader which has been downloaded over 1.5 million times and is credited in Atom format specification it is sad to see such a core part of the RSS ecosystem come tumbling down.

The biggest surprise for many is how far Google seems to have strayed from its core mission as epitomized by this comment on Hacker News

ivank 1 day ago | link | parent

Organizing the World's Information... and setting it on fire 8 years later.

Like many companies before it, the company has done a great job of separating its public persona from the gritty reality of its business practices. Google’s business mission is to organize all the world’s information and use that to sell advertising. The great thing about this business model is that for a long time it has been a win-win for every one involved in the ecosystem. Consumers get great web services and products that are expensive to build and run for free such as Google Search, Gmail, Chrome and Android. Advertisers get to target consumers in new ways which have a better return on investment than they traditionally have been able to get from print and other media. While Google has minted hundreds of millionaires with the profits they’ve gotten from creating this ecosystem.

Unfortunately, Google has also hit the same problems that successful companies before it have had to face. The laws of large numbers mean that for Google to continue to be a great business for its shareholders (which includes its employees and executives) then it needs to find ways to eke even more money out of its customer base (advertisers) by giving them even more product (consumers with rich demographic and behavioral information) or by growing into new markets. Companies like Apple and Microsoft have faced this problem by growing new multi-billion dollar businesses. Apple went from a computer company to the iPod company and now it’s the iPhone company which is transitioning to becoming the iPad company. Microsoft started off as the BASIC company, transitioned to being the Windows & Office company and today has several billion dollar businesses from game consoles to database software. Google has decided to face this challenge by doubling down on their advertising business which means trying to suck in even more data via acquisitions like Waze and ITA as well as extracting more revenue from advertisers by removing consumer targeting options.

For the most part Google has diverted tech press scrutiny from this ongoing attempt to monopolize data sources and hover up personal information about users for later resale using misdirection. A few years ago the Google playbook was to release some technology branded as “open” such as OpenSocial which aimed to paint Facebook as “evil”, open sourced Android which paints Apple as “evil” or Google Chrome which was released with an excellent comic book explaining why it was so open. How could one complain about Google with a straight face when they were giving all of these free services to consumers and “open” technologies to the industry? No one would buy it. This was the strategy of the old Google under Eric Schmidt and it worked beautifully.

It has been an interesting hallmark of Google under Larry Page that it doesn’t play these games anymore. Pursuing Google’s business strategies at the expense of its public image as the good guy of the technology industry is now par for the course. I was taken aback when Google announced that it was going to fork Webkit since this was a blatant political move which many technology savvy geeks would see through. For days afterwards, the article A Short Translation from Bullshit to English of Selected Portions of the Google Chrome Blink Developer FAQ was a viral hit among developers on Twitter with excerpts such as

1.2 Is this new browser engine going to fragment the web platform's compatibility more?

Yes.

We intend to distract people from this obvious problem by continually implying that our as-yet unwritten replacement is somehow much better and more sophisticated than the rendering engine that until yesterday was more than good enough to permit us to achieve total dominance of the Windows desktop browsing market in less than two years.

This strategy has worked extremely well for Netscape, Microsoft, Apple and us in previous iterations of the browser wars, and we firmly believe that everyone in this industry was born yesterday and they will not recognise this for the total bullshit it so clearly is.

1.10 Is this going to be open source?

Not really.

While you can certainly read the source code, we're fully aware that actually tracking and understanding a modern HTML renderer is extremely difficult. In addition, the first changes we will make are intended specifically to break compatibility with WebKit, so the only organisation with sufficient resources to track our changes will no longer be able to do so.

In practice, this allows us to call the project "open" while simultaneously ensuring Google will be the only effective contributor to the Chrome and Blink source now and in the future. We've had enormous success co-opting the language of open source in the past to imply our products are better, and we aim to continue with that strategy.

So what does all of this have to do with Google Reader? As Marco Arment points out in his excellent post titled Lockdown, Google is in a battle with Facebook and Twitter to suck up user demographic and behavioral data around social signals. Google Reader is literally the opposite of that strategy since it is far too decentralized when compared to the model pioneered by Facebook Pages and Twitter.

Google Reader has been living on borrowed time since Facebook and Twitter became prominent. The only thing that has changed in 2013 is that Google’s management doesn’t think it’s worth it to throw a bone to millions of geeks and early adopters by keeping the lights running on the service with its existing skeleton crew. This new Google doesn’t care if geeks and early adopters just see it as another money hungry corporation that only focuses on the bottom line. Larry Page doesn’t give a shit.

Welcome to the new Google.

Note Now Playing: J.Cole - Can't Get Enough (featuring Trey Songz) Note


 

The problem with Google Reader was that by the time you solve all of the problems with the RSS experience, you’ve effectively invented Twitter.

And Jack Dorsey already did that.

Note Now Playing: Lil Wayne - Love Me (featuring Drake & Future) Note


 

In a past life, I worked on the social news feed for a number of Microsoft products including the Messenger Social feed in Hotmail & Messenger and most recently the People app in Windows 8. When I worked on these products, we strongly believed in the integrity of the user experience and so never considered the social feed as a canvas for showing users ads.

Thus I read a pair of recent posts by Dalton Caldwell, founder of App.net, with some interest. Dalton wrote about the recent moves that both Twitter and Facebook are making to ensure that the social feeds on their sites become a great canvas for showing ads.

In his post Understanding Like-gate Dalton writes

The best ad is indistinguishable from content

We can expect to see Facebook deemphasizing traditional advertising units in favor of promoted news stories in your stream. The reason is that the very best advertising is content. Blurring the lines between advertising and content is one of the most ambitious goals a marketer could have.

Bringing earnings expectations into this, the key to Facebook “fixing” their mobile advertising problem is not to create a new ad-unit that performs better on mobile. Rather, it is for them to sell the placement of stories in the omnipresent single column newsfeed. If they are able to nail end-to-end promoted stories system, then their current monetization issues on mobile disappear.

In his post Twitter is pivoting Dalton writes

Predicting the future

In this paradigm, Twitter’s business model is to help brands “amplify their reach”. A brand participating in Twitter can certainly distribute their content for free and get free organic traffic, but if they want to increase their reach, they need to pay.

It’s no accident that this sounds exactly like the emerging Facebook business model. As discussed in that link, algorithmically filtered primary feeds are vastly easier to advertise against vs unfiltered feeds. The issue for Twitter is that Facebook already has a far larger userbase which is already trained to read an algorithmically filtered feed.

In a twist, I wouldn’t have predicted a few years ago it is now a regular occurrence for both users of Facebook and Twitter to see ads in their feeds. Twitter has gone as far as effectively crippling new Twitter apps to ensure that every Twitter user gets an ads-heavy unfiltered Twitter experience. The reason for this is straightforward. Both companies have sky high expectations from investors as evidenced by Facebook's $100 billion valuation it has failed to meet and Twitter's $8 - $10 billion valuation on $100 million in revenues. The challenge for both services is that investors are expecting Google-like returns on investment but neither of these companies have a Google-like business model.

The problem with ad supported online businesses is that for the most part their business models suck. In a traditional business, if you focus on building a great product or service that provides an excellent customer experience then you will be making money hand over fist. In most ad supported online businesses, your business is selling your product’s audience as opposed to the product itself. That means if you want to make more money you have to pimp out your audience often in disrespectful and intrusive ways to eke out that extra dollar.

The one place where this is different is online search (i.e. Google’s primary business). In the web search, the ads aren’t just indistinguishable from content but in the most lucrative cases, the ads are better than the content. As an example take a look at these searches

Since we may get different results, I’m including a screenshot below

There are 8 ads in this screenshot and 2 search results. However instead of being irritated as I would be if the ratio of ads to content was 4:1 in a YouTube video or Facebook feed, the ads are actually more relevant than the organic search results. This is the holy grail that Twitter and Facebook are trying to achieve.

As Dalton points out, Facebook has already socialized its users to the notion that brands will post commercial messages to the feed. In addition, brands have grown so entitled to it then when asked to pay for them since they are ads, they get outraged. However Facebook has been boiling this particular frog for a while. Facebook encourages advertisers to create Pages and users to like Pages so that they can stay connected to the brands they care about. Content in your feed from people and brands you don’t follow snuck in under the aegis of showing you content your friends interacted with. Finally not only has Facebook had promoted posts for brands for a while, they now also allow users to promote their personal posts to friends for $7 a pop.

Without really thinking about it much, we’re halfway to a future where a significant percentage of the content of your Facebook feed is paid. Since the posts go through the same ranking algorithm as your regular feed, they are more likely to be relevant to you than the traditional ad products that Facebook and other online properties are known for today. When the goal is to be entertained, do you really care if that viral video of the day being shared via a friend is a paid impression or not? 

Twitter is playing catch up here but if they don’t, the flop that was Facebook’s IPO will look tame in comparison.

Note Now Playing: Macklemore & Ryan Lewis - Can't Hold Us Feat. Ray Dalton Note


 

November 13, 2012
@ 01:38 PM

I’ve had about four hours of sleep but can’t seem to go back to sleep. There’s a pain of loss that feels like a death in the family and I hope writing this down helps in some way of dealing with it.

Yesterday it was announced that Steven Sinofsky is leaving Microsoft. As someone who considered Steven to be a role model of executive leadership and a source of my faith in the future of Microsoft this is a big shock. Part of me acknowledges that change is a natural part of life and nothing lasts forever but this is still a difficult incident to digest. Steven was a leader who understood how to leverage the strengths of an organization to build world class products while protecting the organizations from its inherent self defeating nature. As the saying goes a group is its own worst enemy.

When I think about Steven Sinofsky’s leadership style, I’m reminded of Joel Spolsky’s guide to interviewing which has the following succinct description of a great hire

In principle, it’s simple. You’re looking for people who are

  1. Smart, and
  2. Get things done.

That’s it. That’s all you’re looking for. Memorize that. Recite it to yourself before you go to bed every night. You don’t have enough time to figure out much more in a short interview, so don’t waste time trying to figure out whether the candidate might be pleasant to be stuck in an airport with, or whether they really know ATL and COM programming or if they’re just faking it.

People who are Smart but don’t Get Things Done often have PhDs and work in big companies where nobody listens to them because they are completely impractical. They would rather mull over something academic about a problem rather than ship on time. These kind of people can be identified because they love to point out the theoretical similarity between two widely divergent concepts. For example, they will say, “Spreadsheets are really just a special case of programming language,” and then go off for a week and write a thrilling, brilliant whitepaper about the theoretical computational linguistic attributes of a spreadsheet as a programming language. Smart, but not useful. The other way to identify these people is that they have a tendency to show up at your office, coffee mug in hand, and try to start a long conversation about the relative merits of Java introspection vs. COM type libraries, on the day you are trying to ship a beta.

People who Get Things Done but are not Smart will do stupid things, seemingly without thinking about them, and somebody else will have to come clean up their mess later. This makes them net liabilities to the company because not only do they fail to contribute, but they soak up good people’s time. They are the kind of people who decide to refactor your core algorithms to use the Visitor Pattern, which they just read about the night before, and completely misunderstood, and instead of simple loops adding up items in an array you’ve got an AdderVistior class (yes, it’s spelled wrong) and a VisitationArrangingOfficer singleton and none of your code works any more.

One of the interesting problems that faces a large software company is that it is very easy to become full of smart people that don’t get things done and then institutionalize this behavior by crowning them software architects or some equivalent. Steven’s leadership style encouraged a process and organizational structure, which you can read about in his book One Strategy: Organization, Planning, and Decision Making, that encourages getting stuff done by limiting the ability of the organization and people within the organization to take up positions where they strayed far from the goals of shipping a valuable product on time and within budget.

There are lots of people who disagreed with his philosophy and approach but it is hard to argue with the results of his efforts. Under him the team that shipped Windows Vista turned around and shipped Windows 7, the big ass table became one of Oprah's favorite things and one that’s close to home is that a mish mash of confusing consumer synchronization products became SkyDrive.

The way things get done in Steven’s organizations is so straightforward it hurts. You spend some time thinking about what you want to build, you write it down so the entire team has a shared vision of what they’re going to build and then you build it. The part where things become contentious is that getting things done (aka shipping) requires discipline. This means not changing your mind unless you have a good reason to after you’ve decided on what to build and knowing when to cut loses if things are coming in late or over budget. A great post about what it is like for an engineer working in a Steven Sinofsky organization that embraces these principles was written by Larry Osterman about Windows 7.

Each of the feature crews I’ve worked on so far has had dramatically different focuses – some of the features I worked on were focused on core audio infrastructure, some were focused almost entirely on UX (user experience) changes, and some features involved much higher level components. Because each of the milestones was separate, I was able to work on a series of dramatically different pieces of the system, something I’ve really never had a chance to do before.

In Windows 7, senior management has been extremely supportive of the various development teams that have had to make the hard decisions to scale back features that were not going to be able to make the quality bar associated with a Windows release – and there absolutely are major features that have gone all the way through planning only to discover that there was too much work associated with the feature to complete it in the time available. In Vista it would have been much harder to convince senior management to abandon features. In Win7 senior management has stood behind the feature teams when they’ve had to make the tough decisions. One of the messages that management has consistently driven home to the teams is “cutting is shipping”, and they’re right. If a feature isn’t coming together, it’s usually far better to decide NOT to deliver a particular feature then to have that feature jeopardize the ability to ship the whole system. In a typical Windows release there are thousands of features and it would be a real shame if one or two of those features ended up delaying the entire system because they really weren’t ready.

The process of building 7 has also been dramatically more transparent – even sitting at the bottom of the stack, I feel that I’ve got a good idea about how decisions are being made. And that increased transparency in turn means that as an individual contributor I’m able to make better decisions about scheduling. This transparency is actually a direct fallout of management’s decision to let the various feature teams make their own decisions – by letting the feature teams deeper inside the planning process, the teams naturally make better decisions.

Of course that transparency works both ways. Not only were teams allowed to see more about what was happening in the planning process, but because management introduced standardized reporting mechanisms across the product, the leads at every level of the hierarchy were able to track progress against plan at a level that we’ve never had before. From an individual developer’s standpoint, the overhead wasn’t too onerous – basically once a week, you were asked to update your progress against plan on each of your work items. That status was then rolled up into a series of spreadsheets and web pages that allowed each manager to track all the teams’ progress against plan. This allowed management to easily and quickly identify which teams were having issues and take appropriate action to ensure that the schedules were met (either by simplifying designs, assigning more developers, or whatever).

Transparency was also a cornerstone of Steven’s leadership style. The level of transparency into the organization’s decision making process via formalized mechanisms as described above as well as his personal decision making process has been unprecedented in my experience at Microsoft. It may not be as transparent as Google’s TGIF but on the other hand, I don’t think there’s anywhere else at Microsoft where visibility into how and why decisions are made was as clear as in the Windows organization.

At the end of the day, I’ll miss Steven and his influence on Microsoft. I’d like to think I became a better manager and leader from my time working spent working in his organization as well as the multiple exchanges we had over the years. Thanks for the memories.

Note Now Playing: Fall Out Boy - Thnks fr th MmrsNote


 

Categories: Life in the B0rg Cube

September 22, 2012
@ 03:06 PM

Last month I read Mike Arrington’s Why I Changed My Mind On Klout (And Invested) and thought to myself that I’d similarly changed my perspective on the much maligned social influence measuring service, Klout. My road to changing my mind on Klout was due to two unrelated sets of occurrences.

The first step to changing my mind were the high profile acquisitions of Vitrue by Oracle for $300 million and Buddy Media by Salesforce for $689 million. Both companies were sold for hundreds of millions of dollars for building enterprise versions ping.fm, tools for managing a company’s social media profile across social networks like Facebook, Twitter and Google+. The lesson from these is that just because something sounds dumb as a consumer play doesn’t mean it isn’t a great enterprise play. More importantly it made clear that helping companies figure out social media is a serious business.

The second incident contributed to my rethought perspective on Klout was Facebook's acquisition of Karma. Karma was co-founded by Lee Linden after his initial success with Tapjoy which grew to a company with a $100 million in revenues. Lee was a friend of mine during his Microsoft days and I can still remember him as this hyperactive guy who couldn’t stop talking about starting his own company and taking advantage of the opportunities in the industry. I remember him telling me about his idea for a mobile startup that would be an ad exchange which would help mobile devs maximize the revenue they got from ads in their free apps. I thought the idea had a low barrier to entry but don’t remember actively pointing that out. A few pivots later, the idea evolved into a pay-per-install ad network that was pulling in a $100 million a year. The lesson from this was that once a good team actually learns the challenges particular businesses face in an area, they can pivot their product to better serve those customers.

So how do these things apply to Klout? Klout tries to figure out who the experts are at particular topics in various social networks. This is valuable to at least two interesting players

  1. Social CRM products: The companies acquired by Oracle and Salesforce now can sell products to companies that help them better manage their Facebook and Twitter profiles but there is still a missing piece. The next logical step is helping companies figure out who their most valuable followers are on those sites and helping them target those customers. Helping a local business like the Pro Sports Club in Bellevue (for example) to figure out a one stop shop for creating and posting to a Facebook, Twitter and Google+ is cool. But even better would be telling them which of their customers they should give a few perks who they could be confident have a lot of “clout” with their audience on things like fitness recommendations. Helping the Pro Sports Club find the budding Jillian Michaels in their customer base would be worth a ton of money to them.

  2. Twitter: The gripes about how bad the targeted ads are on Twitter are the stuff of legend. Personally I have grown tired of the number of times I’ve seen ads for women’s hair products or home installations of air conditioning systems in my Twitter stream. Even though it is a crude approximation, the inferred topics of influence on my Klout profile would be a much better basis for Twitter to decide to use for showing me ads than whatever algorithms is using today. From Twitter’s perspective, Klout is sitting on a goldmine of information. An attempt to acquire Klout by Twitter sounds as inevitable as their acquisition of Summize to beef up their search product a few years ago.

In short, after thinking about it I’m convinced Klout provides a valuable service that is worth a lot of money to certain players in the industry. That said, as a social media user I do think it’s unfortunate that there is a service that provides a score for one’s participation in social media since it creates a set of incentives that may lead to unsavory behavior that harms the ecosystem as a whole.

Note Now Playing: Trey Songz - 2 Reasons (featuring T.I.)Note


 

Categories: Social Software

I’ve spent the last two years leading a PM team that has been part of building software experiences that make me immensely proud. The team has built software experiences that millions of people will use in Windows 8 and a developer platform will enable thousands of developers to build great software. Over the course of the past year we’ve delivered

  1. The social experiences in the Windows 8 People app. With Windows 8 you get a great browse and share your friends’ updates on Facebook and Twitter. The feedback we’ve received about this functionality has been extremely positive which has been quite humbling.

  2. A straightforward way for Metro style apps to take advantage of Single Sign On in Windows with the Live SDK.

  3. A developer platform for SkyDrive which has enabled developers to integrate SkyDrive across multiple apps, websites and devices.

This has been one of the most exciting and fulfilling times of my career. After about eight years working in the same organization at Microsoft, first as part of MSN and now Windows Live I’ve decided to move to another part of the company.

Over the course of the past few years, I’ve looked on at Microsoft’s search competition with Google and often wondered why although there’s been a lot of focus on beating or matching Google in search relevance and experience, there hasn’t been as much heard about how we’d compete with AdWords especially since that’s actually how we make money in the space.

Recently I was giving one of my friends who works in our ads space feedback after using a number of ads products including Facebook ads, Google ads and Microsoft’s. He asked if instead of complaining about what I wouldn’t rather just come join the team and actually help out. I thought “why not?” and since then I’ve become a lead program manager on the Bing Ads team.

My new team will be responsible for a number of things including the Bing Ads API. Regular readers of my blog shouldn’t expect any changes. If anything I’ll try to increase my pace of posting once I’m ramped up in my new gig and can come up with a sane blog posting schedule.

Note Now Playing: Big Boi - Fo Yo Sorrows (featuring George Clinton and Too Short) Note


 

Categories: Personal