According to the RSS Bandit roadmap the time draws nigh for the next release of RSS Bandit codenamed Nightcrawler. As with the previous release we will have an alpha version which will be mostly feature complete, followed by a beta version which will be feature complete and then the final release. Last week, Torsten and I agreed on the following plan for the alpha version of Nightcrawler.

Release Date: August 31, 2005

New Features:

• NNTP Newsgroups support
• Downloading of Enclosures/Podcasts
• Subscription Wizard replaces Add New Feed dialog
• Fast Mode (shutting off comment threading which uses a lot of CPU)
• Synchronization with Newsgator Online
• Atom 1.0 Support
• Extensibility Framework to Enable Richer Plugins
• Item Manipulation from Newspaper Views (e.g. Mark As Read, Flagging)
• Tip of the Day on Startup

There's also a persistent bug that has been bothering some of our users where posts from different feeds end up being mixed up. We haven't located the source of this bug but have added some tracing to the build which will be enabled in the alpha. Users who end up with mixed up feeds after the alpha can send us the trace files which should help us narrow down the source of the problem.

There are a couple of features I'd like to see in the final version such as "Comment Watching" so I can tell when a post I am interested in gets new comments. However we need to start locking down for the next release so that feature isn't likely to make it in unless I can sneak it in before the beta.  If there are other small, nice to have features you'd like to see in Nightcrawler please file a feature request in SourceForge and we'll see what we can get to before the final release.

Any comments or other feedback would be greatly appreciated. 

Nick Bradbury has a post entitled AttentionTrust.org in which he talks about a new non-profit entity that has been formed by Steve Gillmor, Seth Goldstein and a few others. Nick writes

In a nutshell, the idea is that your attention data - that is, data that describes what you're paying attention to - has value, and because it has value, when you give someone your attention you should expect to be given something in return. And just because you give someone your attention, it doesn't mean that they own it. You should expect to get it back.

I know that sounds a little weird - it took me a while to grok it, too. So I'll use an example that's familiar to many of us: Netflix ratings and recommendations. By telling Netflix how you rate a specific movie you're telling them what you're paying attention to, and in return they can recommend additional DVDs to you based on how other people rated the same movie. In return for giving them your attention data - which is of great value to them - they provide you features such as recommendations that they hope will be valuable to you. In my mind, this is a fair trade.

But what if Netflix collected this information without your knowledge, and rather than using it to give you added value they sold it to another service instead? I imagine that many people wouldn't like that idea - chances are, you'd want to be given the opportunity to decide who this information can be shared with. This is one of the goals of AttentionTrust.org: to leave you in charge of what's done with your attention data.

But what about this whole idea of mobility, as mentioned on the AttentionTrust.org site? What's the benefit of making this stuff mobile? Dave Winer provides a nice example: suppose you could share your Netflix attention data with a dating site such as Match.com, so you could find possible partners who like the same movies as you? For that sort of thing to be possible, you'd need to be able to get your attention data back from any service which collects it. (As an aside, this also means you could share your Netflix queue with any new DVD rental service that comes down the pike - so my guess is that smaller, up-and-coming sites will be more willing to share attention data than the more entrenched sites will.).

The attention data is what separates the giants in the Web world like Amazon & Netflix from their competitors. It is in their best interests to collect as much data as possible about what users are interested in so they can target their users better. The fact that [for example] fans of G-Unit also like 50 Cent is data that makes Amazon a bunch of money since they can offer bundle deals and recommendations which lead to more sales. Additionally record labels and concert organizers are also interested customers in the aggregate data of where people's musical interests lie. It is arguable that this is also beneficial to customers since it makes it more likely that their favorite artists will appear in concert together (for example). Similar concepts exist in the physical world such as supermarket loyalty cards.

How much data websites can store about users can vary widely depending on what jurisdiction they are in. Working at MSN, I know first hand some of the legal and privacy hurdles we have to clear in various markets before we can collect data and how we must make users aware of the data we collect. All this is documented in the MSN Privacy policy. To better target user's we'd love to collect as much data as possible but instead adhere to strict policies informed by laws from various countries and guidelines from various privacy bureaus.

It currently isn't clear to me whether AttentionTrust.org plans to become another privacy body like TRUSTe or whether they plan to be a grassroots evangelization body like the WaSP. Either approach can be effective although they require different skill sets. I'll be interested in seeing how much impact they'll have on online retailers.

As to why I called this the "Return of Hailstorm" in the title of this blog post? It's all in the 2001 Microsoft press release entitled "Hailstorm" on the Horizon which among other things stated

"HailStorm" is designed to place individuals at the center of their computing experience and take control over the technology in their lives and better protect the privacy of their personal information. "HailStorm" services will allow unprecedented collaboration and integration between the users' devices, their software and their personal data. With "HailStorm", users will have even greater and more specific control over what people, businesses and technologies have access to their personal information.

Of course we all know how that turned out. The notion of mobile attention data basically requires Web companies like Netflix & Amazon to give up what for them is a key competitive advantage. It makes no business sense for them to want to that. I wish Steve Gillmor and company luck with their new endeavors but unless they plan to lobby lawmakers I don't see them getting much traction with some of their ideas.

I recently stumbled upon a blog post entitled Why MSN is lost again... from Guillaume Belfiore which claimed that MSN is lost because we copy features from competitors without having a roadmap for where we want to go. He uses a specific example of the recent announcement that MSN Spaces will have a social networking feature as proof and claims that we are simply copying Yahoo! 360.

I was going to write a response but then realized that Jeremy Zawodny of Yahoo! had written a post about this topic which is a generic answer to posts like Guillaume's. In his post Secrets of Product Development and What Journalists Write Jeremy wrote

Before I came out to California to work at Yahoo, I watched the business and culture of Silicon Valley from a distance. I read lots of the trade rags, tech web sites, and books about early Internet companies (the Netscape era).

One of the things that amazed me about Internet companies (usually the portals) was how quickly they built things and were able to react to each others moves with frightening speed. Company X would do something amazing and new only to be leapfrogged by Company Y just a few weeks later.

They were putting on one hell of a show and it was all amplified by the crazy bubble of the late 90s. I loved it.

The tech and business press would say things like "in response to Company X, Company Y has just..." or "in an effort to defend their business from Company Y, Company X today launched a new..."

I saw headlines like that all the time and still see them today.

Today there's one important difference: I'm on the inside now. For the last five and a half years, I have had a front row seat to the inner workings of what I used to imagine (with the help of a small army of journalists and reports).

Now I see it first hand and hear about it from coworkers and friends at other companies. And you know what? It's even more insane than it looked from the outside.

So I'm going to let you in on a little secret about how products are developed at large companies--even large Internet companies that some people think are fast on their feet.

Larger companies rarely can respond that quickly to each other. It almost never happens. Sure, they may talk a good game, but it's just talk. Building things on the scale that Microsoft, Google, AOL, or Yahoo do is a complex process. It takes time.

Journalists like to paint this as a rapidly moving chess game in which we're all waiting for the next move so that we can quickly respond. But the truth is that most product development goes on in parallel. Usually there are people at several companies who all have the same idea, or at least very similar ones. The real race is to see who can build it faster and better than the others.

Think about this the next time a news story makes it sound like Yahoo is trying to one-up Google. Or MSN is "responding" to last week's launch of a new AOL service.

It's easy to get caught up in the drama of it all. But reality is often quite different than what you read.

Just because the media likes to paint it as if web companies respond to each other's development efforts in the twinkling of an eye as part of an eternal game of one upmanship doesn't mean this is the case. Although folks like to paint Web development as simply tweaking HTML pages, as Jeremy points out it takes a lot longer than one would expect to build and deploy services that will be utilized by millions of people.

The social networking aspects of Spaces have always been part of the vision and in fact when I was hired at MSN my boss told me that I'd be working on three things; a blogging platform, a social networking platform and an RSS platform. At the time, it wasn't clear my team would own the RSS piece so my [future] boss was worried that I'd be upset if I started on the team and the RSS piece moved elsewhere. Of course, since I already work on RSS Bandit in my free time I didn't mind if I didn't get to work on RSS as part of my day job. It turned out he was right and the RSS pieces ended up being driven by the http://www.start.com/myw3b and http://my.msn.com folks.

Don't believe the hype.

I've been in Nigeria for almost a week and so far it's been great. I've spent a bunch of time with family and friends, eaten a bunch of stuff I haven't hafd in years and decided I like MTV in Africa better than what we get in the United States. I've also been taking pictures of everyday life which I'll post to the photo album on my Space once I get back.

Below is a random grab bag of impressions I've had during my trip

• The traffic scares me. A lot. When being driven in Lagos & Abuja I tend to clench my fists while expecting we'll be in an accident at any minute. I can't get over the fact that as a teenager I used to be able to drive in this chaos and never had an accident. :)

• The proliferation of mobile phones is insane. There seem to be about half a dozen mobile phone carriers and almost everyone on the streets is carrying one. I was talking to my dad and he said the Nigerian mobile phone market is the second fastest growing in the world after to China. About two years ago when I was last here I saw more people downloading ringtones and texting than I'd seen in Seattle & Redmond, the trend has only continued. I have a bunch of pics of mobile phone ads on the sides of buildings and street hawkers selling pay-as-you-go recharge cards which I'll post once I get back.

• There is now a large local movie & hip hop scene. The movie scene was blowing up just before I left for college but it now seems to have matured quite a bit. It seems we export movies all over Africa. Folks have started calling the Nigerian movie scene "Nollywood". There are also a ton of local hip hop acts including one of my high school friends is now a rapper called Big Lo. About a decade ago he and I were part of a rap group called R.I.G. and I still have some of our tracks on my iPod. It's great to see that at least one of us is living our teenage dream of being a famous rap star.

• The newspaper headlines seem to focus exclusively on the goings on of the government & politicians or on tragedies involving loss of life. The contrast between that and the kind of stuff I usually see on the cover of USA Today is striking.

• Inflation is crazy especially in Abuja. Everything seems to cost a couple of hundred or thousands of naira. I still remember when you could get a bottle of Coke or a newspaper for under one naira. Then again, that was about two decades ago.

• There are a lot of billboards about HIV/AIDS prevention in the capital city in Abuja but almost none in Lagos (the former capital and commercial center). I'll try and get some pics of the billboards before I get back.

• Almost every PC I've used so far has been infested with spyware. Except for the Powerbook...

• The London bombings are on people's minds in my social circle. One of my mom's friends lost her only soon in the July 7th attacks. My sister and dad were in London during the first bombing and I was pretty rattled when it happened. It's good to see the British police have caught all the suspects from the second attack. 

• The local airline business seems to be thriving as well. Here's another place where there seems to be at least half a dozen competitors driving down prices. It looks like the government airline, Nigeria Airways, is finally out of commission. Good riddance.

• I miss Nigerian food.

Mike Torres has a blog post entitled  On "MSN: Social Networking Edition" where he points to news stories about a core piece of a future version of MSN Spaces he and I have been working on for the past few months. Mike writes

On "MSN: Social Networking Edition"

Wow, is this actually true?  If so, I wonder how this will change the way people find and communicate with others in the future.

Quote (emphasis mine)

Microsoft Monitor: MSN: Social Networking Edition
A forthcoming "friend of friends" feature will add personal networks of friends to a MSN Space. In someone's My Space [sic], there will appear pictures of the friends, which can include friends of other friends. Friends can be added from people known or associated with friends or from MSN Spaces searches. Blake used the example of searching for golf blogs. If friends are on MSN Messenger, an icon indicates so.

More on Blake, Yusuf, MSN Messenger file sharing, and "Microsoft's new Web-based mail system" as well.  It is an interesting read... 

Funniest part of the post: "small consumer adoption [of Spaces]".  Joe is usually quick to take a shot at Microsoft without actually learning the facts first.  Come on, Joe...  I think the most recent public number was 17 million spaces created worldwide.  Compare that to any of our competitors in their first 3 years of existence.  Yeah...  small consumer adoption.  Maybe 17 million Roombas created those spaces :)

Mary Jo Foley has a little bit more here:  Microsoft Watch.  My favorite part:

"Our ability to enter, differentiate and compete has never been stronger," Mehdi told the Wall Street analysts and media representatives who attended the analyst meeting.

Couldn't agree more

Although I've primarily been talking about my work on getting a public API for MSN Spaces off the ground, I also work on our social software platform on the back end as well. Once we ship the next version I'll be able to talk a bit more about some of the design decisions we made and I'll get to see how users end up utilizing the features we've been working in.

I love my day job.

I've been reading some of the hype around microformats in certain blogs with some amusement. I have been ignoring microformats but now I see that some of its proponents have started claiming that using XML on the Web is bad and instead HTML is the only markup language we'll ever need.

In her post Why generic XML on the Web is a bad idea Anne van Kesteren writes

Of course, using XML or even RDF serialized as XML you can describe your content much better and in far more detail, but there is no search engine out there that will understand you. For RDF there is a chance one day they will. Generic XML on the other hand will always fail to work. (Semantics will not be extracted.)

An example that shows the difference more clearly:

<em>Look at me when I talk to you!</em>

… and:

<angry>Look at me when I talk to you!</angry>

The latter element describes the content probably more accurately, but on ‘the web’ it means close to nothing. Because on the web it’s not humans who come by and try to parse the text, they already know how to read something correctly. No, software comes along and tries to make something meaningful of the above. As the latter is in a namespace no software will know and the latter is also not specified somewhere in a specification it will be ignored. The former however has been here since the beginning of HTML — even before it’s often wrongly considered presentational equivalent I — and will be recognized by software.

This post in itself isn't that bad, if anything it is just somewhat misguided. However Tantek Celik followed it up with his post Avoiding plain XML and presentational markup which boggled my mind. Tantek wrote

The marketing message of XML has been for people to develop their own tags to express whatever they wanted, rather than being stuck with the limited predefined tag set in HTML. This approach has often been labeled "plain XML" or "generic XML" or "SGML, but easier, better, and designed just for the Web".

The problem with this approach is that while having the freedom to make up all your own tags and attributes sounds like a huge improvement over the (mostly perceived) limits of HTML, making up your own XML has numerous problems, both for the author, and for users / readers, especially when sharing with others (e.g. anything you publish on the Web) is important.

This post by no means contains a complete set of arguments against plain/generic XML and presentational markup, nor are the arguments presented as definitive proofs. Mostly I wanted to share a bunch of reinforcing resources in one place. Readers are encouraged to improve upon the arguments made here.

The original impetus for creating XML was to enable SGML on the Web. People had become frustrated with the limited tag set in HTML and the solution was to create a language that enabled content creators to create their own tags yet have them still readable in browsers via stylesheet technologies (e.g. CSS). Over time, XML has failed to take off as a generic document format used by content authors for creating human readable documents on the Web but has become popular as a data format used for machine to machine communications on the Web(RSS, XML-RPC, SOAP, etc) .

Thus any arguments against XML usage on the Web today are really arguing about using XML as a data format since it isn't really used as a document format except for XHTML [and even that is only by markup geeks like Tantek & Anne].

Anyway let's look at some of Tantek's arguments against using XML on the Web...

Tower of Babel Problem

If everyone invents their own tags and attributes, pretty soon you get people calling the same thing by different names and different things by the same name. While avoid both of those occurences completely is very difficult (many of the microformats principles are design to help avoid those problems), downright encouraging authors to make up their own tags and attributes makes it much worse and all you end up with are a bunch of documents that give you the illusion of self-description.

Didn't the XML world solve this with XML namespaces like six or seven years ago?

Temptation of Presentational Markup

What happens all too often when authors or developers make up their own tags is that they choose tags that are tightly tied to a specific presentation rather than abstracting them with semantics. Quite similar to the phenomenon of authors picking presentational class names.

As a casual user of HTML, I personally haven't seen a good explanation of why <strong> is better than <b> so arguments whose entire basis is "presentational markup is evil" don't carry much weight in my book. If I come up with a custom markup format and it has a <bold> element, is that really so evil? I'm pretty sure that the XML formats used by OpenOffice or Microsoft Office contain markup that is presentational in nature whether it is setting font sizes, text colors or paragraph alignemnt. Are they evil or does the fact that they aren't intended for the Web give them a pass?

Preferring Semantic Richness

Sometimes something is a bad idea not just in absolute terms, but also relative to other approaches and solutions.

A while ago I wrote about a semantic richness spectrum on the www-style mailing list which went into a bit more detail. Håkon Wium Lie wrote a paper that both predated my rough summary by a couple of years, and provided a much more thorough analysis.

 Languages with well-known semantics are preferred to proprietary/made-up XML. This is for many reasons, including accessibility, cross-device support, and future user agent support.

This seems to be arguing that instead of cooking up your own custom format you should pick an established format with the semantics you want if one exists. This is regularly practiced in the XML world especially when it comes to the Web so I don't see how this is an argument against using XML.

--

Seriously, I feel like I am in some bizarre alternate universe if having aggregators subscribe to HTML web pages is being advocated as being a better idea than using specialized XML formats like RSS & Atom.

That's it...I'm going back to my vacation. The world has gone too loopy for me.

It looks like MSN Virtual Earth is live. In an attempt to beat everyone to the punch when it comes to building an application using the MSN Virtual Earth API, I present http://www.25hoursaday.com/moviefinder

It requires Javascript (duh) and only works in IE beecause I didn't have time to figure out the Firefox equivalent of innerHTML. Enjoy.

Pardon my brevity, I'm posting this from a kiosk at Heathrow Airport

I'll be leaving for Nigeria in the next couple of days and should be gone for a few weeks. Going home is always fun, I'll have my mom asking me when I'm going to settle down with a nice Catholic Nigerian girl while my dad wonders when I'm going to stop goofing off at Microsoft and go back to school to finish my education.

Of course, the best part about going on vacation is the mountain of email and spec bugs I know will be waiting for me when I get back. :)

A couple of MSN betas snuck out into the world this week.

In his post MSN Shopping Officially Launches New Beta Site, Chris Jolley talks about the new http://beta.shopping.msn.com. Some of the new features include

• RSS Feeds – it remains to be seen how aggressively consumers take to this, but I think it's super-cool how we are embracing an emerging technology.    
• Ratings & Reviews - allows consumers to see what other consumers think of a product, providing them with increased confidence in their purchase decisions
• Ability to browse our complete selection through an easy-to-understand taxonomy and powerful search
• Refine searches with expanded, relevant criteria
  •  By price, rating, popularity, name, best match
  • Multiple Views - Results can be viewed in three different views
• Usability enhancements:
  • Recently viewed– helpful little personalization feature that helps continuity between sessions
  • Compare prices across stores, find out who has free shipping, hot deals, etc.
  • Clean and consistent UI – The UI is clean and simple and designed to NOT get in the user's way
• Useful Merchandising – Comprehensive set of merchandising, including the Gift Center , Seasonal Shops & Guides , and more

I met with the PM who owns the RSS feeds feature a few months ago to talk about my scenarios and its great to see that it's finally out there. I use the Amazon RSS feeds all the time and can definitely see myself getting some use out of the news feeds from MSN Shopping. All I can say about the ratings & reviews feature is...FINALLY. I've already begun to scheme about how I can convince Mike that we should totally gank learn from the feature in Yahoo 360° where you can include your reviews on your "space".

The MSN Screensaver beta is also out. Some of the features include

Personalize with background photos and news and weather information from MSN or any RSS feeds from websites you choose.

Search the Web and click news headlines directly from the Screen Saver.

Stay connected with Hotmail, MSN Messenger, and MSN Spaces. Track how many unread Hotmail messages and current Messenger conversations you have, and display blogs and photos from your friends’ MSN Spaces.

Some of the features of the screen saver are so useful I'm wondering why they are wasted on an app that typically runs when the user is away from the computer.

And finally, http://www.start.com/myw3b has some new features. Sanaz Ahari has a blog post entitled new things on start.com... where she writes

first and foremost, check out our new search results:

- we now have tabbed results that include web, news and rss , so you don't need to do seperate types of search we just give you all the results and it's all ajax based of course

- the rss resutls are very cool, cause you can just subcribe by clicking on subscribe and they'll get added to your feed

 

we've also added themes : ice, granite and ocean... ice is my favorite :)

super easy to navigate, needless to say all in place and no refreshes required.

 

we got rid of our not so useful home link and folder view - so now when you click on a folder we don't replace your dashboard with the content of that folder... yes, we listen to our user feedback :)

 

lastly you can now hit the esc button to close your overlay modules - super usable...

There's a followup post on the start.com weblog entitled more on start: OPML support and popular feeds which states that they added support for importing OPML files as well.  

Postscript: Anyone notice what all three of these betas have in common? (Hint: Starts with R ends with S and has an S in the middle).

Shelley Powers posted a comment to a recent post of mine about her experience interviewing for a Microsoft position. She wrote

Tod, if I sound angry, I am. Bill Gates went in front of Congress and said we should have unlimited H1Bs, because Microsoft just can't find 'good' people in this country.

I interviewed with Microsoft. As soon as I started talking with the guy, I knew he wasn't interested. Within the first five minutes. I've been around long enough to know when someone isn't interested.

Microsoft doesn't hire 50 year old men, must less 50 year old women.

As for interviewing, yes I have interviewed people. And hired them. I've never once had a bad hire. You can tell when you talk to people, their enthusiasm, how they respond to what you say, if they're a good fit. Do they have to have all of the tech background? Not a bit. Oh, I wanted certain things: interest and background with a specific programming language, experience with relational databases, and so on. But as for the nits, most of us can do something called 'learn'. I looked for motivation, interest, fit with the team, experience, but more importantly interest in the job.

I can understand Shelley's frustration around not being made an offer for a position she feels qualified to handle. Last year, when I decided to leave the XML team I interviewed with a bunch of teams at Microsoft including the newly formed RSS team within the Internet Explorer group. The job would have involved community evangelization around the Longhorn/IE RSS efforts and working on platform APIs for RSS. Given my work on RSS Bandit and the fact that the job I was leaving was designing the core XML APIs in the .NET Framework I thought I was a shoo-in for the job. I wasn't. I didn't get an offer because I wasn't "passionate" or "experienced" enough according to the feedback I got when I inquired as to why I didn't get the job.

I could have decided to give up because "Microsoft doesn't hire black men" or "Microsoft doesn't hire people in their 20s for high visibility positions" but didn't. Instead I kept interviewing and ended up at MSN working on social software. Personally I think I'm happier at MSN than I would have been on the RSS team, I've gotten to broaden my technical horizons and work with a more diverse set of individuals on a prettier campus. Also I still get to work on RSS stuff both in my personal time and with different teams at Microsoft in an advisor role.

It's easy to give in to despair. Don't.