@follow - Scaling the Service

July 19th, 2009 by rvanwell

@follow has gone from concept to market - at least in alpha/beta form - in less than 8 weeks.

As Tom has pointed out, it became obvious to us that the desire to follow ‘the conversation’ threads across multiple networks - news, social and otherwise was not just something that we wanted.  However, after we committed to building @follow and the initial ‘irrational exuberance’ period had passed and we started some hard thinking and design work, we realized that one of the reasons we had not come across another service like @follow were the dual problems of scale and cost, particularly when you start throwing around the term ‘real time’.

@follow faces quite a few challenges with scaling the service in order to live up to our original goal, which was to provide a simple, easy to consume stream of information from many sources in near real time available to millions of users/consumers on the web or mobile device .

As the service grows it will have to support:

  • capturing content from more networks
  • capturing content at faster rates - approaching real time
  • supporting more users, with more updates/refreshes more often
  • supporting more ways and devices to view/follow content (web, mobile, xml/rss, api)
  • new and virtually instant demands for service driven by growth or unpredictable spikes in interest surrounding a topic (i.e. catastrophe, politically motivated uprisings or the death of a mega celebrity)

The challenge isn’t just technical, it’s also economic.  That is, given enough resources (money, time, expertise) these problems can all be readily solved, but the primary objective in scaling modern web applications is to be able to meet new demands made by users, growth and the marketplace quickly without service interruption, without reducing service quality, without reducing features and without growing costs.   In fact, effective scaling will see quality improve, performance improve, feature sets grow while the cost per user dramatically decrease.


Tags: ,
Posted under Uncategorized

Leave a Reply