5 minute read.

Facebook’s Muhammad Ali approach to Product Development.

Paul Anthony / October 3, 2012

Posted in: Archive

Muhammad Ali’s championship fight the notable ‘Rumble in the Jungle‘ against George Foreman, is arguably one of the most captivating displays of tactical genius known in the history of boxing. It’s a classic tale of brains versus brawn, with Ali using his ‘rope a dope‘ tactic to absorb the body blows given by his much larger opponent. Foreman was the clear favourite before the fight, visibly the larger of the two men, with more strength and devastating haymaker punches, whilst Ali was the more nimble, with faster jabs and movements evident in his fighting style.

Float like a butterfly, sting like a bee. His hands can’t hit what his eyes can’t see

As the fight progressed, Ali’s quick jabs to his opponents face, became increasingly disorientating, and as the rounds progressed, his ability to land punches that mattered lessened being absorbed by Ali (and more importantly the ropes) with minimal damage.

Although Foreman had strength on his side, Ali had speed, and was able to deliver both a higher volume of punches, as well as choosing where and when to place them.  As many of you will already be aware, Ali won by knockout in the 8th round, taking the heavyweight championship having tired his opponent sufficiently.

In a technology race, it is the agile teams, who can release faster, and get to market quicker that win out over those that take months to get their latest iteration in front of their audience. One of the masters of continuous iteration and constant change are larger tech companies such as Facebook.

In their IPO filing Mark Zuckerberg spelt out some of the internal culture and product ethos at Facebook, affectionately dubbed ‘The Hacker Way.’

The Hacker Way is an approach to building that involves continuous improvement and iteration. Hackers believe that something can always be better, and that nothing is ever complete. They just have to go fix it — often in the face of people who say it’s impossible or are content with the status quo. 

Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once. To support this, we have built a testing framework that at any given time can try out thousands of versions of Facebook. We have the words “Done is better than perfect” painted on our walls to remind ourselves to always keep shipping.

That isn’t just a clever marketing line from Zuckerberg designed to make stock brokers excited. Facebook currently push new code live every single day, to millions of users, and yet maintain the level of stability that they do.  The below technical talk from Facebook goes into some of the technical and cultural things in place at Facebook to make that happen and is a great watch for any engineer curious as to how deployment works in larger organisations.

Using data and feedback directly from live traffic on features that may or may not be appreciated by users lets you see whether you are heading in the right direction quickly. A/B testing isn’t just a tool for increasing your conversion rate – in many cases it provides valuable feedback on product direction and interestingly Facebook do this at scale.

One of the more awesome features that this video illustrates, is an internal tool called ‘Gatekeeper’ which is software layer which facilitates certain users being able to see certain changes prior to it being pushed to the entire Facebook audience.

For example, there are conditionals in place so that only Facebook employees see certain changes, they can use IP whitelists, blacklists – geographical area – you name it, including a fun ‘Everyone but Techcrunch’ rule which exposes a different version to different audiences at a time minimizing disruption, and helping to test code whilst in production. They can ease in traffic onto newly released code live on the fly, providing it with 1% of the traffic, then 5% iterating the whole way to 100% of traffic so load testing is also much safer and log files and events can be watched. In essence, you could be sitting on a new feature without even knowing it, if you aren’t inside a particular bucket testing segment.

Facebook also have people responsible for examining key business metrics when a new release goes live. Key metrics such as Ad Revenue, Interactions, Email signups – are all monitored and tracked on individual dashboards by a designated employee after every release goes live.  This isn’t something left to chance, there are multiple data points scattered throughout their internal systems, many of which are more impressive to hear about than some of the public facing stuff.

Visualization of this data happens at both a systems and operations level in real time, so if a fault occurs, they can both find the problem and resolve quicker.

Essentially most of the tooling at Facebook goes towards minimising deployment risk, and getting things out there quicker and seeing if they are working. There are also clearly significant development resources being applied to developing internal tools. The faster you release, the more risk you carry, so automation, collation of data and working smarter rather than harder more often than not wins the fight and helps to prevent the application from falling over.

There are many parallels to be drawn between the smarts of an underdog boxer, and more rapidly deploying code to market. Like Facebook, Ali not only worked faster, but also smarter to win the fight. He released quicker than his opponent, timed his shots cleverly, adapted to the challenges of the competition, and strategically minimised his risk using the tools available to him.

  • facebook
  • product development
  • rapid iteration
  • speed
  • tools
  • visualisation