How Klout is Harnessing the Twitter Firehose to Keep Pace with Their Growing User Base

The Highlights

• API volume tripled in less than one month
• Twitter influence scoring coverage increased by 50%
• Preprocessed data saved Klout a full-time engineer
• Data delivery technology built for exponential scaling

The Company

Klout measures influence across the social web. Using its core technology, Klout analyzes social network user data and identifies influential individuals based on the impact of their opinions, links, recommendations and other online content. Klout then provides innovative tools to allow companies, including major brands such as Disney, Audi, Nike and Fox, to interact with and interpret this valuable influence data. Klout is a privately held company based in San Francisco, California.

The Background

As our friendship and professional connections have moved online, the social web has created a forum where an individual's influence can be measured for the first time. Recognizing this unique opportunity, Joe Fernandez and Binh Tran along with their team of scientists and engineers developed the Klout Score, a measurement of a user's overall online influence, designed to put real metrics around social influence.

The Problem

Providing Klout Scores for every individual in the exponentially ever-growing base of Twitter users was the task at hand for Matthew Thomson, VP of Platform at Klout. With massive amounts of data flowing in by the second, Thomson and Klout's scientists and engineers needed a fast and reliable solution for processing, filtering, and eliminating data from the Twitter Firehose that was unnecessary for calculating and assigning Twitter users' Klout Scores.

The Solution

Seeking a solution that could handle the back-end data scrubbing of the Twitter Firehose, in December of 2010 Klout turned to Gnip, the leading provider of social media data for enterprise solutions. Klout began using Gnip's Twitter User Mention Stream product, a realtime social media data feed designed to deliver all Tweets that mention a specific user, such as @replies and Retweets. Using this feed, the platform team at Klout no longer spent their days processing and filtering data, but rather shifted their efforts to focus on their core competency: providing influence scores to users of the social web.

With the transition from in-house data cleanup to working with Gnip, Klout gained the ability to focus solely on providing increased influence scoring coverage among Twitter users. The platform team no longer had to worry about filtering out items of the Twitter Firehose unnecessary for calculating Klout Scores, as the data had already been processed prior to delivery from Gnip.

Using Gnip's Twitter User Mention Stream, Klout receives a filtered stream of 100 percent of Tweets that mention Twitter users, extracted from the full Twitter Firehose. As with all Gnip feeds, data is gathered and delivered in compliance with every publisher's Terms of Use and Privacy Policies and is always equipped with standard data enrichments including URL expansion and Boolean support.

The Results

By selecting Gnip as their trusted Twitter data delivery partner, Klout instantly gained the ability to cast a wider net of influence scores among Twitter users. When asked for an influence score, Klout's servers immediately went from returning a Klout Score only 55 percent of the time to 95 percent of the time. In turn, with this increase in coverage Klout tripled the number of successful inquiries for Klout scores coming into their application programming interface (API) in less than one month.

• » Download PDF