Digital Door: The Next Sexy Job

 Hal Varian, chief economist at Google, was interviewed for McKinsey Quarterly in 2009 about how managers need to better understand how technology empowers innovation and the cheap, ubiquity of data. During the interview, Hal states,

I keep saying the sexy job in the next ten years will be statisticians. People think I'm joking, but who would've guessed that computer engineers would've been the sexy job of the 1990s?

Hal Varian

He goes on to say, "The ability to take data - to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it's going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it."

"I think statisticians are part of it, but it's just a part. You also want to be able to visualize the data, communicate the data, and utilize it effectively. But I do think those skills - of being able to access, understand, and communicate the insights you get from data analysis - are going to be extremely important. Managers need to be able to access and understand the data themselves."

Steve Lohr, a technology, business and economics columnist for the New York Times, also in 2009, also wrote about statistics, stating,

The rising stature of statisticians, who can earn $125,000 at top companies in their first year after getting a doctorate, is a byproduct of the recent explosion of digital data. In field after field, computing and the Web are creating new realms of data to explore — sensor signals, surveillance tapes, social network chatter, public records and more. And the digital data surge only promises to accelerate, rising fivefold by 2012, according to a projection by IDC, a research firm. 

Yet data is merely the raw material of knowledge. 'We’re rapidly entering a world where everything can be monitored and measured,' said Erik Brynjolfsson, an economist and director of the Massachusetts Institute of Technology’s Center for Digital Business. 'But the big problem is going to be the ability of humans to use, analyze and make sense of the data.'

The new breed of statisticians tackle that problem. They use powerful computers and sophisticated mathematical models to hunt for meaningful patterns and insights in vast troves of data. The applications are as diverse as improving Internet search and online advertising, culling gene sequencing information for cancer research and analyzing sensor and location data to optimize the handling of food shipments.

The entire article can be read here [read article]. These predictions seem to be even more relevant in 2011. CIO Magazine reported its list of the Hottest New Jobs in IT in June, and Data Scientist made the list at #2.

Big data -- that is, the glut of unstructured or semi-structured information generated by Web clickstreams, system logs, and other event-driven activities -- represents a huge opportunity. Buried in that mountain of data may be invaluable nuggets about customer behavior, security risks, potential system failures, and more. But when you're talking terabytes that double in volume every 18 months, where do you start? That's where the data scientist comes in.

On the business side, data scientists can open up new opportunities by uncovering hidden patterns in unstructured data, such as customer behavior or market cycles. On the dev side, a data scientist can use deep data trends to optimize websites for better customer retention. Within the IT department, a skilled data scientist can spot potential storage cluster failures early or track down security threats through forensic analysis.

"There's now an intellectual consensus in business that the only way to run an enterprise is to use analytics with data scientists to find opportunities," says Norman Nie, CEO of Revolution Analytics, which produces the first commercial application to bring the R data analysis programming language into the business world. Because of the immense opportunity for strategic insight buried in all that data, says Nie, "corporations now have an unlimited demand for people with background in quantitative analysis."

The R programming language is just one tool in the data scientist's toolbox. Others range from business analytics software from established providers like SAS Institute to IBM's new InfoSphere platform to analytics technology acquired in EMC's recent acquisitions of Greenplum and Isilon Systems. Just last May, EMC Greenplum hosted the first ever Data Scientist Summit.

According to Nie, data science jobs will require workers with a spectrum of skills, from entry-level data cleaners to the high-level statisticians, yielding a range of opportunities for newcomers to the field. As the business world gets increasingly social, the demand for people to plumb the depths of all that social networking clickstream data will only increase. The cliché going around is that "data is the new oil." A career in refining that raw material sounds like a good bet.