Big Data Problem Plagues Government Agencies

11.06.2012
Big Data has the potential to transform the work of government agencies, unlocking advancements in efficiency, the speed and accuracy of decisions and the capability to forecast. But despite the potential benefits, most federal government agencies are struggling to leverage Big Data.

These agencies lack the data storage/access, computational power and personnel they need to make use of Big Data, according to a recent report, "The Big Data Gap," by . MeriTalk is a community network for government IT developed as a partnership by the Federal Business Council, Federal Employee Defense Services, Federal Managers Association, GovLoop, National Treasury Employees Union, USO and WTOP/WFED radio.

"Government has a gold mine of data at its fingertips," says Mark Weber, president of U.S. Public Sector for , underwriter of MeriTalk's report. "The key is turning that data into high-quality information that can increase efficiencies and inform decisions. Agencies need to look at Big Data solutions that can help them efficiently process, analyze, manage and access data, enabling them to more effectively execute their missions."

The government is collecting data: 87 percent of government IT professionals say their stored data grew in the last two years and 96 percent expect their data to grow in the next two years (by an average of 64 percent). Unstructured data makes up 31 percent of data held by the government, and the percentage is on the rise.

On average, government agencies store 1.61 petabytes of data, but expect to be storing 2.63 petabytes within the next two years. These data include: reports from other government agencies at various levels, reports generated by field staff, transactional business data, scientific research, imagery/video, Web interaction data and reports filed by non-government agencies.

The majority of IT professionals (64 percent) say their agency's data management system could be expanded or upgraded to cope with this growth, but they estimate it will take an average of 10 months to double their short- to medium-term capacity.