State and local government agencies would love to get their hands around big-data. All they lack is adequate data storage and computer power, and enough staff.
That's how 150 IT pros from the agencies see it anyway. They were surveyed by MeriTalk, the online government IT community, for data management and storage vendor NetApp in a follow-up to last year's study of federal government's "Big Data Gap." You can download a copy of the report here.
On the one hand, SLG authorities seem to have their collective heads in the sand. As many as 44 percent are not even "discussing" big-data, while 39 percent are just starting to learn about it, and only 11 percent are designing strategies to cope with it.
At the same time, the importance of big-data analytics seems to be understood: 57 percent expect successful management of big-data to improve efficiency; 54 percent anticipate increased speed and accuracy in decision-making; and 37 percent think it would help in understanding citizens' needs.
So what gives? I spoke with Regina Kunkle, NetApps's Vice-President for State and Local Government. She explained that two factors are primarily responsible for the apparent tension in the survey results. "One is definitional. 'Big-data' has been a term of art only for the last 18 months. It's new, like cloud computing was three years ago. People who say they're not discussing big-data might actually be tackling big-data challenges, but not using the term."
The second factor also relates to the newsness of the concept. SLG CIOs understand big-data's transformational potential, but are only at the very early stages of coming to grips with it.
One problem facing SLG CIOs, according to Kunkle, is that they find themselves in highly federated, siloed technology environments. Individual state agencies have their own, heritage systems, and there are big political obstacles to seeking consolidation. Even where consolidation has taken place, said Kunkle, "it is usually only partial."
If anything, smaller states, and the less-populous states of the midwest (Utah, Montana, and Wyoming, for example) are making most progress in the adoption of big-data analytics. Governments in these states have fewer agencies, and fewer political barriers to analyzing data across departments.
Of course, it's the larger, more crowded states that might benefit most greatly from being able not just to capture large volumes of data, but also to analyze it and use it as a basis for decision-making. Kunkle gave me a compelling educational example: K-12 common core testing. If all testing was done online, with students from some 125,000 keying in their answers, imagine the data that could be captured.
You could identify pockets of the country that face STEM challenges. You could identity pockets where English is a second language. This is much more meaningful than whether a student got an A or a B, and it could help you channel resources accordingly. You can't do that with a teacher grading papers, then entering the grade on a database.
Kunkle is optimistic that SLG CIOs will continue to push the big-data envelope, although right now some are "a little bit laggard in using the technology, or dealing with the challenges head on." As far as storage problems are concerned, NetApp is one of a number of vendors offering solutions that help clean, compress, and deduplicate data -- to some extent reducing the volume (which is growing, for SLGs, according to the survey, at around 50 percent per year).
At the agency level, it's clear that big-data is being collected and used already, whether the agencies realize it or not. Transportation and agriculture are two examples, but another topical one -- especially after Boston -- is law enforcement. Automated processing of videos or images is a big growth area for police authorities, and is an important example of getting to grips with large volumes of unstructured data.
Pressing social issues we face today, like homeland security, mean that SLGs will have to step up to the big-data challenge.
— Kim Davis , Senior Editor, Internet Evolution