You don’t need petabytes of information to play in the big data league. The low end of the threshold is more like 10 TB, and, in fact, “big” doesn’t really tell the whole story. The many types of data and the speed at which data changes are, along with sheer volume, daunting challenges for businesses struggling to make sense of it all. Volume, variety, velocity – they’re the hallmarks of the big data era we’re now in.
Variety comes in the form of Web logs, wirelessly connected RFID sensors, unstructured textual information from social networks, and myriad other data types. Velocity breeds velocity. Fast-changing data drives demand for deep analytic insights delivered in hours, minutes, or, in extreme cases, seconds, instead of the weekly or monthly reports that once sufficed.
How are IT organizations coming to grips with data volume, variety, and velocity? Specialized databases and data warehouse appliances are part of the answer. Less heralded but also essential are information management tools and techniques for extracting, transforming, integrating, sorting, and manipulating data.
IT shops often break new ground with big data projects, as new data sources emerge and they try unique ways of combining and putting them to use. Database and data management tools are evolving quickly to meet these needs, and some are blurring the line between row and column databases.
Even so, available products don’t fill all the gaps companies encounter in managing big data. IT can’t always turn to commercial products or established best-practices to solve big data problems. But pioneers are proving resourceful. They’re figuring out how and when to apply different tools – from database appliances to NoSQL frameworks and other emerging information management techniques. The goal is to cope with data volume, velocity, and variety to not only prevent storage costs from getting out of control but, more importantly, get better insights faster.
Good observation Paul. Big Data comes with issues of velocity, as the report describes. So real time seems a goal that won't be met for quite awhile. Still, as you note, it is heartening to see so much progress being made. Who knows what the next couple of years might bring?
Im' with you, mhhfive, the growth opportunity is in the expanded use of data that extends the value to business and operational decision making. As this happens, with more simple tools and trained users, the value will increase. We don't as much need more, we just need to learn how to better use the data we are increasingly collecting.
Several decades ago, "big data" meant a spreadsheet or a "very large" spreadsheet.... Then "big data" meant an immense database of transactional information that could only be handled by the largest computers.
At the rate that computer capabilities have grown, "big data" is getting almost unimaginatively huge. Former "big data" projects aren't considered that big anymore... and what's considered "big" is really immense.
But just like spreadsheets made lots of data manageable, there are now tools to make "big data" more manageable. However, there just aren't that many people who really need to use (or have the training to use) the software that can handle insanely large databases.
As people and technology collect more and more information, we'll need to analyze it and store it and make it useful... and making the tools to make data simpler to play with seems like the next big thing for big data.
Thanks for a very educative report on big data. I am quite impressed with the strides IT has make in turning what was perceived to be a huge challenge to an amazing opportunity.
Reading through your report and other online references on big data, I really don't think we can classify current “analytics” efforts as real-time. If by real-time we mean that services act immediately — and intelligently — on information as it streams into the system, then we are far from that at the moment.
"With the paths that go through Hadoop [at Yahoo!], the latency is about fifteen minutes. … [I]t will never be true real-time. It will never be what we call “next click,” where I click and by the time the page loads, the semantic implication of my decision is reflected in the page."
But as you rightly stated, the technology will get better but whether we can actually achieve real-time analytics remains to be seen.
The ThinkerNet does not reflect the views of TechWeb. The ThinkerNet is an informal means of communication to members and visitors of the Internet Evolution site. Individual authors are chosen by Internet Evolution to blog. Neither Internet Evolution nor TechWeb assume responsibility for comments, claims, or opinions made by authors and ThinkerNet bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
The smartphone market reached a significant milestone, a breakthrough that may cause vendors to celebrate but could strain the capabilities of IT service desks.
In the fall of 2011, around 160,000 students in 190 countries enrolled in a Stanford-sponsored online course about artificial intelligence. About 23,000 completed the course and got certificates, including 248 who got a perfect score. The university offered the same course the old-fashioned way to students sitting in Stanford classrooms. None of the those students got a perfect score.
As Mitch Wagner discussed today, Yahoo is acquiring Tumblr. The big Internet debate at the moment is whether Tumblr will be good or bad for Yahoo. Regardless of their stances on the future of Yahoo itself, many claim that Yahoo will somehow ruin Tumblr.
Has China stolen a march on the West, developing an Internet architecture that is not only based on IPv6, but is also inherently secure from both internal and external attack?
New York's Metropolitan Transit Authority is conducting a pilot test of digital kiosks to guide subway users to where they want to go more efficiently and at lower cost.
The whole Amazon.reader debate is a double-stupid. It's stupid to think that there's any e-book buyer who doesn't know Amazon's URL, and it was stupider to let ICANN launch the whole free-form TLD initiative to start with.
While NFC's original goal was to enhance mobile commerce applications, it is finding its way into a number of other uses, which is creating both opportunity as well as challenges for IT departments.
Enterprises would like to move to cloud computing but are hesitant because they are concerned about providers’ ability to secure company data. Here are some tips that help to ensure that if breaches occur, the business is not left holding the bag.
Edmunds separates customers into segments based on the info it collects on its site and from partners, and uses that to push out custom content, said Brian Baron, director of business analytics for Edmunds.com, at Predictive Analytics Innovation Summit.
The automotive website uses propensity modeling to target ads and customer registration forms, said Brian Baron, director of business analytics for Edmunds.com, at Predictive Analytics Innovation Summit.
Ushering in a new era of cognitive computing systems, IBM announced today the IBM Watson Engagement Advisor, a technology breakthrough that allows brands to crunch big data in record time to transform the way they engage clients in key functions such as customer service, marketing, and sales.
Expert Integrated Systems: Changing the Experience & Economics of IT In this e-book, we take an in-depth look at these expert integrated systems -- what they are, how they work, and how they have the potential to help CIOs achieve dramatic savings while restoring IT's role as business innovator. READ THIS eBOOK
your weekly update of news, analysis, and
opinion from Internet Evolution - FREE! REGISTER HERE
Wanted! Site Moderators Internet Evolution is looking for a handful of readers to help moderate the message boards on our site – as well as engaging in high-IQ conversation with the industry mavens on our thinkerNet blogosphere. The job comes with various perks, bags of kudos, and GIANT bragging rights. Interested?
To save this item to your list of favorite Internet Evolution content so you can find it later in your Profile page, click the "Save It" button next to the item.
Email This
Page 1 of 6 
